I'm with Panhandle Pete and Browninglover. Your decision should factor in range envelopes, wind, and sound signature.
I go every summer in a group of 4 dudes, me and my dad, my buddie and his dad.
We started off with a fairly wide variety of calibers, from .17's up to our hunting rifle 30/7mm magnums... and after a few years (and as I built more purpose driven/dedicated rifles rifles for us) we all settled on 223AI with 53g vmax for ~150-400 yards (the vast majority of our shooting) and 6XC with 70 or 80g nosler varmint pills. We get about 3-4K rounds out of a 223AI barrel, and about 1500 rounds out of a 6XC barrel... which means I don't have to rebarrel 8 rifles every year (a huge plus for me)
We will also bring a 17 or two, and my AR for shooting while we are moving the trucks around. Sometimes you just can't pass up a 30-60 yard shot right off your truck window (not of course on public/paved roads!)
We decided we won't shoot anything besides frangible/varmint projectiles. We like pushing our max range, but at long ranges the best we could ever get was a little tumble usually followed by dragging them self's down their hole... it just isn't worth it without the acrobatics. We occasionally play with long shots and VLD's, but usually on "that rock over there" or the like.