I'd run some ballistics at reasonable velocities and see how much you'd gain/lose between both bullets at the distance you'll be shooting/hunting, both in windage, elevation and velocity. In other words, don't be concerned with the ballistics at a mile out if your max distance is 600 yards. I think you'll find that there will be little difference in a reasonable scenario.
In addition, if you have brass and bullets, seat one of each so the bottom of the bearing surface of the bullet, is at the neck/shoulder junction of the case. Then, compare the 2 base to ogive measurements to see the difference. That number will be very close to how much more you will have to seat the longer bullet in the case to get to the same point of touching the lands in the barrel. You might find it to be insignificant and only taking up a small amount of case volume.
At the same time, you can check your OAL to make sure they will fit the magazine.
So, what I am saying is, if it were me, and I didn't have the need or option to shoot very long distances, I would throat it for the 180's. Then if I decided to shoot something heavier for one reason or another, I would take the slightly less case volume and run with it.
It is much easier to cut the throat deeper in the future, then it is to set the barrel back to make the throat shorter.
Edit: I just ran a couple of numbers comparing base to ogive measurements of the 180 VLDH to the 195 Elite. The difference is .107. That alone could take you past magazine length. I sure would check all of the numbers before you make a final decision.
You haven't really said if this is a hunting rifle or mainly a target rifle. If it is a hunting rifle, I would definitely pick a hunting bullet. The only 180 that Berger has for hunting, is the VLD.
3100-3150 with the heavy bullets, especially the 195, would be stretching it, depending on the powder you pick and what the rifle likes. I would also suggest you look at the load data from Hodgdon which is based off of a 24" barrel just like you are having built.
BTW, MidSouth has the 180VLDH on clearance sale right now. $55.43/box cheap.