So, I'm trying to figure out cartridge, bullet, barrel lengths and twist rates in some logical fashion. Kind of makes you dizzy!
First I tend to focus on the animal. I figure to have the best chance there, I need best possible accuracy, expansion, and an exit.
Thinking about accuracy, I'm mostly focused on minimizing recoil which tends to push the cartridge to a minimum bullet weight and case capacity. That works. I'll admit, I shoot a 6.5 Creedmoor better than a 300 WM.
Then expansion. There are definitely cases where expansion is not required. Heart shots in just the right spot, brain, spinal shots. That said, especially at long range, I want an expanded bullet to the vitals. So, I want a bullet running over 1800 fps at max range. My legit max range now is 400. I am working to be good at 600. This has come down quite a bit in the last couple years because I'm more conservative about what shots I will take. As you can imagine, bc, personal effective range, barrel length, case capacity all play a role.
Then, I believe based on hunting experience that I need an exit. I struggle to know by the numbers if a bullet will exit. My thoughts are…., a bullet enters with kinetic energy. This energy does work on the bullet and on the animal. If it lacks enough energy, that will allow the bullet to stop. So, what is a minimum or a target? I go back to a near 0 work example. A 44 mag cast solid, imo will kill a mule deer buck at 120 yards with minimal left over. It will exit. That is about 600 ftlbs. So, I set that as my energy minimum to exit deer. I use 1000 for elk, but I really don't know about that.
So, does this make logical sense? Am I missing something big? Yes, I'm definitely overthinking it!
Given that you have a .300 WSM, that you admit you don't shoot it as well as you do smaller calibers, it seems like you're looking for a cartridge that will be ideal for deer or similar-sized game, and you want to be able to take shots on deer out to 600 yards confidently, then you have some choices within your constraints.
First, expansion. You need energy to get a bullet to expand, and that correlates with velocity. Most hunting bullets reliably expand above 1800 ft/sec, so that would need to be your minimum velocity at 600 yards. You're looking at long, heavy-for-caliber bullets in the .264 to .284 diameter range. I think your best bet is between the 6.5 PRC and the 6.8 Western. Why not 7mms? Too close to what you have in the Sherman and WSM.
What about penetration? As others have pointed out, the more expansion, the less penetration. Yet, the two cartridges above should almost certainly penetrate a deer or elk completely on a broadside chest shot that doesn't hit the shoulders or leg bones. From what I've read the Accubond tends to hold together better than the ELD-X... I'll defer to comments from folks who have shot a deer with these bullets.
The 6.5 PRC/143gr ELD-X Hornady factory load and the 6.8 Western/165 gr Accubond LR Winchester factory load will be within 3 ft/sec at 500 yards, will be above 2,000 ft/sec at 600 yards... they have an almost identical trajectory. The 6.5 PRC will have slightly less recoil, the 6.8 Western will hit harder at long range and thus do more damage. I don't think there's enough difference to make a difference on deer, but I think it would make a difference on elk. The 6.5 PRC is a more specialized long range deer and similar cartridge, the 6.8 Western is marginally better for larger herbivores (elk, moose, African large plains game). Both cartridges are 'modern' designs with diameter-to-height ratios that have been found to increase intrinsic accuracy, both will fit and feed in short (.308-length) actions. In summary, this paragraph explains why Hornady so hates the 6.8... it is a better 6.5 PRC!
Since you already have the .300 WSM, the 6.5 PRC seems to be the right choice. I own .243s, .308s, .30-06s, a .338 Win Mag, a .375 H&H, and a .470 Capstick... I would (and have) picked the 6.8 Western.
I've ordered a magnum bolt face for my new-to-me ARC Mausingfield, and plan to get a 26" CF 6.8 Western barrel for it. I don't plan to run a muzzle brake on it, but I will get it threaded for a suppressor... maybe the new Silencerco Scythe-T. I think that will make the ideal long range hunting rifle... once I choose a stock.