There are multiple reasons for what you experienced. The following is a list of things that can cause POI shifts with changes in position.
1) Don't use those rests. You don't shoot with them in the field and your body position when shooting from them is all jacked up to accommodate the rest. Also, if it's a lead sled and your shooting an unbraked large cartridge rifle off of it it places a lot of stress on your rifles components. It can cause things to loosen and potentially damage scopes. 2) Parallax. Absolutely needs to be adjusted for whatever distance you're shooting, especially the further out you start to reach. 3) Scope shadow/head position. If you have scope shadow it needs to be equal all the way around and your head position/cheek weld should be consistent shot to shot to make this happen. 4) Pressure on the buttstock. Google/YouTube recoil management/controlled recoil. Basically, you only need enough to load your bipod or rest and stabilize it on target. Too much can push your shots around. Too little (like free recoil-which I'm not a fan of) with a heavier cartridge and you will get hit in the face...especially with a light rifle. 5) If you're not shooting with a rear bag you should start. It not only helps to greatly stabilize your shot, but it lets the rifle recoil/ride straight back into you. If you don't shoot with one there's a tendency for the buttstock to move around. Usually down. 6) Your rifle, or at least your optic, isn't level from shot to shot. 7) Even if you have a bubble level or inclinometer on your scope your reticle may not be orthogonal to it. Meaning when the bubble reads level your reticle isn't exactly perpendicular to it and thus when you dial it doesn't dial straight up. There are multiple ways to check this. The easiest is to set it either to a plum line or draw a perfectly vertical line on a large target with a level. 8) Inconsistent trigger pull.
Regardless of the position you're shooting from, try to do the following and I would bet that things tighten up. The fundamental goals are for things to be consistent shot to shot, for you to be straight behind the rifle, and for the rifle to recoil straight back into you.
1) Get square behind the rifle. 2) If prime or shooting off a bench, use a rear bag. 3) Adjust you're parallax. 4) Consistent head position/cheek weld and Make sure scope shadow is equal all the way around. 5) Load your bipod/have solid consistent pressure into the buttstock. 6) Put a bubble level or inclinometer on your scope and make sure that your reticle is set to be perfectly vertical when it reads level. Quickly check your level or inclinometer prior to every shot. 7) Put a good trigger in your rifle and adjust it as low as you feel safe for you. No question lighter triggers let you break shots more consistently, which=less POI shift and better groups. There's lots of options out there. Personally, I use TriggerTechs set at 1.5lbs. For two of my rifles this is perfect. On one of them I wish it was 1lb. 8) Don't be surprised when your rifle goes off. You should know it's going to happen and just be ready for it. If your surprised you're going to jerk.
The bigger the cartridge the more these fundamentals matter. You can't push them around and force them to do things like you can with smaller cartridges.