It's part mechanical and part shooter induced.
As stated by other posters, the chances are that you are not shooting to your full potential.
I think I've heard Darrel Holland say that the bullet will remain in the barrel ~1/24,000 of a second and for about ~3/8" of recoil during which any movement will affect the POI.
If the bore tracks exactly rearward, no big deal. But if it moves left/right/up/down especially with any inconsistency, your groups will reflect it.
I was always told growing up not to dry fire as you could break a firing pin. Perhaps that was due to older rifles and/or safety concerns.
Practicing safely with modern rifles is a good thing. I also find it useful to have my son dry fire before attempting a long shot at game. It helps to settle nerves and verify whether he's got a decent rest/position.
A lot of the long range shooting books and videos go into detail about perfecting your technique. And, I think there are tweaks that competitors use to tune their actions and firing pins etc to squeeze out that extra tiny amount of precision.
...wish I could give specific advice
but, here are some things to ponder (from a non-expert)...
light weight rifle
timing of the bolt
trigger adjustment
trigger control - straight to the rear
don't wrap thumb around stock
bone support/muscle relaxation
stable bench, front bag, bipod
breathing
rear bag placement and square