Inconsistant ignition, untuned load, inferior factory recoil lug, bedding, pressure on barrel from forend inconsistant from heating or rest pressure, inconsistant pressure on shoulder and hold, flinching, jerking the trigger, action screws loosening or wrong torque, scope base or ring movement, bolt misalignment causing contact on lower lug only, etc.
All can contribute, just a process of elimination. I'm not sure what the most common one would be, but there's probably many that fit that statement based on the fact that alot of them are present in factory rifles from the get go.
What's the components being used? More specific help or experiences might be had if we knew more.