The bullet is in the barrel approximately .200 of an inch during rifle recoil, which some refer to as "barrel time".
How you handle recoil, gun placement in the bag, how sturdy the forearm is held in the rest, all come into play with many other issues of gun handling.
Light weight rigs can be difficult to handle without a muzzle break.
When I am shooting two and one flyer, I change primers, and if that does not fix the issue, then I change powders. I am assuming that you have established the best seating depth.