Wildrose, the best analogy I can make is an astronaut in outer space throwing a spear. If you push the spear away from you, you are going to end up going in the opposite direction to the spear. Your speed vs the spears speed is the inverse of the ratio of your mass vs the spears mass. Note that there is no propellant or muzzle blast involved. Action = reaction.
The same would be true if you were pushing a bullet out of a tube by hand. Since our hands are not too fast, imagine trying to push a cannon ball out a cannon barrel. Even if the ball slides in the barrel on an air bearing (no friction), if you accelerate the ball, it is going to be pushing back on your hand. Try that in outer space with no land under your feet and no gravity to hold you down and create friction to keep you stationary and you will be moving back away from the breech while the ball is moving out the breech.
Arguing that all recoil is due only to muzzle blast is wrong and has no basis in math or science. The fact that muzzle blast may be more perceptible than the impulse from the bullet is a completely subjective observation and does not detract from the truth. Since muzzle blast starts only after the bullet has left the barrel, it is highly unlikely to influence point of impact (except for causing the shooter to flinch when he pulls the trigger next time). The impulse of the bullet, however, takes place while the bullet is moving in the barrel.
Considering that a 1" shift in point of impact at 100yds requires only a deflection of 0.006" at the end of a 22" barrel, it seems entirely possible that movement of the weapon due to the way it is held and the force exerted by the impulse of the bullet could contribute towards a degradation in accuracy.
This last paragraph is nothing new, it is what marksmanship instructors drill into you day and night. Assume the "correct" position, develop the right grip, look to see what the muzzle does if you close your eyes, etc etc. But bigger heavier bullets will have a larger impulse, will accelerate more slowly and very likely will travel down a longer barrel, providing more opportunity for point of impact to shift compared to a 32gr Vmax going 4000fps.