The above sentiment is true only if the rifle shooting the "big heavy well constructed bullet moving at high velocity" isnt a direct contributor to all the less than perfect shoots.
We all know people who flinch just thinking about touching the trigger on their hunting piece, a lot of those people are responding to this thread, most if not all of them would never admit it though... this leads to less practice and chronic poor shoot placement.
I hunt with a group of about 15 guys every year, in that group we have one guy who can shoot an un-braked 300Win Mag with the steady hands of a brain surgeon... and another shooting a 7 mag has difficulty bringing in deer even at 100yds and convulses like someone having a seisure if you do the test where you put a dummy round in his magazine while shooting.