This is the same BS argument that we used to see about spin drift, you can't see it shooting from the ground so don't worry about it, come to find out the more we worry about and try to narrow down the noise the smaller and more predictable we shoot.
Why not take everything out you can and just leave it to your shooting ability, that is really the goal of long range shooting isn't it, to narrow down the groups till your at your smallest most consistent. You'll never know what you can do till your the biggest part of the equation, if your rifle is shooting big how can you improve your skill.
I didn't get better than my rifle, I had awesome rifles that would shoot stupid small every time I shoot and cold bore was spot on every time, then I started seeing the patterns in ME and went to work on that and brought it down even farther. Continual improvement should be the goal!!