Since every rifle is unique some will be more affected by primer change than others. It also depends on the potential accuracy of each rifle. The bottom line is, if you can prove - to yourself, at the very least - that one primer has better numbers than other primers, it's pretty clear that's what you should be using. If you have the numbers, it's not opinion, it's fact.
I suspect that where problems may insert themselves is in the consistency of each brand of primer. I know they all strive to make each batch the same as the last one, however some may be better at it than others. Plus it could depend on the makers 'standards' as well. If you're loading on that ragged edge of pressure then it might be wise to do some testing each time you're forced to change lots of both powder and primers. While I don't have numbers to prove it, I suspect that powder has more potential to effect the bottom line than primers. Then again, if you're trying eke out that last 0.01" of accuracy, primers certainly could play a role. Test, test, test is the only way to prove to yourself that the individual lots are the same. It's what keeps many of us at the reloading bench!
Cheers,
crkckr