I dont load too much for the 30-06, but I do load for dozens of other cartridges all the way around it. It seems to me that you've put a few roadblocks in your own way. I can appreciate needing to use the powder you have, so in keeping with that I would not be so stringent on getting top shelf velocity. Slower nodeas are often more accurate than faster nodes, in my experience.
Also, again, I don't load much for 30-06, and certainly not for competition, but the bullet you mentioned is not one that I really ever heard described as a winner in the accuracy world. I could be wrong, but I would consider a different bullet that is know for accuracy, amd eliminate that as the possible cause. If you want to sample a few before making a big investment, I know of a fee great members that might send you a few to try, if you ask.
Definately need to shoot the best brass you can afford. If Win and Rem are it, then I would definately put the work in on a really good prep and sort. Flash hole debur and uniforming, volume sort, anneal, precision trim for length, and consider using an expander mandrel if your interference fit is in excess of .002" .
If you need clarification on any of that, just reach out..