RL-16 works well. I went through the same stuggle figuring out varget isn't great with heavier bullets. I shoot mostly 130 class bullets in my 6.5 CM's and varget is showing to work really well for me. I do have very short barrels though, I use 16", 18" and 20" rifles.
I am using retumbo in a 28 nosler and it works great. Especially if you are shooting a longer barrel. I am running a 26" barrel and shooting 180's right at 3000fps.
The only time I've ever seen a significant change caused by primers was when I tried CCI BR primers instead of magnum primers in my FTR rifle. The magnums worked much better. It went from a sub moa rifle to a 36"-48" rifle at 1000 yards. It was severely worse.
I use 130 ELD-M on white tail and have had outstanding success so far. I shoot a variety of 6.5 bullets and have for about 5 years now and I keep loading them for the rifles that shoot them well.
If you open to other options, I HIGHLY recommend Whidden die sets. They are extremely nice and have been showing to be very accurate. Precise seating depth is a little easier as well with larger indicators on the knob.