Try Berger 168gr VLD Hunting bullets and see if your problem goes away?
I spent months and many dollars trying to get the 162 ELD-X to shoot in my 7mm Rem mag to no avail. I checked scope, rings, bedding, shooting fundamentals, hand loading techniques, etc. On occasion I would get .75" groups but mostly 2.5-3" groups at 100 yds!
One day I thought I would try my old Berger handloads and shot .33"! Every group was under .75".
With some seating depth tweaking I'm shooting .35" fairly consistently.
I've talked to 2 other guys at the range that were getting 2-3" groups with ELD-X bullets also.
I know some can get them to shoot and that is great but they didn't work for me.