This is a great topic.
Before you get too excited or worried, lets look at something.
Think of your rifle with the bolt closed and the striker pulled to the rear. The sear is bearing against the cocking piece and all that spring pressure is trying to drive the striker forward. The bolt has the lugs locked up at the 6 and 12 clock position.
The cocking piece is roughly at a 45* angle so the forward pressure exerted by the spring is trying to drive the bolt up and away from the sear. So what are your lugs doing? The bottom lug is being mashed against the receiver as the top lug is being pushed away. The amount of distance between the two is determined by how much clearance there is between the bolt body and bore of the receiver.
You can machine and lap lugs for the rest of your life and never solve this problem.
Not until you resolve the clearance issue and even then it'll only help so much because you must have some room for the bolt to cycle.
The only two actions I know of that address this issue specifically and account for it are the ones produced by Jim Borden and Nesika Bay Precision. Even still, there are plenty of other high end actions still shoot just as well without the "Borden Bumps". I am a former production manager for Nesika and I have great affection for the product, so know that it's hard for me to admit that fact. But it is the truth. A hundred sloppy old Remmy's and Win M70's on the firing line at Camp Perry proves it.
If your having accuracy issues, this would be the last place I'd get too excited over.
Here's a simple check list for a stubborn rifle.
1. Hold the thing in your hand at its balance point.
2. With your free hand, smack the barrel and listen/feel for a nice "ring" that goes away on its own.
3. What you don't want it doing is sounding like a cracked church bell. That indicates something is touching someplace that it shouldn't. Not very scientific, but it does work. I do it to every rifle I build as part of the QC.
Next test.
Make sure your action screws are tight. If it's a Remmy, I wouldn't go past 40inch pounds. Many will raise eyebrows to that and feel its too low a figure. I've done the research side by side with a man who makes a very nice six figure income designing nothing but aerospace fasteners for Boeing. I have the paper and the findings to qualify the statement. (But it's 7000 miles away in SD right now and I'm in Baghdad Iraq)
Conduct a visual inspection.
Is the crown damaged?
Did something happen to the bore as a result of overzealous cleaning rod usage?
Are the bases tight?
The rings?
Are the rings out of alignment and causing a bound up scope?
Is the scope out of parallax?
Is the scope serviceable or has something internal gone south?
Is the magazine box "floating" between the floor metal and the receiver? If you can't reach into the loading port and wiggle it up and down the rifle will never shoot right.
Are the guard screws binding at all when snugging up the barreled action, floor metal, and stock?
Is the stock inletted deep enough to where the recoil lug isn't bottoming out, preventing the receiver from fully "nesting" in the stock?
I'd go over all this before even batting an eye at the lugs not touching. use a dab of clay or even a piece of chewing gum to determine if the lug is bottoming out.
Assuming everything above is right:
Go buy some good quality factory loaded ammo. Make sure the bullet weight is matched to the twist rate on the gun.
This takes your reloading out of the picture. Think of it as a "second opinion".
Fire some groups. If the result is a nice tight group, then you know that your reloading practices need some revision. If its still a scatter brain, then it's may be time to weigh some more expensive options. (barrel, scope, receiver work, etc. . .) If it shows some improvement, then I think you can tune the gun with careful hand loading.
Good luck.