Thanks for the response. One thing I was told by a builder is that while lower to upper fit generally does not make a difference, when we are talking sub .5 MOA, it matters.
I checked the Aero upper to the Aero lower I have. Unfortunately, it's a somewhat sloppy fit in the front, leaving a lot of lateral movement. My Anderson lower is actually about 4 or 5 thousandanths tigher. I checked an HK 416 lower for comparison, and it's probably 16 thousands tighter or so.
So now I am looking at lots of variables. Looks like the front lugs on all the upper receivers I have are about the same, it slop in the lower that is showing up.
Anyway, I am going to research options on how to fix it, looks like a shim can probably do the trick, but I did find a builder that is going to true the upper and be the whole thing for a decent price, so I'm going to do my best to cover all the possible variables.