siegrisj, questions;
What comparator tool are you measuring with?
What outside diameter do your sized necks measure, and then your loaded necks?
How do you set your die in the press for seating?
This might help.
As mentioned ogive radius variance leads to differing datum distances between seater plug and comparator tool. But what you measure with a comparator is closer to leade contact datum, so this is what your seating needs to be taken to. There really is no getting around this, so accurate seating is not a set-n-forget operation, even with hand dies(which eliminate press tolerances).
You might consider Hornady's 'MicroJust' accessory for your die:
Hornady Manufacturing Company :: Reloading :: Metallic Reloading :: Dies & Die Accessories :: Die Accessories :: Miscellaneous Die Accessories :: MicroJust™ Seating Stem 1 Each
It replaces the adjustment plug at the top of your die & provides micrometer adjustments to your seating. Wit this you can sneak into same seating based on your comparator measure.
Don't fret over the rubber bushing in your die as this just preloads threading to prevent relaxing of settings. It's a good thing.
I use Wilson hand dies for all seating, with micrometer adjustments on them. This is about as accurate as it gets, but I do need to check every single OgvOAL, and occassionally I need to re-seat with a tweak of the mic setting. Sometimes, hatefully, I have to hammer pull the bullet and seat another(something wrong with that one anyway).
Quality consistent bullets really pay off here. Custom benchrest bullets practically take you to set-n-forget, but check anyway.
For a comparator I use a Sinclair 'nut' and caliper. I've tried a few other types but the nut just works best for me. It's chrome plated 10thou under cal holes give me consistent readings.
I had zero luck with the Stoney Point style aluminum caliper attachments.