It could be your brass, and specifically the neck thickness and uniformity. I get a lot better results when not turning necks with quality brass like Lapua etc.
It could be your seating die. I prefer Redding comp seaters.
It could be your sizing die expander ball causing friction and pulling your necks uneven. A lot of people either don't use the ball or use collett dies. On my dies with balls, I use a floating carbide ball and it works great.
It could be out of square case mouths or uneven internal bevel/chamfer on the case mouth.
I know some people are anal about making sure the inside of the case necks are clean before seating. I run mine over a brush chucked up in a drill press just to be sure.
It may not make a difference but if it makes you feel better then do it.
There's a lot of things to check but you may reach a point of diminishing returns if you are getting really picky and shooting a factory gun. But like I said, if it makes you feel better then do it. It's only time involved.