I anneal after every shot fired before the next sizing. I find this helps with consistant neck tension.
Low ES/SD numbers don't always coincide with great accuracy, barrel harmonics make a huge difference to group size regardless of how small the numbers are.
With powders that are borderline for the cartridge expansion ratio, they often show a huge ES but shoot little groups. This can ONLY be attributed to barrel harmonics matching the huge velocity swing in the oscillation of the barrel, when it is still or near to anyway.
There's more to precision loading than just looking at the numbers.
Cheers.