I anneal for every loading. I condition the primer pocket and deburr the inside of the flash hole the first time I prep the brass, then don't touch them. Chamfer the case mouths with a VLD deburring tool.
BUT I found that different neck tension will drastically change ES. So now I use the Redding Imperial Dry Neck Lube instead of the Mica lube. I use a wire brush on my case prep center to remove carbon, then use that lube. ES readings dropped, last time I checked I shot 5 rounds, velocity varied from 3219 to 3226 fps. In previous testing I have seen a 50+ fps increase just due to tight necks when seating the bullets.
Biggest change in ES was seen by switching from Mica to the Imperial Dry Neck Lube. I tumble my brass in untreated corn cob media after sizing the brass too.