Are your shooting precision rounds in a precision rifle?
If the answer is a regular off the shelf hunting rifle, with large tolerances in the chamber, then annealing may make diddly squat difference to your loading and accuracy. Annealing is for brass that gets rotated and shot A LOT.
Sorry to say it does not always need to be done, sloppy neck tolerances, .006" or more, will still overwork your brass every time it's sized.
Annealing may work to get good neck tension, but in a rifle that's MoA or worse, is it worth the trouble?
I only anneal my precision brass every firing, my hunting brass gets annealed every 5th load, and even then it splits necks occasionally.
I think it's necessary only for brass that you can't replace, or for really accurate rifles that get shot a whole bunch.
Cheers.