I've always loaded all of my ammo in a 14' x 24' building separate from my house. I've always stored all of my powder, primers, and a large majority of ammo in those building. While it isn't climate controlled, it is fully insulated, I do have a window unit A/C mounted thru the wall, and use a propane heater to warm the shop in the winter if I'm loading.
I have never had any issues with temperature changes in side the shop affecting any of my powder or primers.
I know the Fire Chief of the local VFD, and he's aware of what I keep in that building. I've told him if there's ever a fire, just stay back and let it burn. And watch me cry!