So, I took my dies up to the gunsmith and had him set them up, apparently I had them adjusted wrong. They are RCBS dies, after adjusting the dies we loaded 3 rounds and tried it, the primers still moved out but not near as much as before. We also shot some Remington 140 gr. factory ammo and had the same result. So, if this rifle wasn't my Dad's and I didn't see deer and bear piled up over the years I'd stop fixing it now. But I'm going to have the barrel removed and the action checked and if possible I'll be hunting with it next year.
Thanks for all the insight and information.