I generally keep shooting long enough to determine a possible load chain the rifle looks to like, and get the sights where I want it. With the black powder guns, most will gradually move poi,(usually downward), with moderns, the jury is still out, but your method of fourty or fifty rounds before fine tuning poa sounds like something worth practicing......especially if most modern rifles exhibit this same thing.
Maybe it is time to rethink the " three shot groups, clean, and do it over and over cold and clean 'till it stays put" method.