I have asked myself numerous times over the last few months...when is this good enough, just as you have. I have a few different perspectives, but all come down to the same decision drivers. 1. Either establish your expectations ahead of time (1moa, 1/2 MOA, or 1/4 MOA, or best it can be, or whatever), and those expectations should be aligned with what 'should be' possible. Examples for me are: Factory Rem 700, with sporter barrel, in replacement stock, with replacement trigger, I set my expectations at 3/4 MOA. When I got there, I realized that every load I tried was sub 1MOA, and 2/3 of them were sub 3/4 MOA. In that instance I told myself that if 8 different loads are sub 3/4" surely there is more potential with this rifle. So I picked a load that was in the velocity flat spot, and I loaded to 4 different seating depths, with only 5 rounds of each. I told myself before I ever left for the range that whichever one is best, is going to be the final load, period. The best one was sub 1/2", and I am done with it. So, I never tried any different primers because the first one I tried got me into the 'zone of acceptance', I never tried any other bullets, because this is a hunting gun, and the ones that I wanted to use were within the 'zone of acceptance'. I don't intend on changing/improving anything unless the components that I am using become unavailable, or if my intended use changes.
On the other end of the spectrum, I have a 300 Win Mag that I built strictly for the purpose of shooting at crazy long distances. I have established that I have to be sub 1/2" with the lowest possible ESs, and the highest velocity powder/primer combination. So I worked the load until I got there , and then I messed with seating depth, and called it good.
To the point that someone else made about shooting out a barrel looking for a load, I think that shooting out a barrel looking for a load, is only a problem if the goal is to be done with load development. While that sounds crazy, part of the fun of this hobby could just be developing loads, and trying different things, and increasing your understanding of the whole cause and effect aspect of reloading/shooting/rifle building. There is nothing wrong with tinkering around, if done safely. We have all benefited from someone that tried something that was unconventional, and got good results, and then told people about it. Just my .02!