If you don't feel like reading a book, skip to the bold print.
I think it depends on the level of confidence you want to have going in to it that you are going to have an excellent rifle. I have put together factory rifles in premier stocks, triggers, ect., Semi custom's where the action was trued and had a premium barrel installed, premium stock, bottom metal, trigger, and so on, and then full on customs.
MOST factory rifles in fancy dresses (premium stocks, bedding, and trigger) I have worked on, after a better stock and bedding and trigger work, .5-1 MOA @100 yds usually consistently achievable, in some rare cases a little better. I have had a couple that simply would not do better than 1.5 MOA consistently, that is the headache you gamble on with a factory rifle. Is it the crown? Is it the Bore? Chamber? Action? Have you just not found the right combo of components? As stated, it is a headache to deal with at times, and generally ends up with a new barrel and trueing job, moving it into the next catagory.
With both semi customs and full customs, I have not had one that I couldn't get sub half MOA groups out of, that includes extended ranges in good conditions. They have all been relatively easy to find good loads for, with poor loads still generally hanging around MOA or just over, and with proper tuning and good shooting form can get into the .2's or less quite often. Now, from a factory action with a good trueing job, and maybe a couple other odds and ends done, in the accuracy department I have not seen a ton of difference from full customs. If I was into BR type stuff, that is where I think full customs with long barrel tenons and overly robust actions would hold value over a worked over factory action. But with guns in the 7-10 lb range shooting chamberings in the 6.5x284 class and bigger and used for hunting, I am generally happy with groups in the .2's-.3's, which if you have a experienced smith do your work, that is achievable with factory actions.
The difference in my opinion between the worked over factory action and full custom action, is more of an overall feeling and performance in the finer levels. The actions generally run smoother, feels tighter in the raceway, and cuts down on overall slop. It's like comparing a 1970 Chevelle with a hot rebuilt 454, tight rebuilt track suspension and sexy paint job, to a 2019 Camero. They both have fantastic performance, may run the quarter at the same speed, but when the road gets really twisty, that Camero just hugs the corners a little better. Also, the Camero may cost a lot, but you put nearly the same amount of money into that sweet looking Chevelle that was once just a rust bucket in someones pasture to get it up to nearly the same performance. But on the other hand, that Chevelle is yours, and built to your spec, so there is some pride in that.
Overall, I think a factory gun with some bolt on's can perform alright, sometimes surprisingly well. But other times, they can be a total flop. A semi custom built by a reputable qualified smith will pretty much always guarantee stellar performance, and will generally be cheaper than a full custom, but not by a huge margin. A full custom doesn't really gain much in the accuracy division for the majority of users needs, however it adds an overall feel of quality, and if absolute top tier performance is a requirement, it's the way to go. Just this guys thoughts.