I think a custom/semi-custom is definitely the way to go and would fully encourage you to take on the task of doing the work yourself. However, I think the guys that are masters of the rifle building craft, guys like Kevin Cram at Montour County Rifles, would probably take some offense to statements like, "How hard can it be?" or any of us thinking that we can put together a rifle that would match the quality of their work. I know I don't have the skills or tools to put together a barreled action and keep everything lined up within .0002" concentricity. Sure, most of us here can buy a Savage rifle, unscrew the factory barrel, screw on a quality aftermarket barrel with a go gauge in the chamber and probably get a rifle more accurate than our abilities if the barreled action is properly bedded in a solid stock. But for me, that still isn't the same as getting custom built rig made from the best components available put together by a good gunsmith. But that's just me...as I said, I don't have the tools to properly true a factory action and install a barrel keeping everything within .0002" the way a rifle from MCRifles would be done, but your tools and skills could be very different than mine.
If you want to do it yourself, your best bet is probably finding a decent deal on a factory gun that has most of the components (action, stock) you want, rather than buying everything individually. For example, I think it would be more budget friendly to buy a Savage 110 FCP or Rem 700 Long range, pull the barrel, put a Savage or Remage barrel on using the barrel nut system, and bed the action with your favorite compound. You'd probably save some dollars doing it that way than buying an action, a stock, bottom metal, etc. individually, unless you are finding great deals on all those things, or if that's the only way to get exactly what you want.
You could also buy something like a 700 LR and take it to your favorite rifle smith to have the action blue printed, new barrel installed to the specs of your choice, and have him do the bedding work.
If you decide to do it yourself, remember to factor in the cost for the tools you will need if you don't already have them. It will be worth the investment if you are either going to go through barrels quickly or will be putting together multiple rigs.
Just my .02. Keep us posted on what you decide, looking forward to seeing the end result.