The majority of my riflles are built or purchased for backpack hunting. Less than ~ 10lbs all decked out. Barrel contours on the smaller/lighter side compared to many rifle builds used by LRHs.
I can't think of a single reason smaller contour barrels will be as stable and provide consistently equivalent precision, than a similar quality larger contour barrel.
I expect to have to work harder to find a load that shoots well from a lightweight barrel, compared to a heavier contour barrel. I also believe powder-puff capacity cartridges are more apt to shoot well in smaller contour barrels than will the larger capacity cartridges. .223 Rem chambered barrels often shoot nicely in many plain-Jane lightweight factory rifles. 300 RUMs not as consistently well.
Just my experiences and opinion... Others will disagree.
Having stated that, you can often find a nice tomato in a bushel of discarded tomatoes. I think it boils down to 1) how demanding and critical is the precision required for your application and 2) how much time and effort are you willing to invest in meeting and maintaining that precision?