I'd be bedding that stock and reinforcing it or better yet changing that stock out if I lost zero by putting it on a bipod or taking it off a bipod. I had a shift on my 308 with a cheap cheesy plastic stock from loading the bipod heavily vs letting it sit on the bipod. It was less than an inch but too much, I bedded and reinforced the stock and that problem is gone. Same zero off a pack, off bags, off the bipod, offhand too. I have other rifles with plastic stocks that didn't have the same issue, even though they are just as flimsy feeling.
Your rifle obviously shoots well and your load is solid, I'd be all over it finding out why it changes like that. Maybe you can borrow a high end bipod and see if that helps? I use an accutac, but have used Harris as well and no issues but for your rifle to lose zero and still group well there is something weird going on. It would drive me nuts.