Totally normal.
I had a MKV in 340Bee, never did shoot as well with the brake on as I would have liked.
Sold it and built a 340Bee on a switch barrel Winchester Model 70 Stainless classic action. Has barrels in 300/340/375Bee. Shoots itty bitty little groups with all 3 barrels.
Love Weatherby cartridges, just not their rifles.
Make a choice, shoot with brake on all the time, or take it off. I've never found the 340 to be harsh in recoil. The 375….well that's a whole different animal.
Cheers.