This question has come up before. And it's always the same debate.
Carrying a marginally lighter rifle on a backpack hunt is irrelevant with respect to the overall weight of the pack. We're essentially talking about shaving off 3 lbs, or 6% of the overall weight. Granted, the weight is significant to the overall rifle weight. But, if I gave you two packs, one weighing 50 lbs and the other at 47 lbs, you'd have a difficult time determining which was the heavier pack. Finding 15% or 20% weight savings is where you'll experience a noticeable difference in the backcountry, which means you'll need to sacrifice more weight than just your rifle.
I spend most of my elk and deer season hiking between 9500 and 12000 ft in some of the most rugged country in Colorado, putting in the same 6 – 7 miles a day, sometimes even longer. I think most who hunt, or have hunted, public land in Colorado will agree that taking quality game requires a lot of time in the backcountry. A number of years back I had the same notion of using a lighter rifle for my backpack hunts. So, I invested in a lightweight rig totaling about 9 lbs. (7 lbs. rifle). My previous rig weighed in at just over 13 lbs. There was little discernable difference with respect to the overall carrying weight. There was absolutely no difference in the quality of my hunt. The terrain still kicked my ***.
My current LR hunting rig weighs in at just under 12.5 pounds. I couldn't stand shooting a 300 mag from a 9 lbs rifle. I think it had more to do with the muzzle break than the weight/recoil.
Depending on where I hunt, and if I'm glassing beyond a mile, I might bring my spotting scope (65mm), with a compact tripod. In all other situations my 24x rifle scope is sufficient. The one thing I learned a long time ago is to ditch the hunting branded backpacks, and stick to the technical packs. You'll typically save a few pounds depending on the pack, and they are almost always more comfortable to carry. As others have said, I'd look inside your pack to see where you can save weight.
Unless you're willing to invest $8k - $10k into the newest lightweight hunting gear, you might just be better off every season offering $500 to a friend to help haul your gear!
From this years self guided hunt. Muley taken near Ute Peak, CO at 10,500 Ft. (612 yds / 35 degree decline).
The long haul out!