A lot can be gained from taking lessons from lightweight backpacking. The two keys are don't take anything you don't need, and try to choose items that serve multiple purposes. I enjoy creating overall systems that work well together, and don't cost s fortune.
With the rifle, I would start by choosing a caliber that is efficient in a shorter barrel. Testing shows there is not much accuracy gained from a longer barrel, but a longer barrel has to have a heavier contour to remain rigid. If you can get by with a 24" #3 contour instead of a 28" varmint you will save significant weight there.
As far as needing weight on the rifle to shoot it accurately there is no need to carry extra weight. Put a butt stock pouch on it and carry your first aid kit there so you have it when you set your pack down to retrieve the game, or stick your water bottle in it when you are ready to shoot. Many people also carry Nalgene bottles, which weigh over 6oz empty, a regular disposable water bottle weighs less than an ounce.
You also might be carrying a pack that weighs 5-6 lbs empty, and its only job is just to carry your gear. The Flash 45 from REI weighs 2 lbs. If you have a spotting scope with a low enough zoom you may be able to leave the binoculars behind.
Evaluate every piece of gear you have, ask if you need it, if it can be made lighter, or if it can be replaced by something that could serve multiple purposes. You may be able to get down to a 9lb rifle and a 25lb pack.