Ryan,
Try to load up some accubonds. The factory Scirocco loads are seated very short. With the accubonds you will be able to get the bullets out near the rifling and in the factory throat should help tighten up groups.
Sadly enough, this is about the level of accuracy Remington will accept in their rifles. With careful handloading you should be able to get groups under 1" as long as the bedding and trigger are in good shape.
As far as a long range factory caliber, there are few better then the big 300 RUM.
My opinion, get rid of the Swift bullets and load up some nosler Accubonds in either 180 or 200 gr form. Powders to use are H-1000, Rl-25 or H-Retumbo.
Good Shooting!!
Kirby Allen(50)