There will be essentially no difference at all. This should be generally obvious but I guess without the math people just don't see it. Simple physics.
recoil energy = (MV*((bullet weight in grains/7000)+1.75*(powder charge weight in grains/7000)))^2
weapon inertia = 64.348*rifle weight in pounds
recoil force remaining = recoil energy / weapon inertia
Recoil is a function of the mass of the powder, the mass of the projectile, the mass of the gun and the velocity of the projectile. Unless you can make a BIG bloody difference in what any ONE of those numbers is doing WITHOUT THE OTHER NUMBERS INCREASING are you will not appreciably affect recoil.
So if you have 180 grain bullet and 77 grains of powder at 2900fps in a 10lbs gun there's total kick of 26lbs/ft/sec. No surprises there. Thing is, smaller bullets tend to be a good bit faster and energy comes from squaring speed not mass so mass changes aren't as helpful as speed changes. So, you can't just go a little lighter to get a noticeable drop in recoil energy. If you drop to a 168gr bullet and pump the speed as it will naturally tend to do, to say 3050fps, then the recoil force actually goes up to 27lbs. If you don't change the velocity upward and leave it at 2900 then you see 24lbs of free recoil. This means that you've made a difference which you will not be able to detect. The small difference is because of the small change in projectile weight. The difference in bullet weight has to be a lot to make a noticeable change. So if you were to go with say, 90gr bullets even though they might be running something silly fast like 3600fps the recoil force would be reduced noticeably to something like 21lbs, as noticeable as the difference between a .308 and a .30-06 with equal weight bullets.