IME going from 24-22 you'll love between 50-75 fps total, we've done actual tests of cutting barrels down. If it'll feel right and balance well and be shootable for you I'd do it in a heart beat. You'll pick up a bit of blast, that would be of more concern for me than a few fps. But, that's just me. My 7WSM has a 23" barrel on it and it's just fine.
When it comes to speeds it's always important to look at data taken from one barrel, IE barrels that were actually cut down. Too many variables when it comes to different guns that have different barrel lengths to be any kind of a basis for how much will be lost.