If you can hit a paper plate size target at that distance consistently you will be fine with either of those calibers. These numbers are straight of hornady's website for 40 smith 165gr and for 45acp 200gr+p load.
Muzzle 50 100 1175/506 1066/416 990/359
Velocity/Energy 4" barrel with the 40, and 5" with the 45
Muzzle 50 100 1055/494 982/428 926/380 If they'll stop a person from getting to you at a distance of 50yd, they'll stop a coyote, no if ands or buts, about it. If you can make a good shot on the dog he will go down, i'd be willing to bet he'll drop.
I remember reading about the MP at Ft Hood who shot and killed a gunman at around 50yd with a 9mm ball rnd which is a 115gr bullet generating only 417ft-lb at the muzzle.
For the record a 40gr v-max from a 22-250 at 400 yd has 410ft-lb of energy, and i've personally seen yotes killed with that load from further that dropped in their tracks...