I honestly have no problem with them as long as they are legal in the area and being ridden responsibly.
By that I mean the rider is aware of others using the area and the effect their machine (and it is a machine) can have on horses etc. and take precautions to ensure they aren't spooking the animals, or riding up on others trying to hunt/stock an animal etc.
I have zero tolerance for people using any wheeled machine illegally and will turn them in if I see it. I've taken pictures of riders (not e-bike specifically) if they are tearing up the trail or making new trails around mud holes etc. and turn that over to LE that has resulted in at least one case I specifically have firsthand knowledge of resulted in fines and lose of license.
I have and continue to use a variety of methods to gain access to different hunting areas based on what is legal. But I also have no problem parking a rig when the terrain makes it no longer safe, or ethical to continue without causing more damage or making a new trail around the obstacle and continuing my hunt on foot.
Unfortunately, we've all seen the few who think it's their right to do whatever they want even when it infringes on our rights, and they will push the limits until they are caught and sometimes that's not even a deterrent anymore.