Horses do cause damage to trails and even more so when it's wet but so would thousands of buffalo if they were still around. I ride as often as I can which is not nearly as often as I would like and as will any group, there are individuals that try to follow the rules and behave but there are always a few that cause problems and damage trails more than necessary.
To answer you question about extremist groups, yes, there are others out there that are causing trouble. There have been reports of a crazy woman hiding behind trees and jumping out at horses & riders as they approach then screams at them for being on the trail. That is, if they're still there. This is not only uncalled for but extremely dangerous for both horse & rider and should this fool meet the wrong people, I imagine it'd be pretty dangerous for her as well. There is talk of bringing a lawsuit against this idiot and the forestry officials are looking into the complaints. This is in southern/central Indiana that this has happened.
In Indiana, we pay $35/year to ride in national forests and $20 for state. I believe bicyclists pay the same. Hikers pay nothing yet they use the same parking lots, picnic tables & facilities as do the rest of us. They also drop and leave their share of garbage on the trails. Can you tell this is a bit of a "saddle sore" for me??