This is something you see often nowadays, especially on social media. Activists get some controversial policy approved, or censor some already-existing content. Yet if you protest it, you get the "dude why do you care" fence-sitters. They'll tell you that it doesn't matter that much and that both sides are silly anyway. Often these people are in complete support of said policy/censorship, but if you point it out they'll just tell you that those extremists don't represent them (because they're the enlightened "centrists", of course), and you get nowhere.
I feel like this is a big problem. How do you deal with this?
It's gaslighting plain and simple. Don't let it derail you. If they ask you why you care, double down on your original message and explain again why you think the topic you're discussing is bad or will have negative consequences. That, or make fun of them.