The very concept of a "hate crime" is nonsensical: it's perfectly legal to dislike a race or gender, in fact if the government tries to punish you for expressing these beliefs it's the one that's breaking the law by violating your 1st Amendment rights.

But if you are known to hold these beliefs and then commit a crime it becomes "more illegaler". This has the effect of chilling free speech, because if you make your views known and ever commit a crime against someone from your disfavored group they can add greatly enhanced penalties, and serves as a way to effectively criminalize disfavored speech.