I believe I have an interesting counter: OK, people of color can't be racist against whites, however they can be racist++. See, racism++ is the same as racism, except that people of color can be racist towards whites. This obviously makes it worse, so why aren't we doing anything to fight racism++?
They are necessary in some roles, for example strip-searching women.