Because for a long time it would literally destroy you, same with sexist/misogynist. You would get completely ostracized and often times outright attacked.

I moved out of the South in high school to a more "progressive" area and the thought that I was racist because I had that accent was enough to get my ass kicked a few times despite me not saying anything and there being no niggers around to even be racist toward to begin with (which I wouldn't have been as my hatred of them hadn't formulated yet).