It's not manifestly unjust to "judge or treat a person according to immutable characteristics". You could even argue it's preferable, depending on context. We don't intuitively feel outrage at slavery or genocide or racial caste systems either. All of the above is learned behavior passed down through culture. Widespread moral opposition to those things (rather than simply hating when it's done to our in-group) is a modern phenomenon brought on by globalization, the spread of Christianity, and expansion of our "circle of empathy."

I only point this relativism out because it's important to realize that our culture is something special that could be completely subverted and turned on its head. There is no limit to the depths that humanity can fall if we don't protect what we consider moral and just, which has been built over generations of civilization.