I just figured in my impersonation that they're called People Of Color because you have to put "people" first, meaning "White people" is just as bad as "Black people" since it puts people second, And you call them People of Color because the electromagnectic spectrum reflects only a portion of the rays off their skin, giving their skin "color", while the closer to "white" you get, the more light you have reflected off your skin. It made sense as the "opposite" term.

I also made sure to say People of Colors, plural, because to assume they're all one color would be exclusionary and not very inclusive-PC.