Because People of Color gives the insinuation that they are unified and it notably only excludes one race to my knowledge which is whites.

While colored people gives the insinuation that they are colored but not necessarily unified and it includes all but whites.

Basically it's power semantics one makes one side sound more normal/correct than the other, But they're every bit as guilty of the "microaggression" they accuse the other side of doing as usual.