Racial diversity always leads to accusations of racism, imaginary or not.

Humans had the solution to that for millenia : your people survives and thrives by living with itself. Interractions with others can be positive, but if you want to keep existing as a people, you need to live in your own place.