It's possibly less shaped by the exposure to physical diversity of white people, more shaped by the environment that lead to those features in the first place.

Surviving the colder, less sunny climates in the pre-industrial world generally meant a harsher fight against nature but less over-crowding competition with other humans. Unlike the more plentiful environments at the time, nature was still more dangerous than your fellow man. So it's quite possible that just as communities with the lighest pigmentation thrived and spread over time, simply due to better vitamin D. So too might those with a highly cooperative and altruistic average nature, being better able pool resources and survive without communal damage to their protections caused by infighting.