This is both accurate and inaccurate. Travel was possible, and even common, in certain contexts. Pilgrimages, for instance, involved people from as far away as England making the trek to Jerusalem (though, much more common, was simply to Canterbury). BUT the difficulty and expense were astronomically high. A pilgrimage to Jerusalem was a life-long goal and/or luxury, and someone would have to plan for decades and arrange their entire lives around an absence of a year or longer to make the journey.

Without some huge incentive driven by religious fervor, the likelihood that someone would make such a sacrifice is very very low. Black Africans, as a whole, had almost no reason to even imagine traveling to Northern Europe for most of history, with the only exceptions being hard-core travelers like pirates or sea-traders, and they did not migrate but simply pass through. So it's fine to depict the unicorns of blacks in, say, London or other port cities, but in the context of visitors and ocean nomads, not permanent settlers.