Swarthy just means darker. And in a region where parts of the country dont experience daylight for a month every year people are going to be very white, so anyone slightly darker than them will be 'swarthy'. The vikings also raided as far as the Mediterranean and even Greenland, so they would come home pretty tanned.