It's not just the slave trade. It's also that they introduced the good part of modernity (like railroads, rule of law and representative government) where their government structures were sound (indirect rule was quite a different matter).

And if you compare empires to nation states, it's not at all clear to me that nation states are superior. Clearly, they have greater viability, or we'd have more empires and fewer nation states, but a great deal of suffering has been imposed on people to create these nation states - expulsions, forced assimilation, language politics. There are people in France, and all other European countries, who have been made second-class citizens due to their native language - which is not a language brought by invaders, but the language (as naturally evolved) that their forefathers spoke, whether Occitan or Flemish. Empires generally respect nationalities, even as they also have dominant nationalities.