Like with everything, it's the economy, stupid.

Always is, always was, always will be.

Also that a lot of people didn't own slaves, because it turns out feeding/housing people isn't cheap, and poor farmers can't afford it (they'd turn out to love tech making their lives easier).