I disliked him from episode one. He didn't get enough time getting his smugness beat down to justify his defection, and not enough on screen showing any "good parts" of his personality to care when he did.

So the fact that the majority of the story revolved around him, his family, and attempting his "redemption" drug it down.

Even Toriyama figured it out, where most of DBZ is Vegeta getting literally shit on over and over until it feels like he has paid for his sins, making his defection seem justified, and a gradual increase in showing his personality having good parts to it, mostly revolving around his family.