Civ is one of those games where the beginning is always the funnest part, and by about half way most people lose the steam to finish. Something it shares with a lot of CRPGs and SPRGs, plus a few others.

You start at equal footing with the CPU and have an abundance of choices in front of you, with a lot of opportunity for organic and unique gameplay to emerge. Even if they are retarded or cheating, lots can happen. By the time everything stabilizes, all the open land is settled it just becomes a slog of either you've already won but need to wait 100 turns to achieve it, or the opposite.

You aren't really ever going to fix that. Even if the CPU get much better, then they will just roll you because they are AI making perfect decisions. Focusing on doing so is a losing fight, as any change to do so (as you described) ruins the game for most people who are playing it just for the "promise" of challenge that isn't fun.

The only solution would be to make multiplayer less cumbersome, so that people are actually willing to sit and play it to the end. But I can't think of a way to do that, even my own family never plays more than 2 sessions on any new map.