It's the same reason why we see even 4x strategy games getting frequently dumbed down.

The main issue with modern Civ is that the AI can't play or pose a challenge within its own environment.

The bare-bones vanilla base game drops and the AI is dogshit, but also represents the pinnacle of PvE play because every subsequent DLC and game update adds new mechanics that the AI simply can't handle.

The player ends up either having to avoid new content or use house rules to keep things even remotely balanced.