The problem I see is that it can only help with solved problems. Anything novel is going to result in wild hallucinations. With everyone becoming reliant on it, the muscles needed for solving problems atrophy and now novel problems become Herculean tasks that mere mortals are no longer equipped to handle. Then we start spiraling.

It's the old problem of, no you're not going to need to remember or use most of what you learn in school, but you needed to learn it all in order to get good at learning new things because that's an important life skill. We're setting up to abandon that skill.

Also as we stop solving novel problems, the pool of knowledge to feed into the AI's training data fails to keep pace with the emerging realities of the world so it can no longer solve things that emerge. ChatGPT can only solve weird issues with Windows registry keys because 20 years ago there was a small army of nerds furiously blogging about it. Now that everyone's relying on ChatGPT no one is writing the blogs to fuel the training data for twenty years from now.