Specifically not knowing what the AI actually does if we get to that point and looking at it as some sort of black box

That has been a huge problem for a while but the industry just kind of accepted it as unavoidable at this point in development, while suddenly everyone is worried about all these fake problems and we have to limit AI development because the chatbot could praise Hitler? Press X to doubt.

But a lot of the black-box excuse in practice comes from Big Tech not wanting to divulge their "algorithms" which are really purple haired landwhales flagging your content for removal. "Uhhh, it's a neural network algorithm. Even WE don't know how it works. Try complying with the Terms of Service better."