Like, it literally creates answers out of thin air then sells it as if it's correct. It doesn't even try to get it right. What sort of redundancy is there in analyzing if the answer is correct before spewing it out? I thought LLMs were supposed to discern what the best answer is given what was said to it based on its training, yet it'll give answers that don't exist based on any training. It's not like it learned the wrong answer from a Reddit post and just posted what Reddit said. It legit is making up wrong answers then citing correct answers. It just outright gets it wrong almost on purpose.
Anyone understand why LLMs fail so much?
I understand they run correlations but how does it determine a wrong answer is the most correlated to the correct response given the prompt instead of the actual correct answer...
https://arxiv.org/abs/2512.01797
Allegedly (coming from Chinese researchers, who as a whole are pretty prone to penning hallucinations themselves) a handful of nodes are responsible for almost all the hallucinations. If they deweight those nodes the answers get more robotic and fewer hallucinations occur, if they increase the weighting of them, the AI gets even more fawning and people pleasing, as well as delusional.
So either they think people want to be lied to sometimes, or they don't want to disappoint by saying "I don't know" and try throwing out any answer instead.
I don't have time to read the full paper right now, but I think it's referencing these kinds of parameters?
https://docs.sillytavern.app/usage/common-settings/#top-k
I can't exactly remember, but there was a slightly more practical reason for why you don't want to just dial it to being autistically strict in its accuracy. I think it was something related to quirks where long sessions with a model can end up "breaking" after a while, loosely related to something happening with the... cached or buffered memory? Or something related to repetition patterns.
In either case, there's a lot of quirks. And I think one particular reason for the "made up answers" is because an LLM generally isn't going to be able to provide a first-shot response saying it doesn't know the answer. Because it's not really capable of knowing that it doesn't know. And if you incorporate training data specifically for it to recognize when it doesn't know something... well that seems kind of backwards doesn't it? I don't know if I'm explaining that even remotely clearly.
Almost mimics humans to a degree. Autistic sperg who tells you the truth that everyone hates in social contexts or the con artist who lies half the time but everyone loves him.
It's more like the NPC that repeats what other people say. If you don't like your LLM results, then real problem is the actual humans it's immitating.