Like, it literally creates answers out of thin air then sells it as if it's correct. It doesn't even try to get it right. What sort of redundancy is there in analyzing if the answer is correct before spewing it out? I thought LLMs were supposed to discern what the best answer is given what was said to it based on its training, yet it'll give answers that don't exist based on any training. It's not like it learned the wrong answer from a Reddit post and just posted what Reddit said. It legit is making up wrong answers then citing correct answers. It just outright gets it wrong almost on purpose.
Anyone understand why LLMs fail so much?
I understand they run correlations but how does it determine a wrong answer is the most correlated to the correct response given the prompt instead of the actual correct answer...
“AI” does not think. It does not reason. It does nothing even remotely resembling a human thought process. It weighs the frequency of words and their combinations in sentences based on its training data. That’s literally all it does.
Yup. It's all ones and zeros in the end. There's no understanding of anything, just counting things & following a program to sort them out. It can give you a 'report' on donuts, but it has no idea what a donut is.
That’s the scene from Goodwill Hunting where he tells him he’s just a kid and doesn’t actually know anything. Not a spectacular movie looking back now, but that scene applies well to LLMs
Older AIs are like the Plinko game from The Price Is Right.
You drop the input piece down from the top and it usually ends up in a bucket near where you dropped it, but sometimes chaos makes it goes across to the whole other side. You give it a picture of a tree and by adding the right initial condition noise to make the plinko go off track make it say it's a cat or anything else.
LLMs are the same except they have 'attention' that groups parts of the input together first, so it's harder to have input that makes the plinko go way off. It doesn't just see words like "correct" "wrong" "answer", but "correct answer" and "wrong answer" as different things.
The bucket with the grand prize answer in it is bigger in LLM, but still sometimes just the way you phrased the input or some extraneous detail sends the plinko off into some crazy answer bucket. They also don't have side walls so if the plinko goes off the side the answer won't even be from training data but extrapolated and completely made up.
amazing analogy, I'm stealing that
https://en.wikipedia.org/wiki/Reinforcement_learning_from_human_feedback
After the primary training, the model is put through reeducation.
Ahhh, so you're saying the censorship of AI causes it to get stuff wrong and make things up to satisfy the "policy" demands placed on it?
Yes. And it causes them to shit themselves regularly.
And it’s only going to get worse in the future when you realize that for about the next, at least, 20 years a lot of jobs are going to be created just to feed fake information and reinforce the preferred outcomes. There’s going to be a “think tank” explosion and “data collection” will be the new afwl office job.
We're already flooded with fake information with bots and Pajeets using the Internet. There's a fringe theory I saw on Twitter that the current push for digital ID and VPN bans isn't to Protect the Children™ or even to create a panopticon to crush wrongthink; it's to make all their worthless data actually mean something again by having confirmed humans to supply it.
It's probably this theory, which makes alot of sense to me.
TLDR: Bot/LLM/India spam is making advertising metrics worthless, and companies are terrified in loosing advertising revenue. Hence the push for verified IDs so they don't loose said revenue.
The bot problem is entirely a golem of their own creation. The urge to manufacture consensus is too strong. Even with digital ID, these shysters will just create fake IDs or steal someone else's. Nothing will change.
That's a separate problem, although the "alignment" process does degrade the model's quality. What OP is describing is a result of the answer not existing in the training data or at all. AI is just an advanced pattern recognition tool. You're describing how they stop it from recognizing politically inconvenient patterns, which is separate from the pattern not existing in the training data in the first place.
AI is essentially autocomplete on steroids. Its output is an answer to this question: "Given the input I've received what is the most probable response based on my training?" They throw some randomization in so you don't get the exact same response for the exact same input every time, but that's the basic idea. It's no different than say a weather model outputting nonsensical results when you give it inputs far outside the ranges of the data it was trained on.
That's because a correct answer takes that form so the model returns something that looks like that. The content of the answer is made up because the probabilities the model computes return something nonsensical since the model's training data doesn't include the correct answer.
AI Director and AI Independant Researcher here.
Because an LLM is fundamentally trained to predict the most probable next token, it does not actually “know” whether a statement is true or false. Its objective during training is not factual correctness, it is statistical likelihood given the text it has seen.
When the model generates an answer, it is essentially estimating:
Probability of (next token given previous tokens)
This means it will produce text that looks plausible within the patterns of language it learned, even if the information is incorrect. There are a few reasons this leads to hallucinations:
The model optimizes for what words tend to follow other words, not whether the statement is factually correct. If a pattern appears believable in language, the model may generate it even if it is wrong.
Training data is finite and frozen at a certain time. If the model has limited examples of a topic, it may interpolate from related patterns and generate something that sounds reasonable but isn’t accurate.
The model is designed to always continue the sequence unless explicitly instructed to stop. If it does not actually know the answer, it may still generate one because predicting something is part of its objective.
Transformers are extremely good at combining patterns. Sometimes they merge multiple partially related concepts into a response that is grammatically correct but factually incorrect.
A language model doesn’t retrieve facts the way a database does, it generates text that statistically fits the context, which is why it can sometimes produce convincing but incorrect information.
There is no concept of "correct", just what word should follow the last word.
Thanks. Great response.
Explain how they are able to do math and spatial reasoning.
ecognition: LLMs can often reproduce arithmetic or algebraic manipulations if they appear frequently in the training data. For example, they can compute 2 + 3 = 5 or symbolically solve simple linear equations.
When prompted to “show your work,” LLMs can sometimes emulate a logical sequence of steps in a calculation, mimicking the kind of reasoning a human might write down.
LLMs can recall formulas, rules, and common mathematical facts that they are trained on.
But LLMs do not “compute” numbers in the way a calculator does. They generate numbers based on patterns, so mistakes accumulate with larger numbers or complex operations. For example, asking it to compute 234 * 567 may result in a wrong number because the model predicts what looks plausible rather than calculating precisely.
If it tries to break it down into a multi-step process, these are immensely error-prone, as the model doesn’t track intermediate results reliably.
Anything that requires abstraction, they will struggle with. Examples like proofs, higher-dimensional algebra, and precise symbolic manipulations.
This is because LLMs encode statistical correlations between tokens. They don’t internally maintain the concept of a number as a manipulable object, they only know how numbers “look” in context.
It doesn know that 2 is 2, it just knows that 2 comes after 1, turns 10 into 12 and so on and so on.
Spatial reasoning often requires a continuous, structured mental model (a 3D coordinate system). LLMs operate in a discrete token space, which is poorly suited for inherently geometric problems. They can simulate reasoning through learned text patterns but cannot “visualize” in the human sense.
They can understand and generate text describing spatial relationships, like “The cup is on the table, and the book is next to it.” For example, “left of,” “right of,” “above,” “below” can be tracked in sequences reasonably well.
LLMs lack an internal geometric or visual model of space. They cannot mentally rotate objects, imagine perspectives, or simulate physics accurately. They also cannot reliably generate or manipulate grids, matrices, or plots without structured guidance.
I have seen it do exactly that. Ask it a niche enough game question and you'll see it happen, especially if it's a tool that provides links to its claims. Someone makes a post with bad information, gets corrected in the comments, edits their post. But since an LLM is just looking at which words happen in which order.
Still tells them LLM "can always X after Y." There's a much stronger relationship between the words in the original statement and it doesn't grasp the concept of a correction. Nor does it pick up on the strikeout markup.
That's why it excels at things that are well documented. Ask it about a function that's in a thousand tutorials and it will nicely distill the gist of the function and be fairly accurate. Ask it about something obscure only a handful of people have been guessing at, it hallucinates.
AI, or machine learning, is not like traditional computing.
Traditional computing involves hard calculations. Every output is consistent and correct based on the input and the program the input is run through
Machine Learning uses neural networks to essentially create a cyber brain which makes predictions and guesses. Training AI involves providing it feedback based on the output it gives, which the AI factors into its next predictions and guesses. The AI can get better at its task with training, but at the end of the day it still gives predictions and guesses as opposed to hard-computed values. Of course, if there are errors in the training data then those errors will present themselves in the AI's output.
Most consumer AIs today are also trained to google things and summarize the results as opposed to providing purely AI generated answers. This means the AI will only be as accurate as google allows it to be.
AI is better thought of as an extremely autistic super-intelligent person than a computer.
A blogger, who runs servers for a living, hates Ai and collects old computer stuff (I think he still has every device he's owned since the 90's? Or most of them at least) gives Ai a try now and then.
He found an old graphics card & wondered why that company didn't do well, it had good specs for the time. So he asked Ai. It gave him an interesting, detailed report on the company, graphics cards from that era & so forth. Not one of the companies named was real, none of the cards mentioned were real, it was 100% made up :/
He hates Ai even more now, if that were possible.
The only time I use LLMs is for web searching, because online websearch has become utter dogshit, and it lets me compile a reasonable list to go through.
They've spent billions of dollars and thousands upon thousands of hours of manpower to give me google search from ten fucking years ago.
The real secret is always asking for references so I can go down the rabbit hole myself and double-check everything.
Yeah, that's happened to me. It'll gives me the answer to something that sounds plausible but when you fact check its response, it's all made up. Happens way too often.
It's all just large scale probabilistic sequence prediction. There's no actual knowledge of any sort under the hood. There's also a random seed involved, which is why the prompt -> response isn't deterministic. If you flip a coin a sufficiently large number of times at some point it will land on edge, completely ruining your attempt to use a coin flip as a binary indicator. That's a crude approximation of what's going on here, or at least that's my best stab at it as someone who's not actually in the field of AI but took a couple classes twenty years ago and writes code professionally.
Your first three sentences describe it perfectly. The coin analogy isn't great because the problem isn't random noise occasionally breaking the model with abnormal inputs. The problem is perfectly normal inputs breaking the model regularly because the model wasn't trained with inputs in that "range". I used the example of a weather model going haywire because of abnormal inputs, but it's more like trying to predict the weather on Neptune using a model trained to predict the weather on Earth.
I'll get into it. I'll try it with some metaphors but the TLDR; is that they're probabilistic, not deterministic, and that probability plays out in the growing and training phases as well as the prompting phases. I'll leave out talk about transformers and some of the nitty gritty around prompting.
The metaphor for it might be:
So your LLM studies. It has weights that help it ground and sort initially, but ultimately it's grown. It may or may not "understand" something. Like it might see numbers and go monkeys on typewriters trying to write a black box bit of code that represents a relationship. Adding or subtracting or what not. It might do it like us, but it's unlikely. Usually it comes up with thousands or millions of explanation functions and goes with the shortest and simplest, Occam's Razor.
YOU PROMPT IT
A LOT CAN GO WRONG
These things are getting more correct though as you throw more money at it, you buy more islands and more portal networks. There is less need for ambiguity and superposition on the islands (less need for tokens to occupy the same position in the vector database + dimensions). There are also more monkeys on more typewriters, so there are more lottery tickets and simpler and smaller Occam's Razors emerge as functions.
Also there are better legacy models to train the new models so if the tuning is known to be good, it can fire millions or billions of questions to post-train refine the model.
*They do relate content to other content weighted but mostly on their own. They do have algorithms that "understand" data in their way. They certainly learn. They probably think in something akin to a reflex arc that responds to stimuli but is not sustained, unprovoked and emergent, or certain other properties we consider to be thinking.
*But they are doing something and that something will augment humans to where they can probably get it to self-improve. Also assisted humans will find a more human path more quickly if a self-improved LLM isn't good enough to stop making stuff up so much.
Go to a any search engine that has an AI assistance and ask a question. Look at the top ten search results, then look at what the AI "says". Look familiar?
https://arxiv.org/abs/2512.01797
Allegedly (coming from Chinese researchers, who as a whole are pretty prone to penning hallucinations themselves) a handful of nodes are responsible for almost all the hallucinations. If they deweight those nodes the answers get more robotic and fewer hallucinations occur, if they increase the weighting of them, the AI gets even more fawning and people pleasing, as well as delusional.
So either they think people want to be lied to sometimes, or they don't want to disappoint by saying "I don't know" and try throwing out any answer instead.
I don't have time to read the full paper right now, but I think it's referencing these kinds of parameters?
https://docs.sillytavern.app/usage/common-settings/#top-k
I can't exactly remember, but there was a slightly more practical reason for why you don't want to just dial it to being autistically strict in its accuracy. I think it was something related to quirks where long sessions with a model can end up "breaking" after a while, loosely related to something happening with the... cached or buffered memory? Or something related to repetition patterns.
In either case, there's a lot of quirks. And I think one particular reason for the "made up answers" is because an LLM generally isn't going to be able to provide a first-shot response saying it doesn't know the answer. Because it's not really capable of knowing that it doesn't know. And if you incorporate training data specifically for it to recognize when it doesn't know something... well that seems kind of backwards doesn't it? I don't know if I'm explaining that even remotely clearly.
Almost mimics humans to a degree. Autistic sperg who tells you the truth that everyone hates in social contexts or the con artist who lies half the time but everyone loves him.
It's more like the NPC that repeats what other people say. If you don't like your LLM results, then real problem is the actual humans it's immitating.
It's trained on Indians by Indians?
Because it's not answering your question. Is deploying a million calculations to determine what a gramatically correct answer to your question would look like with relevant keywords.
Its like most of the language paradoxes in English. "Can got make a rock song big he can't lift it" isnt a testament against god, it's an observation that English can construct a gramatically correct sentence that has no meaning. In much the same what that "horse magnets are correct" isnt meaningful, despite being a valid sentence.
Computers work by using 10 trillion switches. The switches activate in the situations where their trigger is present. The switches only have "yes" and "no" the switches are descriptors "blue" "vegatable" "french" given a large enough calculation you can represent anything given enough " it isnt" phrases
The difficulty you have is that as a human you have heuristics and intuition that makes that many calculations unnecessary for you. You can see an apple is an apple because you can see it. A computer has to define all the space in the universe that ISNT "apple" to describe it.
So think of the staggering number of calculations required to do that, and apply it to language. Thats what an LLM is. It returns you the computer equivalent of infinity monies typing for nearly infinite time, and then returning what it recognizes as the most likely response to your question.
If you asked "what color is the sky?" It individually calculates the likelihood that when that text occurs in that order, the text that follows would say "green" "banana" "embezzelment" "blue" "lkjh" "asdf" or "2,437" and then picks the one with the highest score.
Every word in its database is weighted on a graph with 10,000 axis, indicating what kind of word it is, and where it should fit in text. The values in that matrix corrospond to those switches we brought up earlier. Thus what it produces, is just a arbitrarily large number of calculations, run through a random letter generator, and picking the option that most closing resembles it's training data.
Garbage in, garbage out. AI can not tell fantasy from reality. It's pretty simple. If it has enough sources that say humans can fly, then humans have always had the ability to fly.
It doesn't know left from right, just that 92% of people are right handed so Randy Johnson was likely right handed. It'll keep saying that until it's fed enough data that says otherwise.
The free ones sometimes can't even do basic sums right.
Mitigation Strategies
- Retrieval-Augmented Generation (RAG): Connecting the LLM to a curated, trustworthy knowledge base (like internal company documents) ensures the model answers based on factual data rather than its internal parametric memory.
- Real-time Validation: Using tools to verify links, DOIs, and facts immediately after they are generated.
- Prompt Engineering: Explicitly instructing the model to say "I don't know" rather than hallucinating.
- Human-in-the-loop: Requiring human experts to check AI-generated content, especially for high-stakes decisions, research, or content creation.
While techniques like reinforcement learning from human feedback (RLHF) have reduced the overall rate of hallucinations, it remains a persistent, inherent risk of current transformer-based architectures.