What Is an AI Hallucination? Why Models Invent Facts

An AI hallucination is a confident, fluent, factually wrong output. Not a glitch or a crash: a sentence that reads exactly like the true ones around it and happens to be invented. The fluency is the problem. A model that failed obviously would be easy to work with. A model that fails in the same register it succeeds in requires you to check everything.

The useful thing to understand is that hallucination is not a bug that got left in. It falls out of how these systems are built and, more specifically, how they are graded.

The mechanism, briefly

A language model predicts the next piece of text given everything before it. It is not looking anything up. It has no database of facts to consult and no internal flag that distinguishes "I have seen this many times" from "this seems like the sort of thing that would be true". The output is produced token by token, each one the most plausible continuation given the last.

Plausible is not true. Those two usually coincide, because true statements are common in the training data and false ones are less so. When they diverge, the model has no mechanism that notices.

That explains why hallucinations cluster where they do: specific citations, exact numbers, dates, names of people associated with a niche thing, API methods that ought to exist. In every case the shape of the answer is highly predictable and the content is not.

Why models guess instead of saying they do not know

This is the part that is often left out, and it is the more interesting half.

OpenAI researchers Adam Tauman Kalai and Ofir Nachum argued in Why Language Models Hallucinate that the cause sits in training and evaluation incentives rather than in the architecture alone. Models are optimised to score well on benchmarks. On most benchmarks, a wrong answer and an abstention both score zero, while a guess has some chance of being right. Under that scoring, guessing dominates saying nothing.

The comparison the authors use is a student on a hard exam. If a blank is worth the same as a wrong answer, you never leave a blank. The model learns exactly that policy, and it learns it thoroughly, because it is rewarded for it across the entire training process.

Their proposed fix is on the evaluation side: reward appropriate expressions of uncertainty rather than penalising them, and abstention rates improve. That matters for where this is heading, but it does not help you today. Today you are working with models trained under the old incentive.

Four kinds, and what each responds to

Treating hallucination as one phenomenon leads to one blunt mitigation. Splitting it up gets you further.

The last one is the sneakiest, because the explanation is generated to fit the answer rather than the answer being derived from the explanation. An articulate justification for a wrong conclusion is not evidence of anything.

What reduces it in practice

Five things, roughly in order of effect.

Give it the source material. By far the largest improvement available. A model reasoning over a document you supplied hallucinates far less than one recalling from training. That is the entire case for retrieval-augmented generation, and it is why the first question about any AI answer should be "what was it looking at".

Let it say it does not know. Explicitly permitting "I do not have this information" changes behaviour, because the default assumption baked into the model is that an answer is always expected. It is one sentence in a prompt and it is worth adding.

Ask for uncertainty inline. Requesting that unverified claims be marked as such produces a usable map of where to check. This overlaps with prompting the model to check its own work, which catches some errors before they reach you. Not perfect, and much better than a uniformly confident wall of text.

Narrow the question. Broad questions invite synthesis, and synthesis is where invention creeps in. Specific questions with a small answer space are checked more easily and fail less often. Most of the fixes in how to fix a bad AI prompt work partly through this channel.

Verify anything that carries a number, a name, or a link. The failure rate on those is high enough that treating them as unverified by default is simply correct practice.

Where it bites for people building things

Code is a special case worth calling out. A hallucinated function name fails loudly at runtime, which is the friendliest possible failure. A hallucinated assumption about how a library behaves fails quietly, in production, three weeks later. Why AI writes code that does not work goes through the specific patterns.

For anything customer-facing, the question is not whether your model will hallucinate but what happens when it does. A support assistant that invents a refund policy has created a commitment somebody now has to honour or explain away. The mitigations are boring and effective: ground answers in your actual documents, keep a human in the loop for anything that binds you, and log outputs so you can find out what was said.

Anywhere the answer becomes a record, the retention and access questions in giving AI access to your data apply too, since the log is now your data.

The same evaluation-incentive problem shows up visually too. See how to tell if a photo is AI generated for the current visual tells and provenance signals worth trusting more than your eyes.

Frequently asked questions

Why is it called hallucination?

The term borrows from perception: the system produces something detailed and coherent that does not correspond to anything real. Some researchers prefer "confabulation", which is more accurate, because the model is filling a gap with a plausible construction rather than perceiving something absent. Hallucination stuck.

Do newer models hallucinate less?

Generally yes on well-covered topics, and the improvement is uneven. Better models are also more persuasive when wrong, which can make errors harder to catch. Reduced frequency is not the same as reduced risk.

Can hallucination be eliminated completely?

Not with current architectures. It can be made rare and, more usefully, made visible. Grounding in retrieved sources, requiring citations, and permitting abstention move a system from unpredictably wrong to predictably checkable, which is the achievable goal.

Does asking a model if it is sure help?

Slightly, and less than people expect. A model asked to check itself often produces an equally confident second opinion built from the same faulty basis. External verification against a real source works. Self-assessment mostly does not.

Is a hallucination the same as a wrong answer?

Not quite. A wrong answer can come from bad input, an ambiguous question, or outdated information. A hallucination specifically involves generating content with no basis in the source material or training, delivered with the same confidence as a grounded answer.