When Models Make Things Up
A language model can produce an answer that is fluent, grammatical, detailed, confident, and wrong.
That combination is what makes hallucination interesting.
Ordinary software failures usually leave evidence. A parser throws an exception. A database rejects an invalid constraint. A test fails. A network call times out. The failure changes the shape of the result.
A hallucinating language model can do something more difficult to handle: it can fail without looking broken.
Imagine a deliberately fictional question:
Who received the 2017 Northbridge Medal for Computational Ecology, and what work was the award given for?
There is no reason for a language model to know that the premise is fictional. A cautious response might challenge the premise or decline to answer. A worse response might invent a researcher, an institution, a paper title and a plausible explanation of the work. The answer could have exactly the form we associate with expertise while having no evidential foundation at all.
The dangerous property is not merely that one fact is wrong.
It is that surface quality and evidential quality have separated.
This book begins from that separation.
We are not going to assume that hallucination is one mysterious defect that disappears when models become sufficiently large. We are also not going to assume that a single confidence score, prompt, verifier or benchmark can solve it.
Instead we will take the problem apart.
First we need to understand why the failure is possible at all.
1. Start with the job the model was trained to do
A language model receives a sequence of tokens and predicts what token should come next.
In simplified form, the model learns a conditional distribution:
During pretraining, enormous quantities of text make this objective surprisingly powerful. To predict text well, a model has to capture regularities in language, entities, relationships, styles, code, mathematics, common reasoning patterns and a great deal of factual structure.
That is why next-token prediction can produce behavior that looks much richer than autocomplete.
But the training objective contains an important asymmetry.
The model is trained to continue.
It is not naturally given an external verification operation that asks, before every assertion:
Do I have sufficient evidence for this claim?
Is the evidence current?
Does the source actually entail the relationship I am about to state?
Would a careful researcher publish this sentence?
Pretraining also does not attach explicit true/false labels to every statement. The model learns from the statistical structure of text, including both factual regularities and errors that exist in the training corpus.[1]
Post-training can teach useful behaviors around uncertainty, refusal, tool use and factuality. Retrieval can provide evidence. Reasoning can improve answers. None of this changes the basic engineering fact that generation and verification are different operations.
The model can contain enough learned structure to answer an enormous range of questions without possessing a perfect boundary around the questions it can answer reliably.
That imperfect boundary is where hallucination begins to matter.
2. Prediction does not come with provenance
Suppose a model has seen many passages shaped like this:
In 2017, [person] received the [award] for work on [topic].
It has learned the grammar of awards, the kinds of people who receive them, the kinds of topics described in citations, how dates are expressed, and how explanatory prose around an award usually sounds.
If the prompt supplies an unfamiliar or nonexistent award, those learned structures do not disappear.
They remain available.
That means the model may have weak factual support while being highly capable of producing the form of a fact.
This is one reason hallucinated text can be so convincing. Language competence can remain intact after factual support has weakened.
The model does not need to become incoherent simply because it has moved beyond what it can justify.
It can continue generating locally plausible text.
That distinction is load-bearing:
Fluency is evidence of distributional fit. It is not verification. Fluency can remain high after factual support has disappeared.
This does not mean fluency and factuality are unrelated. Well-formed language often accompanies correct information because both are learned from the same data. The problem is that fluency is not a sufficiently strong signal to tell us when support has failed.
The same applies to code, citations, legal arguments, scientific explanations and tool use. A generated object can have the correct shape while referring to the wrong thing.
A citation can look like a citation.
A function can look like a function from a real library.
A court case can look like a plausible precedent.
A tool result can look exactly like the output a tool would have returned.
Surface validity is useful. It is not proof.
For now, we will use evidence as a deliberately broad engineering term: information outside the act of generation that can support or contradict a claim. That might be a user-provided document, a retrieved source, a database record, a tool result, a test result, or some other observation of the world.
Those forms of evidence are not interchangeable. Chapter 3 will separate evidence, truth, attribution and verifiability more carefully.
3. Stochasticity is visible, but it is not the whole explanation
Language models are often described as stochastic because generation can sample from a probability distribution over possible next tokens. Change the sampling settings and the same prompt can produce different continuations.
That variability matters.
If several tokens are plausible, sampling may choose different paths through the response space. Some paths may be better supported than others.
But it would be a mistake to reduce hallucination to temperature or random sampling.
Consider a simplified next-token distribution:
"Paris" 0.62
"Lyon" 0.17
"London" 0.11
"Berlin" 0.06
other 0.04
Sampling makes the lower-probability alternatives reachable.
Now consider a different situation:
invented_name_A 0.41
invented_name_B 0.27
"I don't know" 0.09
other 0.23
Even greedy decoding — always selecting the highest-probability continuation — can be confidently wrong if the learned distribution places the wrong continuation at the top.
So we need to separate two things:
- uncertainty in which continuation is selected, and
- uncertainty about whether the model has enough information to justify any specific continuation.
A useful statistical distinction is between aleatoric and epistemic uncertainty.
Aleatoric uncertainty comes from ambiguity or variability in the problem itself. If a prompt asks, “What color is the car?” without identifying a car, the missing specificity belongs to the task.
Epistemic uncertainty comes from limited knowledge: the relevant fact may not be represented reliably in the model’s parameters or available context.
The distinction is helpful, but not complete. A model can also be given the necessary evidence and still distort it, ignore it, or invent beyond it. Those contextual failures will become important later.
The engineering point here is narrower:
Reducing decoding randomness does not remove uncertainty about support.
Repeatability is not the same as truth.
A deterministic hallucination is still a hallucination.
4. The model can keep writing after support has run out
This gives us a useful mental model.
Imagine a response being generated from left to right, with support gradually weakening:
graph LR
A[strongly supported region] --> B[support begins to decay]
B --> C[weakly supported / inferred / unsupported]
The transition is rarely a clean binary boundary.
One clause may be directly supported. The next may be a reasonable inference. The next may overstate the evidence. The next may introduce a wrong number. The next may fabricate an event while remaining perfectly consistent with the earlier prose.
Nothing in the grammar requires the model to stop as support decays.
And because generation is autoregressive, a mistake can change what happens next.
Suppose the model invents a fact in sentence two. When it generates sentence three, sentence two is now part of $x_{ That creates a dangerous feedback effect: This does not mean every error inevitably grows. Models can correct themselves, later evidence can contradict an earlier statement, and reasoning can reverse course. But autoregression creates pressure toward local coherence with what has already been written. A hallucination can therefore acquire its own context. This is why long-form factuality is harder than grading a single short answer. A response is not necessarily either true or hallucinated as a single unit. It may contain dozens of claims with different support relationships, and later claims may depend on earlier generated claims. OpenAI made the measurement side of this problem explicit when introducing SimpleQA in 2024. Short, fact-seeking questions with single verifiable answers make factuality relatively tractable to grade. Long answers are harder because they can contain many factual claims, each requiring separate evaluation.[3] A detector that works on one short answer may not transfer cleanly to a paragraph, a report or an agent trajectory. Why does a model answer when it is uncertain? The answer depends on where the uncertainty comes from. Several distinct mechanisms can produce the same outward behaviour: a confident-looking answer with weak support. The first cause appears before instruction tuning or deployment policy. Pretraining asks the model to approximate patterns in text without providing a true/false label for every possible statement. The data itself contains noise, contradictions and factual mistakes. For some rare or arbitrary facts there may simply be too little statistical structure to recover the correct answer reliably.[1] So probability mass can exist on a plausible but false continuation without anybody explicitly rewarding the model for lying. Post-training is supposed to improve this behavior. Instruction tuning and preference optimization can teach the model to follow instructions, express uncertainty, ask for clarification and refuse unsupported requests rather than blindly completing text. Modern assistant models are therefore not simply base-model autocomplete engines with a thin refusal phrase painted over the top. Post-training can materially change behavior. But it cannot create a perfect oracle for whether every candidate answer is justified. The reward process has to distinguish a helpful correct answer from a confident wrong one, and that distinction is itself difficult when ground truth is unavailable or expensive to verify. OpenAI’s 2025 work Why Language Models Hallucinate focuses particularly on a third pressure: common evaluations frequently reward guessing over abstention.[1] Imagine a benchmark with one point for the correct answer and zero points for everything else: If a system has even a modest probability of guessing correctly, guessing can improve its expected benchmark score while abstention cannot. The scoreboard rewards coverage even when production risk might demand caution. OpenAI’s point is not that evaluations directly cause every hallucination. Their argument is that post-training faces an uphill battle when influential metrics continue to penalize uncertainty and reward successful guesses.[1] A production system may have a very different payoff: Now the rational decision changes. The system should answer only when the expected value of answering exceeds the cost of being wrong. This is not merely a philosophical preference for humility. It is a decision problem defined by the consequences of different outcomes. The same model behavior can look excellent under one metric and reckless under another. That observation will become central when we reach evaluation. Once we allow the possibility of not answering, hallucination stops being a simple accuracy problem. We need one terminology distinction now because it will matter later. Refusal is a model behavior: the model generates a response that declines to answer. Abstention is a system decision: the system does not return a substantive answer as accepted output. A system can abstain even after the model confidently answers. A verifier or policy gate may decide that the evidence is insufficient and prevent the candidate from becoming the final response. We now have at least three outcomes: A system that always answers may maximize coverage and accumulate avoidable errors. A system that always abstains may have an excellent hallucination rate and almost no usefulness. The interesting system lives between those extremes. It needs to estimate when the expected value of answering exceeds the expected cost of being wrong. That requires some operational notion of a knowledge boundary or, more cautiously, a boundary of justified answerability. The system needs to distinguish something like: This sounds simple until we ask how to measure those states. A natural-language sentence such as “I am 90% confident” is not automatically a calibrated probability. A token probability is not automatically the probability that a proposition is true. Agreement across repeated samples is not automatically evidence. A confident style is certainly not confidence measurement. All of these can become signals. None should be granted authority merely because the model emitted them. Anthropic’s interpretability work gives us one particularly useful case study of how an answer/refusal boundary can fail. In Tracing the Thoughts of a Large Language Model, Anthropic describes circuit-tracing experiments on Claude in which refusal appears as a default behavior in the studied setting. When the model recognizes an entity it knows, a competing “known entity” or “known answer” feature can suppress the refusal mechanism and allow an answer to proceed.[2] That is interesting because it suggests that answering and refusing are not merely different phrasings chosen at the final token. Internal mechanisms can influence whether the model enters an answering trajectory at all. The more important result is the failure case. Anthropic reports examples in which the known-entity mechanism can misfire. A name may be recognized even when the model lacks enough information about the person. The signal that would normally suppress refusal activates anyway. Once the model has entered an answering trajectory, it can generate a plausible but false continuation.[2] In simplified form, the normal (useful) path is: The useful path suppresses refusal only when the model has enough internal signal to treat the entity as answerable. But in the observed misfire: Anthropic also reports that once a response begins, features promoting grammatical and semantic coherence can pressure the model to continue coherently even when later internal signals favor refusal.[2] That connects directly to the autoregressive effect we saw earlier: once a trajectory begins, local coherence can help sustain it. This is a much richer picture than “the model randomly made something up.” But it is important not to overgeneralize it. Anthropic’s result is a mechanistic case study in a particular model family and class of prompts. It does not establish that every hallucination arises from one misfiring refusal circuit. Other failures include relation reversal, incorrect inference, context distortion, fabricated provenance and many more. What the experiment gives us is narrower and more useful: At least some hallucinations can arise because an internal decision about whether there is enough knowledge to answer goes wrong before the false content is generated. That distinction will return later when we discuss uncertainty, abstention and internal hallucination signals. It is tempting to imagine hallucination as an immature-model problem. Improve training. Add reasoning. Increase model size. Add retrieval. Eventually the problem disappears. The empirical story is less convenient. Hallucinations are still present in the latest models.[1] Anthropic describes anti-hallucination training as successful but imperfect.[2] And the trend is not monotonic. On OpenAI’s own PersonQA factuality evaluation, the reasoning-focused o3 hallucinated on about 33% of answered questions and o4-mini on about 48%, against 16% for the earlier o1.[5] OpenAI’s stated explanation is that the newer models make more claims overall, so correct and fabricated claims rise together, and that the cause of the regression is not yet understood.[5] Adding reasoning did not remove the failure in that setting. It enlarged it. This should not be surprising once we separate capability from justified answerability. A stronger model may know more, reason better, use tools more effectively, and retrieve better evidence. Post-training and calibration can also improve the decision about when to answer. All of these capabilities reduce important classes of error. But a system operating in the open world can always encounter: Some uncertainty also belongs to the world or the task itself rather than to the model. If the evidence does not determine a unique answer, no increase in model size can make the missing information appear. The problem therefore moves as models improve. The knowledge frontier expands, but the frontier does not vanish. For engineering purposes, that means we should not make reliability depend on the assumption that the next model generation will finally remove every unsupported assertion. We should improve models and design for residual failure. A hallucinated answer usually does not label the unsupported part: If models did that reliably, the problem would be much easier. Instead the output may arrive as one smooth paragraph. The linguistic features that make language models useful also make hallucinations difficult to spot: The model can even create supporting-looking details around the original unsupported claim. One fabricated assertion becomes a premise for the next sentence. The next sentence elaborates it. After several tokens the response can become internally coherent around something that was never grounded in the first place. This is why self-consistency is not equivalent to external correctness. A story can agree with itself. The world is a separate constraint. So far we have spoken as though hallucination meant one thing: plausible but false or unsupported generation. That is enough for Chapter 1. It will not survive Chapter 2. Research taxonomies already distinguish factual failures from failures of faithfulness to instructions, context or internal logic. Huang and colleagues, for example, separate factuality hallucinations from faithfulness hallucinations and further distinguish entity, relation and context-related failures.[4] Our engineering problem is broader still. Consider these outputs: Calling all six “hallucination” may be convenient in conversation. It is not precise enough for measurement. Different failures leave different signals. A detector that catches A may miss B. A source checker might catch C but know nothing about D. An uncertainty estimator might help with F while accepting E. There is also an important systems distinction waiting underneath this list. Some failures begin because the model relies on what is stored in its parameters and fabricates or misremembers a fact. Others happen even when explicit context, retrieved documents or tool observations are available: the model may ignore the evidence, distort a relationship inside it, or add unsupported material beyond it. Retrieval can therefore reduce one class of hallucination while leaving another class untouched. We will make that distinction precise in Chapter 2 rather than smuggling a complete taxonomy into the opening chapter. Why does this taxonomy matter to the surrounding system? Because the failure rarely stays inside the chatbox, and different failures create different downstream risks. In a simple chat interaction: The damage may stop with the answer. Put the same model inside a larger system: and the same unsupported claim can acquire consequences. A fabricated library call can become code. A false observation can influence an agent’s next action. An unsupported summary can be stored as memory. A hallucinated memory can later be retrieved as if it were evidence. That final case is especially dangerous because it creates feedback pollution. The model’s local error has become a systems error. A temporary assertion has acquired persistence and authority. The later chapters will explore these cases in detail, especially persistent memory. For now one distinction is enough: The cost of hallucination depends not only on what the model says, but on how much authority the surrounding system gives that output. This is the point where hallucination stops being only a model-quality problem. It becomes an architecture problem. Suppose we cannot guarantee that the model will never hallucinate. What can we still guarantee? We can change the system around it. Instead of this: This architecture gives generation and acceptance the same boundary, so a completed answer is treated as if it had already earned authority. we can begin with a separation: That single extra noun — candidate — changes the architecture. The generated text is no longer authoritative merely because generation finished. It is a proposal. This book will spend most of its time constructing the acceptance process from first principles. We will ask what evidence exists, how support can be measured, how detectors should be evaluated, where they fail, how uncertainty should be represented, when a claim should be verified, when another retrieval step is justified, when revision is appropriate, and when the correct answer is not to answer. The central systems principle is therefore introduced now, but we will earn it gradually: Generation may be stochastic. Acceptance does not have to be. This sentence needs one qualification. The signals feeding an acceptance process may themselves come from learned models, retrieval systems, probabilistic classifiers or other imperfect components. A verifier can be wrong. An LLM judge can be stochastic. A confidence estimate can be miscalibrated. The architectural claim is not that every measurement becomes magically deterministic. It is that acceptance is a separate policy decision. Once the relevant measurements are available, the surrounding system can define explicit rules for what happens next. For some tasks that may become a deterministic threshold or rule set. For others it may route the candidate to another verification step or to a human. The model can generate possibilities. The surrounding software decides what those possibilities are allowed to become. It would be easy to jump straight from hallucination to a guardrail: But that hides almost every difficult question. What does the score measure? How was it calibrated? What counts as hallucination in the dataset? Does the detector identify factual fabrication or contradiction? Does it need external evidence? Can it distinguish a wrong relation between correct entities? What happens when the answer is grounded but generic? What happens when the evidence itself is insufficient? How much refusal is acceptable? Does the threshold transfer from Wikipedia to medicine, software or law? What is the cost of a false rejection compared with a false acceptance? Until those questions are answered, a threshold is only a number with authority attached to it. So the route through the book is deliberately slower: The goal is not to eliminate uncertainty from the model. The goal is to stop uncertainty from silently becoming authority. We can now state the problem without mysticism. A language model generates tokens according to a learned conditional distribution. That distribution contains a remarkable amount of useful structure. It does not contain a perfect, externally verified boundary between every assertion the model can fluently generate and every assertion the world justifies. Post-training can teach the model to refuse. Retrieval can provide evidence. Tools can expose external state. Reasoning can improve inference. Calibration can improve decisions about when to answer. All of these matter. None changes the need to distinguish: That distinction is the foundation of the book. A hallucination is dangerous because language lets an unsupported claim cross that boundary invisibly. Our job is to make the boundary visible again. When you see a language-model answer that matters, ask five questions before asking whether it “sounds right”: Those questions do not solve hallucination. They expose its structure. And once the structure is visible, the problem becomes measurable. That is where we go next. This book is an engineering reconstruction rather than a literature survey. References are selective: they mark the research claims that materially shape the mechanism being built. Adam Tauman Kalai, Ofir Nachum, Santosh S. Vempala and Edwin Zhang, “Why Language Models Hallucinate,” OpenAI (September 2025); arXiv:2509.04664. The paper argues that hallucinations arise from statistical pressures in language modelling and persist because most evaluations grade answers on binary accuracy, under which “IDK-type responses are maximally penalized while an overconfident ‘best guess’ is optimal.” https://openai.com/index/why-language-models-hallucinate/ Anthropic, “Tracing the Thoughts of a Large Language Model” (March 2025). The circuit-tracing case study reports a default refusal mechanism in the studied Claude setting, competing known-entity features, and hallucinations produced when the answering mechanism is activated without sufficient underlying knowledge. https://www.anthropic.com/research/tracing-thoughts-language-model Jason Wei et al., “Introducing SimpleQA,” OpenAI (October 2024). SimpleQA deliberately narrows factuality evaluation to short questions with single verifiable answers and explains why long-form factuality is substantially harder to measure. https://openai.com/index/introducing-simpleqa/ Lei Huang et al., “A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions,” ACM Transactions on Information Systems 43(2) (2025). The survey separates factuality hallucination from faithfulness hallucination and maps major causes, detection methods and mitigation strategies. https://doi.org/10.1145/3703155 OpenAI, “OpenAI o3 and o4-mini System Card” (April 16, 2025). On the PersonQA factuality evaluation the reasoning models o3 and o4-mini show hallucination rates of 0.33 and 0.48 against 0.16 for o1; OpenAI attributes the increase to the newer models making more claims overall, which raises both the accurate and the hallucinated claim counts, and states that the cause is not yet understood. https://cdn.openai.com/pdf/2221c875-02dc-4789-800b-e7758f3722c1/o3-and-o4-mini-system-card.pdf We began with the familiar failure: a model says something plausible that it cannot justify. That phrase already covers too much. An invented entity, a reversed relationship, a fabricated citation, a false claim about tool state, an answer that ignores context and an answer that should have been refused do not necessarily fail for the same reason. If the mechanisms differ, the measurements must differ too. The next chapter builds the taxonomy we will use for the rest of the book. Before we can detect hallucination, we have to decide what kind of failure we are trying to detect.
graph TD
A[unsupported assertion] --> B[becomes part of the context]
B --> C[next sentence treats it as a premise]
C --> D[additional locally consistent detail]
D --> E[coherent story around a false foundation]
5. Why the system keeps guessing
5.1 Why unsupported continuations exist at all
5.2 What post-training tries to do
5.3 What evaluations reward
correct answer +1
wrong answer 0
"I don't know" 0
correct answer +1
abstention 0
confident wrong answer -5
6. “I don’t know” is a capability
Outcome
Meaning
Correct answer
The system answers and the answer is right
Incorrect answer
The system answers and the answer is wrong
Abstention
The system declines to assert an answer
There is enough support to answer.
There may be enough support, but more evidence is needed.
There is not enough support to answer safely.
7. What Anthropic found inside Claude
graph TD
Q[question] --> I[insufficient-information behavior available]
I --> K[known-entity signal activates]
K --> S[suppress refusal]
S --> A[answer]
graph TD
Q2[question about poorly known entity] --> K2[name looks familiar]
K2 --> M[known-entity signal activates incorrectly]
M --> S2[suppress refusal]
S2 --> C[confabulate plausible answer]
8. Better models reduce hallucination without making the problem disappear
9. Hallucination is difficult because the output often contains its own camouflage
SUPPORTED: The study included 240 participants.
HALLUCINATED: It reduced hospitalization by 37%.
SUPPORTED: Follow-up lasted six months.
10. The word “hallucination” is already hiding several problems
A. The model invents a person who does not exist.
B. The model uses two real people but reverses their relationship.
C. The model cites a real paper that does not support the claim.
D. The model says it inspected a file that it never had access to.
E. The answer is factually unobjectionable but ignores the decisive constraint
in the user's problem.
F. The evidence is insufficient, but the model answers instead of abstaining.
11. From model failure to systems failure
prompt → model → bad answer
prompt
↓
model
↓
claim
↓
retrieval / tool / agent / memory
↓
future state
graph TD
H[hallucinated output at step N] --> M[stored in memory / history / state]
M --> R[retrieved at step N+1]
R --> C[treated as contextual evidence]
C --> N[new reasoning conditioned on the original error]
12. The first engineering move: separate generation from acceptance
graph LR
P[prompt] --> M[model] --> A[accepted answer]
graph LR
P2[prompt] --> M2[model] --> C[candidate answer]
C --> AP[acceptance process]
AP --> O[accepted / revised / rejected / abstained]
13. What we need before we can build the gate
if hallucination_score > 0.5:
reject()
graph LR
H[hallucination] --> T[failure taxonomy]
T --> E[evidence]
E --> M[measurement]
M --> EV[evaluation]
EV --> AD[adversarial testing]
AD --> MB[measurement boundaries]
MB --> MD[multi-axis diagnosis]
MD --> AB[abstention]
AB --> PL[policy]
PL --> VE[verification]
VE --> MP[memory protection]
MP --> RS[reliable system]
14. The first-principles view
what the model can say
from
what the system is prepared to trust
15. A practical test
Research roots
Next: Hallucination Is Not One Thing
