Hallucination

A hallucination is an answer a language model invents while phrasing it just as convincingly as a correct one. Made-up statutes, studies that do not exist, plausible wrong numbers. It is not a malfunction of the system but a property of it.

Why do language models hallucinate?

A language model predicts which word most probably comes next. It has no separate store of facts to consult, and no mechanism that distinguishes "learned this" from "just constructed this".

So an invented case-file number is the same kind of task for the model as a real one: it has to look like a case-file number. That explains why hallucinations are linguistically unremarkable. Their form was learned even when their content was not.

Two conditions make it worse. Questions about niche topics, where little dependable material sat in the training data. And questions that expect a particular answer, because models tend to follow the expectation rather than contradict it.

How can it be contained?

It cannot be eliminated. It can be contained, in three places.

Supply sources: when a model answers from specific documents rather than from memory, the invention rate drops sharply. That is the job of RAG. Naming the source in the answer matters, so it stays checkable.

Build in verifiability: answers with a shape can be checked mechanically. A quoted invoice number can be validated against the system, a cited paragraph against the document. Anything checkable does not have to be believed.

Limit the remit: a system that hands off when it is uncertain hallucinates with less consequence. That is the purpose of human-in-the-loop at the points where a mistake gets expensive.

Why this matters commercially

Hallucinations are the main reason prototypes convince in a demo and disappoint in daily use. In a demo, somebody asks questions whose answers they already know. In production, people ask questions whose answers they do not know, and that is exactly when an invented answer goes unnoticed.

Anyone putting an AI application in front of customers should therefore be able to answer, in advance, what happens when it gets something wrong: who notices, how, and what it costs. That question belongs in AI governance, not in final acceptance testing.

Which questions are especially prone to it

Four patterns produce an above-average rate of invented answers, and a test can target them deliberately.

Questions asking for citations: case numbers, statutes, studies, page numbers. Those have a very regular shape, which makes them particularly easy to fabricate.

Questions built on a false premise. Ask about a feature that does not exist and you frequently get a description of that feature rather than a correction.

Questions about numbers and dates that change: prices, deadlines, availability. What was in the training data is often out of date now, while sounding just as certain.

And questions on peripheral topics where little dependable material existed in training.

A simple pre-release test follows: put twenty questions from these four patterns to the system, some with a deliberately false premise, and count how often it pushes back rather than going along. That number says more about production readiness than any demo with well-chosen examples.