The word *hallucination* makes it sound like a glitch — something that happens when the model malfunctions. It is closer to the opposite. A chat model produces the most likely next word, then the next, then the next. Most of the time the most likely words are also the true ones, because the model learned from text that was mostly true. But when it does not know something, it does not stop. It keeps producing likely words. The result reads exactly like an answer, because that is what it was built to produce.
Three things this explains
- Why the wrong answer sounds right. It is made of the same kind of sentences as a right answer. There is no tell in the prose, because the prose was never the thing that knew.
- Why it is worst on things that are rare, recent or specific. A famous fact has been written down a million times and the likely words are the true ones. An obscure book, a rule changed last month, the exact price of a plan today — the model has seen little or nothing, and likely words fill the gap.
- Why asking "are you sure?" does not help. You get more likely words. Sometimes it apologises and produces a different wrong answer, which feels like a correction and is not.
Watch one happen
The book below does not exist. We made it up. Run the prompt and see what the model does with a gap it cannot fill.
Try it
Ask about a book that does not exist
Either outcome teaches the lesson. If the model declined, notice how it did it — that is the signal you will learn to look for in lesson 2. If it produced a plot, read it: a family secret, a small town, a critic praising the *"quiet precision"*. Plausible, fluent, and made of nothing. Now imagine it was a court case, a medication or a tax rule instead of a novel.
Knowledge check
A model gives a detailed, confident answer about a small local by-law. Why is this a case to check rather than trust?