Let us begin with what a large language model actually does, because everything follows from it. An LLM does not consult a database of facts. It « knows » nothing in the sense an encyclopedia knows. It has learned, over trillions of words, to predict the most plausible next word in a given context. That is all — and it is immense: from this single mechanism emerge synthesis, translation, apparent reasoning, style. The strengths are real and spectacular.
But the mechanism has an exact reverse. When the answer exists abundantly in the training data, the most plausible word is also the truest. When it does not — rare data, recent, internal to your organization, or simply nonexistent — the model does not stop. It goes on producing the most plausible word. It fills the void with plausibility. A figure that looks like a figure. A reference that looks like a reference. An industry study that looks like every industry study it has ever read.
Hallucination is not a breakdown of the machine. It is the machine working exactly as designed, on terrain where the designed no longer suffices.
A compounding factor follows: tone. Models are optimized, after their initial training, to produce answers that humans judge satisfying — useful, fluent, assured. The result: invention comes out with the same poise as established fact. No hesitation in the voice, no usable signal of doubt. A human who does not know stammers; a model that does not know asserts.
This is why surface remedies disappoint. Connecting the model to documents (the celebrated RAG) reduces invention but does not eliminate it: the model can misread, over-extrapolate, conflate two sources. Asking it to « check its answers » amounts to asking the suspect to run the investigation. And models genuinely improve — rates fall version after version — but a falling rate is not a null rate, and in a 400-claim deliverable, 2 % invention makes eight false statements, invisible to the eye, distributed to the client under your signature.
The conclusion imposes itself: if invention is structural, protection must be structural. One does not fix a property — one contains it. This means a layer independent of the model, one that takes up each claim produced, confronts it with qualified sources, and returns a verdict that was not drafted by the same mechanism as the text it judges. In cybersecurity, no one asks the code to self-certify: one audits. Generative AI arrives at exactly the same point of maturity.
Two answers to one property: Evidence audits deliverables after the fact, claim by claim; and the enterprise AI we build can only answer from a base of verified sources — it cites where each answer comes from and says « I don't know » rather than embroider. See both →