Try it. Open your favourite AI assistant, ask a precise question, note the answer. Open a new conversation, ask exactly the same question. The answer will be different. Sometimes in form, sometimes in a figure, sometimes in substance.
This is not a malfunction. It is how these systems are designed.
A classical computer program is deterministic: the same inputs produce the same outputs — today, tomorrow, in ten years. That is what makes it auditable: you replay, you compare, you observe.
A language model does not work that way. It does not consult a base of facts: it extends a sequence of words, choosing at each step among the most probable continuations. And that choice includes a deliberate element of chance, tuned by its designers. In conversation, this randomness is a quality — it brings variety, naturalness, momentum. It becomes a problem the day you ask the same system to state what is true.
One might object that a setting exists for this: "temperature", which supposedly needs only to be turned down to zero for the model to always pick the most probable word, hence always the same one. Measured reality says otherwise. A study published in ACM Transactions on Software Engineering and Methodology submitted identical programming tasks to ChatGPT, temperature at zero, several times each: depending on the benchmark, 18 to 44% of tasks still produced different outputs from one run to the next. And in September 2025, Thinking Machines Lab — the laboratory founded by Mira Murati, OpenAI's former chief technology officer — ran the demonstration that marked the community: the same prompt, submitted a thousand times to the same model at temperature zero, produced eighty distinct answers. Their analysis identifies the mechanism: server load changes the size of the batches of requests processed together, which changes the computation path, which changes the result. In other words, your answer partly depends on other people's traffic. OpenAI does not claim otherwise: its own documentation describes the reproducibility of its outputs as "best effort".
A machine that never answers the same way twice cannot, on its own, serve as a reference. Its decision cannot be replayed. It cannot be summoned.
The second phenomenon is less known, and more serious for organisations.
A language model has no inner boundary between what you give it and what it has learned. When you submit a document, it does not read it as a closed file: every word of your text awakens what the model saw of the same kind during training — millions of pages that do not belong to you.
Your internal project is called Meridian. The model's world contains dozens of Meridians: companies, funds, software products, ships. Ask for a summary, and the model may complete your Meridian with facts belonging to the others — a revenue figure, a date, an executive. The sentence will be fluent, plausible, properly built. Nothing will signal that half of it comes from your file and the other half from its memory.
That is the most dangerous hallucination in the enterprise. Not the crude invention you spot with a smile — the silent aggregation, where your vocabulary primes memories that have nothing to do with you. The more specific your field, the rarer your terms, the more treacherous the neighbourhood.
The problem is well known in research, and three public works mark the state of the art. In 2022, a Google Research team (Wang et al., published at ICLR 2023) showed that having the model produce several independent reasoning paths, then keeping the majority answer, clearly improves accuracy — nearly eighteen points gained on the GSM8K arithmetic benchmark. In 2024, Microsoft described in a paper the service that monitors its own products in production: it detects the passages of a summary that contradict the summarised document, and has them rewritten. In 2025, a team at Amazon Web Services (Goel et al., published at EMNLP) proposed confronting models of different architectures with the same question, the gap between their answers serving as a warning signal. Full references are at the end of this essay.
This work is serious, measured, deployed. And it shares one limit that must be named, because it outlines what remains to be built.
Consistent is not true
Detecting that a summary contradicts its document is a check of internal consistency: is the produced text faithful to the supplied text. An indispensable question — and an insufficient one. A summary can be perfectly faithful to a document that is wrong. A reasoning path can be the majority among ten passes and false in the world.
The other question is: is this claim established? Not "does the model agree with itself", nor "do the models agree with each other" — but: which external sources, dated and identifiable, support or contradict what is written. Consistency is checked between the text and itself; veracity is checked between the text and the world. Almost all of today's tooling addresses the first. Organisations, however, sign on the second.
Then comes the temptation of the stamp: true or false. It is misleading, and the reason deserves to be spelled out.
In a real professional document, some claims are established by concordant sources; some are contradicted; and some — often the largest share — sit in between: sources exist on both sides, or no source settles the matter. A system that only knows true and false must file that in-between somewhere, and wherever it files it, it lies — through overconfidence in one direction or the other.
The third verdict is not cosmetic caution. It is the one that says: evidence exists on both sides, or evidence is missing, and the decision belongs to a human being — with the reason for the referral written down and kept. The European AI Act requires effective human oversight; that verdict is what "effective" means once translated into software: the human is not an alibi at the end of the chain, but the designated arbiter of the cases the machine acknowledges it cannot settle.
What we guarantee
At (Urs), we built Evidence on these observations, and here is what we guarantee — in observable effects, not promises.
Every claim in a document receives one of four verdicts: verified, unverifiable, contradicted, or referred to human arbitration with its reason. Every verdict cites its sources — identified, dated, consultable. Nothing is ever rewritten: the original text is preserved to the character, and the decision to correct belongs to its author. And every analysis is archived and replayable identically: two runs of the same file produce the same finding, despite the variance of the models involved — we measure it, run after run.
How do we hold reproducibility on components that, by nature, do not provide it? Part of the answer is the subject of intellectual property filings in progress, and we will publish it when they allow. What we can say today fits in one sentence: we never ask the model to be reliable — we built around it the conditions that make its work verifiable.
An anomaly can only be detected against a reliable background. That is true in medicine, true in auditing, and true for the knowledge of organisations: before amplifying, you must be able to prove.
This is the (Urs) wager: an infrastructure where every claim receives its verdict and its sources, where nothing is rewritten, and where the analysis stays archived and replayable identically. Explore the infrastructure →
Sources. Xuezhi Wang et al., "Self-Consistency Improves Chain of Thought Reasoning in Language Models", arXiv:2203.11171, Google Research, ICLR 2023. Song Wang et al., "Developing a Reliable, Fast, General-Purpose Hallucination Detection and Mitigation Service", arXiv:2407.15441, Microsoft, 2024. Aman Goel et al., "Zero-knowledge LLM hallucination detection and mitigation through fine-grained cross-model consistency", arXiv:2508.14314, Amazon Web Services, EMNLP 2025. Shuyin Ouyang et al., "An Empirical Study of the Non-determinism of ChatGPT in Code Generation", ACM Transactions on Software Engineering and Methodology, vol. 34, 2025 (arXiv:2308.02828). Horace He et al., "Defeating Nondeterminism in LLM Inference", Thinking Machines Lab, September 2025 (thinkingmachines.ai). Regulation (EU) 2024/1689 (the "AI Act"), Articles 12, 14 and 19.