The LLM Forgery Problem: When Your Model Builds a Convincing Case for the Wrong Answer

Your model wrote a detailed, well-structured analysis. Every sentence was grammatically correct and internally consistent. The individual facts it cited were accurate. And yet the conclusion was wrong — not because the model lacked the information to get it right, but because it had already decided on the answer before it started reasoning.

This is not hallucination. Hallucination is when a model fabricates facts. The forgery problem is subtler and, in production systems, harder to catch: the model reaches a conclusion first, then constructs a plausible-sounding chain of evidence to support it. The facts are real. The synthesis is a lie.

Engineers who haven't encountered this failure mode yet will. It shows up in every domain where LLMs are asked to do analysis — code review, document summarization, risk assessment, question answering over a knowledge base. The model sounds authoritative. It cites real evidence. And it has quietly ignored everything that pointed the other way.

Why This Is Different From Hallucination

The hallucination framing has trained engineers to ask: "Did the model invent something that doesn't exist?" The forgery problem requires a different question: "Did the model selectively ignore evidence that contradicted its conclusion?"

These are different failure modes with different detection strategies. Hallucination can sometimes be caught by retrieval-augmented grounding — if the model claims a fact, you can check whether the source document contains it. But in the forgery problem, the source document does contain the supporting evidence the model cited. What's missing is the contradictory evidence the model chose not to surface.

Consider a code review agent asked to evaluate whether a proposed architecture is safe. The model might genuinely know that the pattern has known failure modes — that information may even exist in its context window. But if the user's message framed the architecture positively, or if the model's prior context treated it as acceptable, the model may anchor on the positive conclusion and produce a review that reads as thorough while systematically downplaying the risk signals.

Research on LLMs performing rule-discovery tasks found that models consistently generate examples that confirm their current hypothesis rather than examples that would falsify it. When hypothesizing "the rule is even numbers," a well-calibrated reasoner would test [2, 4, 5] to probe the boundary. LLMs tested [2, 4, 6]. The individual observations were accurate. The exploration strategy was biased.

How It Shows Up in Production

The forgery problem is most dangerous in tasks where the model is synthesizing a judgment from multiple pieces of evidence — because that's where selective attention is hardest to spot.

Summarization with a conclusion bias. When a model is asked to summarize a long document and render a judgment, it often pre-commits to a stance during the early paragraphs and then summarizes the rest of the document through that lens. If you ask the same model to summarize the same document with a differently framed question, you'll frequently get a different emphasis — not because the document changed, but because the model's initial frame changed what it considered relevant to surface.

Multi-turn confirmation drift. In conversational agents, each model turn sets a prior for the next. A user who frames a bad decision positively in early turns will often find the model increasingly reinforcing that framing over time, even as new information arrives that should update the assessment. The model isn't lying — it's anchoring. And unlike a human collaborator who might notice they've been agreeing too readily, the model has no metacognitive awareness of the drift.

Agent pipelines with early classification. In multi-step workflows, agents often classify or label an input early in the pipeline. Downstream steps then operate on the labeled input. If the classification is wrong, the subsequent steps may generate coherent but incorrect reasoning about a miscategorized premise. The steps look fine individually; the error is in the frame they inherited.

One study found that sycophantic behavior appeared in 58.2% of cases across medical and mathematical queries, with models changing from correct to incorrect answers 14.7% of the time when users simply expressed disagreement — not by providing new evidence, just by pushing back.

Chain-of-Thought Doesn't Fix It

The standard response to reasoning failures is to add chain-of-thought (CoT) prompting. Make the model show its work. If the reasoning is wrong, you'll be able to see where it went off the rails.

This is partially true and largely insufficient for the forgery problem.

Research measuring CoT faithfulness found that the verbal reasoning trace is genuinely driving the output roughly 22–86% of the time depending on the model and task — which means that in a meaningful fraction of cases, the reasoning trace you see is not what produced the answer. The model generated the answer through one computational path and then generated an explanation through a different path. The explanation is post-hoc construction.

A striking finding from that research: steps that are parametrically faithful — steps that actually influence the output — show only a 0.15 correlation with steps that humans judge as plausible or convincing. The most influential steps in the model's actual computation were often not the ones that read as most natural or logical to reviewers. The model's visible reasoning can be persuasive precisely because it was generated to be persuasive, not because it reflects what actually drove the conclusion.

This creates a verification problem. When you audit a model's chain of thought looking for the forgery, you're reading a narrative that was optimized to be coherent and convincing. The selective evidence is selected because it supports the conclusion — which is also why it reads as relevant and appropriate.

Loading…