The Chain-of-Thought You Stripped to Save Tokens That Hid an Evidence Requirement

A platform team shipped a prompt refactor that cut average response cost by thirty-two percent. The change was simple: strip the "explain your reasoning" preamble, ask the model to return only the JSON object, and drop the post-processing step that parsed the rationale out of the model's prose. The dashboard turned green. The unit economics page in the quarterly review went from yellow to gold. Nobody on the platform team thought to consult the risk team, because no part of the change touched the answer the customer received.

Two quarters later, a regulated customer's auditor requested the decision rationale for a denied-loan letter from a date six months prior. The team pulled the trace. The input was there. The output was there. The reasoning was gone — not because anyone deleted it, but because it had stopped being produced the day the refactor shipped. The customer's compliance program had been operating on the assumption that the rationale was somewhere in the trace store; the platform team had been operating on the assumption that the rationale was nobody's problem because the customer-facing answer was unchanged. Both assumptions were correct in isolation. Together they cost the customer a regulatory finding and the platform team a contract renewal.

The lesson sounds like a process failure, and it is. But the deeper lesson is structural: reasoning tokens are a dual-use artifact. To the engineering team optimizing per-request cost, they are a line item denominated in dollars per million. To the risk team defending a credit decision, they are the only place the model's "why" lives. When the same byte stream serves two audiences with non-overlapping retention horizons and quality bars, you cannot optimize one without consulting the other — and most platform teams do not even know there is another.

Reasoning Tokens Live in Two Cost Models at Once

A reasoning token costs the same as an output token on the wire — sometimes more, when the provider prices "thinking" tokens at a premium. A model that emits eight hundred reasoning tokens to justify a fifty-token answer has just paid for an eighteen-times multiplier on the visible output. From the unit-economics dashboard, that ratio looks like waste. From the compliance dashboard, that ratio is the entire product.

The two dashboards live on the same byte stream and assign it incompatible values. Engineering wants the ratio at one-to-one. Compliance wants the ratio at whatever-it-takes-to-survive-an-audit. The team that owns the prompt is almost always engineering. The team that depends on the output is almost always not in the room when the prompt changes.

This asymmetry is the source of the bug. If reasoning tokens were billed to a budget line owned by the risk team, no platform engineer would touch them without a conversation. They are not. They are billed to the inference budget, owned by infrastructure, optimized against a target that has nothing to do with auditability. The prompt that asks the model to "answer only with the JSON" is a perfectly rational local optimization that produces a globally indefensible artifact.

The Failure Modes That Look Like Optimization

Three patterns reliably destroy evidence chains without anyone noticing until the auditor arrives.

The clean-output refactor. The prompt is rewritten to drop the "explain your reasoning, then answer" preamble. The model now emits the JSON directly. Cost drops. Latency drops. The reasoning trace is not deleted — it never existed in the first place. The risk team's evidence pipeline was downstream of a string the model used to produce and now doesn't, and the pipeline silently returns empty rationales for every decision shipped after the change. Nobody notices because the rationale is sampled in audits, not in production traffic.

The trace-store retention mismatch. The reasoning is kept in the response, but the team that owns "observability" routes everything in the trace store to the same retention class. Operational traces age out at thirty days because that is the cost-efficient default for debugging. Compliance evidence ages out at thirty days because nobody told the observability team it was anything else. The audit window for a fair-lending review is seven years. The first time anyone discovers the mismatch is when a regulator asks for a rationale from month seven and the storage layer returns a four-hundred-and-four.

The syntactically-present rationale. Someone reads a CFPB bulletin and adds "Briefly state the primary reason for the decision" to the prompt. The model produces a sentence. The sentence is technically a rationale. It says "applicant credit profile" or "insufficient documentation" or some other phrase that satisfies a checkbox and tells a denied applicant nothing actionable. The CFPB has explicitly stated that creditors cannot rely on the sample-form checklist of reasons if those reasons do not specifically and accurately indicate the principal reasons for the adverse action. A one-line "credit profile" rationale is exactly the kind of vague placeholder the bureau is calling out. The model is producing tokens; they are not producing evidence.

In each case, the artifact looks superficially correct on the dimension the engineering team measured. Each fails on the dimension the engineering team did not know was being measured.

The Regulator Does Not Care How You Priced It

The CFPB's position on ECOA and Regulation B is uncommonly direct for a regulator: a creditor cannot justify noncompliance by arguing that the technology making the decision is too complex or too opaque to identify specific reasons for adverse action. If the model is too complex to produce defensible rationales, the model cannot be used. There is no exception for "we removed the reasoning to save costs." There is no exception for "the rationale was in the trace logs but they aged out." Explainability is a precondition of deployment, not a feature you can amortize against inference budget.

The European posture is the same in different language. Article 12 of the EU AI Act requires high-risk systems to automatically log events sufficient to ensure traceability throughout the system's lifecycle, with logs retained appropriately and protected against tampering. Article 18 obliges providers to retain those logs against the audit horizon, which for high-risk systems extends years beyond the operational window any engineering team would choose on cost grounds. The August 2026 compliance deadline for core high-risk requirements has already passed for some categories, and the FCA's 2026 examination posture explicitly emphasizes "principles with proof" — the regulator wants to see the trace, not a description of the trace.

What this means in practice: a model decision that affects a consumer, a patient, a borrower, or a tenant must produce an artifact that, years later, lets a regulator reconstruct why. If the artifact is gone because the prompt no longer asks for it, the regulator does not accept "we were optimizing." If the artifact is gone because retention was scoped to operational needs, the regulator does not accept "that's what the observability stack defaults to." The defense the platform team would like to mount — that they were maximizing the efficiency of a system they were authorized to maximize — is exactly the defense the rules were written to refuse.

A Per-Decision-Class Policy Beats a Per-Prompt Reflex

The discipline that closes the gap is not "always include reasoning." It is a per-decision-class policy that names, before any prompt is written, whether the reasoning trace is a product surface, an audit surface, both, or neither. The four answers have four different consequences.

When reasoning is a product surface — a coding assistant explaining its diff, a search engine showing its citations — the trace is part of the UX and retention can follow the product's logs. When reasoning is an audit surface only — a credit-decision rationale, a medical-triage justification — the trace is the only artifact the regulator will accept and retention has to match the audit window, which is years and not days. When reasoning is both, the system needs two paths: a redacted, user-friendly version for the product and a full version for the audit pipeline, with the second never throttled by the cost pressure that periodically reshapes the first. When reasoning is genuinely neither — an internal classification with no consumer impact and no regulated dimension — strip it freely.

The policy belongs in the same document that names the model, the prompt, and the cost target. If it lives anywhere else, the next refactor will route around it.

Three operational pieces make the policy survive in production. First, the reasoning trace pipeline is separate from the operational trace pipeline. They have different retention classes, different access controls, and different ownership. The risk team owns the reasoning pipeline; nobody else can change its retention without a formal review. Second, an evidence-quality eval runs against the rationales the model actually emits, grading them not on answer correctness but on whether a human auditor would accept the rationale as specifically and accurately describing the principal reason. This eval catches the syntactically-present-substantively-useless failure mode that no engineering metric will. Third, the cost model for the inference budget prices reasoning tokens against the audit value they produce, not against the visible answer. If a reasoning trace prevents a five-million-dollar regulatory finding, paying a few cents per request to preserve it is a trivially obvious trade.

The Architectural Realization

Reasoning tokens are the only place the model's "why" lives. They are also, on most systems, the easiest line item to cut, because they are visible in the cost dashboard and invisible in the user-facing output. The two facts together describe a near-perfect failure mode: a thing whose value is held by a team that does not see the cost, optimized by a team that does not see the value.

A team that deletes reasoning to save thirty percent on inference has also deleted the answer to every question an auditor, a customer, or a postmortem is going to ask. The thirty percent shows up in this quarter's financials. The deletion shows up in the next adverse-action complaint, the next compliance review, the next incident where someone needs to know what the model was thinking and the answer is gone. The financial savings are immediate and the cost is deferred, which is exactly the structure of every decision that looks brilliant in the moment and ruinous in the postmortem.

The right framing is not that reasoning tokens are expensive. It is that reasoning tokens are evidence, evidence is a regulated artifact in any high-stakes domain, and the team optimizing the prompt does not get to unilaterally decide what evidence the company is required to produce. Until that framing makes it into the prompt-review checklist, every cost-savings refactor in a regulated product is a compliance time bomb with a fuse the length of one audit cycle.