Reproducing an AI Decision Eighteen Months Later

A customer disputes a loan denial. A regulator opens an inquiry. A plaintiff's lawyer files discovery. All three arrive with the same deceptively simple question: what did your system decide, and why? The decision happened eighteen months ago. You pull up the case, and every component that produced the original output has since moved on. The hosted model version was deprecated and migrated. The system prompt has been edited nine times. The documents your agent retrieved were re-chunked, re-embedded, and re-ranked into a new index. And the sampling settings that made the whole thing non-deterministic were never logged in the first place.

You cannot reproduce the decision. Not because you were careless, but because nothing in your stack was built to be reproduced. Explainability-after-the-fact turns out to be a reproducibility problem in disguise — and reproducibility is something you either engineer in at decision time or lose forever.

The uncomfortable truth is that most teams discover this gap exactly when they can least afford to. The demand to reconstruct a decision almost never comes during normal operations. It comes attached to a lawsuit, an audit, or an angry customer with a regulator's phone number, and by then the window to have captured the right evidence closed a year and a half ago.

Every Input Has a Different Clock

The reason reconstruction is so hard is that an AI decision isn't a single artifact. It's the output of a pipeline where every stage has its own lifecycle, and those lifecycles are not synchronized with your obligation to explain them.

Consider what actually goes into one inference. The model version is the first thing to rot. Major providers give a hosted model roughly 12 to 18 months before deprecation, and their deprecation notices tell you when a model retires and how to migrate — but say almost nothing about behavioral compatibility. Once the old snapshot is gone, you cannot re-run the exact computation that produced the original answer, full stop. If you pointed at an aliased endpoint rather than a pinned snapshot, the model underneath was already drifting before it was ever formally retired.

The prompt rots on a faster clock. System prompts are edited constantly — a tweak to reduce refusals here, a new guardrail there — and unless each edit is versioned and stamped onto the decision record, you have no way to know which prompt was live on the day in question. "We use this prompt" is a statement about today, not about the moment that matters.

The retrieval context rots most silently of all. If your system uses RAG, the answer depended on specific chunks pulled from a specific index. But indexes get rebuilt. You change your embedding model and reindex everything; you re-chunk with different overlap; you add documents and the ranking shifts. The chunk that anchored the original answer may not exist in the same form anymore — and without logging the retrieved document IDs and their contents at retrieval time, you cannot say what the model was actually looking at.

Finally, the sampling parameters — temperature, top-p, any seed — often aren't captured at all, because at the time they felt like infrastructure trivia rather than evidence.

Four inputs, four independent clocks, none of them wound to keep the time you'll eventually be asked about.

Even the Model Can't Reproduce Itself

Suppose you did everything right. You pinned the exact model snapshot, you archived the exact prompt, you saved the exact retrieved documents and the exact sampling parameters. You re-run the request. You still may not get the same output.

This is the part that surprises engineers who assume a computer is deterministic by default. On a hosted endpoint, temperature zero does not guarantee the same answer. The culprit isn't randomness in the sampler — it's the arithmetic underneath. Floating-point addition is non-associative: (a + b) + c doesn't always equal a + (b + c), and GPU kernels accumulate tensors in whatever order is fastest for the current shape. Change the shape, change the result in the last few decimal places, and occasionally that ripples up into a different token.

The deeper cause, as recent work on inference non-determinism has shown, is batch-size dependence. Your request doesn't run alone; it runs batched with whatever other requests happened to arrive at the same millisecond. The reduction kernels that sum across the batch produce subtly different values depending on how many requests are in flight. Your prompt is identical from run to run, but the batch your prompt lands in is not — and that's enough to make the output non-reproducible even at temperature zero with a fixed seed.

It is possible to defeat this. Batch-invariant kernels for normalization, matrix multiplication, and attention can produce bit-identical outputs across a thousand runs — at roughly a 60% throughput cost that almost no production system pays. Strict determinism is realistically only achievable on open-weights models you run on your own hardware with controlled kernels and single-batch inference. On a shared, hosted endpoint, "reproducible" means statistically consistent, not bit-identical — and your evidence strategy has to account for that gap rather than pretend it doesn't exist.

The practical consequence: your goal usually isn't to regenerate the identical bytes. It's to prove that, given the same inputs, the system behaves within a bounded, explainable range — and to have captured the actual output that was served, because that's the only version that ever affected a real person.

The Decision Record Is the Only Thing That Survives

If you can't reliably re-run the past, you have to have recorded it. The durable answer to "reproduce this decision" is a decision record: an immutable snapshot, written at inference time, of everything needed to reconstruct and defend the outcome. Not logs you hope to piece together later — a purpose-built artifact captured at the moment of the decision.

A defensible decision record captures, at minimum:

• The pinned model identity — the exact snapshot version, not the alias, plus the provider and endpoint.
• The full input context — the exact system prompt, user input, and any tool or function definitions that were in scope, byte for byte.
• The retrieved evidence — the IDs of the documents or chunks that were retrieved and their contents as they existed at retrieval time, plus retrieval scores if you have them.
• The sampling parameters — temperature, top-p, max tokens, seed, and anything else that shaped generation.
• The actual output served — the response the user or downstream system actually received, not a regenerated approximation.
• Provenance metadata — a timestamp, a request ID, and the code or config version of the surrounding application.

This is exactly the direction regulation is pushing. The EU AI Act's Article 12 requires high-risk systems to be built with automatic event logging — the system must generate records without an operator remembering to; manual recording doesn't count. Article 12 explicitly calls for logging that enables traceability of the system's functioning across its lifecycle, and for certain systems, recording the input data that led to a match and the people who verified results. The regulatory bar is not "you kept some logs." It's "the system was designed to make its decisions reconstructable."

The design principle that follows: treat the decision record as a first-class output of every inference, generated automatically and written to append-only, tamper-evident storage. If producing the record is a manual step someone can skip under load, it will be skipped precisely on the decisions you'll later be asked to defend.

The Retention Trap: Log Everything vs. Delete on Request

Here's where good intentions collide with the law. The instinct after reading the above is to log everything, forever. That instinct is itself a compliance violation.

The full input context that makes a decision reconstructable is exactly the personal data that privacy law demands you minimize and delete. GDPR's principles of data minimization and storage limitation say you should collect only what's necessary and keep it only as long as needed; the right to erasure says an individual can demand you delete their data. Meanwhile the EU AI Act tells you to retain logs — at least six months for the automated logs of high-risk systems, and much longer under sector rules like HIPAA's multi-year documentation requirements. One regime says delete; the other says keep. Both apply to the same decision record.

Storing raw customer PII inside an immutable, decade-retained audit trail doesn't satisfy both regimes — it violates one of them by construction. The way out that practitioners are converging on is architectural separation. Raw personal data lives in a store governed by minimization and erasure rules, subject to automated deletion when its operational purpose ends. The long-lived audit trail is constructed from non-personal or irreversibly anonymized assets — hashes, references, pseudonymous IDs, structural metadata — so it can be retained for the AI Act's extended windows without holding personal data hostage to them.

The ordering matters: erasure obligations are honored first on the raw store, and what remains in the audit trail is engineered from the start to contain no recoverable personal data. Getting this separation right is genuinely hard, and it's the part teams most often defer — which is exactly why it's worth designing before the first regulator asks, not after.

Build the Record Before You Need It

The through-line is that reproducibility is not a property you can retrofit. Every component of an AI decision — model, prompt, retrieval, sampling — sits on its own rotation schedule, and hosted inference isn't even bit-deterministic when you freeze all of them. The only decision you'll actually be able to defend eighteen months from now is the one whose evidence you captured at the moment it was made.

Concretely: pin model snapshots instead of aliases, and record which snapshot served each request. Version your prompts and stamp the version onto every decision. Log retrieved document IDs and their contents, not just the final answer. Capture sampling parameters as evidence, not infrastructure noise. Write all of it into an automatic, append-only decision record — and split personal data from the durable audit trail so you can honor deletion and retention at the same time.

None of this is exotic engineering. It's the difference between a system that can explain itself and one that merely worked once. The request to reconstruct a decision is not an edge case you might get unlucky enough to face; for anyone operating in a regulated domain, it's a certainty with an unknown date. The teams that will answer it calmly are the ones treating every inference today as something they may have to reconstruct in a courtroom tomorrow.