8 posts tagged with "reproducibility"

A customer disputes a loan denial. A regulator opens an inquiry. A plaintiff's lawyer files discovery. All three arrive with the same deceptively simple question: what did your system decide, and why? The decision happened eighteen months ago. You pull up the case, and every component that produced the original output has since moved on. The hosted model version was deprecated and migrated. The system prompt has been edited nine times. The documents your agent retrieved were re-chunked, re-embedded, and re-ranked into a new index. And the sampling settings that made the whole thing non-deterministic were never logged in the first place.

You cannot reproduce the decision. Not because you were careless, but because nothing in your stack was built to be reproduced. Explainability-after-the-fact turns out to be a reproducibility problem in disguise — and reproducibility is something you either engineer in at decision time or lose forever.

The uncomfortable truth is that most teams discover this gap exactly when they can least afford to. The demand to reconstruct a decision almost never comes during normal operations. It comes attached to a lawsuit, an audit, or an angry customer with a regulator's phone number, and by then the window to have captured the right evidence closed a year and a half ago.

Give a coding agent a Python module and it operates on solid ground: the file on disk is the program. Read it, edit it, run it, observe the result — the loop closes. Give the same agent a Jupyter notebook and every one of those assumptions quietly breaks. The agent edits cell 12 with full confidence, not knowing that you re-ran cell 3 an hour ago with different data, that a variable defined in a since-deleted cell is still alive in the kernel, and that the output it just read under cell 7 was produced three kernel restarts ago by code that no longer exists.

A notebook is a REPL wearing a file costume. The .ipynb on disk looks like source code, but the thing that actually determines behavior — the kernel's accumulated memory — is invisible, unserialized, and shaped by the exact sequence of human clicks that produced it. Agents are trained on the contract that code determines behavior. Notebooks void that contract, and most agent harnesses don't even know it.

A model gives a customer a wrong, expensive answer. You open the post-mortem, paste the exact prompt back into the exact same endpoint with temperature=0, and you get a different answer. Not a worse one, not a better one — a different one. The bug you are supposed to root-cause refuses to reproduce on demand, and the one knob everyone told you guarantees reproducibility just lied to your face.

This is the moment most teams discover that "set temperature to zero" is folklore, not an engineering control. Temperature zero changes how the model samples — it forces greedy decoding, always taking the highest-probability token. It says nothing about whether the probabilities themselves come out the same twice. And in a production serving stack, they frequently don't.

A fine-tune is not a file. It is the closure of a pipeline over a training set, and the team that ships the file without the closure has built a production dependency whose source code is in someone else's head. The day that person leaves with two weeks of notice and a clean handoff document is the day your bus factor on a revenue feature drops to zero and nobody notices, because the weights are still in the registry and the registry tag is still stable and the model still serves traffic. The reckoning shows up later, in a routine base-model migration that should have taken a sprint and takes a quarter instead.

The pattern is consistent across teams I have watched run into it. An ML engineer spends six months iterating on a fine-tune — data curation, hyperparameter sweeps, behavioral patches evaluated by feel against a held-out set. The final adapter weights get pushed to the model registry with a tag. The training pipeline that produced those weights is a notebook on the engineer's laptop, with hard-coded paths and floating dependencies that resolved to whatever was the latest version on the day each cell was last executed. The team accepts the handoff at face value because the weights work and the eval scores are good and the registry tag is stable. Eighteen months later, the engineer departs. Six months after that, a base-model migration requires regenerating the adapter against an updated base, the notebook runs and produces weights that score three points lower and regress visibly on the hardest customer segment, and the team spends four months trying and failing to reproduce the original artifact.

A platform team ships a quality improvement. An inference-time layer reads the user's recent interactions and silently nudges the response style: more formal here, more terse there, more technical when the history suggests an engineer is asking. The A/B test shows an aggregate satisfaction lift of a couple of points. The launch post goes out under the heading "smarter responses, no API changes required." Nobody flips a flag in the API. Nobody updates the docs. Nothing in the response payload indicates which persona the model just adopted.

Six weeks later an enterprise customer files a support ticket that says, "your model is worse than you advertised." Their internal eval suite — running the same prompts your team published benchmarks against — scores eight points lower. Your team's first move is to verify prompt parity. Prompts match exactly. Decoding parameters match. The model version string matches. The divergence traces to the personalization layer, which infers a "thin-history default persona" for the customer's freshly-provisioned test account and a richer one for the long-lived user accounts your benchmarks were measured against. The conversation about whether the personalization is a feature or a bug stops being a product decision and becomes a contract negotiation.

You set seed=42. You set temperature=0. You logged the run, posted the dashboard, signed off on the model swap. The next morning the rerun returned a different number on the same prompts, and the explanation you reached for — "must be sampling noise" — was wrong twice over: there was no sampling, and the noise was structural. The seed left your client, the gateway threw it away, the kernel batched your request next to seventeen unrelated ones, and the floating-point reduction order changed under you. Your "reproducible" benchmark was always within one batch of being a different benchmark.

This failure mode is quiet because every layer in the stack is technically correct. The SDK accepts the seed. The provider documents the seed. The model returns a system_fingerprint. The eval harness logs all three. Nothing 5xx's, nothing warns, nothing protests. The number on the dashboard just shifts, and the team rationalizes the shift as the kind of jitter that always existed — because they have no instrument that can tell them whether they're looking at stochastic decoding or at a backend rotation that invalidated three weeks of comparisons.

The incident ticket is specific in the way only real incidents are. At 02:14 the support agent closed a customer account that should have been put on a 30-day grace period. The customer noticed. The ticket lands on your desk with a single line under "Steps to reproduce": unknown.

You open the trace. You can see the agent called close_account instead of set_grace_period. You can see the tool succeeded. What you cannot see is why the model chose that branch — and when you replay the same customer message through the same agent, it does the right thing. Twice. The postmortem now has a paragraph-shaped hole where the root cause should be, and the only honest thing you can write is "could not reproduce."

Your agent failed in production last Tuesday. A customer reported a wrong answer. You pull up the logs, see the final output, maybe a few intermediate print statements — and then you're stuck. You can't re-run the agent and get the same failure because the model won't produce the same tokens, the API your tool called now returns different data, and the timestamp embedded in the prompt has moved forward. The bug is gone, and you're left staring at circumstantial evidence.

This is the fundamental debugging problem for AI agents: traditional software is deterministic, so you can reproduce bugs by recreating inputs. Agent systems are not. Every run is a unique snowflake of model sampling, live API responses, and time-dependent state. Without specialized tooling, post-mortem debugging becomes forensic guesswork.

Deterministic replay solves this by recording every source of non-determinism during execution and substituting those recordings during replay — turning your unreproducible agent run into something you can step through like a debugger.