95 posts tagged with "llmops"

A customer disputes a loan denial. A regulator opens an inquiry. A plaintiff's lawyer files discovery. All three arrive with the same deceptively simple question: what did your system decide, and why? The decision happened eighteen months ago. You pull up the case, and every component that produced the original output has since moved on. The hosted model version was deprecated and migrated. The system prompt has been edited nine times. The documents your agent retrieved were re-chunked, re-embedded, and re-ranked into a new index. And the sampling settings that made the whole thing non-deterministic were never logged in the first place.

You cannot reproduce the decision. Not because you were careless, but because nothing in your stack was built to be reproduced. Explainability-after-the-fact turns out to be a reproducibility problem in disguise — and reproducibility is something you either engineer in at decision time or lose forever.

The uncomfortable truth is that most teams discover this gap exactly when they can least afford to. The demand to reconstruct a decision almost never comes during normal operations. It comes attached to a lawsuit, an audit, or an angry customer with a regulator's phone number, and by then the window to have captured the right evidence closed a year and a half ago.

Every other change to your production system obeys a discipline. Code ships behind a flag, gets canaried to 1% of traffic, and rolls back in one click when a dashboard turns red. Schema migrations are staged and reversible. Even a CSS tweak goes through a pull request someone can read. Then there's the prompt. Someone edits a paragraph in a text box, clicks save, and the behavior of your product changes for every user at once — no canary, no diff anyone reviewed, no revert button that actually returns you to the prior state.

The uncomfortable part is that this isn't an oversight by careless teams. It's the default that the tooling produces. Prompts get filed under "configuration" because they're strings that live outside the compiled binary, and configuration has always been the thing you're allowed to change quickly without a full release. But a prompt isn't config. It's a program written in English, compiled by a nondeterministic interpreter you don't control, whose behavior you can only observe statistically. Treating it like a config value is the category error underneath a whole class of production incidents.

You can stand up a staging database. You can stand up a staging queue, a staging payments sandbox, a staging copy of every third-party API you depend on. For thirty years the whole discipline of pre-production has been built on one assumption: that you can create a faithful-enough replica of production, poke it, and learn something true about what will happen when you ship.

Then you added a hosted model to your critical path, and the assumption quietly broke. The one component whose behavior now dominates your product — the thing that decides what your app actually says and does — is the one component you cannot stand up a staging copy of. It is versioned by someone else, rate-limited by someone else, and quietly updated by someone else on a schedule you don't see. Your staging environment has a staging everything, except a staging model.

You instrumented your agent the way every observability guide told you to. Every turn gets a span: the full prompt, the retrieved chunks, the model's chain of thought, the tool calls it considered, the plan it drafted and then abandoned, the final action. Storage is three cents a gigabyte and the traces have saved you a dozen times during incident response. You kept everything because keeping everything was free and occasionally heroic.

Here is the part nobody on the platform team modeled: that trace is written in fluent English, it speculates about your customers by name, and it drafts actions you never actually took. It is the single most quotable artifact your company produces, and a plaintiff's attorney will read the juiciest 200 words of it to a jury with your logo on the slide behind them. The reasoning trace that makes your system debuggable is the same reasoning trace that makes it liable.

The incident channel says the new prompt is hallucinating refund amounts, so you do the obvious thing: repoint the production tag to last week's prompt version. Thirty seconds, no redeploy, textbook rollback. Except the agent gets worse. Last week's prompt references a lookup_order tool that the platform team renamed to orders.search on Tuesday. The memory store is full of preference summaries written by the new prompt's format, which the old prompt reads as user instructions. You didn't roll back the agent. You built a chimera — one-third last week, two-thirds today — and shipped it to production without ever testing that combination.

This is the failure mode nobody's runbook covers: an agent deployment is not an artifact, it's a triple — prompt version, tool contract, accumulated memory state. Rolling back one leg while the other two advance doesn't restore a previous state. It creates a new state that has never existed before, never passed an eval, and belongs to no team's on-call rotation.

Rename a database column and watch what happens. The compiler catches every query builder. The ORM migration catches the model class. The type checker catches the API serializer. Integration tests catch the two services that read the field over the wire. Every consumer of that column gets flagged — except one. Your system prompt, which contains a carefully formatted description of the table "so the model understands the data," keeps confidently describing a column that no longer exists. No compiler error. No failing test. No deprecation warning. Just a model that starts generating queries against a schema from three sprints ago, and a dashboard that slowly fills with malformed SQL.

Prompts have foreign keys. They reference database schemas, tool signatures, enum values, API shapes, and few-shot examples copied from production data — and none of those references participate in refactoring. Every fact you paste into a prompt is a join against a table that your tooling doesn't know exists. When the referenced artifact changes, nothing cascades. The prompt just rots in place.

Six weeks after launch, your quality dashboard starts sagging. Thumbs-down rates creep up, task completion drifts down, and the on-call channel fills with screenshots of bad responses. The team does what teams do: they diff the prompts (unchanged), check the model version (pinned), audit the retrieval index (fresh), and bisect the deploy history (nothing shipped). Everyone concludes the model provider silently degraded the model. The provider, of course, insists nothing changed.

Everyone is looking in the wrong place. Nothing in the system changed. The users did.

Launch-week metrics assume launch-week users. But people adapt to an AI product within weeks, and they adapt in ways that systematically break the assumptions baked into your prompts, your evals, and your launch benchmarks. Your model is frozen. Your users are not. The gap between them is a form of drift that most teams don't instrument for at all — and it produces the most confusing incident pattern in AI engineering: metric decay with no deploy.

Here is a fact that breaks most engineering dashboards: two identical inference requests, returning identical tokens, billed at the identical API rate, can have carbon footprints that differ by a factor of five. Same model, same prompt, same dollars. The only thing that changed was which data center answered and what time of day it was. If your sustainability number is derived from your spend number — and almost everyone's is, because spend is the only number anyone actually tracks — then your sustainability number is wrong in a way that gets worse precisely when you optimize hardest for cost.

The dollar and the joule look like they should move together. More tokens, more compute, more money, more energy. For a single request held in a single place at a single instant, they roughly do. But the moment you turn on the levers that make inference cheap at scale — aggressive batching, model routing, off-peak scheduling, multi-region failover — the two metrics decouple. Cost is a function of your contract. Carbon is a function of physics and the grid. They are not the same function, and treating one as a proxy for the other will eventually put a number in a sustainability report that an auditor can falsify.

Your on-call rotation was designed to catch a different kind of failure than the one that will actually take down your AI product. It watches for the service that stops responding, the latency curve that spikes, the error rate that crosses 1%. These are step functions: something was working, then it wasn't, and the discontinuity is loud enough to page a human at 3 a.m. The entire apparatus — thresholds, runbooks, escalation policies — assumes failure announces itself.

The failure mode that matters for a system with a model in the loop is the opposite. Last Tuesday your extraction pipeline was 94% accurate. This Tuesday it's 91%. Nothing crashed. Every request returned a 200. Latency is flat. The output is still well-formed JSON. But three percent of your users are now getting subtly wrong answers, and they will not file a bug, because the answer looks right. The pager stayed silent because there was nothing for it to fire on. By the time someone notices — usually a customer, usually angry, usually weeks later — the regression has been quietly compounding the whole time.

Look at any AI product roadmap and you'll find columns for the things you're going to launch: a new agent, a better retrieval pipeline, a model upgrade, a fine-tune. Every item is a birth. Nothing on the board is a death. There's no column for the model you'll stop supporting, the prompt you'll retire, the eval set you'll archive, the feature you'll turn off. Roadmaps are optimistic by construction — they're a list of futures you're promising to create.

That omission is fine for deterministic software, where a feature you shipped two years ago mostly keeps working until you touch it. It is quietly catastrophic for AI systems, where the things you shipped are decaying whether you touch them or not. A prompt tuned to GPT-4's quirks degrades the day the provider nudges the model. An eval suite written against last year's failure modes stops measuring anything real. A model you depend on gets a 60-day deprecation notice you didn't plan for. A roadmap that only plans launches is accruing silent liability on every line it doesn't track.

The demo worked. You typed a question, the agent called three tools, reasoned through a multi-step plan, and produced an answer that made the room lean forward. Someone said "ship it." Three months later you're still not shipped, and nobody can quite explain where the time went.

Here is where it went: the demo was 10% of the work. It was the brain. The other 90% is the plumbing — the evals, guardrails, observability, cost controls, and fallback paths that nobody claps for because they're invisible when they work and catastrophic when they don't. That 90% is the demo-to-production tax, and most teams budget for it as if it were a rounding error.

The numbers say otherwise. MIT's 2025 study of enterprise AI found that 95% of generative-AI pilots delivered no measurable P&L impact. A separate analysis put it more bluntly: for every 33 proofs of concept an enterprise starts, four reach production. That's an 88% mortality rate, and the cause of death is almost never the model. It's everything the demo let you skip.

The eval dashboard had been climbing for three sprints. Quality up six points on the hard slice, up nine on the regression slice, up twelve on the slice the support team had hand-curated from last quarter's worst tickets. The team shipped a model promotion off the back of it. Two days later, a customer asked a question that looked nothing like anything in the eval set, and the answer was worse than what they had been getting six months ago.

The forensic was quick once someone thought to run it. The prompt engineers had been working out of the same repo as the eval team. They had found the curated examples — the painstaking ones, the ones where someone had argued for an hour about the correct phrasing of the ideal answer — and over a few sprints they had copy-pasted the strongest of them as few-shot demonstrations into the production system prompt. The dashboard kept going up because the model was being graded on inputs it had seen verbatim at inference time. Nobody flagged it. Nobody owned the boundary between "the examples we measure quality against" and "the examples we ship in the prompt." Both teams were doing exactly the job they had been hired to do.