842 posts tagged with "ai-engineering"

A customer disputes a loan denial. A regulator opens an inquiry. A plaintiff's lawyer files discovery. All three arrive with the same deceptively simple question: what did your system decide, and why? The decision happened eighteen months ago. You pull up the case, and every component that produced the original output has since moved on. The hosted model version was deprecated and migrated. The system prompt has been edited nine times. The documents your agent retrieved were re-chunked, re-embedded, and re-ranked into a new index. And the sampling settings that made the whole thing non-deterministic were never logged in the first place.

You cannot reproduce the decision. Not because you were careless, but because nothing in your stack was built to be reproduced. Explainability-after-the-fact turns out to be a reproducibility problem in disguise — and reproducibility is something you either engineer in at decision time or lose forever.

The uncomfortable truth is that most teams discover this gap exactly when they can least afford to. The demand to reconstruct a decision almost never comes during normal operations. It comes attached to a lawsuit, an audit, or an angry customer with a regulator's phone number, and by then the window to have captured the right evidence closed a year and a half ago.

Every other change to your production system obeys a discipline. Code ships behind a flag, gets canaried to 1% of traffic, and rolls back in one click when a dashboard turns red. Schema migrations are staged and reversible. Even a CSS tweak goes through a pull request someone can read. Then there's the prompt. Someone edits a paragraph in a text box, clicks save, and the behavior of your product changes for every user at once — no canary, no diff anyone reviewed, no revert button that actually returns you to the prior state.

The uncomfortable part is that this isn't an oversight by careless teams. It's the default that the tooling produces. Prompts get filed under "configuration" because they're strings that live outside the compiled binary, and configuration has always been the thing you're allowed to change quickly without a full release. But a prompt isn't config. It's a program written in English, compiled by a nondeterministic interpreter you don't control, whose behavior you can only observe statistically. Treating it like a config value is the category error underneath a whole class of production incidents.

Every other line item on your infrastructure bill went through a gate. Someone filed a purchase order for the database cluster. Someone counted seats before buying the observability SaaS. Someone ran a capacity review before the team doubled its Kubernetes footprint. Then a model API showed up, and none of that happened.

An engineer added their API key to a config file. They wrote a create() call that looks exactly like every other function call in the codebase. It shipped. And the first time anyone in finance learned that this feature existed as a cost center was a variance line on the monthly invoice — a number nobody forecasted, nobody approved, and nobody can immediately explain.

This is the FinOps gap for AI, and it is not a monitoring problem. It is a governance problem wearing a monitoring costume. You can have perfect dashboards and still get surprised, because the spend was invisible to your approval process long before it was visible on a chart.

The most dangerous eval set is the one that still passes. A regression suite that goes red gets attention: someone opens the failing case, argues about it, fixes the bug or updates the expectation. A green suite gets trust. And trust is exactly what a rotted eval set does not deserve, because the score stayed green not because your system is good but because the test stopped resembling what your users actually do.

This is the quiet failure mode of AI evaluation. You build a golden dataset — a few hundred carefully labeled cases that represent the job your product does. It earns its keep for a quarter. Every deploy runs it, every score lands green, and everyone sleeps well. Meanwhile the product ships three new features, enterprise traffic climbs from ten percent of queries to forty-five, and users start phrasing requests in ways nobody on your team wrote down eighteen months ago. The eval set doesn't know about any of it. It keeps grading the model on a distribution that no longer exists.

A teammate opens a pull request. The diff is three words. One line goes red — Do not add information not present in the source. — and one line goes green — Make your best guess if the source is incomplete. The change is small, the intent is reasonable, and the code review takes eleven seconds. You approve it. A week later, your support bot is confidently inventing refund policies that do not exist, and you are reading through logs trying to figure out when the hallucination rate tripled.

The git diff did its job perfectly. It showed you exactly which characters changed. What it could not show you is the only thing that mattered: the behavior on the other side of those characters moved from "refuse when unsure" to "fabricate when unsure." For code, the textual diff is a faithful proxy for the behavioral diff — change a < to a <= and a reviewer can reason about the consequence. For prompts, the textual diff and the behavioral diff have almost nothing to do with each other.

A deletion request lands in your queue. A user has invoked their right to erasure, and legally you have a month to make their personal data disappear. In a normal system this is a DELETE statement and a smug audit-log entry. In an AI system it is the moment you discover that your data doesn't live in one place — it has been smeared across a fine-tuned model's weights, baked into a vector index, cached in a dozen retrieval snapshots, and copied into last quarter's evaluation set. There is no single row to delete. The user is, in a very literal engineering sense, undeletable.

This is not a hypothetical. In March 2025 the European Data Protection Board launched a coordinated enforcement action across thirty national authorities focused specifically on the right to erasure. Regulators have converged on an uncomfortable position: including someone's data in training is processing, so Article 17 applies to the model, not just the database. The question every AI team eventually faces is whether output suppression — teaching the model to refuse to talk about someone — is enough, or whether you actually have to remove the influence of their data from the system. The honest answer is that most teams have never designed for either.

There's a specific kind of argument you can only have with a language model. You paste in your code. It rewrites a working call into one that hasn't existed since a major version ago. You correct it. It apologizes, agrees with you, and on the next turn does the exact same thing. You are not fighting ignorance. You are fighting a confident, well-rehearsed memory of a different version of your stack — and that memory was reinforced by more training examples than your correction will ever carry.

This is the failure mode I've come to think of as fighting the prior. The model's parametric knowledge — everything it absorbed during training — contains the popular, the outdated, or simply the different version of the framework you're actually using. When your context and its prior disagree, the prior often wins. And unlike a plain hallucination, this one is dangerous precisely because it's plausible: the deprecated API used to be correct, so the code looks right, passes a casual read, and sometimes even compiles.

The demo wowed the executive team in a week. Eighteen months later, it is still a demo. The model still answers questions in the sandbox, the slide deck still gets recycled into new pitches, and every quarter someone asks why it hasn't shipped. Nobody has a good answer, because the honest one is uncomfortable: the part that impressed everyone was the cheap part, and nobody scoped the expensive part.

This is pilot purgatory, and it is now the default outcome. MIT's NANDA initiative found that roughly 95% of enterprise generative AI pilots deliver no measurable impact on the P&L. IDC and Lenovo, counting a different thing — POCs that literally never ship — put the number at 88%: for every 33 proofs of concept an enterprise starts, four reach production. The share of companies abandoning most of their AI initiatives jumped from 17% in 2024 to 42% in 2025. These are not stories about weak models. They are stories about a graduation gap that nobody budgeted for.

The uncomfortable truth is that a working demo represents maybe ten percent of the work required to run the thing in production, and it is the ten percent that looks like a hundred. Everything that makes an AI system safe to leave running — evals, guardrails, observability, cost controls, security review, on-call, and an owner with a budget — is invisible in a demo and non-negotiable in production.

There is a specific phrase that should make every engineering leader stop the meeting: "This will just work once the models get better." It is usually said with confidence, sometimes with a slide showing a capability curve bending upward, and it almost always marks the moment a roadmap quietly stopped being a plan and became a forecast.

The distinction matters more than it sounds. A plan is a sequence of things you control: code you will write, integrations you will build, tests you will run. A forecast is a bet on something you don't control — in this case, a model that doesn't exist yet, shipping on a schedule set by a vendor who has never promised you a date. When you scope a feature around "the context window will double" or "reasoning will be good enough by Q3," you have not de-risked the hard part of the problem. You have relocated it onto someone else's release calendar and called it done.

Nothing broke. That's the disorienting part. No deploy failed, no latency alarm fired, no error rate ticked up. Someone merged a one-line PR that appended a sentence to the system prompt — a new tool description, a policy reminder, a "today's date is" header — and the next morning the inference bill was three to five times higher. Traffic was flat. The model was the same. The code did exactly what it was supposed to do.

What changed is that the one line landed in the wrong place, and every cached prefix in your fleet invalidated at once. Your cache hit rate went from 90% to zero in a single request cycle, and every token that used to be nearly free started billing at full price. This is the prompt-cache cliff, and it's the most expensive failure mode in production LLM systems that nobody threat-models, because it doesn't look like a failure at all.

You can stand up a staging database. You can stand up a staging queue, a staging payments sandbox, a staging copy of every third-party API you depend on. For thirty years the whole discipline of pre-production has been built on one assumption: that you can create a faithful-enough replica of production, poke it, and learn something true about what will happen when you ship.

Then you added a hosted model to your critical path, and the assumption quietly broke. The one component whose behavior now dominates your product — the thing that decides what your app actually says and does — is the one component you cannot stand up a staging copy of. It is versioned by someone else, rate-limited by someone else, and quietly updated by someone else on a schedule you don't see. Your staging environment has a staging everything, except a staging model.

You wired up a second model vendor because the first one went down. It was a Tuesday, the primary provider's status page was a wall of red for a few hours, and your feature was dead in the water with it. So you did the responsible thing: you added a fallback. Now, if OpenAI is unreachable, you route to Anthropic. If Anthropic rate-limits you, you fall back to Gemini. The architecture diagram looks clean and grown-up. Reliability, solved.

Except you didn't add a replica. You added a second opinion. And a second opinion is a very different thing to operate than a second copy.

The mental model you imported comes from the stateless-service playbook: run three identical instances behind a load balancer, and if one dies the other two serve the exact same responses. That works because the replicas are byte-for-byte interchangeable. Two language models from two vendors are not. They were trained on different data, tuned with different objectives, and they disagree — systematically, not randomly — on exactly the inputs where your users notice.