990 posts tagged with "insider"

Your RAG eval dashboard is green. Faithfulness is 0.91, answer relevance is 0.88, and the LLM-as-judge harness you spent two sprints building says the system is doing fine. Meanwhile, a user just asked a question whose answer sits in a document your retriever never surfaced, and your model wrote a confident, well-structured, entirely useless response about something adjacent. The judge scored it highly. It read well. It was grounded in the passages it did get. It just answered the wrong question with material that had nothing to do with what the user needed.

This is the quiet structural flaw in how most teams evaluate retrieval-augmented generation: they grade the essay and never check whether the student was handed the right book. A RAG system is two machines bolted together — a retriever that decides what the model gets to see, and a generator that decides what to do with it. Almost every eval harness in production measures only the second machine. The first one, the one that actually determines the ceiling on answer quality, runs unmonitored.

There is a specific kind of outage that never shows up as an outage. The service stays green, latency is flat, error rates are zero, and search quietly starts returning garbage. This is what happens the moment you point a new embedding model at an index built by the old one. Nothing crashes. The results just stop making sense.

The reason is geometry. An embedding model doesn't assign fixed coordinates to a concept — it defines a space, and the same sentence lands in a completely different location depending on which model drew the map. A vector produced by last year's model and a query embedded by this year's model are not "close" or "far." They are measured against different rulers. Cosine similarity between them is a number, and the number is meaningless.

So when someone files a ticket titled "upgrade to the new embedding model," they have not filed a config change. They have filed a full data migration that happens to be disguised as a one-line diff. Treat it like a library bump and you ship the silent outage.

A researcher agent asks a retrieval agent for a document. The retrieval agent, mid-task, decides it needs the researcher to clarify the query before it can search. The researcher, waiting on the document, won't respond until it has the document. Neither one is broken. Neither one is looping. They are both politely, indefinitely, waiting for each other — and your orchestrator, which has no concept of "both of these are blocked on each other," will happily hold that state until a timeout you never configured finally fires, or until a human notices the run has been "in progress" for forty minutes.

This is a deadlock. It is one of the oldest failure modes in computing, and it has nothing to do with how smart your model is. It is a property of how work is coordinated, not how work is done. The uncomfortable finding from the last year of multi-agent research is that most of what breaks in agent swarms breaks here, in the coordination layer, not in the reasoning of any single agent.

Single-agent thinking never surfaces these bugs. When one model runs a loop of tool calls, the worst it does is spin — and a spinning loop is at least visibly spinning. The moment you have two or more agents that can wait on each other, you have inherited the entire back catalog of distributed-systems pathologies: circular wait, livelock, lost messages, premature termination, races on shared state. Nobody sat down and decided to build a distributed system. You built one anyway the day you added a second agent.

When you deprecate an API for human developers, you have a whole ceremony for it. You bump the version, add deprecated: true to the OpenAPI spec, ship a Sunset HTTP header, send an email to the developer mailing list, post to the changelog, and give people six months to migrate. The signal reaches a human who reads it, files a ticket, and updates their client before the old path disappears.

Now point that same deprecation at an agent. The model calling your tool does not read your changelog. It does not subscribe to your mailing list. It never sees the Sunset header unless you deliberately put it somewhere the model looks, and even then it has no reliable habit of acting on it. The deprecation notice you so carefully authored lands in a mailbox with no reader. The agent keeps calling the old shape of the tool until the shape is gone, and then it fails — often silently, often in production, often at 2 a.m.

This is the quiet asymmetry of building tools for agents instead of people. Every discipline we built over two decades of API evolution assumes a human sits between the deprecation and the migration. Take the human out, and the entire mechanism goes dark.

Every other change to your production system obeys a discipline. Code ships behind a flag, gets canaried to 1% of traffic, and rolls back in one click when a dashboard turns red. Schema migrations are staged and reversible. Even a CSS tweak goes through a pull request someone can read. Then there's the prompt. Someone edits a paragraph in a text box, clicks save, and the behavior of your product changes for every user at once — no canary, no diff anyone reviewed, no revert button that actually returns you to the prior state.

The uncomfortable part is that this isn't an oversight by careless teams. It's the default that the tooling produces. Prompts get filed under "configuration" because they're strings that live outside the compiled binary, and configuration has always been the thing you're allowed to change quickly without a full release. But a prompt isn't config. It's a program written in English, compiled by a nondeterministic interpreter you don't control, whose behavior you can only observe statistically. Treating it like a config value is the category error underneath a whole class of production incidents.

Every other line item on your infrastructure bill went through a gate. Someone filed a purchase order for the database cluster. Someone counted seats before buying the observability SaaS. Someone ran a capacity review before the team doubled its Kubernetes footprint. Then a model API showed up, and none of that happened.

An engineer added their API key to a config file. They wrote a create() call that looks exactly like every other function call in the codebase. It shipped. And the first time anyone in finance learned that this feature existed as a cost center was a variance line on the monthly invoice — a number nobody forecasted, nobody approved, and nobody can immediately explain.

This is the FinOps gap for AI, and it is not a monitoring problem. It is a governance problem wearing a monitoring costume. You can have perfect dashboards and still get surprised, because the spend was invisible to your approval process long before it was visible on a chart.

The most dangerous eval set is the one that still passes. A regression suite that goes red gets attention: someone opens the failing case, argues about it, fixes the bug or updates the expectation. A green suite gets trust. And trust is exactly what a rotted eval set does not deserve, because the score stayed green not because your system is good but because the test stopped resembling what your users actually do.

This is the quiet failure mode of AI evaluation. You build a golden dataset — a few hundred carefully labeled cases that represent the job your product does. It earns its keep for a quarter. Every deploy runs it, every score lands green, and everyone sleeps well. Meanwhile the product ships three new features, enterprise traffic climbs from ten percent of queries to forty-five, and users start phrasing requests in ways nobody on your team wrote down eighteen months ago. The eval set doesn't know about any of it. It keeps grading the model on a distribution that no longer exists.

There is a ritual that plays out on almost every team shipping an AI feature. Someone runs a load test, watches the p99 latency climb past two seconds, and files a ticket: "make it faster." The ticket lands on the infra team. They tune batch sizes, add GPUs, argue about the scheduler, and eventually claw the number down to 1.4 seconds. Everyone nods. The p99 is "handled."

The whole exercise is built on a false premise. The premise is that the latency target is a fact of the system — a physical constant the infra team discovers and then optimizes toward. It is not. The target is a choice, and it is a product choice. What counts as "fast enough" depends entirely on what the interface does while the user waits, and the interface is not the infra team's to design.

Nobody decides to give an agent forty tools. It happens the way a garage fills up. You wire in a search tool, then a database reader, then someone on the team ships a Slack integration, then the ticketing MCP server gets installed because it was one config line away. Each addition is individually reasonable. Nobody ever removes anything, because removing a tool feels like taking away a capability, and taking away a capability feels like a regression.

Six months later your agent has a tool belt with three tools it uses constantly, a dozen it uses occasionally, and a long tail of two dozen it has technically never selected in production. That long tail is not free. It is not even cheap. Every unused tool in the catalog is actively making the agent worse at choosing among the ones that matter.

Most teams threat-model the wrong plane. They harden the chat box — rate limits, input validation, a jailbreak classifier watching what the user types — and they treat everything the model reads as inert. The wiki page, the support ticket, the scraped webpage, the calendar invite, the PDF someone uploaded: data, not instructions. Background material for the model to summarize, not commands for it to obey.

That assumption is the vulnerability. The moment your agent retrieves content and drops it into the context window, that content is executing with the same authority as your system prompt. There is no privilege boundary between "here are your instructions" and "here is a document to consider." It's all just tokens, and the model was trained to follow instructions wherever they appear.

Your agent works. It resolves the ticket, issues the refund, updates the record, closes the loop — unattended, at a quality your metrics say beats the median human doing the same job. You are ready to take the human out of the loop. And then the deployment stalls, not in engineering, but in a meeting where someone from legal or finance asks a single question you cannot answer: when it's wrong, who eats the loss?

This is the part of agent autonomy that the capability curve doesn't touch. You can push accuracy from 95% to 99% and the question doesn't move. Because the blocker was never "is the model good enough." The blocker is that a probabilistic system taking irreversible actions is a risk somebody has to hold, and right now nobody wants to hold it. Your errors-and-omissions policy was written for human mistakes and increasingly carves out automated ones. Your model vendor's contract disclaims the whole thing. And no carrier has an actuarial table for a system whose failure rate quietly drifts every time you edit a prompt.

The demo wowed the executive team in a week. Eighteen months later, it is still a demo. The model still answers questions in the sandbox, the slide deck still gets recycled into new pitches, and every quarter someone asks why it hasn't shipped. Nobody has a good answer, because the honest one is uncomfortable: the part that impressed everyone was the cheap part, and nobody scoped the expensive part.

This is pilot purgatory, and it is now the default outcome. MIT's NANDA initiative found that roughly 95% of enterprise generative AI pilots deliver no measurable impact on the P&L. IDC and Lenovo, counting a different thing — POCs that literally never ship — put the number at 88%: for every 33 proofs of concept an enterprise starts, four reach production. The share of companies abandoning most of their AI initiatives jumped from 17% in 2024 to 42% in 2025. These are not stories about weak models. They are stories about a graduation gap that nobody budgeted for.

The uncomfortable truth is that a working demo represents maybe ten percent of the work required to run the thing in production, and it is the ten percent that looks like a hundred. Everything that makes an AI system safe to leave running — evals, guardrails, observability, cost controls, security review, on-call, and an owner with a budget — is invisible in a demo and non-negotiable in production.