990 posts tagged with "insider"

Most teams buy inference the way their predecessors bought EC2 in 2010: everything on-demand, metered to the token, and surprising on two axes at once. The bill is one surprise. The rate limit is the other — the 429 that shows up in the middle of a launch, right when the on-demand pool you never reserved gets contended by everyone else who also chose to pay retail. Then someone opens the pricing page and discovers that the provider has been quietly selling reserved capacity the whole time: provisioned throughput, committed-use discounts, dollars-per-unit-per-hour instead of dollars-per-million-tokens. The reserved-instance decision that took the cloud industry a decade to internalize is sitting right there for tokens, and almost nobody has ported the playbook.

The reason isn't ignorance. It's that the reserved-instance math you learned for compute doesn't transfer cleanly, and the ways it breaks are exactly the ways that punish a naive commitment. A reserved EC2 instance is a bet that you'll still want that instance type in a year. A reserved block of token throughput is a bet that you'll still want that model in a year — and models have a shelf life measured in months, not decades. The commitment structure is familiar. The thing you're committing to is not.

The worst place to discover that you shipped an AI agent is in an incident review. Not the design doc, not the architecture review, not the change ticket — the incident review, where a lawyer is asking why a customer's account got a refund nobody authorized, and an engineer is scrolling through a service that was last meaningfully reviewed eleven months ago, and there, forty lines into a function called enrichTicket, is a model call that reads the customer record, decides on a resolution, and calls the billing API. Nobody diagrammed it. Nobody approved it as an agent, because to the person who wrote it, it wasn't one. It was "just a helper."

This is shadow AI, and it has grown teeth. The first wave was employees pasting company data into consumer chatbots — a data-leakage problem, bad but bounded. The second wave is agents: model calls wired into internal tools, reading real data and taking real actions, sitting inside services that were approved for something else entirely. Roughly half of employees admit to using AI tools their employer never sanctioned, and a striking share of that usage comes from the top — directors and executives are among the biggest culprits, not the interns. When the same instinct reaches your codebase, you don't get a leaked spreadsheet. You get an autonomous actor with production credentials that nobody signed off on.

The uncomfortable part is that shadow agents aren't created by reckless people. They're created by good engineers doing exactly what you asked them to do: ship value fast, reuse existing infrastructure, don't file a ticket for every small change. The governance gap isn't a discipline problem. It's a definitional one — your review process doesn't have a category for what they built.

Your eval dashboard is green. Turn-level accuracy sits at 95%, the LLM judge agrees with your annotators, and every regression test passes on the way to production. Then a user files a bug: the agent recommended a Postgres index on turn nine that directly contradicted the "we're on DynamoDB" constraint the user set on turn one. You pull the transcript. Every single turn, read in isolation, is a reasonable response. The conversation, read as a whole, is a disaster.

This is the central lie of turn-level evaluation. It grades request-response pairs because that's the unit that's cheap to label, and it quietly assumes that a conversation is just a bag of independent turns you can average. It isn't. The response at turn nine is conditioned on everything that came before it, and the failures that actually reach users almost never live inside a single turn — they live in the seams between turns, where state gets dropped, assumptions harden, and small errors compound into a wrong final answer.

The upgrade looked like free money. The provider shipped a newer model that scored higher on every public benchmark, cost less per token, and returned tokens faster. You bumped the model string in one config file, ran the eval suite, watched all 340 cases go green, and shipped it Tuesday afternoon. By Thursday, support tickets were climbing and nobody could point at a single failing test.

This is the most disorienting failure mode in applied LLM work, because it violates the deal every other kind of software makes with you: if the tests pass, the behavior held. Here that contract is void. A model upgrade is not a version bump on a library whose interface you control. It is a silent, wholesale swap of a probabilistic function, and your eval suite only checks the handful of behaviors you thought to write down.

The regressions that hurt live in the behaviors you never encoded — tone, verbosity, format habits, how the model handles the ambiguous middle of a request. Those are precisely the things your users came to rely on, and precisely the things a pass/fail assertion can't see.

The incident starts the way most of these do: a customer-facing feature throwing 429s in production at 2 p.m. on a Tuesday, no deploy, no traffic spike, nothing in the feature's own logs to explain it. The on-call engineer stares at the dashboard for twenty minutes before someone in another channel mentions, almost in passing, that they kicked off a "quick backfill" to re-summarize a few hundred thousand old documents. Two teams, two codebases, two on-call rotations — and one API key's worth of rate limit between them. The backfill ate the budget. The chatbot starved.

This is the noisy neighbor problem, and the twist that makes it so dangerous with model APIs is that the neighbor isn't some anonymous tenant on shared cloud hardware. The neighbor is another team at your own company, and the wall between you is thinner than anyone thinks.

The incident bridge is quiet in a way that means everyone is stuck. A support ticket says the agent told a customer their refund was approved when it wasn't. You have the full trace: the prompt, the retrieved account record, the tool calls, the model's reasoning, the final message. You replay it. The agent does the right thing. You replay it again. Right again. Nine times out of ten, the trace that produced the incident produces a correct answer instead. Someone on the call finally asks the question that the retro template can't handle: so what was the root cause?

There isn't one. Not in the sense the template means. The five-whys chain runs "the agent told the customer the wrong thing" → "because the model generated an approval" → "because it sampled a token sequence that asserted approval" → "because... that's what the probability distribution allowed." The last why bottoms out in a shrug. "The model sampled a bad token" is technically true and operationally useless. It names no fix, assigns no owner, and closes no gap. You can write it in the report, but everyone reading it knows you've documented a coincidence, not a cause.

Nothing broke. That's the disorienting part. No deploy failed, no latency alarm fired, no error rate ticked up. Someone merged a one-line PR that appended a sentence to the system prompt — a new tool description, a policy reminder, a "today's date is" header — and the next morning the inference bill was three to five times higher. Traffic was flat. The model was the same. The code did exactly what it was supposed to do.

What changed is that the one line landed in the wrong place, and every cached prefix in your fleet invalidated at once. Your cache hit rate went from 90% to zero in a single request cycle, and every token that used to be nearly free started billing at full price. This is the prompt-cache cliff, and it's the most expensive failure mode in production LLM systems that nobody threat-models, because it doesn't look like a failure at all.

You can stand up a staging database. You can stand up a staging queue, a staging payments sandbox, a staging copy of every third-party API you depend on. For thirty years the whole discipline of pre-production has been built on one assumption: that you can create a faithful-enough replica of production, poke it, and learn something true about what will happen when you ship.

Then you added a hosted model to your critical path, and the assumption quietly broke. The one component whose behavior now dominates your product — the thing that decides what your app actually says and does — is the one component you cannot stand up a staging copy of. It is versioned by someone else, rate-limited by someone else, and quietly updated by someone else on a schedule you don't see. Your staging environment has a staging everything, except a staging model.

Finance opens the invoice at the end of the month. One vendor. One number. It is bigger than last month, and it will be bigger next month. Then they ask the only question that matters — which feature spent this? — and nobody in the room can answer.

This is the quiet failure mode of running AI in production. Not that the bill is large; large is fine if the value is there. The failure is that the bill is unattributable. It arrives as a single line item — OpenAI, Anthropic, Bedrock, Azure — with none of the dimensions finance actually needs: not per feature, not per team, not per customer, not per successful task. You can see the total go up. You cannot see why, and by the time the invoice lands, the request context that could have explained it is long gone.

A support agent is about to issue a refund. Before it fires the tool call, your team added a gate: only proceed if the model says it's at least 90% confident. The model dutifully returns "confidence: 0.95," the refund goes out, and the citation it used to justify the amount — a policy clause about damaged goods — does not exist. It never did. The model invented the clause and then rated its own invention at 95%.

This is the trap. Teams reach for the model's confidence number because it looks like the probability you'd get from a calibrated classifier — a value where 0.9 means "right nine times out of ten." It is not that. An LLM's self-reported confidence is a fluent-sounding token like any other, shaped by tone, phrasing, and training incentives that have almost nothing to do with whether the underlying claim is true.

If you are gating real actions on that number, you are gating on a vibe.

Your agent fleet opened forty pull requests this week. You reviewed the six that touched payment code, skimmed a few that happened to land while you had the tab open, and merged the rest on green CI. If someone asked you what your review policy is, you would describe something like this — and it is not a policy. It is a mood.

The numbers say most teams are in the same place. A recent large-scale study of agent-authored pull requests in popular open-source repositories found that 61% receive no recorded review activity at all, and of the ones that do get reviewed, most are reviewed only by other agents. Meanwhile the volume keeps climbing: agents now produce PRs, documents, support responses, and tickets at a rate no human review process was designed for. Reviewing everything is impossible. Reviewing nothing is malpractice. So teams improvise somewhere in the middle, with no stated rule, no measured coverage, and no way to say whether the current level of scrutiny is paranoid or reckless.

Manufacturing solved this exact problem a century ago. When Western Electric was churning out telephone equipment in the 1920s, inspecting every unit was economically impossible and shipping uninspected lots was unacceptable — so Bell Labs statisticians built acceptance sampling: a mathematically grounded discipline for deciding how much of a batch to inspect, when to reject the whole batch, and when a supplier has earned lighter scrutiny. It became MIL-STD-105 during World War II, then ANSI/ASQ Z1.4 and ISO 2859-1, and it still governs how a container of goods gets accepted at a port today. The mapping onto agent fleets is almost embarrassingly direct — and almost nobody in AI engineering has picked it up.

Every frontier lab's launch post now carries the same footnote: "with parallel test-time compute." Sonnet's SWE-bench number jumps about five points with it. GPT-5-class models cut errors by double digits with it. Gemini's Deep Think nearly doubles its ARC-AGI-2 score with it. Most engineering teams read that footnote as benchmark seasoning — a way to inflate a leaderboard number that no real system would pay for — and then go back to architecting their product around one attempt from the biggest model they can afford.

That instinct is roughly two years out of date. The labs didn't add parallel sampling to their pro tiers as a marketing garnish; they added it because running N attempts behind a single answer is often the cheapest way to buy quality, and sometimes the only way. When three attempts from a cheap model plus a decent selector beat one attempt from a model that costs ten times more per token, best-of-N stops being a benchmark trick and becomes an architecture decision — one with its own cost model, its own latency profile, and its own signature failure mode. The teams treating it that way are quietly shipping better answers at lower cost than the teams still doing one-shot inference on the flagship model.