778 posts tagged with "llm"

The incident bridge is quiet in a way that means everyone is stuck. A support ticket says the agent told a customer their refund was approved when it wasn't. You have the full trace: the prompt, the retrieved account record, the tool calls, the model's reasoning, the final message. You replay it. The agent does the right thing. You replay it again. Right again. Nine times out of ten, the trace that produced the incident produces a correct answer instead. Someone on the call finally asks the question that the retro template can't handle: so what was the root cause?

There isn't one. Not in the sense the template means. The five-whys chain runs "the agent told the customer the wrong thing" → "because the model generated an approval" → "because it sampled a token sequence that asserted approval" → "because... that's what the probability distribution allowed." The last why bottoms out in a shrug. "The model sampled a bad token" is technically true and operationally useless. It names no fix, assigns no owner, and closes no gap. You can write it in the report, but everyone reading it knows you've documented a coincidence, not a cause.

You wired up a second model vendor because the first one went down. It was a Tuesday, the primary provider's status page was a wall of red for a few hours, and your feature was dead in the water with it. So you did the responsible thing: you added a fallback. Now, if OpenAI is unreachable, you route to Anthropic. If Anthropic rate-limits you, you fall back to Gemini. The architecture diagram looks clean and grown-up. Reliability, solved.

Except you didn't add a replica. You added a second opinion. And a second opinion is a very different thing to operate than a second copy.

The mental model you imported comes from the stateless-service playbook: run three identical instances behind a load balancer, and if one dies the other two serve the exact same responses. That works because the replicas are byte-for-byte interchangeable. Two language models from two vendors are not. They were trained on different data, tuned with different objectives, and they disagree — systematically, not randomly — on exactly the inputs where your users notice.

Finance opens the invoice at the end of the month. One vendor. One number. It is bigger than last month, and it will be bigger next month. Then they ask the only question that matters — which feature spent this? — and nobody in the room can answer.

This is the quiet failure mode of running AI in production. Not that the bill is large; large is fine if the value is there. The failure is that the bill is unattributable. It arrives as a single line item — OpenAI, Anthropic, Bedrock, Azure — with none of the dimensions finance actually needs: not per feature, not per team, not per customer, not per successful task. You can see the total go up. You cannot see why, and by the time the invoice lands, the request context that could have explained it is long gone.

A support agent is about to issue a refund. Before it fires the tool call, your team added a gate: only proceed if the model says it's at least 90% confident. The model dutifully returns "confidence: 0.95," the refund goes out, and the citation it used to justify the amount — a policy clause about damaged goods — does not exist. It never did. The model invented the clause and then rated its own invention at 95%.

This is the trap. Teams reach for the model's confidence number because it looks like the probability you'd get from a calibrated classifier — a value where 0.9 means "right nine times out of ten." It is not that. An LLM's self-reported confidence is a fluent-sounding token like any other, shaped by tone, phrasing, and training incentives that have almost nothing to do with whether the underlying claim is true.

If you are gating real actions on that number, you are gating on a vibe.

A new frontier model ships. It's cheaper, faster, and tops every public leaderboard. You run it through your eval suite — the one you've spent eighteen months building — and it scores worse than the model you're running today. So you keep the incumbent, file the challenger under "not ready," and move on.

Here's the uncomfortable part: that result tells you almost nothing about which model is better. It tells you that your eval suite was built by watching your current model fail, one production incident at a time, and then patched to make those specific failures go away. The suite isn't a neutral measurement of quality. It's a catalog of one model's scar tissue. And a challenger that has different weaknesses will always look worse against a test set assembled from the incumbent's particular weaknesses — even when it's better on the traffic you actually serve.

This is incumbent bias, and it's the switching cost nobody prices into the migration decision. It quietly locks you onto a model long after a better option exists, and it does it while wearing the costume of rigorous engineering.

Your agent fleet opened forty pull requests this week. You reviewed the six that touched payment code, skimmed a few that happened to land while you had the tab open, and merged the rest on green CI. If someone asked you what your review policy is, you would describe something like this — and it is not a policy. It is a mood.

The numbers say most teams are in the same place. A recent large-scale study of agent-authored pull requests in popular open-source repositories found that 61% receive no recorded review activity at all, and of the ones that do get reviewed, most are reviewed only by other agents. Meanwhile the volume keeps climbing: agents now produce PRs, documents, support responses, and tickets at a rate no human review process was designed for. Reviewing everything is impossible. Reviewing nothing is malpractice. So teams improvise somewhere in the middle, with no stated rule, no measured coverage, and no way to say whether the current level of scrutiny is paranoid or reckless.

Manufacturing solved this exact problem a century ago. When Western Electric was churning out telephone equipment in the 1920s, inspecting every unit was economically impossible and shipping uninspected lots was unacceptable — so Bell Labs statisticians built acceptance sampling: a mathematically grounded discipline for deciding how much of a batch to inspect, when to reject the whole batch, and when a supplier has earned lighter scrutiny. It became MIL-STD-105 during World War II, then ANSI/ASQ Z1.4 and ISO 2859-1, and it still governs how a container of goods gets accepted at a port today. The mapping onto agent fleets is almost embarrassingly direct — and almost nobody in AI engineering has picked it up.

Every frontier lab's launch post now carries the same footnote: "with parallel test-time compute." Sonnet's SWE-bench number jumps about five points with it. GPT-5-class models cut errors by double digits with it. Gemini's Deep Think nearly doubles its ARC-AGI-2 score with it. Most engineering teams read that footnote as benchmark seasoning — a way to inflate a leaderboard number that no real system would pay for — and then go back to architecting their product around one attempt from the biggest model they can afford.

That instinct is roughly two years out of date. The labs didn't add parallel sampling to their pro tiers as a marketing garnish; they added it because running N attempts behind a single answer is often the cheapest way to buy quality, and sometimes the only way. When three attempts from a cheap model plus a decent selector beat one attempt from a model that costs ten times more per token, best-of-N stops being a benchmark trick and becomes an architecture decision — one with its own cost model, its own latency profile, and its own signature failure mode. The teams treating it that way are quietly shipping better answers at lower cost than the teams still doing one-shot inference on the flagship model.

You ran the deprecation playbook flawlessly. Announcement email six months out. A migration guide with code samples in four languages. Sunset headers on every v1 response. A banner in the developer dashboard. Two reminder emails. Then you turned off v1 — and instead of the quiet cutover the playbook promised, your error rate went vertical and stayed there. Traffic didn't migrate. It just kept arriving, failing, and retrying, at higher volume than before, because every failure triggered another attempt.

The playbook didn't fail because you executed it badly. It failed because it assumes a human is on the other end — someone who reads email, skims changelogs, and files a Jira ticket to migrate before the deadline. A growing share of your traffic has no such person behind it. It comes from agents whose knowledge of your API is frozen in system prompts, tool schemas, and model training data scattered across thousands of repositories you can't see and will never reach. Your sunset notice was written for a reader. Your biggest client is a prompt.

Your cloud bill can tell you, down to the tag, that a forgotten S3 bucket cost $14,000 last month. Ask the same question of your LLM bill — which feature burned $40,000 in tokens — and the honest answer at most companies is a shrug. The provider invoice has one line per model per API key, three teams share the key, and the "AI costs" row in the finance spreadsheet is allocated by headcount, vibes, or whoever complained least in the last planning cycle.

This is not a small bookkeeping annoyance. When nobody can name the feature behind a dollar of token spend, two failure modes follow. Cheap features get throttled because they share a budget line with an expensive one. And genuinely wasteful features survive forever, because their cost is invisible — smeared across a shared key, a shared cache, and a shared agent loop that serves six different product surfaces.

The FinOps Foundation's 2026 survey found that 98% of organizations now actively manage AI spend, up from 63% a year earlier and 31% the year before — the fastest adoption curve the foundation has ever recorded. Everyone is suddenly doing "FinOps for AI." Far fewer have noticed that the core primitive of cloud FinOps — the resource tag — doesn't exist in the token world, and that three specific mechanics of modern LLM usage actively destroy attribution.

GraphQL's core bet was that clients should compose their own data requirements. For a decade, that bet mostly lost — because the clients were human teams who didn't want to compose anything. Frontend engineers wanted a stable endpoint they could call and forget. The flexibility GraphQL sold was a tax they paid in resolver complexity, caching workarounds, and security review, in exchange for a benefit — per-request field selection — that a known, slow-changing web app barely needed. By 2024, the practitioner consensus had visibly cooled: most internal APIs serve two or three known clients, and a well-shaped REST endpoint or a BFF layer covers them fine.

Then a new kind of client showed up. An AI agent doesn't have a fixed set of screens. It decides, per task, what data it needs, and it pays for every byte of the response — literally, in tokens, and cognitively, in degraded reasoning as the context window fills with fields nobody asked for. The client that actually composes its own data requirements finally exists. It just isn't human.

Here is a migration story that repeats in every team shipping agents. You built your agent on one provider's SDK. Tool definitions live as JSON schemas in the exact shape that provider expects. Tool results get formatted into that provider's message structure. Then something forces a change — a better model ships from a competitor, procurement mandates a second provider for redundancy, an MCP server replaces a hand-rolled integration — and you discover the real inventory of the migration: it isn't one API client. It's every tool definition, every result formatter, every retry handler, and every test fixture in the codebase.

The failure isn't that you chose the wrong provider. It's that you let someone else's serialization format become your internal architecture. There is a twenty-year-old answer to exactly this problem — Alistair Cockburn's hexagonal architecture, better known as ports and adapters — and agent systems are the most compelling new use case it has had in years.

In 1980, Jon Postel wrote a sentence into the TCP specification that became a founding principle of the internet: "be conservative in what you do, be liberal in what you accept from others." For forty years, engineers have applied it everywhere — parsers that tolerate trailing commas, APIs that coerce "10" into 10, HTML renderers that quietly repair broken markup. The web arguably exists because browsers forgave everyone's mistakes.

Then the caller stopped being a human, and the advice inverted. When an AI agent calls your tool with a stringly-typed number, a mis-nested JSON object, or an enum value that's almost right, the tolerant parser that saves a human developer twenty minutes of debugging does something much worse to the agent: it confirms that the sloppy call was correct. The agent's only training signal inside a loop is the feedback your tool returns. Accept garbage, and you are teaching the model — right now, in this episode — that garbage works.