22 posts tagged with "llm-cost"

Nothing broke. That's the disorienting part. No deploy failed, no latency alarm fired, no error rate ticked up. Someone merged a one-line PR that appended a sentence to the system prompt — a new tool description, a policy reminder, a "today's date is" header — and the next morning the inference bill was three to five times higher. Traffic was flat. The model was the same. The code did exactly what it was supposed to do.

What changed is that the one line landed in the wrong place, and every cached prefix in your fleet invalidated at once. Your cache hit rate went from 90% to zero in a single request cycle, and every token that used to be nearly free started billing at full price. This is the prompt-cache cliff, and it's the most expensive failure mode in production LLM systems that nobody threat-models, because it doesn't look like a failure at all.

Your application opens a long conversation with a forty-thousand-token system prompt and a full tool inventory. Turn 1 pays the prefix at write rates and the provider's KV cache warms up. Turn 2 arrives ninety seconds later. You assume it's a cache hit. Sometimes it is. Sometimes the same forty thousand tokens land on your invoice again at uncached prices, and nothing in your code changed between turn 1 and turn 2.

The thing that changed was somebody else's traffic. The KV cache is shared infrastructure. Your tenant was colocated on a serving node that, in the ninety seconds between your two turns, took on enough other tenants to evict your prefix from memory. The provider's dashboard will describe this as "cache pressure." Your finance team will describe it as a line item that doubled. Both descriptions are accurate. Neither is in your code.

Your finance team filed a dispute and lost. The line item is "output tokens" and it exceeds your sum-of-delivered-tokens metric by fourteen percent. The provider's support engineer closed the ticket as "expected behavior under streaming cancellation," with a link to a documentation page that says "cancellation stops billing at the last delivered token." Both sentences are true, and the gap between them is the line of code you have not written.

The contract you read says one thing. The inference scheduler does another. The mismatch is not a bug, not a billing error, and not malice — it is a layered system in which the cancellation signal travels through three boundaries (browser, edge, GPU) and the billing meter sits at the third boundary while your "stop generating" button sits at the first. Closing the gap is an engineering project with a finance owner.

The release notes were two lines long. "Improved multilingual tokenization. No breaking changes to model outputs." Nine words. Your evals confirmed it: same prompts, same completions, same scores. Your platform team signed off on the upgrade Friday afternoon. By Tuesday morning your cache hit rate had collapsed from 80% to 4%, your daily inference bill had quadrupled, and the on-call engineer who paged you at 6am could not find a single line of your code that had changed.

Nothing in your code had changed. The provider had shipped a new tokenizer that split one Unicode glyph one byte differently than the old one. Every cached prefix in your system was now fingerprinted against a token sequence that no longer existed. The model behaved identically — that was true. The cache layer, which the release notes did not mention, paid the bill in full.

Finance sent a memo on a Monday. By Friday, every product team had shipped a prompt change, and on the following Tuesday the support queue grew by a third. Nobody had touched the model. Nobody had touched the product. The only thing that had changed was that the LLM bill was now flowing back to the teams that issued the calls — and the teams had responded the way any rational cost center responds to a new line item on its P&L. They cut it.

The story that gets told inside the company afterwards is a story about prompt engineering, or about the model regressing, or about a noisy week of user traffic. The truer story is that finance, through a chargeback policy, had quietly become a product manager. The cost-attribution dashboard was a product-quality lever that nobody had reviewed, nobody had instrumented for, and nobody owned. When it moved, every prompt in the company moved with it, and the trade-offs that produced the quality regression were never seen by the people whose job it was to see them.

You taught the agent to say "I don't know" when the context was thin and called it a safety win. The OpenAI bill went down. Everyone agreed it was the responsible move. Three months later your VP of Support is asking why headcount projections are off by 40% and nobody in the AI org has an answer, because the metric you tracked was abstention rate and the metric that moved was tickets-per-week — and nobody owned the line that summed them.

This is the abstention tax. It's not a model cost. It doesn't show up on the inference invoice. It shows up downstream, in the queue depth of the human team that catches every "I cannot answer," in the second model call that runs against the enriched context the human had to assemble, in the customer who churned during the wait. The model-only cost frame quietly hides it. And the org seam where the AI team owns abstention and the ops team owns the queue means nobody is incentivized to see it.

The migration looked clean. Evals were re-anchored against the new model version. Judge prompts were re-calibrated. Two weeks of shadow traffic showed behavior parity within tolerance. p50 and p99 latency were inside the budget. The rollout call signed off on Thursday afternoon and the team went home.

By Friday morning, the inference bill was 3x normal. Eval scores were still fine. Latency was still fine. No one on the rollout call had thought to instrument the cache hit rate, because the prefix had not changed — the system prompt was byte-identical, the tool definitions were byte-identical, the conversation framing was byte-identical. What had changed was the model version in the request body, and the provider keys its prefix cache on (prefix bytes + model version). Every request after the cutover landed on a cold cache. The warm-up curve took six weeks of organic traffic to recover, and the team paid full input-token rates for every token on every request for the duration.

Somewhere in your company, four people each believe a fifth person owns the inference bill. Engineering treats it as a cloud line item. The AI team treats it as the price of building. Finance treats it as a variable margin input that someone in engineering must already be managing. Product treats it as overhead that engineering absorbs. The bill keeps growing, and the only thing everyone agrees on is that it isn't theirs.

This is not a budgeting problem. It is an ownership vacuum, and it surfaces the first time the line item gets large enough for a CFO to ask about it on a board call. By then, the answers people improvise — "we'll optimize," "we'll cache more," "we'll switch models" — describe interventions without naming an owner. The conversation that should have happened a year earlier was not about how to lower the bill. It was about whose P&L the bill belonged to in the first place.

The shift is structural. Inference moved from 15% of enterprise AI spend in 2024 to roughly 85% in 2026, and the average enterprise AI budget grew from $1.2M to around $7M over the same window. A line item that was once rounding error is now the kind of number a board notices, and the org chart written before that shift has no row for it.

A failed agent run is cheap. It misroutes a query, hits a dead end, returns "I couldn't help with that," and burns maybe a few hundred tokens doing it. A successful run is the disaster. It retrieves context, reflects on it, calls three tools, reflects again, and stitches together a confident multi-paragraph answer — fifty times the token spend of the failure that cost you nothing.

This is the uncomfortable shape of agent economics: your happy path is your expensive path. The outcome you are selling, the one your marketing page promises, the one users thank you for, is the single most costly thing your system can do. And if you priced the product the way SaaS has been priced for fifteen years — a flat monthly fee per seat — then every time the agent does its job well, it quietly erodes your margin.

Most teams discover this backwards. They watch cost dashboards, see failures are cheap, and conclude that reliability work will save money. It won't. Raising your success rate raises your bill.

Finance asks one question about every feature you ship: "What does it cost per user?" For a traditional feature, the answer is a number. A page render, a database query, a push notification — each has a marginal cost that barely moves from one request to the next. You measure it once, multiply by your user count, and the forecast holds.

An AI feature breaks that contract. Ask "what does this agent cost per request" and the honest answer is not a number, it's a histogram. The same agent that resolves one ticket for two cents will burn four dollars on the next one, because that user asked a vague question, the agent looped through eleven tool calls, and each call dragged the entire growing conversation back through the model. The mean of those two requests — two dollars — describes neither of them, and it definitely doesn't describe the bill.

That is the trap. When you hand finance a single average cost, you are not simplifying a messy reality. You are reporting a number that is wrong in a specific, expensive direction.

Connect a single GitHub MCP server to your agent and you've already spent twelve to forty thousand tokens before the user types a word. Connect a filesystem server, a calendar, a database, an internal CRM, and a third-party tool catalog, and a heavy desktop configuration has been measured at sixty-six thousand tokens of pure tool disclosure — nearly a third of Claude Sonnet's 200K window, paid every single planning turn. The agent hasn't done anything yet. The user hasn't asked anything yet. The bill is already running.

This is the disclosure tax, and it is the most underpriced line item in agentic systems shipping right now. Teams add MCP servers the way teams once added microservices — each integration looks like a free composition primitive, the procurement story writes itself ("more tools = more capability"), and the unit economics dashboard never surfaces the per-server cost because the cost lives inside a token bucket nobody attributes back to the connector. The result is an agent that gets slower, dumber, and more expensive every time someone adds another integration, and a team that explains the regression by re-tuning prompts and chasing the model vendor for a new version.

The conversation looks healthy on the dashboard. Average tokens per call is sane, the p50 input length is comfortably inside the cached prefix, the provider invoice ticks up at the rate finance approved. Then someone exports a single 200-turn coding session and the line item for that one user is larger than the rest of the team's daily traffic combined. The dashboard wasn't lying — it was averaging. The bill comes from the long tail, and the long tail does not scale linearly with turn count.

Every multi-turn AI feature eventually meets this surprise. The per-call token count is the wrong unit of measurement, because the cost of a 30-turn conversation is not 30 times the cost of a single turn — it's something between 50× and 200×, depending on how the history is structured, how the prompt cache decays, and what tier the request lands in once the input crosses 200K tokens. The team that priced the feature off the per-call number is underwriting a tail it never modeled.