76 posts tagged with "cost-optimization"

Every other line item on your infrastructure bill went through a gate. Someone filed a purchase order for the database cluster. Someone counted seats before buying the observability SaaS. Someone ran a capacity review before the team doubled its Kubernetes footprint. Then a model API showed up, and none of that happened.

An engineer added their API key to a config file. They wrote a create() call that looks exactly like every other function call in the codebase. It shipped. And the first time anyone in finance learned that this feature existed as a cost center was a variance line on the monthly invoice — a number nobody forecasted, nobody approved, and nobody can immediately explain.

This is the FinOps gap for AI, and it is not a monitoring problem. It is a governance problem wearing a monitoring costume. You can have perfect dashboards and still get surprised, because the spend was invisible to your approval process long before it was visible on a chart.

Most teams buy inference the way their predecessors bought EC2 in 2010: everything on-demand, metered to the token, and surprising on two axes at once. The bill is one surprise. The rate limit is the other — the 429 that shows up in the middle of a launch, right when the on-demand pool you never reserved gets contended by everyone else who also chose to pay retail. Then someone opens the pricing page and discovers that the provider has been quietly selling reserved capacity the whole time: provisioned throughput, committed-use discounts, dollars-per-unit-per-hour instead of dollars-per-million-tokens. The reserved-instance decision that took the cloud industry a decade to internalize is sitting right there for tokens, and almost nobody has ported the playbook.

The reason isn't ignorance. It's that the reserved-instance math you learned for compute doesn't transfer cleanly, and the ways it breaks are exactly the ways that punish a naive commitment. A reserved EC2 instance is a bet that you'll still want that instance type in a year. A reserved block of token throughput is a bet that you'll still want that model in a year — and models have a shelf life measured in months, not decades. The commitment structure is familiar. The thing you're committing to is not.

Your cloud bill can tell you, down to the tag, that a forgotten S3 bucket cost $14,000 last month. Ask the same question of your LLM bill — which feature burned $40,000 in tokens — and the honest answer at most companies is a shrug. The provider invoice has one line per model per API key, three teams share the key, and the "AI costs" row in the finance spreadsheet is allocated by headcount, vibes, or whoever complained least in the last planning cycle.

This is not a small bookkeeping annoyance. When nobody can name the feature behind a dollar of token spend, two failure modes follow. Cheap features get throttled because they share a budget line with an expensive one. And genuinely wasteful features survive forever, because their cost is invisible — smeared across a shared key, a shared cache, and a shared agent loop that serves six different product surfaces.

The FinOps Foundation's 2026 survey found that 98% of organizations now actively manage AI spend, up from 63% a year earlier and 31% the year before — the fastest adoption curve the foundation has ever recorded. Everyone is suddenly doing "FinOps for AI." Far fewer have noticed that the core primitive of cloud FinOps — the resource tag — doesn't exist in the token world, and that three specific mechanics of modern LLM usage actively destroy attribution.

Pull up your token dashboard and ask one question about every workload on it: was a human waiting for this response? For most teams running agents in production, the honest answer is no for half the bill or more. Eval suites, embedding backfills, nightly report generation, bulk classification, overnight code migration, summarization of yesterday's tickets — none of it has a user staring at a spinner. Yet nearly all of it flows through the interactive endpoint, at full price, competing for the same capacity as the requests that actually are latency-critical.

Every major provider will run that deferrable work for half the cost. OpenAI's Batch API, Anthropic's Message Batches, and Gemini's batch mode all price asynchronous jobs at a flat 50% discount in exchange for a 24-hour completion window. The discount requires no negotiation, no committed spend, no engineering heroics. It requires only that you admit, in your architecture, that some work can wait — and most teams have never made that admission, because nobody made deferral a design decision.

We have seen this movie before. Spot instances offered 60–90% off cloud compute for years while most teams kept everything on on-demand, not because the savings were unreal but because using them forced an uncomfortable question: which of our workloads can tolerate interruption? Teams that answered it built checkpointing and cut their compute bills by more than half. Teams that didn't kept paying the "everything is urgent" tax. The batch tier is the same fork in the road, except the axis is latency tolerance instead of interruption tolerance — and agentic workloads, which burn 5–30x more tokens per task than chatbots, make the price of not choosing much steeper.

Every major model provider will sell you the same tokens for half price. OpenAI, Anthropic, and Google all run batch APIs that charge 50% of the synchronous rate — same models, same prompts, same outputs — in exchange for one concession: you accept a 24-hour completion window instead of an answer in seconds. For a team spending $50,000 a month on inference, that is $25,000 sitting on the table, claimable without changing a single prompt.

Most teams never claim it. Not because the discount is hidden — it's on every pricing page — but because claiming it requires answering a question nobody in the org has asked: which of our inference calls actually need an answer now? That question turns out to be an architecture question, and the honest answer at most companies is "we never classified them, so everything runs in the interactive lane by default." The batch discount isn't a pricing footnote. It's a test of whether your system knows its own latency requirements — and most systems fail it.

A team I talked to recently spent three engineer-weeks shaving their average prompt from 4,000 tokens to 2,600. They were proud of it — a clean 35% reduction, real numbers, a nice graph in the deck. Then someone did the arithmetic the other direction. The savings came to roughly $1,800 a month. The three engineer-weeks they spent getting there cost something like $25,000 in fully-loaded salary. At that monthly run rate, the optimization pays for itself in about fourteen months — assuming the prompt never changes, the model never gets cheaper, and those engineers had nothing more valuable to build.

None of those assumptions held. The prompt changed twice in the next quarter. The model they were on dropped its input price by 40% on its own. And the feature those engineers didn't ship that month was the one the biggest customer had been asking about.

Most finance teams filed their first LLM bill in the wrong drawer. It arrived during the experimentation phase, when a handful of engineers were prototyping with an API key, and it looked like exactly what it was at the time: research and development. A few thousand dollars a month to figure out whether the technology worked. So it went into R&D, mentally and sometimes literally, and nobody thought harder about it.

Then the feature shipped, usage climbed, and the same line item that was a rounding error in Q1 became the fastest-growing cost on the cloud invoice by Q4. The problem was never the dollar amount. The problem was that the cost had quietly changed categories — from a fixed bet on building something to a variable cost of serving every customer who used it — and the mental model hadn't moved with it.

That misclassification is the single most expensive accounting mistake in AI products right now, and it's not really an accounting mistake at all. It's a forecasting one.

A cache that is being written to but never read from is not a cache. It is a logging system with extra latency, billed by the kilobyte. And the cruelest version of this failure mode is the one where the cache looks healthy from every angle except the one that matters: the set calls succeed, the get calls return quickly, the keys are well-formed, the values are valid, the TTLs are sensible. The only thing wrong is that no get call ever finds the key a previous set call wrote, because a single field in the key changes every time it is computed.

This is the story of a debugging session that added a timestamp to a cache key "so I can tell which cache entry I'm looking at," and the system that quietly paid for fourteen extra LLM calls per conversation for two weeks before anyone noticed.

Your incident timeline shows no deploys. Your code did not change. Your traffic mix did not change. Your prompts did not change. And yet your p99 output length doubled inside a week, your downstream rendering layer started clipping responses, and your output-token bill rose 38% on traffic that wasn't asking for longer answers. The change was real, the regression was measurable, and nothing in your version control system records it — because the value that moved was one your code never sent.

The provider raised an implicit default. The release notes filed it under "improved long-form behavior." The parameter in question was max_tokens, which your application has been omitting since day one because the documented default was generous and your outputs rarely came close. The default moved from 4096 to 8192 to accommodate longer reasoning in the provider's newer models. Your application got the new default whether you wanted it or not, because the absence of a parameter is itself a configuration choice — and the provider owns the right to change the value behind it.

This is the failure mode where a "no-op" release on the provider's side propagates through your system as a behavior change, a cost change, and a UX change all at once, and your team's only diagnostic signal is the bill arriving at the end of the month.

A platform team shipped a prompt refactor that cut average response cost by thirty-two percent. The change was simple: strip the "explain your reasoning" preamble, ask the model to return only the JSON object, and drop the post-processing step that parsed the rationale out of the model's prose. The dashboard turned green. The unit economics page in the quarterly review went from yellow to gold. Nobody on the platform team thought to consult the risk team, because no part of the change touched the answer the customer received.

Two quarters later, a regulated customer's auditor requested the decision rationale for a denied-loan letter from a date six months prior. The team pulled the trace. The input was there. The output was there. The reasoning was gone — not because anyone deleted it, but because it had stopped being produced the day the refactor shipped. The customer's compliance program had been operating on the assumption that the rationale was somewhere in the trace store; the platform team had been operating on the assumption that the rationale was nobody's problem because the customer-facing answer was unchanged. Both assumptions were correct in isolation. Together they cost the customer a regulatory finding and the platform team a contract renewal.

The product team ships personalization. The agent now greets the user by name, tunes its response length to their stated preference, knows the user works in healthcare, and respects the user's timezone for any date it mentions. The satisfaction lift is real and measurable — the A/B is a four-point win on thumbs-up rate and the rollout goes to one hundred percent. Three weeks later, finance flags that inference spend has roughly tripled, and nobody on the AI team can immediately explain why.

The explanation is one line of code change buried in the system-prompt builder. Per-user context — name, preferred response length, industry, timezone — got prepended to the system prompt so the model would see it on every turn. That made every user's prompt unique from the first token. Your provider's prompt cache, which had been serving roughly ninety percent of your input tokens at one-tenth the standard price, stopped hitting. Latency barely moved, so the perf dashboard stayed green. The billing dashboard caught up at month-end.

Your planner emits five tool calls. On paper, it reads like a clean solution: lookup_user, search_documents, call_external_api, spawn_sub_agent, request_human_approval. The trace looks elegant, the logic is sound, the agent will arrive at the right answer. In production, those five steps take 12 milliseconds, 800 milliseconds, 4 seconds, 2 minutes, and 6 hours respectively. The planner never noticed that its five-step plan spans nine orders of magnitude in cost.


This is not a hallucination. The model picked the right tools. It picked them in a sensible order. What it could not do — what the tool schema gave it no way to do — was reason about the fact that the last step in its plan is qualitatively different from the first one. To the planner, a tool is a tool. Every node in the plan graph has weight one.