47 posts tagged with "finops"

Every other line item on your infrastructure bill went through a gate. Someone filed a purchase order for the database cluster. Someone counted seats before buying the observability SaaS. Someone ran a capacity review before the team doubled its Kubernetes footprint. Then a model API showed up, and none of that happened.

An engineer added their API key to a config file. They wrote a create() call that looks exactly like every other function call in the codebase. It shipped. And the first time anyone in finance learned that this feature existed as a cost center was a variance line on the monthly invoice — a number nobody forecasted, nobody approved, and nobody can immediately explain.

This is the FinOps gap for AI, and it is not a monitoring problem. It is a governance problem wearing a monitoring costume. You can have perfect dashboards and still get surprised, because the spend was invisible to your approval process long before it was visible on a chart.

Most teams buy inference the way their predecessors bought EC2 in 2010: everything on-demand, metered to the token, and surprising on two axes at once. The bill is one surprise. The rate limit is the other — the 429 that shows up in the middle of a launch, right when the on-demand pool you never reserved gets contended by everyone else who also chose to pay retail. Then someone opens the pricing page and discovers that the provider has been quietly selling reserved capacity the whole time: provisioned throughput, committed-use discounts, dollars-per-unit-per-hour instead of dollars-per-million-tokens. The reserved-instance decision that took the cloud industry a decade to internalize is sitting right there for tokens, and almost nobody has ported the playbook.

The reason isn't ignorance. It's that the reserved-instance math you learned for compute doesn't transfer cleanly, and the ways it breaks are exactly the ways that punish a naive commitment. A reserved EC2 instance is a bet that you'll still want that instance type in a year. A reserved block of token throughput is a bet that you'll still want that model in a year — and models have a shelf life measured in months, not decades. The commitment structure is familiar. The thing you're committing to is not.

Nothing broke. That's the disorienting part. No deploy failed, no latency alarm fired, no error rate ticked up. Someone merged a one-line PR that appended a sentence to the system prompt — a new tool description, a policy reminder, a "today's date is" header — and the next morning the inference bill was three to five times higher. Traffic was flat. The model was the same. The code did exactly what it was supposed to do.

What changed is that the one line landed in the wrong place, and every cached prefix in your fleet invalidated at once. Your cache hit rate went from 90% to zero in a single request cycle, and every token that used to be nearly free started billing at full price. This is the prompt-cache cliff, and it's the most expensive failure mode in production LLM systems that nobody threat-models, because it doesn't look like a failure at all.

Finance opens the invoice at the end of the month. One vendor. One number. It is bigger than last month, and it will be bigger next month. Then they ask the only question that matters — which feature spent this? — and nobody in the room can answer.

This is the quiet failure mode of running AI in production. Not that the bill is large; large is fine if the value is there. The failure is that the bill is unattributable. It arrives as a single line item — OpenAI, Anthropic, Bedrock, Azure — with none of the dimensions finance actually needs: not per feature, not per team, not per customer, not per successful task. You can see the total go up. You cannot see why, and by the time the invoice lands, the request context that could have explained it is long gone.

Your cloud bill can tell you, down to the tag, that a forgotten S3 bucket cost $14,000 last month. Ask the same question of your LLM bill — which feature burned $40,000 in tokens — and the honest answer at most companies is a shrug. The provider invoice has one line per model per API key, three teams share the key, and the "AI costs" row in the finance spreadsheet is allocated by headcount, vibes, or whoever complained least in the last planning cycle.

This is not a small bookkeeping annoyance. When nobody can name the feature behind a dollar of token spend, two failure modes follow. Cheap features get throttled because they share a budget line with an expensive one. And genuinely wasteful features survive forever, because their cost is invisible — smeared across a shared key, a shared cache, and a shared agent loop that serves six different product surfaces.

The FinOps Foundation's 2026 survey found that 98% of organizations now actively manage AI spend, up from 63% a year earlier and 31% the year before — the fastest adoption curve the foundation has ever recorded. Everyone is suddenly doing "FinOps for AI." Far fewer have noticed that the core primitive of cloud FinOps — the resource tag — doesn't exist in the token world, and that three specific mechanics of modern LLM usage actively destroy attribution.

Every internal LLM platform goes through the same arc. In month one, inference is free: the platform team eats the bill, product teams experiment wildly, and everyone celebrates adoption curves. By month six, the bill has grown 10x, finance is asking pointed questions, and the platform team discovers that three teams account for 80% of spend — one of them running a nightly batch job nobody remembers approving. The instinctive response is to install a meter and start charging. That instinct, applied naively, is how you kill your platform.

Here's the uncomfortable truth: per-token chargeback punishes exactly the behavior you built the platform to encourage. The team prototyping an agent that might transform your support workflow burns tokens like a furnace — agentic workloads consume 5 to 30 times more tokens per task than a simple chat completion. Bill them list price from day one and they stop prototyping. Meanwhile, the team running a mature, optimized feature pays pennies and looks virtuous. You've built a pricing system that taxes learning and rewards stagnation.

Here is a fact that breaks most engineering dashboards: two identical inference requests, returning identical tokens, billed at the identical API rate, can have carbon footprints that differ by a factor of five. Same model, same prompt, same dollars. The only thing that changed was which data center answered and what time of day it was. If your sustainability number is derived from your spend number — and almost everyone's is, because spend is the only number anyone actually tracks — then your sustainability number is wrong in a way that gets worse precisely when you optimize hardest for cost.

The dollar and the joule look like they should move together. More tokens, more compute, more money, more energy. For a single request held in a single place at a single instant, they roughly do. But the moment you turn on the levers that make inference cheap at scale — aggressive batching, model routing, off-peak scheduling, multi-region failover — the two metrics decouple. Cost is a function of your contract. Carbon is a function of physics and the grid. They are not the same function, and treating one as a proxy for the other will eventually put a number in a sustainability report that an auditor can falsify.

Your finance team built a beautiful AI cost dashboard. Token spend, sliced by feature. Embedding spend, sliced by provider. Every quarter, the per-feature pane gets reviewed in a leadership meeting and somebody asks why the support-chat workflow is up 12%, and a product manager has a defensible answer. Every quarter, the per-provider pane gets reviewed in an infra meeting and somebody asks why OpenAI is up 8%, and a platform engineer has a defensible answer. And every quarter, the line that actually doubles your AI bill — the corpus re-index — lands in a third bucket called "infrastructure" that nobody reviews because nobody owns it.

That bucket is where forty percent of your AI spend goes to die unattributed. The teams who could have optimized it never see it. The teams who see it can't tell you which feature it serves. The dashboard is honest about every cost it can explain and silent about the cost it can't, which is exactly the cost that matters most.

Your finance team filed a dispute and lost. The line item is "output tokens" and it exceeds your sum-of-delivered-tokens metric by fourteen percent. The provider's support engineer closed the ticket as "expected behavior under streaming cancellation," with a link to a documentation page that says "cancellation stops billing at the last delivered token." Both sentences are true, and the gap between them is the line of code you have not written.

The contract you read says one thing. The inference scheduler does another. The mismatch is not a bug, not a billing error, and not malice — it is a layered system in which the cancellation signal travels through three boundaries (browser, edge, GPU) and the billing meter sits at the third boundary while your "stop generating" button sits at the first. Closing the gap is an engineering project with a finance owner.

A quarter into the rollout, the AI team reported a 25% reduction in average cost-per-API-call. The support team reported that average handle time on AI-routed tickets had drifted from four turns to seven. Both numbers were correct. Both teams were measuring the system they had been told to optimize. The finance team, sitting between them, could not reconcile the dashboards because neither one was denominated in the thing the customer was actually paying for: a resolved ticket. The cost-per-call had gone down. The cost-per-resolved-task had gone up 40%. Nobody owned that number, so nobody was watching it move.

This is the most common unit-economics failure I see in agentic deployments, and it is not a measurement bug. It is a definitional one. The vendor's pricing page exposes cost-per-call because that is the unit they bill. The spreadsheet line item inherits that unit because it fits in a cell. The engineering team optimizes against the unit they were given. By the time the gap between API economics and business economics becomes visible, it has been compounding for a quarter, and the agent has been quietly trained on the wrong loss function the entire time.

Finance sent a memo on a Monday. By Friday, every product team had shipped a prompt change, and on the following Tuesday the support queue grew by a third. Nobody had touched the model. Nobody had touched the product. The only thing that had changed was that the LLM bill was now flowing back to the teams that issued the calls — and the teams had responded the way any rational cost center responds to a new line item on its P&L. They cut it.

The story that gets told inside the company afterwards is a story about prompt engineering, or about the model regressing, or about a noisy week of user traffic. The truer story is that finance, through a chargeback policy, had quietly become a product manager. The cost-attribution dashboard was a product-quality lever that nobody had reviewed, nobody had instrumented for, and nobody owned. When it moved, every prompt in the company moved with it, and the trade-offs that produced the quality regression were never seen by the people whose job it was to see them.

The line item climbed 130% over six weeks and nobody on the engineering team noticed. PRs were landing faster. Per-PR CI cost on the dashboard looked the same as last quarter. The agent's branches went green on the first try more often than the humans' branches did, which actually pulled the median CI duration down. Finance found it during quarterly review, flagged it as an unexplained variance, and asked engineering for the postmortem. Engineering had nothing to write — no incident, no regression, no failed deploy. Just a budget line that had quietly doubled while every dashboard reported normal.

That postmortem-shaped hole is the artifact. The cost shifted from a labor-dominant curve to an infrastructure-dominant curve, and the team that owned the labor budget was not the team that owned the infrastructure budget. The agent didn't break anything. It just changed which line on the P&L absorbed the work.