The Inference Budget Committee: Governance When Token Spend Crosses Seven Figures

At $50,000 a month, the "compute + tokens" line on your infra bill is rounding error. At $5,000,000 a month, it is a CFO question. The transition between those two states is not gradual — it is a phase change in how an organization talks about model spend, and most engineering orgs are unprepared for the social and political work that follows. The bill stays a single line; the conversation around it does not.

What changes is who has standing to ask "why." When three product teams share one API key and one capacity reservation, every quota argument has the same structure: someone is currently winning at the expense of someone else, and there is no neutral party to call it. The first time a team's launch is throttled because another team shipped a chatty agent, the absence of a governance body is felt by the entire engineering org at once. Calling a meeting and inventing a process under pressure is the worst time to design one.

This piece is about the body that owns those decisions: the inference budget committee. It is part finance, part platform, part political. It is the thing that turns "we should optimize tokens" into a recurring quarterly review with names attached. And it is the unglamorous artifact that distinguishes the orgs whose AI spend curve flattens from the orgs whose curve goes vertical and stays there.

Why the bill stops being a line item

The composition of the bill matters more than the number on it. By 2026, inference accounts for roughly 85% of enterprise AI spending — not training, not data prep, not labeling. Once you cross seven figures monthly, three structural facts of inference economics stop being abstractions and start setting your roadmap.

The first is that agentic loops compound. A single user-visible action can fan out into dozens of LLM calls — planning, tool selection, retrieval reranking, self-verification, reflection. Each of those is a token bill. The product team that ships an agent does not always know how many internal turns the agent will take on a real production trace, and the gap between dev-environment cost and production cost is often 20x.

The second is that retrieval-augmented generation is a latent multiplier on every prompt. The question "what is our token cost per request" presupposes that there is a single answer. There isn't. A request that hits a fresh document with 30 chunks of context costs more than a cached one, and the ratio between cheapest and most expensive requests of the same shape can easily be 50:1.

The third is that always-on intelligence — background agents, scheduled scans, continuous monitoring — hides behind the absence of a user. Nobody is staring at a load test for these workloads. They simply consume capacity continuously, and the only signal that something has gone wrong is the bill.

Each of these dynamics turns "what does the LLM cost" from a back-of-envelope estimate into a real forecasting problem. And once it is a real forecasting problem, finance has standing to ask who owns it.

The shared API key is a tragedy of the commons

In the early days of an AI initiative, one team gets an API key from a provider, and a second team asks for access. The platform team, helpful by default, hands out a sub-key or a wrapper. By the time three or four teams are using the same provider account, the shared key is the org's largest unmanaged shared resource — and it has all the failure modes of any unmanaged shared resource.

Quotas at the provider level apply to the whole account, not to teams. If one team accidentally enters a retry loop or ships a regression that doubles its prompt length, the rate limit fires for everyone. The team that wakes up to a paged on-call has nothing to do with the team that caused the problem. The blameless postmortem turns into "we should have isolated capacity better," which is true, but does not change the fact that the platform team is now the bottleneck for every product team's incident response.

Cost attribution is similarly broken. The provider invoice arrives once a month with one number on it. Without per-team tagging at the gateway layer, the only way to allocate is by self-report — which is to say, not at all. Finance ends up dividing by team headcount or last-quarter usage, which rewards the teams that under-report and penalizes the ones that instrument honestly.

The fix is structural, not exhortatory. You put an internal LLM gateway in front of the provider, every request carries a team identifier in a header, the gateway enforces per-team token-per-minute limits, and the chargeback report is a database query rather than a debate. Open-source projects like LiteLLM and commercial gateways from Portkey, TrueFoundry, and Kong all converged on roughly this shape because there is exactly one shape that works.

But a gateway by itself is plumbing. The gateway answers "who used what." It does not answer "who should get how much." That second question is what the committee exists for.

What a working committee actually does

The temptation, when you stand up a body to govern AI spend, is to make it look like a steering committee. Quarterly meetings, slide decks, no decisions. Resist this. The inference budget committee is closer to a capacity planning meeting than a strategy review, and its outputs are concrete: quota allocations, burst windows, exception requests, and a small number of standing policies.

A reasonable membership: one engineering leader from each major consuming team, the platform team lead who owns the gateway, a finance partner who can authoritatively map token cost to budget categories, and a chair who is empowered to break ties without escalation. Five to seven people total. Bigger than this and decisions stop happening; smaller and a single team dominates.

The committee owns four artifacts:

• The capacity pool. Total provisioned throughput purchased from each provider, reviewed monthly. This is the inventory the committee allocates from.

Loading…