The Inference Budget Committee: Governance When Token Spend Crosses Seven Figures

At $50,000 a month, the "compute + tokens" line on your infra bill is rounding error. At $5,000,000 a month, it is a CFO question. The transition between those two states is not gradual — it is a phase change in how an organization talks about model spend, and most engineering orgs are unprepared for the social and political work that follows. The bill stays a single line; the conversation around it does not.

What changes is who has standing to ask "why." When three product teams share one API key and one capacity reservation, every quota argument has the same structure: someone is currently winning at the expense of someone else, and there is no neutral party to call it. The first time a team's launch is throttled because another team shipped a chatty agent, the absence of a governance body is felt by the entire engineering org at once. Calling a meeting and inventing a process under pressure is the worst time to design one.

This piece is about the body that owns those decisions: the inference budget committee. It is part finance, part platform, part political. It is the thing that turns "we should optimize tokens" into a recurring quarterly review with names attached. And it is the unglamorous artifact that distinguishes the orgs whose AI spend curve flattens from the orgs whose curve goes vertical and stays there.

Why the bill stops being a line item

The composition of the bill matters more than the number on it. By 2026, inference accounts for roughly 85% of enterprise AI spending — not training, not data prep, not labeling. Once you cross seven figures monthly, three structural facts of inference economics stop being abstractions and start setting your roadmap.

The first is that agentic loops compound. A single user-visible action can fan out into dozens of LLM calls — planning, tool selection, retrieval reranking, self-verification, reflection. Each of those is a token bill. The product team that ships an agent does not always know how many internal turns the agent will take on a real production trace, and the gap between dev-environment cost and production cost is often 20x.

The second is that retrieval-augmented generation is a latent multiplier on every prompt. The question "what is our token cost per request" presupposes that there is a single answer. There isn't. A request that hits a fresh document with 30 chunks of context costs more than a cached one, and the ratio between cheapest and most expensive requests of the same shape can easily be 50:1.

The third is that always-on intelligence — background agents, scheduled scans, continuous monitoring — hides behind the absence of a user. Nobody is staring at a load test for these workloads. They simply consume capacity continuously, and the only signal that something has gone wrong is the bill.

Each of these dynamics turns "what does the LLM cost" from a back-of-envelope estimate into a real forecasting problem. And once it is a real forecasting problem, finance has standing to ask who owns it.

The shared API key is a tragedy of the commons

In the early days of an AI initiative, one team gets an API key from a provider, and a second team asks for access. The platform team, helpful by default, hands out a sub-key or a wrapper. By the time three or four teams are using the same provider account, the shared key is the org's largest unmanaged shared resource — and it has all the failure modes of any unmanaged shared resource.

Quotas at the provider level apply to the whole account, not to teams. If one team accidentally enters a retry loop or ships a regression that doubles its prompt length, the rate limit fires for everyone. The team that wakes up to a paged on-call has nothing to do with the team that caused the problem. The blameless postmortem turns into "we should have isolated capacity better," which is true, but does not change the fact that the platform team is now the bottleneck for every product team's incident response.

Cost attribution is similarly broken. The provider invoice arrives once a month with one number on it. Without per-team tagging at the gateway layer, the only way to allocate is by self-report — which is to say, not at all. Finance ends up dividing by team headcount or last-quarter usage, which rewards the teams that under-report and penalizes the ones that instrument honestly.

The fix is structural, not exhortatory. You put an internal LLM gateway in front of the provider, every request carries a team identifier in a header, the gateway enforces per-team token-per-minute limits, and the chargeback report is a database query rather than a debate. Open-source projects like LiteLLM and commercial gateways from Portkey, TrueFoundry, and Kong all converged on roughly this shape because there is exactly one shape that works.

But a gateway by itself is plumbing. The gateway answers "who used what." It does not answer "who should get how much." That second question is what the committee exists for.

What a working committee actually does

The temptation, when you stand up a body to govern AI spend, is to make it look like a steering committee. Quarterly meetings, slide decks, no decisions. Resist this. The inference budget committee is closer to a capacity planning meeting than a strategy review, and its outputs are concrete: quota allocations, burst windows, exception requests, and a small number of standing policies.

A reasonable membership: one engineering leader from each major consuming team, the platform team lead who owns the gateway, a finance partner who can authoritatively map token cost to budget categories, and a chair who is empowered to break ties without escalation. Five to seven people total. Bigger than this and decisions stop happening; smaller and a single team dominates.

The committee owns four artifacts:

• The capacity pool. Total provisioned throughput purchased from each provider, reviewed monthly. This is the inventory the committee allocates from.
• The allocation table. Per-team token-per-minute quotas, with explicit burst allowances for launches and seasonal events. Published; not negotiable in real time.
• The chargeback report. Monthly per-team consumption, mapped to the team's stated cost-per-outcome metric. This is the input to the next quarter's allocation.
• The exception log. A short list of recent exception requests, who approved them, and what condition was attached. This is the institutional memory that prevents the same emergency-approval pattern from happening every quarter.

None of these are exotic. They are the same artifacts a Kubernetes platform team produces for compute quota or a database team produces for shared cluster capacity. The novel part is that the underlying resource is provider tokens, and the underlying cost driver is application behavior that the platform team cannot directly observe.

Chargeback by outcome, not by token

The single most consequential decision the committee makes is the chargeback model. Naive chargeback is by raw token consumption: each team sees its share of the bill, and the team that uses the most tokens pays the most. This sounds fair and is in fact disastrous.

Charging by tokens incentivizes prompt compression contests. A team that spends a quarter rewriting prompts to be terser will report lower spend without producing more value. A team that switches to a smaller, cheaper model that produces worse outputs will look like a cost hero on the dashboard while quietly degrading the product. The metric that punishes spending without measuring outcome punishes good engineering and rewards illegible cuts.

The functional alternative is cost per validated business outcome. Each consuming team picks one or two outcome metrics — cost per resolved support ticket, cost per qualified lead, cost per document processed, cost per generated test case that survives review. The chargeback report shows token spend, outcomes delivered, and the ratio. The committee reviews the ratio, not the raw spend.

This reframing is the entire point. A team whose cost per resolved ticket is $0.40 and falling is a team that should get more capacity. A team whose cost per resolved ticket is $4 and rising — even if its raw token consumption is small — is a team that should be having a different conversation. Tying chargeback to outcomes is what turns the committee from a budget-cutting exercise into a portfolio review.

It also forces honesty about which AI features are not paying for themselves. The dirty secret of many enterprise AI rollouts is that some features generate negative ROI in production once token costs are honestly attributed. The cost-per-outcome view surfaces this. Token-only attribution buries it.

Forecasting bursty-correlated demand

Capacity planning for traditional services rests on an assumption: per-user demand is approximately independent. If 10,000 users each have a 1% chance of making a request in a given second, the variance of total load is well-behaved and the central limit theorem is your friend.

LLM workloads break this assumption in two ways. First, agentic flows mean that a single user action triggers many model calls in rapid succession — the unit of demand is not "request" but "session," and sessions are bursty within themselves. Second, traffic across teams is correlated by external events. Marketing's lead-scoring agent and Sales's call-summarization agent both spike on the Tuesday after a campaign launch. Treating these as independent Poisson processes will leave you under-provisioned by a wide margin during exactly the windows that matter most.

The forecasting discipline that holds up looks more like financial portfolio risk than queue theory. Each team submits a monthly demand profile that includes a baseline, a peak multiplier, and the events that drive the peak. The committee aggregates these into a covariance-aware forecast that accounts for which peaks are likely to coincide. Provisioned throughput is sized to the 95th-percentile aggregate load, not the sum of per-team peaks.

When peaks do coincide, the burst allowance in each team's allocation is what absorbs the shock. When the burst allowance is insufficient — when two teams genuinely need the same incremental capacity on the same launch day — the committee is the tiebreaker. This is why the chair must be someone who can decide and have the decision stick. If the chair has to escalate every conflict to a VP, the committee has no actual authority and its meetings are theater.

The political surface

Most of the friction in running an inference budget committee is not technical. It is the discomfort of saying no to a peer team during a launch they have been planning for a quarter. It is the awkwardness of asking a team to defend its cost-per-outcome metric in front of finance when last month's number was bad. It is the recurring conversation with a senior leader who wants to override the allocation table because their team's feature is "strategic."

The orgs that get this right have a few habits in common. They publish the allocation table widely and refer to it by name in conflicts, so the answer to "why can't I have more capacity" is "because the allocation table says so, and the next review is in three weeks." They keep exception requests rare and require written justification, so that the cost of asking for an exception scales with the cost of granting one. They rotate the committee chair annually so that no single person becomes the perpetual bad cop. And they make the committee's decisions reversible — every allocation has a review date, and any team can request reconsideration with new data.

The orgs that get this wrong let the committee become advisory. The allocation table exists but nobody enforces it. Exceptions are approved by Slack DM. The cost-per-outcome metric is reported quarterly but never actually used to reweight allocations. Within two quarters, the committee is a meeting nobody attends, and the next time inference spend doubles unexpectedly, there is no body in place to triage it.

The leadership shift

The most interesting change in orgs that have been through this transition is not the process — it is how senior leadership talks about the bill. Inference budget stops being a finance line buried under "infrastructure" and becomes a planning category at the same level of granularity as headcount and cloud spend. Roadmaps get written with token budget constraints in them. Feature proposals include an estimated cost-per-outcome alongside engineering estimates. Hiring plans for AI-heavy teams include the inference capacity those teams will need to be productive.

This is the leadership shift that the committee is in service of. The committee is the mechanism, but the deeper change is that the org has internalized inference as a constrained resource that requires governance, not a hope that the usage curve will stay linear because the unit price has been falling. Unit price has been falling for years; total spend has been climbing the whole time. The leaders who notice the difference are the ones who staff a committee before the seven-figure month, not after it.

The forward-looking version of this is straightforward: every org that ships AI features at scale will eventually own an inference budget committee, in the same way every org that runs production infrastructure eventually owns a change advisory board. The only question is whether you build it deliberately at the $500K/month mark, or improvise it under pressure at the $5M/month mark, after a finance escalation has already made the conversation harder than it needed to be. The work is the same either way; the cost of doing it late is paid in trust.