The Internal Capacity Market: Rationing Scarce Inference Between Teams

At 4:50 PM on a Friday, someone on the data team kicks off an eval sweep: forty thousand prompts against the company's shared model deployment, scheduled to finish over the weekend. At 5:10 PM, the customer-facing chat assistant starts timing out. The on-call engineer spends two hours staring at dashboards that show the provider returning 429s before someone thinks to ask who else is using the account. Nothing is broken. The system is doing exactly what it was configured to do, which is nothing, because nobody configured it to do anything.

This is the shape of a new class of incident, and it has a property that makes it nastier than an ordinary outage: there is no bug to fix. The eval sweep was legitimate work. The chat assistant's traffic was legitimate work. The failure is that two teams with different urgency profiles were drawing from one undifferentiated pool of inference capacity, and the pool had no opinion about who mattered more. Capacity allocation stopped being an infrastructure detail the moment your company had more than one team shipping against the same provider account — it became a political problem, and the pager inherited it.

The instinct is to treat this as a rate-limiting problem, buy a bigger quota, and move on. But quota growth just delays the collision. As agent workloads multiply — every team now has a batch backfill, a nightly eval suite, a CI job that calls the model, and at least one agent that occasionally gets stuck in a retry loop — demand grows faster than any quota negotiation cycle. The durable fix isn't more capacity. It's a mechanism for deciding, ahead of time and in code, whose tokens get served when there isn't enough for everyone. In other words: an internal capacity market.

One Pool, Many Masters

The default architecture at most companies is a single provider account, or a single provisioned-throughput deployment, fronted by a shared API key or a thin gateway. It's the natural starting point — one contract to negotiate, one bill to pay, one integration to build. It's also a commons with no fences.

The workloads drawing from that commons are wildly heterogeneous:

• Interactive, user-facing traffic — the chat assistant, the in-product copilot. Latency-sensitive, revenue-adjacent, spiky in ways that follow user behavior.
• Batch and backfill jobs — embedding regeneration, document classification sweeps, data enrichment. Enormous token volume, zero urgency. Nobody cares if they finish at 2 AM or 6 AM.
• Evals and CI — bursty by construction. An eval suite is designed to fire thousands of requests at once, and it runs whenever someone merges.
• Development and experimentation — prototypes, notebooks, and agents under construction, including the occasional runaway loop that can eat an entire day's budget in minutes.

These have fundamentally different tolerance for delay, different business value per token, and different failure modes. When they share one undifferentiated pool, the pool's behavior under contention is decided by accident — usually by whoever sends requests fastest. A shared key can't tell the platform team which consumer is responsible for the burn; it just shows a number going up. The noisy-neighbor problem from multi-tenant infrastructure has reappeared, except the resource being contended isn't CPU or disk IOPS, it's tokens per minute, and the tenants are your own coworkers.

What makes this politically toxic rather than merely technical is that every resolution path runs through an org chart. When the eval sweep starves the chat product, the fix isn't a code change — it's a negotiation between two directors about whose work matters more, conducted at incident-retro tempo, with the on-call engineer as the involuntary mediator. The whole point of building an allocation mechanism is to have that negotiation once, in a design doc, instead of every Friday night.

The Mainframe Already Solved This

None of this is new. It is, almost point for point, the problem that time-sharing systems solved in the 1960s. A mainframe was a scarce, expensive, centrally-owned compute resource that every department wanted a slice of. The answer wasn't to give each department its own mainframe — nobody could afford that — it was to build an operating system that multiplexed the machine across users, metered consumption in CPU-seconds and kilobyte-seconds of memory, and billed each department for what it used.

Metering changed behavior: when departments paid for CPU time, programmers optimized their code. The market did what memos couldn't.

The lineage runs straight through to modern cluster schedulers. Google's Borg — the system that shaped Kubernetes — organizes every job into non-overlapping priority bands: monitoring, production, batch, best-effort. A production job can preempt a batch job when the cluster is tight. Quota is a vector of resource quantities at a given priority for a period of time, negotiated ahead of demand.

And there's a rule in Borg that is directly transferable to inference allocation: production jobs do not preempt other production jobs, because allowing that creates preemption cascades — a chain reaction where each eviction triggers another.

The lesson from both eras is the same. Scarce shared compute needs three things: priority classes that encode whose work yields to whose, quotas negotiated before contention rather than during it, and metering that makes consumption visible and attributable. Every company now running multiple AI teams against one inference pool is rediscovering this, usually one incident at a time. You can skip ahead by stealing the design.

Priority Tiers and Preemption for Tokens

The concrete version of this for inference has two layers: the primitives your provider gives you, and the policy your gateway enforces.

Providers have already productized the priority spectrum, because they face the same multiplexing problem internally. OpenAI sells priority processing at a per-token premium for latency-critical traffic, standard pay-as-you-go in the middle, and flex and batch processing at roughly half price for workloads that can tolerate slow responses or occasional unavailability. Anthropic offers a priority tier with committed capacity that falls back to standard when you exceed it. Azure's provisioned-throughput deployments support spillover, where requests that would 429 against your reserved capacity are redirected to pay-as-you-go instead of failing. The pricing is the message: the market rate for urgency is roughly a 2–4x spread between "serve this now" and "serve this whenever."

The mistake most platform teams make is buying these primitives and then routing everything through the default tier anyway, because no internal mechanism decides which workload deserves which tier. That decision is the actual work. A workable starting policy, translated from Borg's bands:

• Critical (interactive, user-facing): reserved capacity — a guaranteed floor of tokens-per-minute that no other class can touch, sized to peak production traffic. This is your provisioned throughput or committed priority tier.
• Production batch: runs against the shared pool with elevated priority, allowed to burst into idle reserved capacity, first to be shed when the critical class needs headroom.
• Evals and CI: capped concurrency, routed to flex or batch tiers by default. An eval suite should never be able to consume more than a fixed fraction of shared capacity, no matter how many merges land at once.
• Development and experimentation: best-effort, hard per-key budgets, throttled first. A stuck agent loop in this class should hit its own ceiling long before anyone else notices it exists.

The gateway pattern that implements this is reserved plus burstable: each class gets a guaranteed slice, and a shared burst pool absorbs spikes when others are quiet. The reserved slice protects fairness; the burst pool keeps utilization high — because the dual failure mode of naive static partitioning is paying for reserved capacity that sits idle while another team queues.

Preemption for inference looks different than for cluster jobs — you don't kill an in-flight request, you stop admitting new ones. Demotion under pressure means batch traffic spills to the provider's cheap-and-slow tier, eval concurrency drops, dev keys get throttled, and the critical class keeps its floor. Borg's cascade rule carries over intact: two production-critical classes should never preempt each other. If your chat assistant and your fraud-review agent are both critical and their combined peak exceeds the floor, the answer is buying a bigger floor, not letting them fight — a fight between two critical services is an outage with extra steps.

Chargeback: Making Teams Feel Their Own Load

Priority tiers control when work runs. They don't control how much work teams generate — and without a feedback loop, demand from every class grows until it hits the ceiling. The mainframe answer was billing, and it still works, but LLM consumption has a property that breaks the standard cloud playbook: there's no resource to tag. An API call is a transaction, not an asset. Cost attribution has to be captured at the application layer, at call time, as metadata on every request.

In practice this means the gateway stamps every call with a small, fixed set of dimensions — team, project, environment, model, cost center covers nearly every allocation question you'll ever be asked — and meters tokens against them. Do this from day one; retrofitting attribution onto six months of anonymous shared-key traffic is archaeology.

Then sequence the rollout the way GPU platform teams learned to:

1. Showback first. Publish per-team consumption — tokens, cost, share of contention-hour capacity — for a month or two before money moves. Showback finds your attribution gaps, and it changes the conversation: quota negotiations stop being about feelings and start being about a dashboard both sides can see.
2. Chargeback once coverage is trustworthy. When teams' actual budgets absorb their actual token spend, the incentives fix themselves. The team running the 40,000-prompt eval sweep on the priority tier discovers, on their own P&L, that batch-tier processing costs half as much. Nobody has to send the memo.
3. Price the tiers differently. Internal pricing should mirror provider pricing: priority tokens cost more than standard, standard more than batch. If urgency is free, everyone claims it — every workload is critical until its owner sees the bill.

The failure mode to avoid is the opposite extreme: chargeback so aggressive that teams start hoarding quota, padding their reservations against future need, or building shadow accounts on personal credit cards to escape the internal market. Sizing reservations against measured P99 usage — not against what teams request — keeps the hoarding honest. If the internal market is more painful than going around it, people will go around it, and you'll have traded a noisy-neighbor problem for an unmetered shadow-IT problem.

Dedicated Capacity or Shared-Pool Priority?

The last recurring decision is when a team graduates from shared-pool priority to its own dedicated capacity — its own provisioned deployment, its own provider account, its own negotiated contract. Dedicated capacity is the org-chart solution to a scheduling problem, and like most org-chart solutions, it's sometimes correct.

Give a team dedicated capacity when:

• Their traffic is the reason everyone else pages. If one workload's spikes routinely force every other class to shed, isolation is cheaper than refereeing.
• Their utilization would keep a reservation warm. Provisioned capacity bills by the hour whether used or not; a team below roughly half sustained utilization is paying a large premium for isolation they could get from a priority floor instead.
• Their requirements genuinely diverge — a different model, a compliance boundary, a latency SLO the shared gateway can't honor, a blast-radius argument for keeping their failures out of everyone else's.

Keep them on shared-pool priority when their need is ordinary urgency, because the shared pool has a structural advantage no collection of dedicated silos can match: statistical multiplexing. Ten teams with spiky demand sharing one pool need far less total capacity than the sum of ten individually-provisioned peaks, and idle reservations in a fragmented setup can't be lent to whoever is bursting — capacity trapped in team A's silo is worthless to team B's spike, even while A sleeps. Every dedicated carve-out shrinks the pool and weakens the smoothing for everyone who remains. Fragmentation is a tax the whole org pays for one team's autonomy; make them justify it with utilization math, not seniority.

The Pool Is the Product

The uncomfortable summary is that if your company runs more than a handful of AI workloads, you are operating a capacity market whether you designed one or not. The undesigned version allocates by speed of request submission and settles disputes by incident retro. The designed version allocates by priority classes agreed in advance, meters consumption per team, prices urgency honestly, and reserves the political fights for genuinely new questions.

Start smaller than you think you need to: split traffic into three classes (critical, batch, best-effort), give the critical class a reserved floor, cap dev keys, and turn on showback. That's a few weeks of gateway configuration, and it converts the next Friday-evening eval sweep from an outage into a scheduling event nobody notices. The mainframe generation learned this in the era of billed CPU-seconds; the cluster generation encoded it in priority bands and quota vectors. The inference generation gets to inherit both playbooks — the only question is whether you adopt them before or after your own 4:50 PM Friday incident.