The Batch Tier Is the New Spot Instance

Pull up your token dashboard and ask one question about every workload on it: was a human waiting for this response? For most teams running agents in production, the honest answer is no for half the bill or more. Eval suites, embedding backfills, nightly report generation, bulk classification, overnight code migration, summarization of yesterday's tickets — none of it has a user staring at a spinner. Yet nearly all of it flows through the interactive endpoint, at full price, competing for the same capacity as the requests that actually are latency-critical.

Every major provider will run that deferrable work for half the cost. OpenAI's Batch API, Anthropic's Message Batches, and Gemini's batch mode all price asynchronous jobs at a flat 50% discount in exchange for a 24-hour completion window. The discount requires no negotiation, no committed spend, no engineering heroics. It requires only that you admit, in your architecture, that some work can wait — and most teams have never made that admission, because nobody made deferral a design decision.

We have seen this movie before. Spot instances offered 60–90% off cloud compute for years while most teams kept everything on on-demand, not because the savings were unreal but because using them forced an uncomfortable question: which of our workloads can tolerate interruption? Teams that answered it built checkpointing and cut their compute bills by more than half. Teams that didn't kept paying the "everything is urgent" tax. The batch tier is the same fork in the road, except the axis is latency tolerance instead of interruption tolerance — and agentic workloads, which burn 5–30x more tokens per task than chatbots, make the price of not choosing much steeper.

Classify by Latency Tolerance, Not Importance

The reason deferrable work ends up on the interactive endpoint is a category error: teams classify workloads by how much they matter instead of by how long the result can wait. Your nightly eval run is important — it gates tomorrow's deploy. Your compliance summarization is important — legal reads it every morning. Importance triggers the instinct to give the work the "best" infrastructure, which everyone assumes means the fastest path.

But importance and urgency are orthogonal. The eval run is important and can tolerate six hours of delay. The autocomplete request is comparatively trivial and must return in 800 milliseconds.

A cleaner sorting question: who or what is blocked until this response arrives, and for how long can they stay blocked? That yields roughly four tiers.

• Interactive (seconds): a human is watching. Chat turns, in-IDE completions, live support agents. This is the only tier where time-to-first-token matters.
• Responsive (minutes): a human is nearby but not staring. A ticket triage agent, a PR review bot, a document-processing step inside a workflow someone checks periodically.
• Deferrable (hours): the consumer is a schedule, not a person. Eval suites, data enrichment, embedding refreshes, agent-driven code migrations that land as a morning PR.
• Whenever (a day or more): backfills, corpus re-processing, periodic re-scoring of historical data.

The last two tiers belong on the batch tier or its equivalents, and in most agent-heavy shops they are 40–60% of token volume. Background inference — monitoring agents, document watchers, compliance surveillance — is the fastest-growing slice of enterprise LLM spend precisely because it runs continuously with no human in the loop. It is the most deferrable traffic you have, and it is usually paying the interactive premium.

The Tier Menu Nobody Reads

The pricing landscape has quietly become a real menu, and "standard" is now the middle of it, not the default.

At the top, OpenAI sells priority processing at a premium per token for consistently faster and more stable generation during peak demand — explicitly aimed at the interactive tier where p99 latency is a product feature. At the bottom, the batch APIs give a flat 50% off with a 24-hour SLA: you upload a file of requests (up to 50,000 for OpenAI, tens of thousands per batch for Anthropic), the provider schedules them into spare capacity, and you collect results when they're done.

In between, OpenAI's flex processing offers roughly half price on the synchronous API in exchange for slower responses and occasional 429s when capacity is tight — a spot-like tier where you keep the request/response programming model but accept best-effort scheduling. DeepSeek went a different route and priced the clock directly: historically 50–75% off during off-peak UTC hours, turning "run it tonight" into a literal discount window. Routers like OpenRouter now expose provider service tiers as a per-request field.

Look at what this menu actually is: providers are load-shaping, exactly the way power utilities and cloud spot markets do. Inference capacity is provisioned for peak interactive demand, which means overnight and off-peak troughs are full of idle accelerators. The 50% discount is the provider paying you to move your flexible load into their valleys. That framing matters because it tells you the discount is structural, not promotional — as long as interactive demand is spiky and GPUs are expensive, someone will pay you to be flexible. Flexibility is now a monetizable property of your architecture, and an architecture that cannot express "this can wait" is leaving that money on the table permanently.

The economics compound, too. Batch discounts stack with prompt caching on providers that support both, pushing effective per-call cost toward a quarter of list price for repetitive, template-heavy workloads — which is precisely what evals and backfills are.

Surviving the 24-Hour Window: Spot Lessons Apply Directly

The reason teams bounce off the batch tier after one bad experience is the same reason they bounced off spot instances: they moved the workload without redesigning it for the failure model. Spot taught a generation of infrastructure engineers that cheap capacity is only cheap if you can be interrupted gracefully — checkpoint progress externally, handle the two-minute warning, resume instead of restart. The batch tier has its own failure model, and it demands the same discipline.

First: batch jobs fail partially, not atomically. A 100,000-row batch comes back with completed rows, provider-errored rows, content-filtered rows, and timed-out rows mixed together. If your pipeline treats the batch as one unit — "did the job succeed?" — a 4% error rate either fails the whole run (wasting the 96%) or gets silently ignored (corrupting downstream data). Design row-level: track completion status per request, and decide upstream what happens to failed rows — drop, retry in the next batch, or recover through the synchronous API at full price. The honest metric is cost per completed row, including retries, not the invoice.

Second: checkpoint across the window, because the window is real. Results can land anywhere from minutes to a full day after submission, and your submitting process will not be alive the whole time. Persist batch IDs and per-row state to durable storage the moment you submit, write checkpoints atomically, and make the collector idempotent so a crashed poller can resume rather than resubmit. This is exactly the spot checkpointing pattern — externalize progress, assume the worker dies — applied to a job whose "interruption" is simply the passage of hours.

Third: the 24-hour SLA breaks feedback loops that assume same-hour results, and this is the failure mode that actually burns teams. If your eval suite gates deploys and you move it to batch, the gate now lands on yesterday's prompt against today's model. The working pattern is a synchronous canary: run 1–2% of rows through the interactive endpoint to gate the decision now, and let the full batch land overnight for the comprehensive next-day view. Cheap bulk coverage and fast gating are different jobs; split them instead of forcing one path to do both.

Fourth: audit quality across paths. Practitioners report the same model snapshot scoring a few points lower through the batch path than the sync path on identical rubrics — capacity-pool routing and mid-window snapshot rotation are real. A periodic paired audit of a few hundred rows through both paths tells you whether the discount is quietly costing you quality. Trust, but sample.

The Smell Test: An "Urgent" Queue That Is 90% Deferrable

Here is the architectural smell worth grepping for: a single request path where everything is implicitly urgent. If every LLM call in your codebase goes through one client wrapper with one endpoint and no notion of deadline, then urgency was never decided — it was defaulted. The queue looks urgent because urgency is the only state it can express.

The fix is small and boring: make latency tolerance a required field on every LLM task, the same way spot-era job schedulers made interruption tolerance a required field. When an engineer enqueues agent work, they declare a deadline — interactive, 1h, 24h — and a router maps that declaration to a tier: priority or standard for interactive, flex or off-peak for the one-hour bucket, batch for the rest. Three things happen once the field exists:

• The default flips. Absent a human waiting, work drifts to the cheap tier instead of the expensive one, because engineers must actively claim urgency rather than passively inherit it.
• Cost review becomes an urgency audit. Finance asking "why is this workload on the interactive tier?" is a much better conversation than "why is the AI bill up 40%?" — the first has an owner and an answer.
• Capacity planning gets honest. Your true interactive QPS — the number that determines rate-limit negotiations and failover design — turns out to be a fraction of what your raw request volume implied.

There is an organizational payoff hiding here as well. Deferral pressure-tests your agents' autonomy. Work that can wait 24 hours must also run unattended: no human to approve a tool call, no one to nudge a stuck loop. Teams that route agent work to the batch tier are forced to build the checkpointing, idempotency, and self-recovery that fully autonomous agents need anyway. The discount funds the discipline.

Deadlines Are the New Design Primitive

The interactive endpoint is not the "real" API with batch as a discount gimmick — it is the premium tier of a maturing market that prices latency the way cloud priced interruption a decade ago. Spot instances rewarded teams whose workloads could say "I can be interrupted"; the batch tier rewards workloads that can say "I can wait." Both are the same underlying trade: providers pay you to absorb their scheduling flexibility, and the payment goes only to architectures capable of accepting it.

So make deferral a decision instead of an accident. Add the latency-tolerance field, classify your existing agent workloads against the four tiers, and move the two obvious candidates — evals and backfills — this quarter; they are high-volume, template-shaped, and nobody is waiting. Build the row-level bookkeeping and the sync canary before trusting a gate to a 24-hour window. Then check the dashboard again. If half your tokens were deferrable all along, you were paying double for the privilege of never asking the question — and the question was one field away.