The Carbon Line Item: Budgeting the Energy Cost of Inference

Here is a fact that breaks most engineering dashboards: two identical inference requests, returning identical tokens, billed at the identical API rate, can have carbon footprints that differ by a factor of five. Same model, same prompt, same dollars. The only thing that changed was which data center answered and what time of day it was. If your sustainability number is derived from your spend number — and almost everyone's is, because spend is the only number anyone actually tracks — then your sustainability number is wrong in a way that gets worse precisely when you optimize hardest for cost.

The dollar and the joule look like they should move together. More tokens, more compute, more money, more energy. For a single request held in a single place at a single instant, they roughly do. But the moment you turn on the levers that make inference cheap at scale — aggressive batching, model routing, off-peak scheduling, multi-region failover — the two metrics decouple. Cost is a function of your contract. Carbon is a function of physics and the grid. They are not the same function, and treating one as a proxy for the other will eventually put a number in a sustainability report that an auditor can falsify.

This post is about why those two metrics diverge, where the divergence is largest, and how to put energy on the same dashboard as latency and spend instead of bolting it on once a year for the ESG slide.

What a Single Inference Actually Costs in Joules

Start with the unit economics, because most teams have never measured them. A short query to a frontier-class model lands somewhere around 0.3 watt-hours, with measured estimates for GPT-4o-class models clustering near 0.3 Wh for a typical interaction and rising toward 0.4 Wh for longer ones. Per output token, that's roughly 0.0003 Wh — on the order of a couple of joules of work to produce each token you read.

That number feels trivially small, and per request it is. A single query costs about as much energy as running an LED bulb for a few minutes. The problem is never the single request. The problem is that a moderately successful product does tens of millions of these per day, and the per-token figure is multiplied by a volume that grows faster than anyone's efficiency gains. At that scale, a 25% swing in energy per token is the difference between needing one more megawatt of contracted capacity or not.

To turn watt-hours into carbon you multiply by the grid's carbon intensity, and this is where the convenient assumption that "energy is energy" falls apart. A kilowatt-hour drawn in a region running mostly hydro and nuclear might carry 100 grams of CO2. The same kilowatt-hour drawn from a grid leaning on gas peakers carries 400 to 500 grams. U.S. hyperscale facilities have been measured at an electricity-weighted average around 545 gCO2/kWh — meaningfully dirtier than the national grid average, because they are often sited for cheap power and tax incentives rather than clean power. The carbon math has a second multiplier the dollar math does not: where the electrons came from.

Why Batching Helps the Dollar and the Joule — Until It Doesn't

Batching is the first place the metrics seem to agree and then quietly stop agreeing. When you batch more requests onto a GPU, fixed costs — kernel launches, memory transfers, the model weights already resident in VRAM — get amortized across more tokens. Measurements show the steepest efficiency gains arrive as batch size climbs from 32 to 256, where a large model can shed roughly a quarter of its per-token energy as the GPU moves from under 50% utilization to nearly saturated. Beyond a batch of 256 the curve flattens: power draw stays roughly constant while each additional token buys less marginal efficiency.

For cost, this is an unambiguous win — better utilization means fewer GPU-hours per million tokens, and GPU-hours are what you pay for. For carbon, it is also a win, for the same reason. So far the two metrics agree.

The divergence is in the third axis nobody puts on the same chart: latency. Bigger batches mean requests wait longer to be scheduled. The lever that improves both your cost-per-token and your carbon-per-token degrades your p99. Teams that optimize for tail latency run smaller batches and leave GPUs under-utilized, which raises energy per token and dollars per token at the same time — here cost and carbon move together, both in the wrong direction. The point is not that batching is good or bad. The point is that the cost number alone can't tell you which regime you're in, because two of the three forces it's balancing are invisible to it.

Model Routing: Where Spend and Carbon Fully Decouple

Model routing — sending easy requests to a small cheap model and hard ones to a large expensive one — is sold as the universal efficiency win, and for spend it usually is. But routing is exactly where the dollar and the joule stop being correlated at all, for three independent reasons.

First, price is set by your provider's margin, not by physics. A vendor can price a small model at a tenth of a large one while the actual energy ratio is a third, or a half, depending on architecture and how well each is batched on their fleet. You are optimizing a number the provider chose, not the joules the silicon burned. Cutting your bill in half does not reliably cut your energy in half.

Second, routing changes where and when work runs. A cheaper model may live in a different region or get scheduled to a different fleet with a different grid mix. You can route to the cheaper option and increase your carbon, because the cheaper option happens to run on a dirtier grid. Spend went down; emissions went up. Nothing about the dollar figure would warn you.

Third, the embodied-carbon accounting is different from the depreciation accounting. The newest accelerator is more energy-efficient per token and often more cost-efficient per token — but it carries enormous manufacturing emissions that are already spent the moment it's built. Your finance system amortizes that hardware as a smooth depreciation line. The carbon doesn't amortize the same way; it was emitted up front regardless of how many tokens you eventually serve. A fleet that looks cost-optimal on a depreciation schedule can be carbon-suboptimal if it's churning hardware to chase efficiency.

The Green Software Foundation's Software Carbon Intensity specification — now an ISO standard — encodes exactly this structure: a carbon score is energy times the carbon intensity of the grid at that place and time, plus embodied emissions of the hardware, all divided by a unit of useful work. Three terms. Your cost model has one and a half of them. That gap is the divergence, written as an equation.

Making Energy a First-Class Number

The fix is not a heroic measurement project. It's treating energy and carbon the way you already treat latency and spend: instrumented continuously, attached to every request, visible on the same dashboard, and budgeted.

A few moves get you most of the way:

• Estimate per-request energy at the call site. You don't need a wattmeter on every GPU. Model size, input and output token counts, and a per-model joules-per-token coefficient — measured once on your serving stack — give you a usable estimate you can attach to each request, the same way you already attach a token count and a dollar cost.

• Multiply by a real-time grid signal, not an annual average. Carbon intensity for most grids is available as a live or near-live feed. Using the marginal intensity — the carbon of the plant that actually spun up to serve your incremental load — is more honest than the grid-average number that flatters everyone. This is the difference between attributional and consequential accounting, and it's the difference between a number you can defend and a number you can't.

• Exploit temporal and spatial flexibility, but be honest about the ceiling. Batch and asynchronous workloads — evals, embeddings backfills, offline summarization — can be shifted to cleaner hours or cleaner regions. This genuinely helps. But the research on carbon-aware shifting is clear that the realistic gains are modest, often a low-double-digit percentage, and that simple policies capture most of it. Interactive traffic, which can't wait, barely moves. Don't promise a 50% reduction the grid can't deliver.

• Include PUE and embodied emissions, or you'll undercount. Your IT load isn't your facility load. Even a best-in-class hyperscaler runs a PUE near 1.1, and much of the industry sits closer to 1.5 — that's a 50% surcharge on every joule before it touches a chip. And the hardware's manufacturing emissions are real whether or not your depreciation schedule notices them.

Put It on the Dashboard

The reason carbon is wrong in most organizations is not that it's hard to measure. It's that it lives in a different system than the one engineers look at, owned by a different team, reported on a different cadence. Spend is on the dashboard because finance made it a first-class number; latency is there because SRE did. Energy will stay a once-a-year estimate until someone gives it the same treatment: a per-request value, a budget, an alert when it regresses, and a place on the screen next to the two numbers it keeps getting confused with.

The teams that get this right won't do it for the ESG report. They'll do it because energy turns out to be a leading indicator of cost they couldn't see — the under-utilized GPU, the request routed to a dirty-and-not-actually-cheaper region, the hardware churn that looked efficient on paper. Carbon and dollars diverge often enough to mislead you, but they overlap often enough that watching both makes you better at each. The line item was always there. It just wasn't on the dashboard.