Token Budgets Are a Headcount Decision in Disguise

A team I talked to recently spent three engineer-weeks shaving their average prompt from 4,000 tokens to 2,600. They were proud of it — a clean 35% reduction, real numbers, a nice graph in the deck. Then someone did the arithmetic the other direction. The savings came to roughly $1,800 a month. The three engineer-weeks they spent getting there cost something like $25,000 in fully-loaded salary. At that monthly run rate, the optimization pays for itself in about fourteen months — assuming the prompt never changes, the model never gets cheaper, and those engineers had nothing more valuable to build.

None of those assumptions held. The prompt changed twice in the next quarter. The model they were on dropped its input price by 40% on its own. And the feature those engineers didn't ship that month was the one the biggest customer had been asking about.

This is the trap hiding inside every token budget. We treat prompt size as a cost-of-goods problem — fewer tokens, smaller bill, obviously good. But the bill you pay a provider is almost never the expensive part. The expensive part is the engineer time you spend chasing it, the support load you create when a leaner prompt degrades quality, and the infra complexity you take on to claw back pennies per call. Token budgets look like a line on the inference invoice. They are actually a headcount decision wearing a cost-optimization costume.

The ratio nobody puts on the slide

Start with the number that reframes everything: for most software organizations, cloud and infrastructure spend is 20–30% of total technical cost. Engineering salaries are the other 70%. A CTO can grind an AWS bill from $60K to $40K a month while paying $200K a month in engineering compensation — which, as one FinOps writer put it, is like obsessing over the grocery bill while ignoring the mortgage.

LLM inference is a new line on the infra side of that split, and it inherits the same lopsided economics. Your token spend is real, it's variable, and it shows up in a dashboard that makes it feel urgent. But the people staring at that dashboard cost three to four times more than the dashboard's worst month. Every hour they spend optimizing tokens is an hour priced at $80–$150 fully loaded, spent to reduce a number that is structurally smaller than their own salary.

This doesn't mean token optimization is never worth it. It means the worthwhile-ness is a function of two quantities that almost never appear together: the dollars you save per month, and the engineer-hours you burn to save them. Most teams track the first religiously and the second not at all. So optimization always looks free, because the cost is paid out of a budget — payroll — that nobody is charging it against.

Three things you're actually spending

When you decide how many tokens a prompt gets, you're not just buying inference. You're making an implicit purchase across three accounts at once, and only one of them shows up on the invoice.

• Inference dollars. The obvious one. Input tokens times input price, output tokens times output price, with output running 4–5x more expensive than input on flagship models. This scales linearly with traffic and is the easiest to forecast.
• Engineer time. The cost of shrinking the prompt — and, crucially, the cost of keeping it shrunk. A lean prompt is more brittle. It has less margin for the next edge case, so every new requirement means re-tuning, re-evaluating, and re-testing against a tighter token budget. This cost is recurring and invisible.
• Support and quality load. When you cut tokens, you usually cut context, examples, or guardrails. Sometimes quality holds. Sometimes it degrades in ways that surface two weeks later as support tickets, bad outputs, and a trust problem with the very customers the feature was supposed to delight. This cost lands on a different team's budget, which is exactly why it gets ignored at decision time.

The mistake isn't spending on any one of these. It's spending on one without pricing the others. A 35% token cut that adds a recurring eval burden and a slow drip of quality complaints can easily be net-negative even though the inference graph went down and to the right.

When the trade is obviously worth it

There's a clean category where token optimization is close to free money, and it has a defining feature: the engineering cost is near-zero or one-time, and the savings recur forever.

Prompt caching is the canonical example. Both major providers now bill cached input reads at roughly 10% of the standard rate — a 90% discount on the cached portion. For an agent or a chat product with a large static system prompt repeated on every call, organizing that prompt into a stable cached prefix can cut input costs 70–90% in real applications. The engineering effort is often a single restructuring of how you assemble the prompt, sometimes nothing at all because provider-side caching kicks in automatically above a token threshold. One-time cost, permanent savings, no quality hit. Take it every time.

The same logic applies to a handful of other levers:

• Structured outputs. Asking for JSON against a schema constrains both content and length. Responses get shorter and more parseable at once, so you save tokens and reduce downstream parsing code. The cost is writing the schema once.
• Cheap model routing for easy traffic. If a measurable slice of requests is trivial, routing it to a smaller model is a config change with a large recurring payoff. The catch is that "measurable" word — more on that below.
• Killing dead context. Few-shot examples that no longer earn their keep, instructions duplicated three times, retrieved chunks nobody reads. Deleting these is pure upside; the prompt gets cheaper and often better, because models attend worse to bloated context.

The unifying test: if the optimization is structural and one-time, do it. You pay the engineer cost once and amortize it across every future call. The arithmetic that sank the team in my opening example only goes wrong when the engineering cost is recurring — when you're hand-tuning, babysitting evals, and re-testing on every change.

When the trade quietly loses money

The losing trades share the opposite shape: ongoing engineering effort chasing a saving that's smaller than the effort, often while shifting cost onto a budget you don't see.

The clearest example is per-prompt manual optimization at low volume. If a prompt fires a few thousand times a month, the entire monthly spend on it might be a few hundred dollars. There is no optimization on earth worth a senior engineer's week against a $300 line item. But teams do it anyway, because the token number is visible and the engineer's time isn't charged to the same ledger.

The subtler trap is the evaluation tax on aggressive cost cuts. To know whether a cheaper model or a leaner prompt is "good enough," you have to define good enough for each task, build evals, collect representative data, and run comparisons. That's real engineering work that ships no features. Skip it and you're flying blind — you cut tokens, quality silently drops, and the support queue absorbs the difference. Do it properly and you may find the eval infrastructure cost more than years of the savings it was protecting. The honest move is to scope the eval work before committing to the cut, and let its size veto small-stakes optimizations.

And then there's premature optimization of a moving target. Model prices have been falling fast enough that a meaningful share of "savings" arrives for free if you simply wait. Spending a sprint to manually offset a cost that the next price drop or the next cheaper model would have erased is the LLM-era version of optimizing code the compiler was about to optimize for you.

A framework: make the trade explicit before you make it

The fix is not a policy of "always optimize" or "never optimize." It's a habit of writing down both sides of the trade before you start, so the decision stops being free by default. Four questions force the math into the open:

1. What's the monthly inference cost of this prompt or path, today? Not the unit cost — the aggregate. A prompt that's expensive per call but rarely fires is not your problem. Find your actual top spenders before touching anything.
2. Is the engineering cost one-time or recurring? One-time structural wins (caching, schemas, dead-context deletion) are almost always worth it. Recurring hand-tuning almost never is unless the volume is enormous.
3. What does this push onto another budget? If a token cut trades inference dollars for support tickets or eval-maintenance time, name that explicitly. A saving that just relocates cost to a team you don't manage is not a saving.
4. Would waiting one quarter make this free? If falling prices or a cheaper model are likely to absorb the cost on their own, the highest-ROI move is to do nothing and spend the engineer-week on product.

Run those four questions and the population of "obvious" optimizations thins out dramatically. What survives is a short list of structural, high-volume, one-time wins — which is exactly where the real money is, and exactly the list most teams never get to because they exhausted themselves on the small stuff first.

The reframe

The deepest version of this isn't a cost-accounting tip. It's a recognition that in an AI product, your two scarcest resources — engineer attention and inference budget — are convertible into each other, and the exchange rate is terrible in one direction. You can almost always spend engineer time to buy down tokens. You can far more rarely spend tokens to buy back engineer time, and when you can — a slightly larger prompt that removes a class of edge-case bugs, a more generous context window that kills a brittle retrieval pipeline — that's often the better trade, because it's paying down the expensive resource with the cheap one.

So the next time someone proposes trimming a prompt, ask what it costs in the currency that actually dominates your budget. Sometimes the answer is "nothing, it's a one-line cache change, ship it." Sometimes it's "three engineer-weeks to save eighteen hundred dollars a month against a price that's dropping anyway." Those are not the same decision, and the only thing that ever made them look the same was refusing to put both numbers on the same slide. A token budget that doesn't account for headcount isn't a budget. It's a guess about which resource you can afford to waste.