Renting Intelligence: A CFO's Mental Model for LLM Spend

Most finance teams filed their first LLM bill in the wrong drawer. It arrived during the experimentation phase, when a handful of engineers were prototyping with an API key, and it looked like exactly what it was at the time: research and development. A few thousand dollars a month to figure out whether the technology worked. So it went into R&D, mentally and sometimes literally, and nobody thought harder about it.

Then the feature shipped, usage climbed, and the same line item that was a rounding error in Q1 became the fastest-growing cost on the cloud invoice by Q4. The problem was never the dollar amount. The problem was that the cost had quietly changed categories — from a fixed bet on building something to a variable cost of serving every customer who used it — and the mental model hadn't moved with it.

That misclassification is the single most expensive accounting mistake in AI products right now, and it's not really an accounting mistake at all. It's a forecasting one.

The companies that get blindsided aren't the ones spending too much. They're the ones who modeled a usage-scaling cost as if it were a fixed one, priced their product on that assumption, and discovered the gap only when the invoice arrived. OpenAI itself is the cautionary tale at scale: in 2025 it generated an estimated $3.7 billion in revenue and lost roughly $5 billion, spending around $1.35 for every dollar it earned — driven not by headcount or training, but by the raw cost of serving inference to billions of requests. You don't have to be at that scale for the dynamic to hurt. You just have to misjudge which kind of cost you're carrying.

Tokens Are COGS, Not R&D

The cleanest mental model is the oldest one: think like a manufacturer, not a software company.

Traditional SaaS earned its 80-percent-plus gross margins because the marginal cost of serving one more customer was effectively zero. You built the software once and sold it a million times. The cost structure was almost entirely fixed — engineering salaries, a flat cloud bill — and revenue scaled while costs barely moved. That's what made software such a beautiful business.

Inference breaks that. Every query runs the model again. Every model run consumes GPU compute, memory bandwidth, and energy that someone has to pay for. There is no "build once, sell infinitely" trick, because every transaction carries a real, variable cost. That makes token spend cost of goods sold in the truest sense — usage-linked, not user-linked. It rises with prompt length, response length, retries, and concurrency, and the marginal cost of the next request is never zero, even after caching and volume discounts.

The numbers reflect the shift. ICONIQ's 2026 data puts the average AI product gross margin at 52 percent — up from 41 percent in 2024, but still a world away from the 80-percent benchmark that defined mature SaaS. At scaling-stage AI B2B companies, inference alone consumes roughly 23 percent of revenue. Public companies disclosing AI-driven margin pressure are reporting gross margins ten to seventeen points below their pre-AI baselines. When a quarter of your revenue is being eaten by a cost that grows with every successful customer interaction, that cost belongs above the gross-margin line, modeled and watched like the COGS it is.

The practical consequence: if your token spend lives in R&D on the P&L, your gross margin looks healthier than it is, and you'll keep making pricing and growth decisions on a number that's lying to you.

When Renting Is the Smart Trade

Calling it COGS doesn't mean it's a problem to be eliminated. The whole point of paying per token is that you're renting intelligence instead of buying it, and renting is frequently the correct financial decision.

When you call a provider's API, you carry no capex, no GPU procurement, no depreciation schedule, no idle-capacity risk. You pay only for what you use, and the cost scales down to zero when usage does. For a product still finding its shape — uncertain demand, evolving workloads, a model landscape that reshuffles every quarter — that optionality is worth a lot. You're buying the right to change your mind cheaply. Committing to owned infrastructure to shave the per-token rate is the equivalent of buying a factory before you know what you're manufacturing.

There's a second gift hidden in the rental model: the price of the thing you're renting keeps collapsing. The cost to achieve a given level of capability has fallen roughly 280x over two years — from around $20 per million tokens for GPT-3.5-class performance down to about seven cents. API pricing now spans a 600x range across models, from a dime per million tokens at the cheap end to sixty dollars for the frontier reasoning models. A workload that was uneconomical eighteen months ago may be comfortably profitable today without you changing a line of code. Owners of fixed infrastructure don't get that deflation handed to them for free; renters do.

So the honest version of the framework isn't "variable cost bad, fixed cost good." It's: rent while the workload is uncertain and the technology is moving, and revisit only when usage becomes large and stable enough that the per-unit savings of owning clear the operational cost of running it. The rule of thumb that's emerged in practice is that companies stay on the API until monthly inference spend reaches somewhere in the $50K–$100K range, then evaluate dedicated or self-hosted capacity. Below that line, the rental premium is cheaper than the engineering it would take to escape it.

Loading…