39 posts tagged with "inference"

Most teams buy inference the way their predecessors bought EC2 in 2010: everything on-demand, metered to the token, and surprising on two axes at once. The bill is one surprise. The rate limit is the other — the 429 that shows up in the middle of a launch, right when the on-demand pool you never reserved gets contended by everyone else who also chose to pay retail. Then someone opens the pricing page and discovers that the provider has been quietly selling reserved capacity the whole time: provisioned throughput, committed-use discounts, dollars-per-unit-per-hour instead of dollars-per-million-tokens. The reserved-instance decision that took the cloud industry a decade to internalize is sitting right there for tokens, and almost nobody has ported the playbook.

The reason isn't ignorance. It's that the reserved-instance math you learned for compute doesn't transfer cleanly, and the ways it breaks are exactly the ways that punish a naive commitment. A reserved EC2 instance is a bet that you'll still want that instance type in a year. A reserved block of token throughput is a bet that you'll still want that model in a year — and models have a shelf life measured in months, not decades. The commitment structure is familiar. The thing you're committing to is not.

Nothing broke. That's the disorienting part. No deploy failed, no latency alarm fired, no error rate ticked up. Someone merged a one-line PR that appended a sentence to the system prompt — a new tool description, a policy reminder, a "today's date is" header — and the next morning the inference bill was three to five times higher. Traffic was flat. The model was the same. The code did exactly what it was supposed to do.

What changed is that the one line landed in the wrong place, and every cached prefix in your fleet invalidated at once. Your cache hit rate went from 90% to zero in a single request cycle, and every token that used to be nearly free started billing at full price. This is the prompt-cache cliff, and it's the most expensive failure mode in production LLM systems that nobody threat-models, because it doesn't look like a failure at all.

Every frontier lab's launch post now carries the same footnote: "with parallel test-time compute." Sonnet's SWE-bench number jumps about five points with it. GPT-5-class models cut errors by double digits with it. Gemini's Deep Think nearly doubles its ARC-AGI-2 score with it. Most engineering teams read that footnote as benchmark seasoning — a way to inflate a leaderboard number that no real system would pay for — and then go back to architecting their product around one attempt from the biggest model they can afford.

That instinct is roughly two years out of date. The labs didn't add parallel sampling to their pro tiers as a marketing garnish; they added it because running N attempts behind a single answer is often the cheapest way to buy quality, and sometimes the only way. When three attempts from a cheap model plus a decent selector beat one attempt from a model that costs ten times more per token, best-of-N stops being a benchmark trick and becomes an architecture decision — one with its own cost model, its own latency profile, and its own signature failure mode. The teams treating it that way are quietly shipping better answers at lower cost than the teams still doing one-shot inference on the flagship model.

Somewhere in your inference fleet, there is probably a chip that computes wrong answers. Not crashes — wrong answers. It passed manufacturing tests, it passes health checks, and under a specific combination of instruction sequence, data values, voltage, and temperature, it returns a number that is simply incorrect. The hyperscalers have documented this at scale: roughly one device in a thousand silently corrupts data, a rate several orders of magnitude worse than the cosmic-ray bit flips we used to worry about.

For fifty years, software had an immune system against this: determinism. Same input, same output — so you could checksum, replay, and compare against golden results, and lying hardware eventually got caught. LLM inference is the first major workload where that immune system is gone. When a model gives a slightly worse answer, was it the sampler being a sampler, or a degraded GPU flipping bits in your KV cache? Nobody can tell by inspection. A flaky accelerator can quietly drag down your quality metrics for weeks while every dashboard stays green.

Every latency playbook you own was written for systems where the work per request was roughly constant. A database lookup takes what it takes. An image resize scales with pixel count, which you know in advance. Even classic LLM completions had a knowable cost envelope: input tokens in, bounded output tokens out. Reasoning models quietly deleted that assumption. When the model decides at runtime how long to think — and it decides based on how hard the problem turns out to be — response time stops being a property of your infrastructure and becomes a property of the question.

The consequence shows up first in your percentiles. Teams running reasoning models in production report p99 latencies spiking three to five times above p50, not because a host got slow or a cache went cold, but because one request in a hundred happened to be genuinely difficult. Your autoscaler, your timeout policy, and your SLO dashboard were all tuned for a world where that spread meant something was broken. Now it means the system is working as designed — and every tool you have for managing the tail is pointed at the wrong cause.

A model gives a customer a wrong, expensive answer. You open the post-mortem, paste the exact prompt back into the exact same endpoint with temperature=0, and you get a different answer. Not a worse one, not a better one — a different one. The bug you are supposed to root-cause refuses to reproduce on demand, and the one knob everyone told you guarantees reproducibility just lied to your face.

This is the moment most teams discover that "set temperature to zero" is folklore, not an engineering control. Temperature zero changes how the model samples — it forces greedy decoding, always taking the highest-probability token. It says nothing about whether the probabilities themselves come out the same twice. And in a production serving stack, they frequently don't.

Your finance team filed a dispute and lost. The line item is "output tokens" and it exceeds your sum-of-delivered-tokens metric by fourteen percent. The provider's support engineer closed the ticket as "expected behavior under streaming cancellation," with a link to a documentation page that says "cancellation stops billing at the last delivered token." Both sentences are true, and the gap between them is the line of code you have not written.

The contract you read says one thing. The inference scheduler does another. The mismatch is not a bug, not a billing error, and not malice — it is a layered system in which the cancellation signal travels through three boundaries (browser, edge, GPU) and the billing meter sits at the third boundary while your "stop generating" button sits at the first. Closing the gap is an engineering project with a finance owner.

The cluster did exactly what we told it to. Traffic dropped to zero for forty-five seconds, the queue-depth metric flatlined, KEDA flipped the replica count from one to zero, and the node autoscaler reclaimed the H100 pod ninety seconds later. The graph looked clean. The Slack channel was quiet. The cost dashboard ticked down half a cent.

An hour and twelve minutes later, a customer support ticket arrived: a long-running document-analysis job — a 180k-token reasoning task that was budgeted for twenty-eight minutes of decode — had vanished. No error in their client SDK. No exception in our application logs. Only a single 499 line buried in the gateway access log, timestamped roughly when the scheduler had decided the pod was idle and reaped it.

You set seed=42. You set temperature=0. You logged the run, posted the dashboard, signed off on the model swap. The next morning the rerun returned a different number on the same prompts, and the explanation you reached for — "must be sampling noise" — was wrong twice over: there was no sampling, and the noise was structural. The seed left your client, the gateway threw it away, the kernel batched your request next to seventeen unrelated ones, and the floating-point reduction order changed under you. Your "reproducible" benchmark was always within one batch of being a different benchmark.

This failure mode is quiet because every layer in the stack is technically correct. The SDK accepts the seed. The provider documents the seed. The model returns a system_fingerprint. The eval harness logs all three. Nothing 5xx's, nothing warns, nothing protests. The number on the dashboard just shifts, and the team rationalizes the shift as the kind of jitter that always existed — because they have no instrument that can tell them whether they're looking at stochastic decoding or at a backend rotation that invalidated three weeks of comparisons.

The CI test was a single assertion: same model, same temperature, same prompt, same seed, same output string. It passed on every developer's laptop, passed on the first hundred CI runs, and then flaked once every fifty runs for three weeks before anyone admitted the pattern was real. The first hypothesis was the obvious one — a non-deterministic dependency somewhere in the test harness — and three days of investigation found nothing. The actual cause was sitting in a footnote on the provider's API reference: "seed provides best-effort determinism." The team had read the parameter name and assumed a contract. The provider had documented a hint.

This is a specific failure mode of hosted inference that catches teams who design test infrastructure around a single mental model: the model is a pure function of its inputs, and the seed is what makes the function reproducible. Both halves of that model are wrong in production, and the gap between the API surface and the underlying physics is wide enough that teams build entire eval and regression-test stacks on top of an assumption their provider explicitly disclaimed.

The nightly fine-tune job starts at 02:00 UTC. It walks into the shared GPU pool, takes every slot it can find, and holds them. By 09:30, when the first inference traffic of the business day arrives, the autoscaler tries to claim capacity that has been continuously occupied for seven and a half hours. The first ninety minutes of the morning run at roughly four times the baseline p99 latency. The dashboard reports a "noisy morning tail" that the inference team attributes to user behavior, because the actual contention lives in a job queue nobody on the inference team owns.

This is the GPU-sharing failure mode that the cost-attribution slide in your capacity review does not capture. The sharing was sold as a utilization win — train at night, serve in the day, fill the trough. What actually shipped was a latency tail you cannot escape until the pool is partitioned by latency class, not by team or by clock.

A conversation has been moving along on a single Claude session for forty minutes. Eleven turns, each averaging 800ms time-to-first-token, each cheap because the 28,000-token prefix is hitting the prompt cache. Turn twelve arrives and TTFT is 3.4 seconds. The transcript hasn't changed shape. The model didn't switch. The network is fine. Cached input tokens drop from 27,800 to 0. The next turn's prefill bill is paid in full, from the first token.

You go looking for the cause in your traces and find nothing that names it. There is no event in your logs labeled "another tenant's burst evicted you." The only honest reading of the spike is that some other customer's prompt, somewhere on the same GPU pool, made the scheduler decide your warm prefix was the cheapest thing to drop. You cannot replay the turn. You cannot prove the eviction. The cache state at that moment was a function of strangers' traffic, and that traffic is not in your trace because it was never yours to see.