778 posts tagged with "llm"

Your pricing page says every user pays the same. Your cost dashboard says otherwise. The same AI feature — same prompt template, same model, same feature flag — costs 55% more to serve a Spanish user, roughly double for a Japanese user, and over 3x for an Arabic or Bengali user. Meanwhile, the quality those users receive is measurably worse: on identical benchmark questions translated across languages, frontier models drop 13 to 24 percentage points when you leave the English distribution.

Most teams shipping AI features globally have never measured either number. They have per-locale pricing, per-locale support SLAs, per-locale legal review — and a single English eval suite standing in for the experience of every user on the planet.

This is the tokenizer tax, and it compounds with a quality gap that scale alone doesn't close. Both are invisible in your dashboards until you slice by language, and both were decided years before you wrote your first prompt, by the training corpus of a tokenizer you don't control.

Every internal LLM platform goes through the same arc. In month one, inference is free: the platform team eats the bill, product teams experiment wildly, and everyone celebrates adoption curves. By month six, the bill has grown 10x, finance is asking pointed questions, and the platform team discovers that three teams account for 80% of spend — one of them running a nightly batch job nobody remembers approving. The instinctive response is to install a meter and start charging. That instinct, applied naively, is how you kill your platform.

Here's the uncomfortable truth: per-token chargeback punishes exactly the behavior you built the platform to encourage. The team prototyping an agent that might transform your support workflow burns tokens like a furnace — agentic workloads consume 5 to 30 times more tokens per task than a simple chat completion. Bill them list price from day one and they stop prototyping. Meanwhile, the team running a mature, optimized feature pays pennies and looks virtuous. You've built a pricing system that taxes learning and rewards stagnation.

Every progress bar you have ever shipped rests on one assumption: you know the denominator. Uploading a 40 MB file? The denominator is 40 MB. Installing 212 packages? The denominator is 212. The bar is honest because the total work is known before the work begins.

An agent breaks this assumption at the root. It doesn't execute a predetermined list of steps — it discovers its remaining work as it goes. It reads a file, which reveals three more files worth reading. It runs the tests, which fail, which spawns a debugging detour nobody planned. Step 4 of 7 becomes step 4 of 19 becomes, occasionally, step 4 of 4 because the last three turned out to be unnecessary. Percent-complete for an agent isn't hard to compute. It's undefined. There is no denominator until the work is finished, at which point the answer is always 100%.

Yet look at what we ship: spinners that promise imminent completion, bars that creep to 90% and stall, labels that say "almost done…" on minute two of a task that takes eight. These are all small lies, and users catch them. The interesting design question isn't how to fake progress more convincingly — it's what honest reassurance looks like when the duration of the work is structurally unknowable.

Fifteen years ago, we learned to fear the N+1 query: an ORM that looked innocent in code review would fire one query for a list and then one more query per row, and a page that should have made two database calls made two hundred. We fixed it with eager loading, batching, and a generation of linters. Then we built AI agents, and we shipped the same bug at a much more expensive layer.

A single agent task — "reconcile these invoices," "triage this incident" — routinely makes dozens of serial tool calls. Each one is a full network round trip: agent to tool, tool to data store, data back through the tool, result serialized into the model's context, another inference pass to decide the next call. If your inference runs in one cloud and your data lives in another, every one of those hops crosses a metered, high-latency boundary. The dominant cost term of your agent system is no longer the model. It's the geography.

Nobody itemized this. Latency budgets were written per call, egress was a rounding error on the storage bill, and the model invoice got all the scrutiny. Meanwhile the tool loop quietly became the chattiest client your infrastructure has ever served.

Persistent memory is the feature everyone adds to their agent and almost nobody maintains. The pitch is irresistible: the agent remembers your schema, your preferences, the decision from last Tuesday, and every session starts smarter than the last. The failure mode is quieter: memory grows monotonically by default, and an append-only store of facts about a changing world is a slow poisoning. The API that got migrated, the team that got reorged, the architectural decision that got reversed — all of it sits in the store next to fresh facts, retrieved with equal authority, injected into context with equal confidence.

A stateless agent makes isolated mistakes. A memory-equipped agent can turn one mistake into a recurring one, because it stores the error and then retrieves it later as evidence. One confidently-written wrong memory — "the payments service owns refund logic" — contaminates every future run that recalls it, and each run that acts on it may write new memories derived from it. That's not a storage problem. That's a garbage collection problem, and most agent memory systems ship without a collector.

Every major model provider will sell you the same tokens for half price. OpenAI, Anthropic, and Google all run batch APIs that charge 50% of the synchronous rate — same models, same prompts, same outputs — in exchange for one concession: you accept a 24-hour completion window instead of an answer in seconds. For a team spending $50,000 a month on inference, that is $25,000 sitting on the table, claimable without changing a single prompt.

Most teams never claim it. Not because the discount is hidden — it's on every pricing page — but because claiming it requires answering a question nobody in the org has asked: which of our inference calls actually need an answer now? That question turns out to be an architecture question, and the honest answer at most companies is "we never classified them, so everything runs in the interactive lane by default." The batch discount isn't a pricing footnote. It's a test of whether your system knows its own latency requirements — and most systems fail it.

Text-to-SQL demos never die on syntax. The model writes fluent SQL — better than most junior analysts, honestly — and the query runs, and a number comes back. The demo dies three weeks later, in production, when the CFO notices that the agent's "Q2 revenue" doesn't match the board deck. Not because the SQL was malformed, but because the warehouse contains three defensible definitions of revenue — bookings, recognized, and net-of-refunds — and the model confidently picked one. Just not the one finance uses.

This is the failure mode that matters, and it's invisible to every benchmark you've seen. The fix isn't a better model or a longer prompt. It's a piece of infrastructure most data teams already half-built and then abandoned: the semantic layer. The metrics definitions you wrote for BI dashboards — dbt metrics, LookML, cube definitions — turn out to be the missing tool contract for data agents. The teams shipping reliable agents figured out that the build order is inverted from what everyone assumed: semantic layer first, agent second.

The budget meeting for a fine-tuning project always prices the wrong thing. Teams estimate the data pipeline, the training runs, the eval passes — a one-time investment with a clear finish line. Then the model ships, the accuracy chart goes up and to the right, and everyone moves on. Six months later an email arrives: the base model your adapter is welded to has a retirement date. Nothing about your system changed. Everything about its foundation did.

This is the part nobody prices in: a fine-tune is not a product you finished. It is a fork of someone else's codebase, and every base-model release is an upstream rebase you didn't schedule. Anyone who has carried private patches against a fast-moving open-source project knows exactly how this story goes — the fork is cheap to create and expensive to keep.

Here is a postmortem pattern that is becoming a genre. The primary model was healthy all night. Latency was flat, token throughput normal, provider status page green. And yet every user request failed for forty minutes — because the safety classifier sitting in front of the model timed out, and the middleware wrapped that timeout in a generic exception, and the exception handler returned a refusal. Your model didn't go down. Your gate went down, and the gate was wired to fail closed by an engineer who never thought of it as a decision.

The uncomfortable truth is that most teams run a second machine-learning system in production without admitting it. The moderation classifier, the jailbreak detector, the PII scrubber, the topical filter — each one is a model, with its own latency distribution, its own error rates, its own training-data assumptions quietly rotting under drift, and its own failure modes. But because it's called a "guardrail," it gets treated like a config file: set once, never monitored, absent from the dashboard, missing from the on-call runbook. You would never ship your primary model without an SLO. Most teams ship their guardrail without even a health check.

Ask a coding agent to build a React component and it writes idiomatic, hook-shaped, accessibility-annotated code on the first try. Ask the same agent to use your in-house ORM — the one your platform team has maintained for six years, the one with excellent docs and a hundred internal consumers — and it hallucinates methods that don't exist, invents configuration options from some other library, and confidently ships code that compiles against an API it made up.

The difference isn't quality. Your ORM might be better-designed than half the open-source libraries the model handles flawlessly. The difference is training data. React has millions of public repositories behind it; your framework has zero. In the vocabulary of natural language processing, your internal framework is a low-resource language — and every consequence NLP researchers documented for low-resource languages now applies to your codebase.

There is a class of AI-generated bug that passes code review almost every time, and it isn't the hallucinated function or the fabricated package. It's the perfectly idiomatic code — idiomatic for the version of your stack that existed when the model's training data was frozen. The model writes a tailwind.config.js for a project running Tailwind v4, reaches for a class component lifecycle method in a hooks codebase, or calls an API that was deprecated three minor versions ago and removed in the one your lockfile actually pins. Nothing about the code looks wrong. It looks like code from a well-regarded tutorial. The tutorial is just from 2024.

Call it the training-cutoff bug class: defects that exist not because the model reasoned badly, but because the model's knowledge of your dependencies has a timestamp and your lockfile doesn't care. An ICSE 2025 study of seven code models across eight popular Python libraries found deprecated-API usage rates of 25–38% in plausible completions — and when the surrounding code already contained outdated patterns, that rate climbed to 70–90%. These aren't rare edge cases. They're the default failure mode of asking a frozen artifact to write for a moving target.

We spent two decades teaching services to retry safely. The playbook is well worn: a client generates a unique idempotency key, attaches it to the request, and the server records the key alongside the result inside the same transaction that performs the work. A dropped connection, a timeout, a 500 — the client retries with the same key, the server recognizes it, and returns the recorded result instead of charging the card twice. Stripe shipped this pattern years ago and it became table stakes for any API that touches money.

That entire design rests on one assumption nobody wrote down: the caller repeats its request byte-for-byte. The retry carries the same key because the retry is the same code path re-executing with the same variables. Break that assumption and the whole scheme quietly stops working.