842 posts tagged with "ai-engineering"

Every latency playbook you own was written for systems where the work per request was roughly constant. A database lookup takes what it takes. An image resize scales with pixel count, which you know in advance. Even classic LLM completions had a knowable cost envelope: input tokens in, bounded output tokens out. Reasoning models quietly deleted that assumption. When the model decides at runtime how long to think — and it decides based on how hard the problem turns out to be — response time stops being a property of your infrastructure and becomes a property of the question.

The consequence shows up first in your percentiles. Teams running reasoning models in production report p99 latencies spiking three to five times above p50, not because a host got slow or a cache went cold, but because one request in a hundred happened to be genuinely difficult. Your autoscaler, your timeout policy, and your SLO dashboard were all tuned for a world where that spread meant something was broken. Now it means the system is working as designed — and every tool you have for managing the tail is pointed at the wrong cause.

Every security team has a mental model of typosquatting: an attacker registers requets and waits for someone to fat-finger requests. It works, but it's a scattershot bet on human clumsiness. Slopsquatting is worse, because the "typo" isn't random. Language models invent plausible-but-nonexistent package names in predictable, repeatable patterns — and attackers can query the same models you use, harvest the names they invent, and register exactly those packages on PyPI and npm. The hallucination becomes a preorder. Your coding agent, running with autonomous install rights, is the customer who picks it up.

This isn't hypothetical. The largest study of the phenomenon generated 2.23 million code samples across sixteen models and found that 19.7 percent of recommended packages didn't exist — 205,474 unique fabricated names. And when a security researcher registered one of the most commonly hallucinated Python packages as a harmless empty shell, it was downloaded more than 30,000 times in three months and ended up in the install instructions of a major tech company's open-source repository. The supply chain attack that vibe coding made possible is already running its proof of concept.

When a dependency breaks, your build fails. When a workaround becomes unnecessary, nothing happens at all. That asymmetry is why every production LLM system older than a year is carrying scaffolding it no longer needs — retry choreography, output-repair parsers, forced chain-of-thought, elaborate task decomposition, chunking heuristics — each one built as a compensation for a specific model's specific weakness, and each one silently outliving the weakness it compensated for.

The uncomfortable part is that this isn't a hygiene problem, like stale feature flags. Obsolete scaffolding doesn't just sit there costing you latency and tokens. In the worst case it actively constrains the new model to the old model's ceiling: your decomposition logic chops a task into six steps because the 2024 model couldn't hold the whole thing, and the 2026 model — which could have one-shotted it — now inherits six opportunities to lose context at the seams you built.

Somewhere in your company there is a slide with one number on it. "AI quality: 87." Last quarter it said 85, so the slide is green. Meanwhile, your on-call channel is full of screenshots of the assistant confidently inventing refund policies for your largest enterprise customer. Both things are true at once, and the slide is the one lying.

The executive request behind that slide is completely reasonable: give me one score I can track, so I know whether the thing is getting better or worse. It works for revenue. It works for uptime. It does not work for an AI feature, because an AI feature's quality is not a scalar — it is a distribution over inputs, users, and time. Averaging that distribution into a single number doesn't summarize it; it destroys precisely the information a decision-maker needs.

This post is about the gap between those two facts: why the mean of your eval suite hides the regressions that actually hurt you, what to report instead, and how to present a legitimately noisy metric to a board-deck audience without torching your credibility the first week it moves down.

Every framework migration has the same shape. You write a codemod, run it across the repository, and it cleanly converts 80% of the files — the ones that follow the patterns the codemod's author anticipated. Then you hit the long tail: the test file where someone monkey-patched the renderer, the component that reaches into framework internals, the module written in 2017 by an engineer who has since left, using an idiom nobody else ever adopted. The codemod parses these files fine. It just has no rule that applies. So the last 20% of the migration consumes 80% of the calendar, done by hand, file by file, by engineers who would rather be doing anything else.

Coding agents invert this economics. The hand-written weirdness that defeats a deterministic AST transform is exactly what a model handles well — it reads the file, understands intent, and rewrites it the way a human would, without needing an explicit rule for every variant. Airbnb proved the point at scale: nearly 3,500 Enzyme test files migrated to React Testing Library in six weeks, against an original estimate of 1.5 years of manual work. But here's what gets lost in the headline: the hard part wasn't the prompting. Once you point a fleet of agents at ten thousand files, the engineering problem stops being an AI problem and becomes a batch-operations problem — sharding, verification, quarantine, and merge strategy. The right mental model is a MapReduce job whose mapper is stochastic.

Pull up your request logs from the last time you hit sustained 429s. You will probably find something odd: the errors don't arrive as a steady stream. They arrive in waves — a burst of 429s, a quiet gap, a bigger burst, another gap. The provider's quota didn't change between waves. Your traffic didn't spike. What you are looking at is your own retry logic, synchronized against itself. Every client that failed at second zero computed the same backoff delay, slept the same duration, and woke up at the same instant to fail together again.

This is the thundering herd, and the punchline is that the standard fix — exponential backoff — does not prevent it. Deterministic exponential backoff organizes the herd. It takes a crowd of clients that failed at roughly the same moment and marches them forward in lockstep: everyone retries at 1 second, then everyone at 2, then everyone at 4. The load spikes get farther apart, but each spike is just as tall as the first. If the spike is what triggered your rate limit, you have built a metronome that re-triggers it forever.

Your pricing page says every user pays the same. Your cost dashboard says otherwise. The same AI feature — same prompt template, same model, same feature flag — costs 55% more to serve a Spanish user, roughly double for a Japanese user, and over 3x for an Arabic or Bengali user. Meanwhile, the quality those users receive is measurably worse: on identical benchmark questions translated across languages, frontier models drop 13 to 24 percentage points when you leave the English distribution.

Most teams shipping AI features globally have never measured either number. They have per-locale pricing, per-locale support SLAs, per-locale legal review — and a single English eval suite standing in for the experience of every user on the planet.

This is the tokenizer tax, and it compounds with a quality gap that scale alone doesn't close. Both are invisible in your dashboards until you slice by language, and both were decided years before you wrote your first prompt, by the training corpus of a tokenizer you don't control.

Most build-vs-buy arguments in AI engineering are fought one component at a time. Should we build our own model gateway or use OpenRouter? Write our own agent loop or adopt a framework? Fine-tune or prompt? Each debate gets its own meeting, its own spreadsheet, its own loudest voice. And each one quietly assumes the landscape will hold still long enough for the decision to pay off.

It won't. The ground under your AI stack is moving faster than any stack has moved before. Inference prices for a fixed capability level have been falling somewhere between 9x and 900x per year depending on the capability threshold you track — GPT-3-level output that cost $60 per million tokens in late 2021 now costs pennies. Agent frameworks have gone through multiple breaking rewrites in three years. The gateway layer went from "we should build one" to a crowded commodity market with managed and self-hosted options in about eighteen months. Deciding component-by-component in a landscape like this is navigating a river by staring at your feet.

There's a sixty-year-old tool built for exactly this problem: the Wardley map. It won't tell you what to build. It will tell you something more useful — which of your components are about to become someone else's cheap product, and which are moving the other way.

Every internal LLM platform goes through the same arc. In month one, inference is free: the platform team eats the bill, product teams experiment wildly, and everyone celebrates adoption curves. By month six, the bill has grown 10x, finance is asking pointed questions, and the platform team discovers that three teams account for 80% of spend — one of them running a nightly batch job nobody remembers approving. The instinctive response is to install a meter and start charging. That instinct, applied naively, is how you kill your platform.

Here's the uncomfortable truth: per-token chargeback punishes exactly the behavior you built the platform to encourage. The team prototyping an agent that might transform your support workflow burns tokens like a furnace — agentic workloads consume 5 to 30 times more tokens per task than a simple chat completion. Bill them list price from day one and they stop prototyping. Meanwhile, the team running a mature, optimized feature pays pennies and looks virtuous. You've built a pricing system that taxes learning and rewards stagnation.

Persistent memory is the feature everyone adds to their agent and almost nobody maintains. The pitch is irresistible: the agent remembers your schema, your preferences, the decision from last Tuesday, and every session starts smarter than the last. The failure mode is quieter: memory grows monotonically by default, and an append-only store of facts about a changing world is a slow poisoning. The API that got migrated, the team that got reorged, the architectural decision that got reversed — all of it sits in the store next to fresh facts, retrieved with equal authority, injected into context with equal confidence.

A stateless agent makes isolated mistakes. A memory-equipped agent can turn one mistake into a recurring one, because it stores the error and then retrieves it later as evidence. One confidently-written wrong memory — "the payments service owns refund logic" — contaminates every future run that recalls it, and each run that acts on it may write new memories derived from it. That's not a storage problem. That's a garbage collection problem, and most agent memory systems ship without a collector.

Every major model provider will sell you the same tokens for half price. OpenAI, Anthropic, and Google all run batch APIs that charge 50% of the synchronous rate — same models, same prompts, same outputs — in exchange for one concession: you accept a 24-hour completion window instead of an answer in seconds. For a team spending $50,000 a month on inference, that is $25,000 sitting on the table, claimable without changing a single prompt.

Most teams never claim it. Not because the discount is hidden — it's on every pricing page — but because claiming it requires answering a question nobody in the org has asked: which of our inference calls actually need an answer now? That question turns out to be an architecture question, and the honest answer at most companies is "we never classified them, so everything runs in the interactive lane by default." The batch discount isn't a pricing footnote. It's a test of whether your system knows its own latency requirements — and most systems fail it.

Your agent answered a customer's billing question with last quarter's pricing, and the postmortem will blame the model. It shouldn't. The prompt was assembled correctly, the retrieval scored well, the model reasoned soundly over everything it was given — and everything it was given was true three days ago. Somewhere between the CRM export, the docs sync, and the vector index rebuild, "current state of the world" quietly became "state of the world as of Tuesday," and nothing in your stack was measuring the difference.

Data engineers solved this class of problem years ago. A downstream dashboard consuming ten upstream tables gets lineage, freshness checks, and an on-call rotation that pages when the nightly job slips. The context window your agent consumes is the same thing — a materialized view joined from docs, tickets, code, CRM, and memory — except nobody owns the join, nothing measures its staleness, and when it serves yesterday's truth the failure gets filed as "the model hallucinated."