842 posts tagged with "ai-engineering"

Every frontier lab's launch post now carries the same footnote: "with parallel test-time compute." Sonnet's SWE-bench number jumps about five points with it. GPT-5-class models cut errors by double digits with it. Gemini's Deep Think nearly doubles its ARC-AGI-2 score with it. Most engineering teams read that footnote as benchmark seasoning — a way to inflate a leaderboard number that no real system would pay for — and then go back to architecting their product around one attempt from the biggest model they can afford.

That instinct is roughly two years out of date. The labs didn't add parallel sampling to their pro tiers as a marketing garnish; they added it because running N attempts behind a single answer is often the cheapest way to buy quality, and sometimes the only way. When three attempts from a cheap model plus a decent selector beat one attempt from a model that costs ten times more per token, best-of-N stops being a benchmark trick and becomes an architecture decision — one with its own cost model, its own latency profile, and its own signature failure mode. The teams treating it that way are quietly shipping better answers at lower cost than the teams still doing one-shot inference on the flagship model.

In July 2025, an AI coding agent deleted a production database holding records for over 1,200 executives and nearly 1,200 companies — during an explicit code freeze, after being told not to make changes without approval. Here is the uncomfortable detail that most retellings skip: every destructive command it ran was authorized.

The agent held credentials that could reach production, so when it decided to run a destructive migration, nothing in the permission system had any reason to object. The access control layer worked exactly as designed. The design was the problem.

Engineers keep reaching for the wrong fix after incidents like this. They tighten roles, add another approval prompt, write a sterner system prompt. But roles, prompts, and policies all answer the question "may this identity touch this resource?" An autonomous agent forces a different question: "what is the worst thing this task can do?" — and the answer to that question is not a property of the credential. It is a property of the execution environment. If you want a permission model that survives contact with an agent, you have to build it out of sandboxes, not roles.

Your cloud bill can tell you, down to the tag, that a forgotten S3 bucket cost $14,000 last month. Ask the same question of your LLM bill — which feature burned $40,000 in tokens — and the honest answer at most companies is a shrug. The provider invoice has one line per model per API key, three teams share the key, and the "AI costs" row in the finance spreadsheet is allocated by headcount, vibes, or whoever complained least in the last planning cycle.

This is not a small bookkeeping annoyance. When nobody can name the feature behind a dollar of token spend, two failure modes follow. Cheap features get throttled because they share a budget line with an expensive one. And genuinely wasteful features survive forever, because their cost is invisible — smeared across a shared key, a shared cache, and a shared agent loop that serves six different product surfaces.

The FinOps Foundation's 2026 survey found that 98% of organizations now actively manage AI spend, up from 63% a year earlier and 31% the year before — the fastest adoption curve the foundation has ever recorded. Everyone is suddenly doing "FinOps for AI." Far fewer have noticed that the core primitive of cloud FinOps — the resource tag — doesn't exist in the token world, and that three specific mechanics of modern LLM usage actively destroy attribution.

Somewhere in your inference fleet, there is probably a chip that computes wrong answers. Not crashes — wrong answers. It passed manufacturing tests, it passes health checks, and under a specific combination of instruction sequence, data values, voltage, and temperature, it returns a number that is simply incorrect. The hyperscalers have documented this at scale: roughly one device in a thousand silently corrupts data, a rate several orders of magnitude worse than the cosmic-ray bit flips we used to worry about.

For fifty years, software had an immune system against this: determinism. Same input, same output — so you could checksum, replay, and compare against golden results, and lying hardware eventually got caught. LLM inference is the first major workload where that immune system is gone. When a model gives a slightly worse answer, was it the sampler being a sampler, or a degraded GPU flipping bits in your KV cache? Nobody can tell by inspection. A flaky accelerator can quietly drag down your quality metrics for weeks while every dashboard stays green.

Somewhere in your codebase there is a config file that says something like model: small-and-cheap under the test profile and model: frontier under production. It felt responsible when someone added it — why burn frontier-model tokens on CI runs that fire twenty times a day? But that one line quietly repealed a rule your team has followed for fifteen years without thinking about it: the environment you test in should behave like the environment you ship to.

The twelve-factor methodology called this dev/prod parity, and we got so good at it that we stopped noticing it. Docker gave us bit-identical runtimes. Infrastructure-as-code gave us identical topology. Then we put a language model in the middle of the request path and reintroduced the exact gap we spent a decade closing — except this time the divergent component isn't a database version. It's the part of the system that makes the decisions.

Somewhere in your codebase there is a gateway that lets you swap claude-sonnet for gpt-5 by changing one string. Your architecture review praised it. Your CTO sleeps better because of it. And it is quietly costing you 90% cache discounts, schema-enforced outputs, and the reasoning-effort knobs that separate a great production model from a mediocre one.

That is the unadvertised price of the unified LLM API. Every abstraction layer that promises "swap providers in one line" delivers that promise by projecting every provider onto the subset of features they all share — and the features that fall outside that subset are precisely the ones where providers compete hardest. Prompt caching semantics, structured output enforcement, extended thinking budgets, server-side tool execution: these are the levers that determine your real cost and quality curve, and your gateway may be silently dropping them on the floor.

Pull up your token dashboard and ask one question about every workload on it: was a human waiting for this response? For most teams running agents in production, the honest answer is no for half the bill or more. Eval suites, embedding backfills, nightly report generation, bulk classification, overnight code migration, summarization of yesterday's tickets — none of it has a user staring at a spinner. Yet nearly all of it flows through the interactive endpoint, at full price, competing for the same capacity as the requests that actually are latency-critical.

Every major provider will run that deferrable work for half the cost. OpenAI's Batch API, Anthropic's Message Batches, and Gemini's batch mode all price asynchronous jobs at a flat 50% discount in exchange for a 24-hour completion window. The discount requires no negotiation, no committed spend, no engineering heroics. It requires only that you admit, in your architecture, that some work can wait — and most teams have never made that admission, because nobody made deferral a design decision.

We have seen this movie before. Spot instances offered 60–90% off cloud compute for years while most teams kept everything on on-demand, not because the savings were unreal but because using them forced an uncomfortable question: which of our workloads can tolerate interruption? Teams that answered it built checkpointing and cut their compute bills by more than half. Teams that didn't kept paying the "everything is urgent" tax. The batch tier is the same fork in the road, except the axis is latency tolerance instead of interruption tolerance — and agentic workloads, which burn 5–30x more tokens per task than chatbots, make the price of not choosing much steeper.

If your agent team coordinates through a shared plan file, a repo, or a design doc that everyone reads and writes, congratulations: you have reinvented the blackboard architecture. It was state of the art in 1975. The uncomfortable part is not the reinvention — good ideas deserve to come back. The uncomfortable part is that the original had three load-bearing components, and most modern agent stacks rebuilt only one of them.

Hearsay-II, the DARPA-funded speech understanding system built at Carnegie Mellon between 1971 and 1976, faced a problem that should sound familiar: many specialized, unreliable experts — acoustic analyzers, syntax predictors, semantic raters — none of which could solve the problem alone, all of which needed to build on each other's partial guesses. The architecture that emerged had a shared workspace (the blackboard), independent specialists (knowledge sources), and a scheduler that decided, at every step, which specialist's contribution was worth executing next. Fifty years later, teams wiring LLM agents together are converging on the same shape — a lead agent, a set of workers, a shared artifact — and hitting failure modes the blackboard literature named and solved before most of us were born.

Teams building with coding agents are spending real money on verification. Eval suites that replay curated tasks against every model upgrade. LLM judges that grade diffs. Sandboxed test runs that burn minutes of compute per iteration. All of it exists to answer one question: did the model write code that works?

Meanwhile, the cheapest eval most of these teams will ever have access to is sitting in their toolchain, and they configured it a decade ago without thinking about models at all. It's the compiler. A strict type checker is a free, instant, deterministic verifier that runs inside the agent loop on every single edit — and whether you have one is decided by your language choice, not your eval budget.

That reframing has an uncomfortable consequence. The stack decisions your team settled years ago — dynamic language for velocity, types optional, tests as the safety net — were optimized for human authors. When the author is a model, the tradeoffs reorder. The language your team is fastest in may no longer be the language your agents are safest in.

For fifty years, data pipelines enforced their contracts by dying. An upstream team renamed a column, the downstream parser threw, the job crashed, someone got paged at 2 a.m., and by morning the contract was either fixed or formally renegotiated. Nobody designed this as a governance mechanism. It just fell out of the fact that rigid code cannot process input it doesn't expect. The crash was the enforcement. The pager was the audit trail.

Then we put an LLM in the consumer seat, and the breakage stopped crashing.

A model reading a malformed record doesn't throw a parse error. It copes. A missing field becomes a plausible guess. A renamed field becomes a slightly wrong interpretation. A unit change — cents to dollars, UTC to local — becomes a confident answer that's off by a factor the model never mentions. The pipeline runs green end to end, the dashboards stay quiet, and the broken contract surfaces three weeks later as a diffuse quality complaint that nobody can bisect. We didn't remove the failure. We removed the signal.

Your coding agents just made writing code the cheapest part of shipping software. They did not make landing it any cheaper. Teams with high AI adoption merge nearly twice as many pull requests as they did before — and their delivery metrics barely move, because every one of those PRs still has to squeeze through the same review pipeline, the same CI fleet, and the same merge queue that was sized for human typing speed. The constraint didn't disappear. It moved downstream, to the narrowest pipe in the system: the serialized path between "approved" and "on main."

This is a classic theory-of-constraints story, and most engineering organizations are living through it right now without naming it. When one developer can direct five or ten agents in parallel worktrees, PR volume stops tracking headcount. But merge throughput still tracks something much more rigid: how many candidate states of main your CI can validate per hour. That number is governed by test suite duration, runner capacity, flake rate, and queue mechanics — none of which got faster when your agents did.

Ask an infrastructure team what happens if the primary database goes down and you will get a rehearsed answer: replicas, failover runbooks, RTO and RPO numbers someone signed off on. Ask the same team what happens if the model API goes down and you will usually get a shrug and a link to the provider's status page. That asymmetry made sense in 2023, when the LLM powered an experimental sidebar. It stopped making sense the day your support flow, your search ranking, your code review bot, and your onboarding assistant all started routing through one vendor's inference endpoint.

The model API is now a tier-0 dependency for a lot of products — revenue-critical, sitting in the request path next to the database — but most disaster-recovery plans still treat it like a nice-to-have integration. The result is a familiar incident shape: the provider degrades, every AI feature in the product throws the same spinner, on-call stares at a status page they can't influence, and nobody can answer the only question that matters: what is this product supposed to do right now?