990 posts tagged with "insider"

Somewhere in your inference fleet, there is probably a chip that computes wrong answers. Not crashes — wrong answers. It passed manufacturing tests, it passes health checks, and under a specific combination of instruction sequence, data values, voltage, and temperature, it returns a number that is simply incorrect. The hyperscalers have documented this at scale: roughly one device in a thousand silently corrupts data, a rate several orders of magnitude worse than the cosmic-ray bit flips we used to worry about.

For fifty years, software had an immune system against this: determinism. Same input, same output — so you could checksum, replay, and compare against golden results, and lying hardware eventually got caught. LLM inference is the first major workload where that immune system is gone. When a model gives a slightly worse answer, was it the sampler being a sampler, or a degraded GPU flipping bits in your KV cache? Nobody can tell by inspection. A flaky accelerator can quietly drag down your quality metrics for weeks while every dashboard stays green.

Every team building agents eventually hits the same wall. The system prompt started at 400 tokens. Then someone added the database migration checklist. Then the postmortem template, the deploy runbook, the style guide for customer emails. Eighteen months later it's a 9,000-token monolith that nobody dares to edit, because changing the line about rollback procedures somehow degrades the agent's tone in support tickets. You've built the prompt equivalent of a single 50,000-line main.c — and everyone is statically linking against it.

The instinct is to reach for RAG: chunk the runbooks, embed them, retrieve on demand. That fails in a subtler way. RAG is built to retrieve facts, and facts degrade gracefully when fragmented — three out of five relevant chunks about your billing model still tells the agent most of what it needs. Procedures don't degrade gracefully. A database migration runbook retrieved at 60% is not 60% useful; it's a production incident. Steps 1 through 4 without step 5 ("verify replication lag before cutting over") is worse than no runbook at all, because the agent now acts with confidence it hasn't earned.

Somewhere in your codebase there is a config file that says something like model: small-and-cheap under the test profile and model: frontier under production. It felt responsible when someone added it — why burn frontier-model tokens on CI runs that fire twenty times a day? But that one line quietly repealed a rule your team has followed for fifteen years without thinking about it: the environment you test in should behave like the environment you ship to.

The twelve-factor methodology called this dev/prod parity, and we got so good at it that we stopped noticing it. Docker gave us bit-identical runtimes. Infrastructure-as-code gave us identical topology. Then we put a language model in the middle of the request path and reintroduced the exact gap we spent a decade closing — except this time the divergent component isn't a database version. It's the part of the system that makes the decisions.

If your agent team coordinates through a shared plan file, a repo, or a design doc that everyone reads and writes, congratulations: you have reinvented the blackboard architecture. It was state of the art in 1975. The uncomfortable part is not the reinvention — good ideas deserve to come back. The uncomfortable part is that the original had three load-bearing components, and most modern agent stacks rebuilt only one of them.

Hearsay-II, the DARPA-funded speech understanding system built at Carnegie Mellon between 1971 and 1976, faced a problem that should sound familiar: many specialized, unreliable experts — acoustic analyzers, syntax predictors, semantic raters — none of which could solve the problem alone, all of which needed to build on each other's partial guesses. The architecture that emerged had a shared workspace (the blackboard), independent specialists (knowledge sources), and a scheduler that decided, at every step, which specialist's contribution was worth executing next. Fifty years later, teams wiring LLM agents together are converging on the same shape — a lead agent, a set of workers, a shared artifact — and hitting failure modes the blackboard literature named and solved before most of us were born.

At 4:50 PM on a Friday, someone on the data team kicks off an eval sweep: forty thousand prompts against the company's shared model deployment, scheduled to finish over the weekend. At 5:10 PM, the customer-facing chat assistant starts timing out. The on-call engineer spends two hours staring at dashboards that show the provider returning 429s before someone thinks to ask who else is using the account. Nothing is broken. The system is doing exactly what it was configured to do, which is nothing, because nobody configured it to do anything.

This is the shape of a new class of incident, and it has a property that makes it nastier than an ordinary outage: there is no bug to fix. The eval sweep was legitimate work. The chat assistant's traffic was legitimate work. The failure is that two teams with different urgency profiles were drawing from one undifferentiated pool of inference capacity, and the pool had no opinion about who mattered more. Capacity allocation stopped being an infrastructure detail the moment your company had more than one team shipping against the same provider account — it became a political problem, and the pager inherited it.

Your coding agents just made writing code the cheapest part of shipping software. They did not make landing it any cheaper. Teams with high AI adoption merge nearly twice as many pull requests as they did before — and their delivery metrics barely move, because every one of those PRs still has to squeeze through the same review pipeline, the same CI fleet, and the same merge queue that was sized for human typing speed. The constraint didn't disappear. It moved downstream, to the narrowest pipe in the system: the serialized path between "approved" and "on main."

This is a classic theory-of-constraints story, and most engineering organizations are living through it right now without naming it. When one developer can direct five or ten agents in parallel worktrees, PR volume stops tracking headcount. But merge throughput still tracks something much more rigid: how many candidate states of main your CI can validate per hour. That number is governed by test suite duration, runner capacity, flake rate, and queue mechanics — none of which got faster when your agents did.

Somewhere, right now, an AI assistant is explaining your product to a prospect. It is quoting a price you changed eighteen months ago, recommending an integration you sunset last quarter, and suggesting an API endpoint that returns 410 Gone. You will never see this conversation. No analytics event fires. No session recording exists. The prospect either believes the wrong answer and files a confused support ticket, or believes the wrong answer and quietly buys from the competitor the model mentioned in the same breath.

This is not a hypothetical future problem. AI referrals already account for meaningful traffic — up to 5–8% for some technology and e-commerce sites — and the answers behind those referrals are generated from whatever the model absorbed about you, whenever it absorbed it. Your marketing team spent a decade learning to monitor brand search, review sites, and social mentions. Almost nobody is monitoring the surface that is growing fastest: what the models say when someone asks about you.

The pitch for overnight coding agents is seductive: you go to sleep, the fleet works, and you wake up to finished pull requests. What actually happens is subtler and more expensive. You wake up to a queue — six branches, two failed runs, a dependency bump you didn't ask for, and a refactor that is either brilliant or subtly wrong. The agents did produce code. But the deliverable that landed on your desk isn't code. It's a triage problem, and most teams have no workflow for it.

The numbers say this isn't a niche complaint. A telemetry study of more than 10,000 developers across 1,255 teams found that high-AI-adoption teams merged 98% more pull requests — while review time rose 91% and average PR size grew 154%. Follow-up data from 2026 is worse: production incidents per PR roughly tripled, and 31% more PRs now merge with no human review at all. The bottleneck didn't disappear when agents started working the night shift. It moved to 9 a.m., concentrated into the first ninety minutes of your day, and got a name: the morning review queue.

Every latency playbook you own was written for systems where the work per request was roughly constant. A database lookup takes what it takes. An image resize scales with pixel count, which you know in advance. Even classic LLM completions had a knowable cost envelope: input tokens in, bounded output tokens out. Reasoning models quietly deleted that assumption. When the model decides at runtime how long to think — and it decides based on how hard the problem turns out to be — response time stops being a property of your infrastructure and becomes a property of the question.

The consequence shows up first in your percentiles. Teams running reasoning models in production report p99 latencies spiking three to five times above p50, not because a host got slow or a cache went cold, but because one request in a hundred happened to be genuinely difficult. Your autoscaler, your timeout policy, and your SLO dashboard were all tuned for a world where that spread meant something was broken. Now it means the system is working as designed — and every tool you have for managing the tail is pointed at the wrong cause.

Every security team has a mental model of typosquatting: an attacker registers requets and waits for someone to fat-finger requests. It works, but it's a scattershot bet on human clumsiness. Slopsquatting is worse, because the "typo" isn't random. Language models invent plausible-but-nonexistent package names in predictable, repeatable patterns — and attackers can query the same models you use, harvest the names they invent, and register exactly those packages on PyPI and npm. The hallucination becomes a preorder. Your coding agent, running with autonomous install rights, is the customer who picks it up.

This isn't hypothetical. The largest study of the phenomenon generated 2.23 million code samples across sixteen models and found that 19.7 percent of recommended packages didn't exist — 205,474 unique fabricated names. And when a security researcher registered one of the most commonly hallucinated Python packages as a harmless empty shell, it was downloaded more than 30,000 times in three months and ended up in the install instructions of a major tech company's open-source repository. The supply chain attack that vibe coding made possible is already running its proof of concept.

Ask an engineer where their company's business logic lives and they'll point at a Git repository. Ask the finance team, the ops team, or the sales team, and the honest answer is a file called pricing_model_v7_FINAL_final.xlsx. The pricing formulas, the headcount plan, the commission calculations, the month-end reconciliation macros — the operational core of most companies runs on Excel and Google Sheets. It has no types, no tests, no code review, and no deploy pipeline. It is production infrastructure maintained like a napkin sketch.

For thirty years this mostly worked, because the only things reading and writing those files were humans — slow, cautious, and equipped with an intuition for "that number looks wrong." That era just ended. MCP servers for Google Sheets and Excel now expose full create-read-update-delete access as first-class agent tools, and every agent platform ships a spreadsheet connector because that's where the customers' data actually is. We have connected the fastest writers ever built to the most fragile production system ever deployed, and most teams did it in an afternoon without a single design review.

The incident channel says the new prompt is hallucinating refund amounts, so you do the obvious thing: repoint the production tag to last week's prompt version. Thirty seconds, no redeploy, textbook rollback. Except the agent gets worse. Last week's prompt references a lookup_order tool that the platform team renamed to orders.search on Tuesday. The memory store is full of preference summaries written by the new prompt's format, which the old prompt reads as user instructions. You didn't roll back the agent. You built a chimera — one-third last week, two-thirds today — and shipped it to production without ever testing that combination.

This is the failure mode nobody's runbook covers: an agent deployment is not an artifact, it's a triple — prompt version, tool contract, accumulated memory state. Rolling back one leg while the other two advance doesn't restore a previous state. It creates a new state that has never existed before, never passed an eval, and belongs to no team's on-call rotation.