35 posts tagged with "coding-agents"

Teams building with coding agents are spending real money on verification. Eval suites that replay curated tasks against every model upgrade. LLM judges that grade diffs. Sandboxed test runs that burn minutes of compute per iteration. All of it exists to answer one question: did the model write code that works?

Meanwhile, the cheapest eval most of these teams will ever have access to is sitting in their toolchain, and they configured it a decade ago without thinking about models at all. It's the compiler. A strict type checker is a free, instant, deterministic verifier that runs inside the agent loop on every single edit — and whether you have one is decided by your language choice, not your eval budget.

That reframing has an uncomfortable consequence. The stack decisions your team settled years ago — dynamic language for velocity, types optional, tests as the safety net — were optimized for human authors. When the author is a model, the tradeoffs reorder. The language your team is fastest in may no longer be the language your agents are safest in.

Your coding agents just made writing code the cheapest part of shipping software. They did not make landing it any cheaper. Teams with high AI adoption merge nearly twice as many pull requests as they did before — and their delivery metrics barely move, because every one of those PRs still has to squeeze through the same review pipeline, the same CI fleet, and the same merge queue that was sized for human typing speed. The constraint didn't disappear. It moved downstream, to the narrowest pipe in the system: the serialized path between "approved" and "on main."

This is a classic theory-of-constraints story, and most engineering organizations are living through it right now without naming it. When one developer can direct five or ten agents in parallel worktrees, PR volume stops tracking headcount. But merge throughput still tracks something much more rigid: how many candidate states of main your CI can validate per hour. That number is governed by test suite duration, runner capacity, flake rate, and queue mechanics — none of which got faster when your agents did.

Every security team has a mental model of typosquatting: an attacker registers requets and waits for someone to fat-finger requests. It works, but it's a scattershot bet on human clumsiness. Slopsquatting is worse, because the "typo" isn't random. Language models invent plausible-but-nonexistent package names in predictable, repeatable patterns — and attackers can query the same models you use, harvest the names they invent, and register exactly those packages on PyPI and npm. The hallucination becomes a preorder. Your coding agent, running with autonomous install rights, is the customer who picks it up.

This isn't hypothetical. The largest study of the phenomenon generated 2.23 million code samples across sixteen models and found that 19.7 percent of recommended packages didn't exist — 205,474 unique fabricated names. And when a security researcher registered one of the most commonly hallucinated Python packages as a harmless empty shell, it was downloaded more than 30,000 times in three months and ended up in the install instructions of a major tech company's open-source repository. The supply chain attack that vibe coding made possible is already running its proof of concept.

For twenty-five years, "never rewrite from scratch" has been the closest thing software engineering has to a commandment. The canonical argument rested on a cost structure everyone took for granted: rewriting means re-reading, re-understanding, and re-typing years of accumulated code, and while you do that, the old system keeps moving and your competitors keep shipping. The rewrite was forbidden because it was slow.

Coding agents just deleted the slow part. An agent can translate a hundred-thousand-line codebase from one language or framework to another in days, not quarters. Teams that ran the numbers on mainframe modernization watched average program costs drop from $9.1 million in 2024 to $7.2 million in 2025, largely on the back of AI-assisted conversion, and consultancies now report 40–50% acceleration on modernization timelines. So the commandment is dead, right? If the expensive part of the rewrite got cheap, the rewrite is back on the menu.

Here is the problem: the typing was never the expensive part. It was just the visible part. The organizations that failed at big migrations rarely failed because the code was hard to convert. They failed on everything around the code — undocumented behavior, data migration, integration cutover, and the operational muscle memory that nobody thought to write down. Agents made the rewrite cheap to start. They did much less for what made rewrites dangerous to finish.

Every framework migration has the same shape. You write a codemod, run it across the repository, and it cleanly converts 80% of the files — the ones that follow the patterns the codemod's author anticipated. Then you hit the long tail: the test file where someone monkey-patched the renderer, the component that reaches into framework internals, the module written in 2017 by an engineer who has since left, using an idiom nobody else ever adopted. The codemod parses these files fine. It just has no rule that applies. So the last 20% of the migration consumes 80% of the calendar, done by hand, file by file, by engineers who would rather be doing anything else.

Coding agents invert this economics. The hand-written weirdness that defeats a deterministic AST transform is exactly what a model handles well — it reads the file, understands intent, and rewrites it the way a human would, without needing an explicit rule for every variant. Airbnb proved the point at scale: nearly 3,500 Enzyme test files migrated to React Testing Library in six weeks, against an original estimate of 1.5 years of manual work. But here's what gets lost in the headline: the hard part wasn't the prompting. Once you point a fleet of agents at ten thousand files, the engineering problem stops being an AI problem and becomes a batch-operations problem — sharding, verification, quarantine, and merge strategy. The right mental model is a MapReduce job whose mapper is stochastic.

Your design system survived the last decade because humans absorbed it slowly. New engineers learned the button variants through Figma files, PR nitpicks, and the one designer who always caught the wrong shade of gray. That absorption pipeline had a throughput of maybe a few components per engineer per week — slow enough that a design team could police the edges by hand.

Coding agents just broke the pipeline. An agent fleet can generate fifty slightly-wrong buttons before lunch: each one plausible, each one using a hex value that's two shades off, a padding that's 14px instead of your 16px scale step, a border radius invented on the spot. No designer reviews at that speed. And the instruction you reached for first — "follow the style guide" in the system prompt — decays exactly like every other soft instruction: it loses statistical weight as the context window fills, and the model reverts to the generic CSS patterns it learned from a million other people's codebases.

The fix is not a better prompt. It's a category change: your design system has to stop being documentation that humans interpret and become a contract that machines enforce. Documentation asks. Compilers refuse.

Ask a coding agent to build a React component and it writes idiomatic, hook-shaped, accessibility-annotated code on the first try. Ask the same agent to use your in-house ORM — the one your platform team has maintained for six years, the one with excellent docs and a hundred internal consumers — and it hallucinates methods that don't exist, invents configuration options from some other library, and confidently ships code that compiles against an API it made up.

The difference isn't quality. Your ORM might be better-designed than half the open-source libraries the model handles flawlessly. The difference is training data. React has millions of public repositories behind it; your framework has zero. In the vocabulary of natural language processing, your internal framework is a low-resource language — and every consequence NLP researchers documented for low-resource languages now applies to your codebase.

There is a class of AI-generated bug that passes code review almost every time, and it isn't the hallucinated function or the fabricated package. It's the perfectly idiomatic code — idiomatic for the version of your stack that existed when the model's training data was frozen. The model writes a tailwind.config.js for a project running Tailwind v4, reaches for a class component lifecycle method in a hooks codebase, or calls an API that was deprecated three minor versions ago and removed in the one your lockfile actually pins. Nothing about the code looks wrong. It looks like code from a well-regarded tutorial. The tutorial is just from 2024.

Call it the training-cutoff bug class: defects that exist not because the model reasoned badly, but because the model's knowledge of your dependencies has a timestamp and your lockfile doesn't care. An ICSE 2025 study of seven code models across eight popular Python libraries found deprecated-API usage rates of 25–38% in plausible completions — and when the surrounding code already contained outdated patterns, that rate climbed to 70–90%. These aren't rare edge cases. They're the default failure mode of asking a frozen artifact to write for a moving target.

For two decades, documentation was where good intentions went to die. You wrote the README during the first sprint, when the architecture was clean and your enthusiasm was high. Nobody read it. By the third sprint it was lying about the build command, and by the sixth it described a service that had been deleted. Documentation was a tax everyone agreed to pay and nobody actually paid — a moral imperative with no feedback loop. Write bad docs and nothing happened. Write no docs and nothing happened either, because the senior engineers carried the architecture in their heads.

Then we pointed coding agents at our repositories, and the feedback loop arrived overnight. The README is now the single highest-leverage file you own — not because anyone gave a motivational talk about documentation hygiene, but because the quality of that file now visibly determines whether your agent ships correct code or confidently hallucinates an architecture that no longer exists.

This is the documentation renaissance, and it has almost nothing to do with the documentation we used to write.

A coding agent on your team opens a pull request that calls parseUserToken() four times across two files. The function does not exist in the repository, has not existed for nineteen days, and was replaced by decodeSessionClaim() in a commit your engineers all remember reviewing. The agent did not invent the name. It read the name from its semantic index — a vector store rebuilt from a working copy that was twenty-one days behind main. The agent's edit step, by contrast, ran git pull at session start and operated on fresh code. Two views of the same repository, three weeks apart, and the agent confidently bridged them with code that does not compile against anything real.

This is the failure mode that doesn't announce itself. The agent ran. The tests appeared to pass. The PR landed. The first reviewer noticed only because a stubbed-out function shared a name with an unrelated helper and tripped the linter. By then the agent had spent a full sprint writing against a phantom version of the codebase, and no one on the team — including the agent — had any signal that something was wrong.

Your coding agent opened a pull request at 3:14 on a Tuesday afternoon. The PR description was clean, the diff was small, the CI was green. It got squash-merged twenty minutes later. The teammate who came back from lunch at 1:20 the next day saw a notification: "PR #1247 was closed." Not merged. Closed. The branch was gone. The seventy-two review comments she'd left on it the previous week were gone too — collapsed under an "outdated" label on a PR that no longer existed in any active list. A senior engineer's design decisions, two rounds of back-and-forth with the security reviewer, and a careful migration plan that took a week to negotiate, all vanished into a footnote on a different PR that nobody had read closely. The squash commit's only trace of what happened was a one-line tag at the bottom: Closed by #1893.

This is the failure mode of trusting a coding agent to write its own pull request metadata. Not the code — the metadata. The diff was fine. The agent did good work. What it could not do was distinguish a fresh discussion from a stale one, and GitHub's auto-close machinery treats every closing keyword the agent writes as a load-bearing instruction. Your agent reads the comments to gather context, infers from a six-month-old reply that its work supersedes an older PR, writes Closes #1247 in the description it generates, and the merge does the rest — silently, mechanically, irrevocably from the perspective of anyone who wasn't watching the diff at the moment of squash.

The line item climbed 130% over six weeks and nobody on the engineering team noticed. PRs were landing faster. Per-PR CI cost on the dashboard looked the same as last quarter. The agent's branches went green on the first try more often than the humans' branches did, which actually pulled the median CI duration down. Finance found it during quarterly review, flagged it as an unexplained variance, and asked engineering for the postmortem. Engineering had nothing to write — no incident, no regression, no failed deploy. Just a budget line that had quietly doubled while every dashboard reported normal.

That postmortem-shaped hole is the artifact. The cost shifted from a labor-dominant curve to an infrastructure-dominant curve, and the team that owned the labor budget was not the team that owned the infrastructure budget. The agent didn't break anything. It just changed which line on the P&L absorbed the work.