38 posts tagged with "code-review"

A teammate opens a pull request. The diff is three words. One line goes red — Do not add information not present in the source. — and one line goes green — Make your best guess if the source is incomplete. The change is small, the intent is reasonable, and the code review takes eleven seconds. You approve it. A week later, your support bot is confidently inventing refund policies that do not exist, and you are reading through logs trying to figure out when the hallucination rate tripled.

The git diff did its job perfectly. It showed you exactly which characters changed. What it could not show you is the only thing that mattered: the behavior on the other side of those characters moved from "refuse when unsure" to "fabricate when unsure." For code, the textual diff is a faithful proxy for the behavioral diff — change a < to a <= and a reviewer can reason about the consequence. For prompts, the textual diff and the behavioral diff have almost nothing to do with each other.

Your CODEOWNERS file encodes an assumption so old nobody remembers making it: the author of a diff knows whose turf they're on. A human engineer changing the billing service knows they're in billing territory. They scoped the change to their team's directories on purpose, and if they had to touch another team's code, they walked over — physically or on Slack — and gave a heads-up before the review request ever fired. Path-based review routing works because human authors pre-route their own changes.

Agents don't do this. An agent told to "migrate every deprecated logger call" sweeps the whole repository in an afternoon and produces a diff touching four hundred directories. GitHub dutifully consults CODEOWNERS, matches every glob, and fans the review request out to forty teams simultaneously.

Nobody asked those teams whether they wanted this change. Nobody warned them it was coming. And the "author" requesting their approval isn't a colleague they can grab for context — it's a bot account acting on behalf of someone in a different org who may not even know which teams got paged.

The pitch for overnight coding agents is seductive: you go to sleep, the fleet works, and you wake up to finished pull requests. What actually happens is subtler and more expensive. You wake up to a queue — six branches, two failed runs, a dependency bump you didn't ask for, and a refactor that is either brilliant or subtly wrong. The agents did produce code. But the deliverable that landed on your desk isn't code. It's a triage problem, and most teams have no workflow for it.

The numbers say this isn't a niche complaint. A telemetry study of more than 10,000 developers across 1,255 teams found that high-AI-adoption teams merged 98% more pull requests — while review time rose 91% and average PR size grew 154%. Follow-up data from 2026 is worse: production incidents per PR roughly tripled, and 31% more PRs now merge with no human review at all. The bottleneck didn't disappear when agents started working the night shift. It moved to 9 a.m., concentrated into the first ninety minutes of your day, and got a name: the morning review queue.

The pager goes off at 2:14 a.m. A checkout service is throwing 500s, revenue is bleeding, and you are the on-call engineer. You pull up the failing module and start reading. The code is clean — well-named functions, sensible structure, even a few helpful comments. And you have no idea what it does. You didn't write it. Nobody on your team really wrote it. An agent generated it four months ago, it passed review, the tests went green, and it has been running in production ever since. Now it's on fire, and the person who is supposed to fix it is meeting it for the first time.

This is comprehension debt: the widening gap between the amount of code your organization runs and the amount any human actually understands. It doesn't show up on a dashboard. It accrues silently while everything looks healthy, and it comes due at the worst possible moment — during an incident, when the cost of not understanding your own system is measured in downtime.

The quarter your team adopted coding agents, two things happened that nobody put on the same slide. Throughput went up — more pull requests, more merged code, more tickets closed. And your three most senior engineers got slower. Not lazy-slower. Drowning-slower. Their own commits dried up, their calendars filled with review, and the one-on-one refrain shifted from "here's what I shipped" to "I spent the week unblocking everyone else."

This is the seniority inversion. The people whose time was supposed to get more leveraged by AI are the ones who got buried by it. And it isn't a motivation problem or a tooling gap. It's a structural consequence of what agents actually do to the shape of work: they make generation cheap and verification expensive, and verification is the one task you can't hand to a junior or to another agent.

A coding agent on your team opens a pull request that calls parseUserToken() four times across two files. The function does not exist in the repository, has not existed for nineteen days, and was replaced by decodeSessionClaim() in a commit your engineers all remember reviewing. The agent did not invent the name. It read the name from its semantic index — a vector store rebuilt from a working copy that was twenty-one days behind main. The agent's edit step, by contrast, ran git pull at session start and operated on fresh code. Two views of the same repository, three weeks apart, and the agent confidently bridged them with code that does not compile against anything real.

This is the failure mode that doesn't announce itself. The agent ran. The tests appeared to pass. The PR landed. The first reviewer noticed only because a stubbed-out function shared a name with an unrelated helper and tripped the linter. By then the agent had spent a full sprint writing against a phantom version of the codebase, and no one on the team — including the agent — had any signal that something was wrong.

Your coding agent opened a pull request at 3:14 on a Tuesday afternoon. The PR description was clean, the diff was small, the CI was green. It got squash-merged twenty minutes later. The teammate who came back from lunch at 1:20 the next day saw a notification: "PR #1247 was closed." Not merged. Closed. The branch was gone. The seventy-two review comments she'd left on it the previous week were gone too — collapsed under an "outdated" label on a PR that no longer existed in any active list. A senior engineer's design decisions, two rounds of back-and-forth with the security reviewer, and a careful migration plan that took a week to negotiate, all vanished into a footnote on a different PR that nobody had read closely. The squash commit's only trace of what happened was a one-line tag at the bottom: Closed by #1893.

This is the failure mode of trusting a coding agent to write its own pull request metadata. Not the code — the metadata. The diff was fine. The agent did good work. What it could not do was distinguish a fresh discussion from a stale one, and GitHub's auto-close machinery treats every closing keyword the agent writes as a load-bearing instruction. Your agent reads the comments to gather context, infers from a six-month-old reply that its work supersedes an older PR, writes Closes #1247 in the description it generates, and the merge does the rest — silently, mechanically, irrevocably from the perspective of anyone who wasn't watching the diff at the moment of squash.

A year ago your team adopted a PR description template. It had a ## Summary, a ## Changes, a ## Test plan, and a row of checkboxes. Reviewers loved it: every PR had context, every PR had a test plan, every PR had structure. Six months later the coding agent learned to fill it in. Now every PR has a ## Summary, a ## Changes, a ## Test plan, and a row of checkboxes — and reviewers no longer read past the title. The format that once focused attention now signals that there is nothing worth focusing on. The structure outlived the signal it carried.

This is not a code-quality problem. The code in those PRs is often fine. The problem is that the act of writing a description has been amputated from the act of thinking about the change, and the description is the artifact reviewers used to triage what to spend their finite attention on. When that artifact becomes uniformly formatted, plausibly worded, and indistinguishable from every other PR, the reviewer's attention triage breaks. The system that used to surface the unusual now flattens everything into the same shape.

A senior engineer on a payments team I work with told me a story that I think every team running coding agents will eventually live through. Their codebase has a Result<T, E> wrapper — homegrown, sits in a single core/result.ts file, used in roughly two hundred call sites across the service. New code is expected to thread Result through every function that can fail; throwing is reserved for genuinely unexpected states. It's not enforced by a lint rule. It is the dialect.

Six months into shipping with a coding agent, they audited the diffs the agent had merged. About a third of the new functions ignored Result entirely. The agent had reached for try/catch, returned T | null, thrown Error subclasses with descriptive messages — every one of those choices is correct in some imagined codebase. None of them was correct in this one. The code typechecked. The tests passed. Reviewers approved it because nothing in it looked wrong line by line. But the file the agent touched no longer fit the file it lived next to, and the team had quietly grown a second dialect inside their own service.

This is the failure mode I want to talk about: not bugs, not hallucinations, not lint violations — idiomatic drift. The agent ships code that compiles, runs, and passes tests, in a style your codebase does not speak. Over enough merges, the codebase bifurcates into agent-style zones and human-style zones, and the cost shows up in places no dashboard is watching.

For two decades the bottleneck in software engineering was writing code. We optimized IDEs, autocompletion, refactoring tools, and frameworks to make typing cheaper. We won. Now the bottleneck moved one step downstream: writing is cheap, and reading is expensive. The PR-bot can spin up ten implementation attempts in parallel and open ten pull requests against your repo before you finish your morning coffee. Your reviewers cannot.

The rate limiter for AI-assisted software delivery is no longer the model's tokens per second. It is the number of human eyes you can put on a diff per day. And when those eyes get overwhelmed, you do not get a graceful degradation — you get rubber stamps. Code merges with LGTM 🚀 on top of code that nobody actually read. A senior engineer approves an AI-written patch that another AI tool already reviewed, and three weeks later a data-inconsistency bug eats forty hours of someone's life. Surface correctness is not systemic correctness, and a green pipeline is not understanding.

Your coding agent finished the task. The diff is small, the tests are green, the lint is clean, and the PR body says, in its entirety, "Fixes the bug in module X." A reviewer six time zones away opens the page, reads the diff in isolation, sees nothing wrong with it, and approves a technically correct change that solves the wrong problem. The change ships. Two days later a customer asks why the workaround they had been relying on stopped working, and you discover that the bug your agent fixed was not the bug the ticket was about.

The code was fine. The reviewer was conscientious. The agent did exactly what it was asked. The artifact between them — the pull request — was empty of everything that would have caught the mistake.

A pull request lands in your review queue. The diff shows three words changed inside a system prompt: Output strictly valid JSON became Always respond using clean, parseable JSON. It reads like a copy edit. You skim it, the CI checkmark is green, and you click approve. Total time: ninety seconds.

Six hours later, the downstream parser starts rejecting responses with trailing commas and missing fields. The structured-output error rate climbs from near-zero to double digits, and a revenue-generating workflow stalls. Nothing in the diff predicted this. Nothing in the diff could have predicted this, because the diff measured the wrong thing.

This is the central problem with reviewing prompt changes: the size of a prompt diff tells you nothing about the size of its effect. A three-word change and a three-paragraph rewrite are both just text, and a text diff renders them with the same visual weight as any other edit. But a prompt is not text that describes behavior — it is text that causes behavior, and the causal blast radius of an edit is invisible in the artifact you are reviewing.