Performance Reviews Measure the Fleet, Not the Engineer

Your next calibration meeting has a measurement problem nobody wants to name. The packet in front of the committee says one engineer shipped 340 PRs this half and another shipped 90. Five years ago that gap meant something. Today it mostly tells you who has the better agent harness, the more permissive review culture on their team, and the higher tolerance for merging generated code. The number on the slide measures the fleet. The committee is supposed to be evaluating the human.

This isn't a hypothetical drift that will arrive someday. Industry analyses estimate AI now writes on the order of 30% of committed code at companies that have adopted agents seriously, and a longitudinal study of 300 engineers found teams generating 98% more pull requests after agent adoption. Every output metric your review process inherited from the pre-agent era — lines changed, PRs merged, story points, velocity — is now a blended measurement of human judgment and machine throughput, with no attribution boundary between them. Calibration committees are comparing blended numbers as if they were still measuring people.

Output Metrics Became Tool-Adoption Metrics

The old metrics were never great, but they were at least correlated with something real. An engineer who merged twice as many PRs as their peer was usually doing more work, or at least more visible work. That correlation is what agents severed.

Consider what each classic signal actually measures now:

• Lines of code and diff size measure how much the engineer let the agent write. GitClear's analysis of 211 million changed lines found copy-paste-style duplication rising sharply as AI adoption grew — bigger diffs increasingly signal less curation, not more effort.
• PR count and merge velocity measure harness quality. An engineer who has invested in good agent scaffolding — task templates, eval loops, auto-fixing CI — can 5x their PR count in a quarter without their judgment improving at all.
• Commit frequency and activity graphs measure how often the fleet runs, which is a budget line, not a performance signal.

The deeper problem is that these metrics don't just become noisy — they become adversarially noisy. Once engineers know throughput is agent-inflatable and reviews still reward throughput, the rational move is to tune the harness for volume. You get exactly what you measure: more PRs, bigger diffs, and a review queue that grows 91% while task completion grows 21%, which is what the 300-engineer study observed. The bottleneck moves to the humans reviewing the flood, and the people creating the flood get the promotion packet.

There's a second-order distortion worth naming: self-assessment broke too. METR's randomized study of experienced open-source developers found they believed AI made them 20% faster while it actually made them 19% slower on those tasks. If your review process leans on self-reported impact — and most do — you're now aggregating perception errors that run 40 percentage points in the wrong direction. Neither the metrics nor the narratives are trustworthy on their own.

What's Left That Actually Measures the Human

Strip away everything the fleet can inflate and a short list survives. It's a good list, because it's the list that always mattered — the agent era just forces you to stop pretending the proxies were the substance.

Problem selection. Which problems did this engineer choose to spend fleet capacity on? Agents make execution cheap, which makes direction the scarce input. An engineer who pointed their agents at the retention-killing latency issue created more value than one who generated forty refactoring PRs nobody asked for. This is observable: look at what they chose, not how much came out.

Review judgment on agent output. The engineer's real fingerprint is no longer in the diff — it's in what they rejected, what they caught, and what they insisted on before merging. Qodo's survey of practitioners found the top pain with AI code isn't syntax errors, it's plausible-but-wrong logic that sails through inattentive review. The engineers who catch those are doing the load-bearing work. Your tooling can surface this: comments on agent PRs, rejection rates, post-merge defect rates on code they approved.

Spec and eval quality. Agents are a compiler for intent. Garbage specs compile into garbage systems, quickly and at volume. The engineer who writes the task definition tight enough that the agent's first attempt is usually right, and who builds the eval that catches the failure mode before production does, is the one multiplying the whole team. This artifact trail — specs, eval suites, harness configs — is reviewable in a way "wrote lots of code" never was.

Durability of what shipped. Change failure rate and post-merge churn survive the agent era almost intact, because they measure consequences rather than activity. GitClear found the share of code revised within two weeks of landing climbed steadily alongside AI adoption. An engineer whose merged work stays merged is telling you their judgment gate works. One whose output churns at 1.5x the baseline is shipping the agent's first draft.

None of these are as cheap to collect as a PR count. That's the point. The cheap signals are exactly the ones that stopped meaning anything.

Loading…