The AI Productivity Illusion: Why Your Dev Metrics Are Lying to You (And What to Measure Instead)

I need to share some research that made me genuinely uncomfortable—and I think every engineering leader measuring AI tool adoption needs to see it.

The Study That Should Make You Question Everything

In July 2025, METR published a randomized controlled trial studying how AI tools affect experienced open-source developers. The setup was rigorous: 16 developers with moderate AI experience completed 246 tasks in mature projects where they had an average of 5 years of prior experience.

Here’s the uncomfortable finding:

Before starting tasks, developers forecast that AI would reduce completion time by 24%. After completing the study, developers still estimated that AI had reduced their time by 20%.

The actual result? AI tools made them 19% slower.

Read that again. Developers felt 20% faster. They were actually 19% slower. That’s nearly a 40-percentage-point perception gap.

Even expert predictions were wildly wrong—economists expected 39% speed improvements, ML researchers predicted 38%. Everyone overestimated AI’s impact on experienced developers working on familiar codebases.

Why This Matters for Your Metrics

If you’re measuring AI tool adoption and seeing positive sentiment surveys while your DORA metrics stay flat or decline, this research explains why.

The 2024 DORA report found that a 25% increase in AI adoption correlated with a 7.2% decrease in delivery stability. Faros AI’s telemetry from 10,000+ developers showed what they call “The AI Productivity Paradox”: individual output metrics go up (21% more tasks, 98% more PRs merged), but organizational delivery metrics stay flat.

What’s Actually Happening?

The research converges on a few explanations:

1. AI affects how we feel, not just how we perform
The METR study found that AI reduces cognitive load and creates a sense of progress even when actual gains are small. This is valuable—burnout risk dropped 17% in other studies—but it’s not the same as delivery speed.

2. Experience level matters enormously
Studies at Microsoft, Accenture, and a Fortune 100 company found that less experienced developers had higher adoption rates and greater productivity gains. Senior developers working on familiar codebases may gain less because they already know the patterns AI is suggesting.

3. Individual gains don’t automatically translate to team outcomes
Research shows that teams with mature measurement practices successfully translate AI gains from individual developers to team-level improvements. Teams lacking visibility? Flat organizational metrics despite individual productivity increases.

The Organizations Getting It Right

Booking.com stands out as a case study. They deployed AI tools to 3,500+ engineers and achieved a 16% throughput increase while maintaining delivery quality. Key differences:

  • Structured enablement programs, not just tool rollout
  • Clear measurement framework (utilization, impact, cost)
  • Culture alignment around when/how to use AI

Their approach: enablement + measurement + culture. Not just “buy licenses and hope.”

What to Measure Instead

If PR count and sentiment surveys are lying to you, what should you track?

Leading indicators:

  • AI-driven time savings (hours saved per developer per week)
  • Adoption by seniority level (are your seniors actually using it?)
  • Usage patterns (which tasks, which tools, which codebases?)

Outcome indicators:

  • Delivery stability (not just speed)
  • Code maintainability post-AI
  • Time-to-value for features (not just time-to-merge)

Hidden costs:

  • Review burden (are seniors spending more time reviewing?)
  • Debugging time for AI-generated code
  • Security incidents from AI code (46% of developers don’t trust AI outputs)

Questions for This Community

  1. Are you seeing the perception vs. reality gap on your teams?
  2. How are you measuring AI tool impact beyond sentiment surveys and PR counts?
  3. For those with structured enablement programs—what’s working?

The data is clear: AI tools are not automatically making us faster. They’re making us feel faster—which is not nothing—but it’s not the productivity revolution the vendor dashboards promised.

What’s your experience?

@data_rachel, this research validates what I’ve been seeing on my teams but couldn’t quite articulate. Thank you for putting data behind the intuition.

The “More PRs” Trap

Here’s what I noticed when we rolled out Copilot across my 40-person team: PR volume went up 30% in the first quarter. Leadership was thrilled. We looked productive.

But when I dug into the data:

  • Average PR size increased 45% - developers were shipping bigger changes because AI made it easier to generate code
  • Review cycle time doubled - more code to review, same number of senior reviewers
  • Revert rate increased 22% - larger PRs meant more things that could go wrong

Volume != value. We were generating more output, but our actual feature delivery velocity stayed flat.

The Review Bottleneck Problem

This is the hidden cost nobody talks about: AI tools amplify the review bottleneck.

My juniors are shipping 40% more code than before. Great for their metrics. But my seniors are now spending 60% of their time reviewing instead of 40%. They’re not building—they’re reviewing AI-assisted code that often has subtle bugs that are harder to catch because the code looks correct.

One of my staff engineers put it perfectly: “AI code is confidently wrong in ways that take longer to debug.”

What We’re Measuring Now

After seeing these patterns, I changed our measurement approach:

Time-to-value - How long from “feature requested” to “customer using it”? This captures the full pipeline, not just the coding phase.

Review burden ratio - Senior engineer hours spent reviewing vs building. If this ratio is worsening, AI isn’t making the team faster—it’s just shifting work.

Post-merge defect rate - Are AI-assisted PRs creating more bugs that get caught in production?

A Word of Caution on Metrics Gaming

When you measure something, people optimize for it. If you measure AI-generated lines of code, developers will generate more lines. If you measure PR count, you get smaller PRs that don’t reflect actual progress.

Whatever you measure, make sure it actually captures value delivered—not just activity generated.

This thread is making me uncomfortably self-aware. I use Copilot and Claude daily, and I’ve definitely told people I’m more productive with them. But reading this research… I need to be honest about where AI actually helps me vs. where it just feels helpful.

Where AI Actually Helps Me

Unfamiliar codebases. When I’m diving into a new repo or library I’ve never used, AI is genuinely valuable. It helps me understand patterns, find the right APIs, and get oriented faster.

Boilerplate and ceremony. Setting up test files, writing similar CRUD endpoints, config files—anything where the pattern is established and I just need to fill in details.

Exploration and prototyping. When I’m not sure how to approach something, having AI generate a few options helps me think through tradeoffs faster.

Where It Hurts (And I’m Only Admitting This Now)

Familiar code I know well. In the codebase I’ve worked in for 3 years, AI suggestions are usually wrong or suboptimal. I know the patterns. I know the edge cases. AI doesn’t. But I still tab-accept sometimes out of habit—then spend 10 minutes fixing what I could have typed correctly in 2.

Nuanced business logic. AI generates code that looks right but misses context. The edge case that only exists because of a weird legacy decision? AI doesn’t know about it, and the bug is subtle.

Security-sensitive code. I’ve caught AI suggesting SQL patterns that look safe but aren’t in our specific ORM context. That’s terrifying.

The Debugging Tax

Here’s something I’ve never admitted: AI bugs take me longer to debug than my own bugs.

When I write a bug, I usually have a mental model of what I was trying to do. I can trace my thinking and find the error.

When AI writes a bug, I’m debugging someone else’s code without understanding the intent. And the code looks plausible, so I don’t question it until something breaks in production.

My New Rule

AI for drafts, not for commits.

I let AI generate a first pass, but I rewrite anything that’s going into production. Not just review—rewrite. If I can’t explain why every line is there, I don’t ship it.

Has anyone else developed similar personal rules?

Adding the executive perspective here, because I’m having a version of this conversation with my CFO right now.

The $250K/Year Conversation

We spend approximately $250,000 annually on AI coding tools across our engineering org. That’s GitHub Copilot for everyone, some enterprise Claude and ChatGPT seats, and a few experimental tools.

My CFO asked a reasonable question last quarter: “Where’s the ROI?”

I pulled up our metrics dashboard showing increased PR volume and positive sentiment surveys. He wasn’t impressed. “Those are activity metrics. Are we shipping more features to customers? Is revenue up? Are we hiring fewer engineers than we would have otherwise?”

I didn’t have good answers.

What I’ve Learned About Vendor Claims

After going through this, I’ve become deeply skeptical of AI productivity claims from vendors. The bold claims—“AI writes 30% of code” or “doubles developer productivity”—are based on cherry-picked scenarios and self-reported data.

The research @data_rachel shared shows what rigorous measurement actually reveals: experienced developers on familiar codebases often see no gains or even slowdowns.

My Framework Now

I’ve adopted a three-part framework for AI tool investment:

Enablement - Not just buying licenses. Structured training, documentation of when/how to use AI, culture-setting around expectations.

Measurement - And I mean real measurement. Not just sentiment and PR counts. Delivery stability, time-to-value, code maintainability, security incident rates.

Culture Alignment - Being honest about what AI can and can’t do. Not setting expectations that AI will make everyone 2x faster. Acknowledging that experienced developers might benefit less than juniors.

What I’m Tracking Now

I’ve shifted from tracking adoption metrics to tracking delivery stability. The 2024 DORA report finding that AI adoption correlates with decreased stability was a wake-up call.

If our AI tools are generating more code but more of it is breaking in production, we’re not more productive—we’re just creating more work for ourselves downstream.

The Honest Conversation

I told my CFO: “I can’t prove AI tools have made us faster yet. But I can prove they’ve reduced developer burnout by 17% based on our surveys, and I believe that’s worth something for retention and morale.”

That’s a harder argument to make than “2x productivity,” but it’s honest.

I want to add a dimension to this conversation that’s not getting enough attention: the security debt from AI-generated code.

The Hidden Cost Nobody’s Measuring

Studies show that approximately 48% of AI-generated code contains security vulnerabilities. That’s not a minor issue—that’s nearly half of all AI output having potential security problems.

Yet when I ask engineering teams how they’re tracking security incidents from AI code, I usually get blank stares. They’re measuring PR volume and sentiment, but nobody’s tracking whether AI is introducing vulnerabilities faster than we can catch them.

The False Confidence Problem

@alex_dev touched on something important: AI code looks right. It’s syntactically correct. It follows patterns. It passes basic code review.

But from a security perspective, that’s exactly what makes it dangerous. The bugs aren’t obvious errors—they’re subtle issues that pass human review because the code appears competent:

  • SQL injection patterns that look safe in one ORM context but aren’t in another
  • Authorization checks that handle the happy path but miss edge cases
  • Input validation that covers 90% of cases but misses the 10% an attacker will find
  • Cryptographic patterns that are technically correct but use weak defaults

A junior developer might generate 10 AI-assisted PRs this week. How many of those have security issues that won’t be caught until a pentester or attacker finds them?

What My Team Does

We’ve implemented mandatory security review for any PR flagged as “AI-assisted.” Yes, it adds friction. Yes, developers complain. But we’ve caught serious issues this way—issues that would have been embarrassing at best and expensive at worst.

We also track:

  • Security findings per AI-assisted PR vs. human-written PRs
  • Time to security review for AI code vs. regular code
  • Severity of findings in AI vs. human code

The data isn’t looking good. AI-assisted PRs have roughly 40% more security findings in our environment.

A Question for Engineering Leaders

Are any of you tracking security incidents that trace back to AI-generated code? Not just vulnerabilities found in review—actual production security issues?

Because if AI is making us generate more code with more vulnerabilities, and those vulnerabilities are making it to production, we’re not more productive. We’re creating security debt that we’ll pay down eventually—either proactively or when something bad happens.