Developers Think They're 24% Faster With AI, But Studies Show 19% Slower - The Perception Gap Is Our Real Problem

General
1

Last quarter, we ran an AI coding tools pilot that should be a case study in how perception diverges from reality.

We gave half our engineering team access to GitHub Copilot. Classic A/B test. After three months, we surveyed both groups.

The Copilot group reported:

  • 72% said they felt “significantly more productive”
  • Average self-reported speed increase: 24%
  • 89% wanted to keep using it permanently
  • Overwhelmingly positive sentiment

Great, right? Ship it to everyone?

Then we looked at the actual DORA metrics: deployment frequency, lead time for changes, change failure rate, time to restore service.

No statistically significant difference between the two groups.

Zero. After three months of developers feeling 24% faster, the business outcomes were identical.

How is that even possible?

Then I found the research showing developers thought AI sped them up 24%, but studies measured them as actually 19% slower. Our data matched perfectly. Self-reporting is completely unreliable when people are excited about new tools.

Here’s my current theory on what’s happening:

1. We’re measuring velocity, not value

AWS research shows teams with AI complete 21% more tasks and create 98% more pull requests per developer. Sounds amazing until you realize: more PRs doesn’t mean better software.

If AI helps you ship 30% more features but they’re the wrong features, or if they introduce subtle bugs that slow down everything downstream, the net impact is zero or negative.

2. The bottleneck shifted

Our seniors report being overwhelmed with code review. Junior velocity went up, but now everything piles up waiting for experienced engineers to review AI-generated code that’s “almost right but not quite.”

Product managers spend more time clarifying requirements because AI can generate the wrong thing very quickly.

3. We’re measuring the wrong phase

Traditional metrics track coding speed. But AI changes the time distribution. What I’m seeing:

  • Initial coding: 40% faster (feels amazing)
  • Debugging time: 60% longer (feels terrible)
  • Net result: slower overall, but feels faster because writing code is more fun than debugging

What I’m trying to measure instead:

  1. Time from idea to validated learning - not shipped code, but code that proves/disproves a hypothesis about user behavior
  2. Rework rate - how often do we have to redo AI-generated work because it didn’t solve the actual problem
  3. Debugging time ratio - time writing code vs time fixing code
  4. Senior engineer cognitive load - are our most expensive people spending more time reviewing or less?

Early findings: AI tools optimize for the wrong part of the value stream. They make the easy parts faster and the hard parts harder.

The executive problem:

How do I justify AI tool investments when developers love them but business metrics don’t improve? How do I push back when everyone FEELS more productive even if they’re not?

And the bigger question: if we can’t measure AI impact accurately, how do we make any investment decisions around these tools? Developer sentiment? Lines of code? Vibes?

Michelle’s right about skill debt accumulating silently. We won’t measure it until it causes a crisis. But we also can’t measure current impact reliably.

Anyone built a measurement framework that actually works for AI tool ROI?

2

David, this is the conversation I’ve been trying to have with my board for six months.

The measurement crisis is real, and it’s forcing me to make multi-million dollar decisions on incomplete data.

Your finding that DORA metrics showed no improvement despite 72% feeling more productive mirrors what I’m seeing. But here’s what changed for us that traditional metrics don’t capture:

1. Bug severity distribution shifted

Total bug count stayed roughly the same. But the TYPES of bugs changed. Fewer syntax errors and off-by-one mistakes. More subtle logic errors, race conditions, and security vulnerabilities that take longer to debug and have higher business impact.

2. Code review became the bottleneck

Our seniors went from reviewing 8-10 PRs per day to 12-15. Sounds like improvement until you realize review depth decreased. They’re catching fewer issues because volume overwhelmed capacity.

I added a metric: “Issues found in production that passed code review.” That number went up 31% after AI adoption.

3. Onboarding time paradox

Juniors ship their first feature 40% faster with AI. But their 6-month competency assessment scores are 22% lower. They can deliver with AI assistance, but they can’t debug without it.

What I’m measuring now:

  • Time to competent (not time to first commit) - can engineer solve novel problems without AI?
  • Blast radius of failures - are AI-generated bugs more expensive to fix?
  • Review effectiveness rate - issues caught in review vs production
  • Senior engineer retention - are our best people burning out from increased review load?

The competitive pressure David mentioned in that other thread is the knife at my throat. If I slow down to measure properly, we lose market position. If I don’t measure, we might be accumulating risk we can’t see.

Here’s my uncomfortable truth: I’m making AI tool decisions based on strategic bets, not data. Because the data doesn’t exist yet.

3

The bottleneck shifting is EXACTLY what we experienced.

Before AI tools: coding was the constraint, code review flowed smoothly.
After AI tools: coding is fast, code review is drowning.

Our senior engineers (the ones who actually understand the system architecture) went from maybe 60% utilization on reviews to 120%. Not sustainable.

David’s “debugging time ratio” metric is brilliant. We started tracking it last month:

Pre-AI baseline:

  • 65% time writing code
  • 35% time debugging

Post-AI reality:

  • 75% time writing code (feels great!)
  • 45% time debugging (wait, that’s 120% total?)

The math works because the phases overlap now. You’re writing new code while still debugging the last AI-generated feature. Context switching between “create mode” and “fix mode” constantly.

What changed our measurement approach:

We stopped tracking individual developer productivity and started tracking value stream flow.

From business idea to customer value delivery - that’s the only metric that matters. If AI speeds up coding but slows down everything else (requirements clarification, review, debugging, deployment), the total cycle time increases.

Lead time for changes actually went UP by 12% after AI adoption, even though coding time went down by 35%. The other stages in the pipeline absorbed all the gains and then some.

The financial services reality:

In my world, compliance and security reviews are mandatory. AI code generates 23.7% more security vulnerabilities according to research. That means more security review cycles, more rework, more delay.

AI made us faster at creating technical debt, not faster at delivering business value.

I’m running the same unaided learning sprints I mentioned before specifically because I need engineers who can debug complex distributed systems when AI can’t. That skill is now the constraint.

Product’s question about measuring skill degradation before it causes outages - I don’t have a good answer. The first time we’ll know we have a problem is when something breaks and nobody can fix it.

4

The design parallel is wild.

When Figma made iteration cheap, critique became expensive. We went from 5 carefully considered design options to 20 variants generated in minutes. Great, except now design review takes 4x longer because there’s so much more to evaluate.

Sound familiar?

Quantity increased, but decision quality didn’t improve. We shipped faster, but not better.

The hidden cost nobody talks about:

Decision fatigue. When tools make it easy to generate options, the cognitive load shifts from creation to evaluation. That’s actually harder and requires deeper expertise.

Junior designers using AI generate tons of component variants. But when I ask “which one is better for accessibility?” or “which one scales to mobile?” they can’t answer. They can create faster than they can think.

The 40% faster coding / 60% longer debugging split Luis described matches what I see in design: 50% faster mockups, 80% longer critique sessions.

What changed how we measure:

We track “Time to quality decision” not “time to options generated.”

Decision quality metrics:

  • How many iterations before stakeholder approval?
  • How often do we revisit decisions we thought were done?
  • How many design system breaks happen from new components?

Early finding: AI helps us generate more wrong answers faster. The bottleneck is knowing which answer is right.

Michelle’s point about skill debt is exactly what I’m worried about for my junior designers. If they never learn to evaluate design quality because AI makes creation so easy, they’ll never become senior designers who can lead critique.

The tools are optimizing for the wrong skill.

5

“Organizations aren’t ready for the output AI creates” - this should be tattooed on every CTO’s forehead.

CircleCI research showed 59% throughput increase possible with AI, but most companies are “leaving gains on the table” because systems haven’t caught up.

What’s not ready:

  1. Code review process - designed for 10 PRs/day, now handling 20-30
  2. Security scanning - more code = more surface area to scan
  3. Testing pipeline - CI/CD systems overwhelmed by volume
  4. Documentation - AI writes code faster than we can document it
  5. Monitoring - more features = more things to monitor

The productivity gains exist, but they’re bottlenecked by organizational readiness.

Here’s what I’m investing in:

  • Automated code review tools (ironically, more AI to handle AI output)
  • Expanded security team to handle increased vulnerability surface
  • Senior engineer hiring specifically for review capacity
  • Better telemetry and monitoring to catch AI-generated bugs faster

David’s measurement framework is critical: Time from idea to validated learning. Not shipped code, but validated learning.

If AI helps us learn we’re building the wrong thing faster, that’s actually valuable. If it helps us build the wrong thing faster without learning, that’s expensive.

Luis, your point about this being a marathon bet is exactly right. I’m defending slower short-term velocity to build long-term capability. But I need 18-24 months before I can prove I was right.

The risk: in 18 months, if I’m wrong, I’ve lost competitive position AND I don’t have the skills to catch up.