40% Increase in Coding Efficiency, 25% Reduction in Bugs With AI Tools—But Teams Report No DORA Improvement. Where's the Disconnect?

General
1

I just walked out of an executive meeting where our CFO challenged our entire AI tooling investment. The data he presented was stark: we’ve rolled out AI coding assistants to 85% of engineering, developers self-report 40% efficiency gains and 25% fewer bugs, but our DORA metrics are flat—or worse. Deployment frequency hasn’t budged. Lead time is up 8%. And our change failure rate actually increased 7.2% since AI adoption began.

He asked point-blank: “If developers are so much more productive, why isn’t it showing up in our delivery metrics?”

I didn’t have a good answer.

The Individual vs Team Paradox

Here’s what we’re seeing at the individual level:

  • Developers complete 21% more tasks per sprint
  • Pull requests merged per developer up 98%
  • Self-reported satisfaction with coding tools at all-time high
  • Time saved on boilerplate, tests, and refactoring is real and measurable

But at the team and organizational level:

  • Delivery stability down 7.2% according to our DORA tracking
  • PR review times up 91% creating a massive bottleneck
  • Average PR size increased 154% making reviews harder
  • Bugs per developer up 9% despite individual claims of fewer bugs

The data doesn’t lie, but it also doesn’t make sense.

Where Is the Disconnect?

I’ve been digging into the research, and the pattern is consistent across the industry. Multiple studies show that while individual developers feel more productive, companies aren’t seeing measurable improvement in delivery velocity or business outcomes when you aggregate the data.

Some hypotheses I’m exploring:

1. We’re measuring the wrong things. DORA metrics reflect team capabilities, not individual code generation speed. Maybe we need new metrics for the AI era that capture creative problem-solving vs code volume.

2. The bottleneck shifted. We optimized one part of the assembly line (code writing) while leaving others untouched (review, testing, deployment). Now we have a massive pile-up at the review stage—senior engineers drowning in PRs they can’t approve fast enough.

3. Quality is degrading invisibly. Research shows AI-assisted code has 23.7% more security vulnerabilities. We’re moving faster, but are we building sustainable systems or creating tomorrow’s technical debt?

4. AI amplifies existing dysfunction. Teams with strong processes see gains. Teams with weak organizational capabilities see more chaos. The research suggests AI acts as an “amplifier”—magnifying whatever you already have, good or bad.

The CFO’s Real Question

What he’s really asking is: “Are we confusing activity with impact?”

29-41% of our code is now AI-generated. Developer productivity is up only 3.6%. That’s a massive efficiency gap. Where did the productivity go?

More code churned doesn’t automatically mean more customer value delivered. It might just mean we’re busy—writing more code, reviewing more code, debugging more code—without actually solving more problems or shipping more features.

How Are You Explaining This?

I need to go back to the exec team with a better answer. For those of you facing similar scrutiny:

  • How are you measuring AI impact beyond self-reported developer satisfaction?
  • Have you seen DORA improvements, or are you experiencing the same paradox?
  • What organizational changes did you make to capture individual productivity gains at the team level?
  • How do you communicate this complexity to finance and business leaders who just want to see ROI?

I’m open to the possibility that AI tools aren’t delivering on the productivity promise—or that we’re implementing them wrong. But I also suspect we’re dealing with a measurement and organizational design problem, not just a tooling problem.

What’s your experience?

2

Michelle, this hits close to home. We’re experiencing the exact same pattern in my fintech org.

The Review Bottleneck Is Real

Your hypothesis #2 about shifted bottlenecks is what we’re living through right now. Since rolling out AI assistants six months ago:

  • PR volume per developer: +154% (almost exactly your number)
  • Average PR review time: 3.2 days → 6.1 days
  • Senior engineers spending 60% of their week on reviews (up from 35%)

We essentially turbocharged code generation without upgrading our review capacity. It’s like adding more lanes to a highway but keeping the same number of toll booths—you just create a bigger traffic jam.

What We Tried (With Mixed Results)

Attempt #1: AI code review tools

We piloted GitHub Copilot for Pull Requests and CodeRabbit. The promise was to automate the easy stuff so humans could focus on architecture and business logic.

Reality: 66% of AI review suggestions were “almost right, but not quite.” Senior engineers ended up spending MORE time reviewing both the original PR and the AI’s review comments. We actually made the problem worse.

Attempt #2: Rebalance sprint work

We looked at where AI showed the biggest gains: tests (90% time savings), refactoring (75% savings), boilerplate (80% savings).

So we shifted our sprint mix—more test coverage work, more technical debt paydown, less greenfield feature development. The logic was: let AI do what it’s good at, reduce the review burden by keeping PRs focused.

Result: DORA metrics improved slightly (deployment frequency +12%, lead time -8%), but feature velocity dropped 15%. Product wasn’t happy.

The Uncomfortable Truth

Here’s what I think is happening: AI isn’t making us more productive at solving hard problems. It’s just making us faster at writing code.

The creative work—understanding requirements, system design, debugging complex interactions—still takes the same amount of time. But now we’re generating 2-3x more code to review, test, and maintain.

It’s like having a really fast typist who still needs to think about what to write.

The Question I’m Wrestling With

Should we be measuring “velocity” differently in the AI era?

Traditional DORA assumes code writing is the constraint. But if code generation is now near-free, maybe we should be tracking:

  • Problem-solving throughput: Features shipped, not PRs merged
  • Architecture decisions per week: The creative work that AI can’t do
  • Review capacity utilization: Are we maxing out our senior engineers?
  • Code maintainability over time: Is AI-generated code creating future burden?

I don’t have answers yet, but I’m increasingly convinced that optimizing for “more code faster” is the wrong goal when code writing isn’t the bottleneck anymore.

Michelle, how is your team thinking about adjusting sprint planning to account for this new reality?

3

Coming at this from the product side, and I think we’re making a classic measurement mistake: we’re tracking outputs (code, PRs, tasks) instead of outcomes (customer value, features shipped, business impact).

The Assembly Line Analogy Is Perfect

Luis nailed it with the highway metaphor. Let me extend it with a factory analogy:

Imagine you speed up one machine on an assembly line—say, the widget stamping machine—from 100 units/hour to 140 units/hour. What happens?

You don’t get 40% more finished products. You get a massive pile of half-finished widgets sitting in front of the next bottleneck (quality control, packaging, whatever).

The factory looks busier. Everyone’s working harder. But output hasn’t changed.

That’s what we’re seeing with AI coding tools.

Our Real Numbers

Michelle, here’s what happened in my org over the last 6 months:

  • Code commits: +42%
  • PRs merged: +98%
  • Lines of code in production: +67%

Sounds great, right?

But when we looked at actual product outcomes:

  • Features shipped to customers: +2% (basically flat)
  • Customer-facing improvements: -8% (yes, DOWN)
  • Sprint goals completed: 73% → 71% (slightly worse)

So where did all that code go? We generated a ton of activity, but not impact.

The Busy Work Problem

After digging into specific PRs, we found patterns:

  1. Over-engineering: Developers using AI to generate comprehensive error handling, logging, edge cases for simple features. Good in theory, but adds review burden and complexity for minimal value.

  2. Code churn: AI makes it easy to refactor, so developers refactor more often. 3 different implementations of the same feature in the same sprint because “AI suggested a better way.”

  3. Scope creep: When code is easy to generate, developers expand scope. “While I’m at it, let me also add…” Now PRs are bigger, reviews take longer, and we’re building features we didn’t need.

  4. Test proliferation: AI generates tons of tests. Sounds good, but 40% of them test trivial paths or duplicate coverage. More maintenance burden, not more quality.

What Actually Matters

Here’s my controversial take: Individual developer “productivity” is a vanity metric.

What we should be tracking:

  • Customer value delivered (features that move business metrics)
  • Time from idea to customer (end-to-end cycle time, not just code)
  • Team throughput (whole team delivering outcomes, not individual task completion)
  • Customer satisfaction and adoption (did the feature actually work?)

In our case, developers feel productive because they’re generating more code. But the product team sees slower feature delivery because all that extra code creates review, testing, deployment, and support overhead.

The Hard Question

Michelle asked: “Are we confusing activity with impact?”

Yes. Absolutely yes.

AI tools optimize for code generation. But code generation was never the constraint for delivering customer value. The constraints are:

  • Understanding what to build (discovery, requirements)
  • Making good design decisions (architecture, tradeoffs)
  • Coordinating across teams (alignment, dependencies)
  • Validating with customers (testing, feedback, iteration)

AI doesn’t help with any of that. It just makes the easy part (typing code) even easier.

My Advice to CFOs

When finance asks about AI ROI, I now show them:

  1. Business outcomes (revenue, retention, NPS) – has AI moved these?
  2. Feature velocity (customer-facing improvements per quarter) – did we ship more value?
  3. Team satisfaction (survey results) – are engineers happier and learning?

If those three are positive, AI is working. If not, we’re just generating more code, not more value.

Michelle, have you tried reframing the CFO conversation around business outcomes rather than engineering metrics?

4

Okay, I’m going to add a perspective that might be uncomfortable: we’re trading speed today for technical debt tomorrow.

What I’m Seeing From Design Systems

I lead design systems, so I see patterns across multiple product teams. Since AI tools rolled out:

The good:

  • Developers implement designs faster
  • Component variations get built quickly
  • UI bugs get fixed in minutes instead of hours

The concerning:

  • Design system adoption dropped 23% because it’s faster to ask AI to generate a similar component than to learn our system
  • Component duplication is through the roof – 7 different button variants that should all be using <Button> from our library
  • Visual consistency is degrading – each AI-generated variation is slightly different (padding, colors, spacing)

We built a design system to create consistency and reduce maintenance burden. AI is undermining both goals because it rewards short-term speed over long-term architecture.

The Architectural Debt Problem

Michelle mentioned research showing 23.7% more security vulnerabilities in AI-assisted code. I’m seeing the same pattern with architectural debt.

AI is great at generating code that works right now. It’s terrible at generating code that will be maintainable in 2 years.

Examples I’ve encountered:

  1. Hardcoded values everywhere – AI doesn’t know about our config system, so it inlines magic numbers and strings that should be constants

  2. Inconsistent patterns – Team A uses one approach, Team B uses another, because AI suggested different solutions to the same problem

  3. Over-abstraction – AI loves generating helper functions and wrapper classes. Now we have 4 layers of abstraction for what should be a direct call.

  4. Missing context – AI-generated code often lacks comments explaining WHY decisions were made, making future changes risky

All of this creates maintenance burden that doesn’t show up in DORA metrics—until it does, 6-12 months later when velocity suddenly drops because the codebase has become a tangled mess.

The Junior Developer Ceiling

Here’s what really worries me: AI is preventing skill development.

I’m seeing junior developers hit a plateau around 18 months. They can generate code with AI tools day one, which feels productive. But they’re not learning foundational skills—debugging, system design, reading code, understanding tradeoffs.

Then they hit a problem AI can’t solve (complex bug, architectural decision, performance optimization) and they’re stuck. They can’t debug code they don’t understand because AI wrote most of it.

Research backs this up: developers using AI scored 17% lower on knowledge assessments despite having access to tools that could generate correct code.

We’re creating a generation of developers who are fast typists but poor problem-solvers.

The Quality vs Speed Tradeoff

David’s point about busy work resonates. But I’d frame it differently: AI optimizes for local speed at the cost of global quality.

  • Fast to generate code → Slow to review thoroughly
  • Fast to implement feature → Slow to maintain over time
  • Fast to fix bug → Slow to understand root cause
  • Fast to ship → Slow to operate in production

Your CFO is asking the right question: where did the productivity go?

My hypothesis: it went into technical debt we haven’t acknowledged yet.

  • More code to maintain
  • More inconsistency to reconcile
  • More bugs to fix downstream
  • More complexity to understand

DORA metrics measure short-term throughput. They don’t measure long-term sustainability.

What I Think We Should Track

If we’re going to use AI tools seriously, we need to start measuring:

  1. Code maintainability over time – How many “wtf” moments per PR review?
  2. Architecture consistency – Are we building coherent systems or patchwork solutions?
  3. Junior developer skill progression – Are people learning or just using tools?
  4. Technical debt accumulation rate – How fast is complexity growing?

Michelle, I don’t have an answer for your CFO yet. But I think the productivity went into creating systems that are harder to maintain, not easier. And that bill is coming due.

Anyone else seeing patterns like this?

5

Michelle, I think you’re asking the wrong question. It’s not “why aren’t DORA metrics improving despite AI adoption?”

The real question is: “What organizational dysfunction is AI exposing?”

AI Is an Amplifier, Not a Solution

Luis, David, and Maya are all describing the same phenomenon from different angles. Let me tie it together with a framework that’s been helpful for me:

AI doesn’t fix broken systems. It amplifies what you already have.

If you have strong code review culture → AI helps teams move faster
If you have weak review culture → AI creates chaos

If you have clear architecture standards → AI generates consistent code
If you have inconsistent patterns → AI makes it worse

If you have good testing practices → AI accelerates quality
If you have poor testing → AI ships bugs faster

Real Example From My EdTech Company

We have two product teams. Same AI tools, same training, same resources. Wildly different outcomes.

Team A (High-performing before AI):

  • DORA metrics improved 15% after AI adoption
  • Deployment frequency up, lead time down, stability maintained
  • Developers love the tools, productivity gains are real

Team B (Struggling before AI):

  • DORA metrics degraded 12% after AI adoption
  • More code, longer reviews, more bugs, slower delivery
  • Developers frustrated, feeling underwater despite “productivity” tools

What’s the difference? Team A had solved their organizational constraints first.

  • Clear ownership and accountability
  • Strong architecture review process
  • Disciplined testing and deployment practices
  • Senior engineers actively coaching juniors
  • Product-engineering alignment on priorities

Team B had none of that. AI just accelerated their existing problems.

The Downstream Bottleneck Problem

Michelle, you said PR review times are up 91%. That’s the symptom, not the disease.

The disease is: your delivery pipeline has bottlenecks AI can’t solve.

Here’s what I’d audit:

  1. Review capacity: Do you have enough senior engineers to review increased PR volume? If not, AI is just creating a bigger backlog.

  2. Testing infrastructure: Can your CI/CD system handle 98% more PRs? Or is it now a 4-hour queue that kills productivity?

  3. Deployment process: Is shipping to production still manual and scary? AI generates code fast, but if deployment is slow, you’re stuck.

  4. Support and operations: Is your on-call team drowning in issues from rushed AI-generated code? That creates fear of deployment, which kills velocity.

  5. Product prioritization: Are you building the right things, or just building more things? AI makes it easy to say yes to everything, which overloads the system.

  6. Knowledge transfer: Are senior engineers too busy reviewing to mentor? That creates the junior dev skill gap Maya described.

AI accelerated code generation. That exposed every other constraint in your delivery system.

What I Actually Did

When I saw Team B struggling, here’s what we fixed (before expecting DORA improvements):

1. Invested in review capacity

  • Added 2 senior engineers to Team B
  • Created review rotations so no one person was the bottleneck
  • Set SLAs for review time (24 hours max)

2. Strengthened architecture governance

  • Required design reviews for PRs over 200 lines (AI tends to generate big PRs)
  • Created clear decision records (ADRs) so AI couldn’t contradict established patterns
  • Paired junior devs with seniors for AI-assisted work

3. Upgraded testing and deployment

  • Automated visual regression tests (Maya’s design consistency problem)
  • Added architectural linting (catch pattern violations before review)
  • Parallelized CI/CD to handle higher volume

4. Reframed success metrics

  • Stopped celebrating “PRs merged” and started tracking “customer value delivered”
  • Added technical debt metrics to sprint reviews
  • Measured learning (skill assessments) alongside productivity

Result: After 3 months, Team B’s DORA metrics started improving. Not because AI suddenly worked better—because we fixed the organizational capacity to absorb AI-generated volume.

Michelle’s CFO Question

Here’s how I’d answer your CFO:

“AI tools have delivered exactly what they promised: faster code generation. The problem is, code generation was only 20% of our delivery process. The other 80%—review, testing, deployment, operations, coordination—hasn’t changed. So we’re seeing 40% improvement on 20% of the work, which nets to ~8% theoretical improvement. But because we didn’t upgrade our capacity in those other areas, we’re actually seeing degradation as bottlenecks get overwhelmed.”

“The ROI will come when we invest in organizational capacity to match the AI code volume. That means more senior engineers, better testing infrastructure, and clearer architecture governance. We need to upgrade the whole system, not just the code generation step.”

“Short version: AI gave us a faster engine, but our transmission, brakes, and steering can’t handle the power yet. We need to upgrade the whole car.”

The Uncomfortable Truth

David’s right: individual developer productivity is a vanity metric.

What matters is organizational throughput—how fast can the entire system (product, engineering, design, operations, support) deliver customer value?

AI optimizes one person’s local output. But if that creates bottlenecks downstream, system throughput goes down, not up.

Michelle, before you invest more in AI tools, I’d recommend:

  1. Map your entire delivery pipeline end-to-end
  2. Identify where AI-generated code gets stuck (review, testing, deployment, etc.)
  3. Measure capacity at each stage
  4. Invest in upgrading bottlenecks before expecting DORA improvements

AI isn’t the problem. Your organizational capacity to absorb AI volume is the problem.

Fix the system, then AI becomes a force multiplier. Keep the system broken, and AI just amplifies the chaos.