30-79% Faster Development Cycles Reported — But Are We Measuring Speed or Outcomes?

General
1

Every AI vendor is publishing productivity studies now. The numbers are impressive:

  • 30-79% faster development cycles (Web and Crafts, 2026)
  • 26% productivity improvement for experienced developers (IT Revolution)
  • Teams merging 98% more PRs (Faros AI)

These stats are being used to justify AI tool investments across the industry. But as someone who has spent 12 years thinking about product-market fit and how we measure success, I have to ask: are these the right metrics?

The Speed Trap

Speed metrics are seductive because they’re easy to measure. Lines of code written, PRs merged, tickets closed, features shipped. These all feel like progress.

But the metrics I care about as a product leader:

  • Customer retention after feature launch
  • Support ticket volume related to new features
  • Time to value for end users
  • Revenue impact per engineering hour

Nobody is publishing studies on these.

The Hidden Assumption

Most AI productivity studies measure: How fast can developers complete tasks?

But they’re not measuring: Are developers completing the right tasks? Are the completed tasks actually working correctly in production? What’s the maintenance burden 6 months later?

The Faros AI study showing 98% more PRs merged is fascinating, but it also notes PR review time increased by 91%. So we’re shipping more code, but reviewers are spending nearly double the time checking it.

A Framework for Real Measurement

If I were designing an AI productivity measurement system, I’d want to track:

Speed (what everyone measures)

  • Time to first PR
  • Cycle time
  • Deployment frequency

Quality (harder to measure, often ignored)

  • Post-deployment bug rate
  • Rollback frequency
  • Production incident rate

Outcomes (what actually matters)

  • Feature adoption rate
  • Customer satisfaction change
  • Time to value for users
  • Revenue per engineering hour

Sustainability (long-term)

  • Codebase maintainability scores
  • Onboarding time for new team members
  • Technical debt accumulation

The Inconvenient Question

What if AI tools are making us faster at building the wrong things?

What if the 30-79% speed improvement is being offset by increased rework, higher maintenance costs, and features that don’t drive user value?

I’m not saying AI isn’t valuable — but I am saying we need more rigorous measurement before claiming victory.

What metrics are your teams actually using to evaluate AI tool impact? Are you seeing outcomes improve, or just velocity?

2

David, this hits close to home. I’ve been on multiple calls with our CFO where engineering leadership presents AI productivity gains using exactly the metrics you’re questioning.

The finance perspective is brutal: we don’t care about PRs merged. We care about:

  • Revenue per engineering FTE
  • Customer acquisition cost changes
  • Support cost per feature
  • Time to break-even on engineering investments

When I look at our AI tool spending vs. measurable business outcomes, the picture is murky at best. We’ve spent $400K on AI tooling this year. Can I point to $400K in value created? Not with confidence.

The 91% review time increase is the scariest stat in this post.

Here’s why: at Series B stage, our most expensive resource is senior engineer time. If AI tools are generating more code that requires more senior review time, we might be net-negative on engineering capacity for senior folks while net-positive for juniors. That’s not obviously good.

What I’d add to your framework: Cost metrics.

  • AI tool spend per FTE
  • Review time cost (senior hours × rate)
  • Rework and bug-fix cost
  • Total cost of ownership for AI-generated features

The reason these vendor studies don’t measure outcomes is obvious: outcomes take time to materialize and are hard to attribute. It’s much easier to say “we saved 40% of coding time” than to prove “this feature drove $2M in revenue and the AI tools reduced time-to-market by 6 weeks which captured $X more market share.”

The CFO in me wants to defer 25% of our AI investment to 2027 until we have better measurement frameworks. I think a lot of finance teams are thinking the same way.

3

I’ll push back a little here because I think there’s a risk of analysis paralysis.

Yes, we should measure outcomes. Yes, the vendor studies are biased toward metrics that make their products look good. But:

We’ve always had bad productivity metrics in software.

Before AI tools, we measured lines of code, story points, velocity, deployment frequency. All of these were imperfect proxies for value. The DORA metrics revolution was explicitly about trying to connect developer activities to outcomes, and even those have well-documented limitations.

The question isn’t “are AI productivity metrics perfect?” The question is “are they worse than what we had before?”

My lived experience: I’ve shipped three major features in the past quarter using AI heavily. Two of them are seeing good adoption (measured by user engagement), one is struggling. That’s about my normal hit rate from before AI tools.

The difference: I shipped them faster, which meant we had more time to iterate based on user feedback. Speed does create option value.

Where I agree with David though: The maintenance question is genuinely unknown. I’m 6 months into heavy AI usage. Will the code I shipped hold up? I genuinely don’t know yet. We won’t know for another year.

A practical suggestion: Instead of trying to build comprehensive measurement frameworks upfront (which will take forever), what if we just tracked a simple ratio?

Value Ratio = (Features Adopted by Users) / (Features Shipped)

If AI tools increase shipping velocity but this ratio drops significantly, that’s a signal. If the ratio stays constant or improves, the speed gains are real.

It’s not perfect, but it’s measurable and it focuses on what actually matters: are we building things users want?

4

Coming from mobile development, I want to add a nuance that often gets lost in these productivity discussions: different domains have different measurement realities.

In mobile, our feedback loops are brutal and fast:

  • App Store ratings update in real-time
  • Crash rates are immediately visible
  • User retention at day 1, 7, 30 is measured automatically
  • Performance regressions show up in our monitoring within hours

So when I use AI tools to ship faster, I know pretty quickly if that speed came at a cost. Last month I shipped a feature 3x faster than I would have manually. Within 48 hours, I knew:

  • Crash rate: unchanged
  • 1-day retention on the feature: 72% (above our threshold)
  • ANR rate: actually improved (AI suggested better async patterns)

For mobile, the measurement problem David describes is less acute because the platform forces feedback.

But I also work with backend teams where the feedback loops are much longer. A microservice that “works” in production might be accumulating latency debt, memory leaks, or subtle correctness bugs that don’t surface for months.

My suggestion: domain-specific measurement frameworks.

Mobile teams should track:

  • Crash-free session rate
  • ANR/freeze rate
  • App start time
  • Feature-specific conversion funnels

Backend teams need different metrics:

  • P99 latency trends
  • Error rates over time
  • Resource utilization patterns
  • Time to debug production issues

Web teams, data teams, infra teams — all different.

The mistake is trying to create a universal “AI productivity” metric. The right question is: what already tells us if our domain is healthy, and is that getting better or worse with AI tools?