We Ship 31% Faster With AI Tools, But My Team Spends 15 Hours/Week Verifying the Output. What's the Real Productivity Number?

General
1

Six months ago, I greenlit GitHub Copilot and similar AI coding tools for my entire engineering organization. The productivity numbers looked incredible on paper: 55% faster task completion, 3.6 hours saved per developer per week. My team was shipping features at a pace we’d never seen before.

Last week, I ran a different analysis. I tracked how much time my engineers spend reviewing, testing, and correcting AI-generated code. The number? 15 hours per week per developer.

We’re shipping 31% faster, but we’re spending nearly double that time second-guessing every line.

The Trust Gap Is Real

Here’s what I’m seeing across my teams:

84% adoption, but only 29% trust. Every single one of my engineers uses AI tools daily. But in our retrospectives, when I ask “do you trust the code it generates?” — the room goes quiet. They use it because it’s fast. They trust it… conditionally. With verification. After review.

The verification burden is crushing. One of my senior engineers told me: “Reviewing AI code takes more cognitive effort than reviewing code from our junior developers. At least I understand how a junior thinks. AI surprises me in ways I can’t predict.”

That hit hard. 38% of developers report that reviewing AI-generated code requires more effort than reviewing human-written code. We’re not just talking about a quick glance — this is deep, careful review work.

The Productivity Paradox

Here’s the math that keeps me up at night:

  • Time saved writing code: 3.6 hours/week
  • Time spent verifying code: 15 hours/week
  • Net productivity change: -11.4 hours/week

And yet, we’re shipping faster. How does that make sense?

The only explanation: we’re skipping verification. The research backs this up — only 48% of developers always check their AI-assisted code before committing it. That means 52% of AI-generated code enters our codebase with incomplete review.

That scares me. We’re in EdTech. Our code serves millions of students. The idea that we’re shipping faster because we’re verifying less — that’s not a productivity win. That’s technical debt with a timer.

The Perception Gap

What really bothers me is the perception-reality gap. A study by METR found that developers using AI tools were actually 19% slower than those coding without assistance, despite believing they were 20% faster.

Think about that. We feel productive. The dopamine hit of watching code appear on screen is real. But if you measure end-to-end delivery time, including review, bug fixes, and rework — we’re slower.

Are we lying to ourselves about the gains?

The Team Morale Impact

This trust gap is affecting team dynamics in ways I didn’t anticipate:

Junior engineers feel lost. They’re using AI to write code they don’t fully understand. When bugs appear, they don’t have the foundational knowledge to debug. One junior dev told me: “I feel like I’m becoming a code reviewer, not a code writer.”

Senior engineers feel overwhelmed. They’re stuck being the “AI validators.” Every PR now comes with an unspoken question: “Did a human write this, or do I need to check every edge case?”

Code review culture is changing. We used to review for design decisions. Now we review for correctness. That’s a regression.

What I’m Trying

I don’t have answers yet, but here’s what we’re experimenting with:

  1. Tiered trust model: Not all AI suggestions are equal. Boilerplate? Ship it. Complex business logic? Mandatory human review.

  2. Verification time as a metric: I’m tracking review time alongside shipping velocity. If verification time grows faster than code output, we’re regressing.

  3. AI literacy training: Teaching engineers how to effectively prompt, review, and validate AI outputs. It’s a skill, not a toggle.

  4. Honest retrospectives: Creating space for engineers to admit when AI hurt their productivity, not just when it helped.

My Question to This Community

How are you measuring net productivity when AI tools are in the mix?

Are you tracking verification time? Review cycles? Bug rates on AI-assisted code?

Or are we all just looking at the “time saved writing code” metric and calling it a win?

I want to believe the 31% productivity gain is real. But right now, it feels like we’re trading speed for trust — and I’m not sure that’s a trade worth making.

What am I missing?

2

@vp_eng_keisha This hits close to home. We run into the same paradox in financial services, but our compliance requirements force us to be more rigorous about measurement.

The perception gap you mentioned is backed by hard data. That METR study is just the tip of the iceberg. When researchers actually measured end-to-end delivery time (not just “time to first code”), developers with AI assistance were 19% slower while believing they were 20% faster.

That’s a 39-point perception gap. We’re not even close to understanding our own productivity.

What We’re Measuring

In my org, we can’t afford verification gaps. A bug in our payment processing system isn’t just technical debt — it’s regulatory exposure. So we implemented mandatory metrics:

  1. Code churn rate: How often do AI-assisted PRs get revised after merge?
  2. Bug origin tracking: We tag every bug with “AI-assisted” or “human-written” in our retrospectives
  3. Review time per line of code: Turns out AI-generated code takes 2.3x longer to review per LOC
  4. Time to production: Not time to PR, but time until code is live and stable

The results? Net productivity gain is closer to 8-12%, not 31%. And that’s only for certain types of work.

The Financial Services Reality

Here’s what works and what doesn’t in our environment:

Where AI wins:

  • Data transformation scripts
  • Test case generation
  • Boilerplate API endpoints
  • Documentation generation

Where AI fails:

  • Complex business logic (our domain is too specific)
  • Security-critical code (too much risk)
  • Edge case handling (AI optimizes for the happy path)
  • Anything touching PII or financial data (compliance won’t allow it)

The uncomfortable truth: We’re using AI for maybe 30% of our codebase. The “low-trust” 70% still requires traditional development.

Are We Measuring the Wrong Things?

Your question about what to measure really resonates. I think the industry is measuring activity (lines of code, PRs merged, velocity points) when we should be measuring outcomes:

  • Customer value delivered: Did this code solve a real problem?
  • Stability: Production incidents per 1000 lines of code
  • Maintainability: How long does it take someone else to understand this code?
  • Team learning: Are engineers growing their skills, or just becoming better AI prompt writers?

That last one keeps me up at night. If junior engineers never struggle through complex implementations, how do they develop the expertise to review those implementations when AI generates them?

My Hypothesis

I think we’re in the “peak of inflated expectations” phase of the AI hype cycle. The 31% productivity gains are real for specific, narrow tasks. But when you factor in:

  • Verification overhead
  • Increased review burden
  • Quality issues
  • Learning opportunity cost
  • Hidden technical debt

…the net gain is much smaller. Maybe even negative for complex domains.

The question isn’t “does AI make us faster?” It’s “does AI make us better at building software that delivers value and lasts?”

I don’t have a good answer yet. But I’m skeptical that faster code generation equals better outcomes.

What metrics are others tracking? I’d love to hear if anyone has found a way to measure this that accounts for the full lifecycle cost.

3

This entire thread is giving me flashbacks to when design tools started offering “AI-powered” features. We went through the exact same cycle: excitement → adoption → disillusionment → grudging acceptance with guardrails.

The Trust Problem Is a UX Problem

Here’s what I think is happening: AI tools have terrible UX for building trust.

When I use Figma’s auto-layout, I can see how it works. I can mentally model the behavior. When it does something unexpected, I can reason about why. There’s a feedback loop that builds understanding over time.

AI code generation is the opposite. It’s a black box that sometimes produces magic and sometimes produces garbage, and the interface gives you no way to understand which you’re getting.

Your senior engineer’s quote nails it: “At least I understand how a junior thinks.”

That’s not about code quality. That’s about mental models. We can build mental models of junior developers. We can’t build mental models of AI.

The Cognitive Load Is Exhausting

You mentioned verification taking 15 hours/week. I want to name what that feels like:

Constant vigilance. Every AI suggestion requires full-brain engagement. You can’t skim. You can’t pattern-match. You have to deeply reason about correctness, edge cases, security implications.

It’s like being in a code review 100% of your working hours.

No wonder people are burned out. No wonder they’re skipping verification. The tool demands a level of sustained attention that’s not humanly sustainable.

A Scary Story From Our Design System

Last month, AI suggested a component pattern for our design system. It looked good. Clean React code, nice props API, good TypeScript types. We almost shipped it.

Then someone on accessibility review caught it: the pattern was completely keyboard-inaccessible.

AI had learned from thousands of components online. Most of them are inaccessible. So it confidently generated inaccessible code that passed all our other reviews.

If we’d shipped it, we would have broken keyboard navigation for thousands of users. All because we trusted the AI’s confident-looking output.

The verification gap isn’t just about time. It’s about coverage. Are we verifying for correctness? Performance? Accessibility? Security? Maintainability?

You can’t verify everything. So which blind spots are we accepting?

What Helps: AI Literacy Training

We started running “AI literacy” workshops for designers and frontend engineers. Not “how to use the tool” but “how to think about AI outputs”:

  1. Assume AI optimizes for common patterns, not correct patterns
  2. AI doesn’t understand your users or your constraints
  3. Confidence ≠ correctness (AI-generated code often looks more polished than it is)
  4. Verification checklist: Don’t just read the code, test edge cases, accessibility, performance

It’s helping. But it’s also another 4 hours of training per person, which… doesn’t show up in the productivity metrics.

The Honest Question

@vp_eng_keisha asked: “What am I missing?”

I don’t think you’re missing anything. I think the AI productivity gains are real for the narrow task of generating code quickly.

But software development isn’t just code generation. It’s:

  • Understanding the problem
  • Designing the solution
  • Implementing correctly
  • Verifying completeness
  • Maintaining over time
  • Teaching the next generation

AI helps with #3. Maybe. Sometimes. With verification overhead.

For everything else? We’re on our own.

Maybe the real productivity question is: What percentage of your job is “generate code quickly”?

If it’s 20%, then a 50% gain on that 20% is a 10% total productivity boost. Minus verification overhead. Minus quality issues. Minus learning opportunity cost.

Suddenly that 31% starts looking more like 5%. Or zero. Or negative.

I don’t have answers. But I think the hype is way ahead of the reality.

4

Reading through this thread, I’m reminded that we’re still in the very early days of understanding how AI tools reshape engineering work. This isn’t the first time we’ve been here.

When we introduced continuous deployment tools in 2010s, the promise was “ship 10x faster.” The reality was more complex: we shipped faster and broke production more often. It took years to develop practices (feature flags, canary deployments, observability) that made the speed gains sustainable.

AI coding tools are following the same pattern. We’re at the “ship faster, break things more often” stage. We haven’t yet developed the organizational practices that make the gains sustainable.

The Tiered Trust Model Is the Right Direction

@vp_eng_keisha - your tiered trust model is exactly where I think this needs to go. But I’d push it further.

We’ve implemented what I call risk-based AI governance:

Tier 1 - High Trust (AI can auto-commit):

  • Test data generation
  • Documentation updates
  • Formatting/linting fixes
  • Dependency updates (with automated testing)

Tier 2 - Supervised Use (AI generates, human reviews):

  • Boilerplate CRUD operations
  • API endpoint scaffolding
  • Database migrations
  • UI component implementation

Tier 3 - Assisted Use (AI suggests, human implements):

  • Complex business logic
  • Security-critical code
  • Performance-sensitive code
  • Novel architectural decisions

Tier 4 - No AI (human only):

  • Authentication/authorization logic
  • Payment processing
  • PII handling
  • Infrastructure security configs

This framework makes the trade-offs explicit. Different teams can set different thresholds based on their risk tolerance.

What We’re Actually Measuring

I pushed my engineering leaders to track three categories of metrics, not just velocity:

1. Output Metrics (what everyone tracks):

  • Lines of code written
  • PRs merged
  • Story points delivered

2. Quality Metrics (what matters):

  • Production incidents per 1000 lines of code
  • Rollback rate
  • Security vulnerabilities detected in review vs production
  • Accessibility issues caught pre/post deploy

3. Capability Metrics (what determines long-term health):

  • Time for junior engineers to ramp to productivity
  • Knowledge sharing (documentation, pair programming, mentoring)
  • Architectural decision quality
  • Technical debt reduction vs accumulation

The results: Output metrics are up 20-25%. Quality metrics are down 5-10%. Capability metrics are mixed - faster ramp time, but less deep learning.

The Strategic Question

Here’s the question I ask my executive team:

“Do we want to be 30% faster at building software the same way we’ve always built it, or do we want to fundamentally rethink how we build software in an AI-enabled world?”

Because right now, we’re using AI to accelerate the old model. That’s incremental.

The transformational approach would be:

  • Rethinking what engineers should spend time on
  • Redesigning code review for AI-assisted code
  • Building new onboarding programs for an AI-first world
  • Developing new quality gates for AI-generated code
  • Redefining what “senior engineer” means when AI handles implementation

That’s hard work. It requires organizational change, not just tool adoption.

The Uncomfortable Truth

@eng_director_luis mentioned the learning opportunity cost. That’s the part that worries me most.

If junior engineers never struggle through complex implementations, they don’t develop the judgment to review those implementations. In 5 years, who will be our senior engineers?

We might be optimizing for short-term velocity at the cost of long-term capability.

That’s not a technical problem. That’s a strategic risk.

My Answer to “What Are You Missing?”

You’re not missing anything, @vp_eng_keisha. You’re seeing the reality that the AI productivity narrative hasn’t caught up with yet.

The 31% gain is real - for a narrow definition of productivity that ignores verification costs, quality impacts, and learning effects.

The question isn’t whether AI makes us faster. It’s whether AI makes us better at delivering value sustainably.

I don’t have that answer yet. But I know the answer isn’t in the “time saved writing code” metric alone.

We need to measure what matters: customer outcomes, system reliability, team capability, and long-term maintainability.

Until we do, we’re optimizing for the wrong target.

5

Coming at this from the product side - this whole thread is making me rethink how we talk about engineering velocity with our board.

The Speed vs. Value Disconnect

We’ve been celebrating our increased sprint velocity. We’re shipping features 25% faster since adopting AI tools. Our board loves it. Investors love it.

But here’s what nobody’s talking about in our board decks: Are we shipping 25% more customer value?

Because if we’re shipping faster but:

  • Creating more bugs (customer frustration)
  • Accumulating technical debt (slower future development)
  • Building features with poor UX (because we didn’t think through edge cases)

…then we’re optimizing for the wrong metric.

Shipping fast means nothing if we’re shipping the wrong things faster.

The Product Debt Nobody Tracks

@cto_michelle mentioned technical debt. I want to add product debt to that conversation.

When engineers are moving faster, they’re making more micro-decisions without full context. AI fills in the gaps with “reasonable defaults” that might not align with our product vision.

Example from last month: An engineer used AI to build an admin dashboard feature. It shipped in 2 days instead of the usual 5. Looked great.

Then our customer success team reported that the workflow didn’t match how admins actually use the product. It solved the ticket, but it didn’t solve the problem.

We had to rebuild it. Net time: 7 days instead of 5.

Fast implementation without deep understanding creates product debt. And product debt is invisible until customers complain.

The Trust Gap Affects Product Decisions

Here’s where the trust gap really hurts product development:

When I’m planning a roadmap, I need to understand the true cost and risk of features. If engineering says “we can build this in 2 weeks with AI assistance,” I need to know:

  • Is that 2 weeks to first code, or 2 weeks to production-ready?
  • What’s the quality risk?
  • What’s the maintainability cost?
  • Will this need to be rebuilt in 6 months?

If the productivity numbers are inflated, my product decisions are wrong.

I might prioritize a complex feature because “AI makes it cheap,” when the true lifecycle cost is actually higher than traditional development.

That’s a strategic risk, not just an engineering productivity question.

What I’m Starting to Ask

In our product-engineering sync meetings, I’ve started asking different questions:

Instead of: “How long will this take to build?”

I ask: “What’s the confidence level that this implementation will be correct, maintainable, and aligned with our product vision?”

Instead of: “How many features can we ship this quarter?”

I ask: “How many features can we ship well this quarter?”

The answers are very different.

The Real Productivity Question

@vp_eng_keisha asked “what’s the real productivity number?”

From a product perspective, I think the real productivity metric is:

Time from customer problem to validated solution

Not time to code. Not time to ship. Time until we’ve verified that what we shipped actually solves the customer’s problem without creating new problems.

When I measure that, the AI productivity gains shrink dramatically. Because:

  • We’re shipping faster but validating at the same speed (or slower)
  • We’re finding more bugs in QA and production
  • We’re doing more post-ship iteration
  • We’re accumulating product debt that slows future development

Net result: Maybe 5-10% faster delivery of validated solutions. Not 31%.

The Question I’m Taking to Our Board

Our next board meeting, I’m planning to reframe the AI productivity conversation:

“We’re generating code 31% faster. But we’re delivering customer value only 8% faster. The gap is verification overhead and quality issues. Should we optimize for code generation speed, or customer value delivery speed?”

I think the answer is obvious. But it means rethinking how we measure engineering productivity.

And honestly, after reading this thread, I’m not sure we’re measuring the right things at all.

What do other product leaders think? Are we celebrating velocity gains that don’t translate to customer outcomes?