I need to share some research that made me genuinely uncomfortable—and I think every engineering leader measuring AI tool adoption needs to see it.
The Study That Should Make You Question Everything
In July 2025, METR published a randomized controlled trial studying how AI tools affect experienced open-source developers. The setup was rigorous: 16 developers with moderate AI experience completed 246 tasks in mature projects where they had an average of 5 years of prior experience.
Here’s the uncomfortable finding:
Before starting tasks, developers forecast that AI would reduce completion time by 24%. After completing the study, developers still estimated that AI had reduced their time by 20%.
The actual result? AI tools made them 19% slower.
Read that again. Developers felt 20% faster. They were actually 19% slower. That’s nearly a 40-percentage-point perception gap.
Even expert predictions were wildly wrong—economists expected 39% speed improvements, ML researchers predicted 38%. Everyone overestimated AI’s impact on experienced developers working on familiar codebases.
Why This Matters for Your Metrics
If you’re measuring AI tool adoption and seeing positive sentiment surveys while your DORA metrics stay flat or decline, this research explains why.
The 2024 DORA report found that a 25% increase in AI adoption correlated with a 7.2% decrease in delivery stability. Faros AI’s telemetry from 10,000+ developers showed what they call “The AI Productivity Paradox”: individual output metrics go up (21% more tasks, 98% more PRs merged), but organizational delivery metrics stay flat.
What’s Actually Happening?
The research converges on a few explanations:
1. AI affects how we feel, not just how we perform
The METR study found that AI reduces cognitive load and creates a sense of progress even when actual gains are small. This is valuable—burnout risk dropped 17% in other studies—but it’s not the same as delivery speed.
2. Experience level matters enormously
Studies at Microsoft, Accenture, and a Fortune 100 company found that less experienced developers had higher adoption rates and greater productivity gains. Senior developers working on familiar codebases may gain less because they already know the patterns AI is suggesting.
3. Individual gains don’t automatically translate to team outcomes
Research shows that teams with mature measurement practices successfully translate AI gains from individual developers to team-level improvements. Teams lacking visibility? Flat organizational metrics despite individual productivity increases.
The Organizations Getting It Right
Booking.com stands out as a case study. They deployed AI tools to 3,500+ engineers and achieved a 16% throughput increase while maintaining delivery quality. Key differences:
- Structured enablement programs, not just tool rollout
- Clear measurement framework (utilization, impact, cost)
- Culture alignment around when/how to use AI
Their approach: enablement + measurement + culture. Not just “buy licenses and hope.”
What to Measure Instead
If PR count and sentiment surveys are lying to you, what should you track?
Leading indicators:
- AI-driven time savings (hours saved per developer per week)
- Adoption by seniority level (are your seniors actually using it?)
- Usage patterns (which tasks, which tools, which codebases?)
Outcome indicators:
- Delivery stability (not just speed)
- Code maintainability post-AI
- Time-to-value for features (not just time-to-merge)
Hidden costs:
- Review burden (are seniors spending more time reviewing?)
- Debugging time for AI-generated code
- Security incidents from AI code (46% of developers don’t trust AI outputs)
Questions for This Community
- Are you seeing the perception vs. reality gap on your teams?
- How are you measuring AI tool impact beyond sentiment surveys and PR counts?
- For those with structured enablement programs—what’s working?
The data is clear: AI tools are not automatically making us faster. They’re making us feel faster—which is not nothing—but it’s not the productivity revolution the vendor dashboards promised.
What’s your experience?