I just read the new Anthropic research on AI coding assistants and I haven’t been able to stop thinking about it. The headline finding: developers using AI assistance scored 17% lower on comprehension tests compared to those coding manually. And here’s the kicker—the productivity gains weren’t even statistically significant.
As someone scaling an engineering org from 25 to 80+ people, I’m watching this play out in real-time with our junior hires.
What the Research Found
Anthropic studied 52 junior software engineers working with Python. The results are sobering:
- Quiz scores: AI-assisted group averaged 50%, manual coding group averaged 67%
- Time saved: Only ~2 minutes faster (not statistically significant)
- Biggest gap: Debugging questions—the very skills needed to validate AI-generated code
- Critical insight: HOW developers used AI mattered more than IF they used it
- Those using AI for conceptual inquiry: 65%+ scores
- Those delegating code generation to AI: <40% scores
What I’m Seeing In Practice
At our EdTech startup, I’ve noticed our newest junior engineers struggle when AI tools go down or when they hit edge cases AI can’t handle. They can ship features quickly with Claude or Copilot, but ask them to debug a production issue without AI assistance and they freeze.
Last month, we had a junior spend 3 hours debugging an API integration error that a mid-level engineer solved in 20 minutes. The junior had used AI to generate the integration code but couldn’t reason about the actual data flow.
This isn’t about individual capability—these are smart, motivated engineers. This is about skill formation in an AI-native environment.
The Leadership Dilemma
We’re facing some hard questions:
- Should we limit AI tools during onboarding? Create “training wheels” periods where juniors code manually?
- How do we measure fundamental competency? PR velocity doesn’t tell us if someone understands their code.
- What’s the trade-off? Slower initial velocity vs stronger long-term foundation
- Competitive pressure: Other companies let juniors use AI from day one. Are we handicapping our recruiting?
The Anthropic research suggests we’re trading short-term productivity for long-term skill mastery. But we’re also in a market where:
- 85% of developers already use AI tools regularly
- 41% of code written in 2025 was AI-generated (crossing 50% by late 2026)
- New grads expect AI tools as standard equipment
The Security Angle
Here’s what keeps me up at night: research shows a 23.7% increase in security vulnerabilities from AI-assisted code. If our juniors can’t debug their own code, how can they validate it’s secure?
We’re seeing this in code reviews—juniors often can’t explain why their AI-generated code works, which means they definitely can’t explain why it might fail.
Questions for This Community
For other engineering leaders:
- How are you handling AI tools in your onboarding process?
- Have you seen similar skill gaps with junior engineers?
- What metrics are you using to assess fundamental coding competency?
For ICs who started their careers recently:
- How do you balance learning with AI vs learning fundamentals?
- Do you feel like AI tools helped or hindered your skill development?
The meta question:
Are we creating a generation of engineers who are incredible at directing AI but can’t code when the AI fails? And if so, is that actually a problem—or just the new normal we need to adapt to?
I don’t have answers yet, but I think this is one of the most important conversations we can have as an industry right now.
Sources: