Building Effective AI Code Verification Workflows: Beyond 'Just Review Everything'

eng_director_luis · March 15, 2026, 6:01am

Following up on the discussion about AI code review mandates, I want to share what we’ve learned implementing verification workflows in financial services. The short version: “Review everything manually” doesn’t scale, but you can’t just trust blindly either.

The Problem We Faced

Like many teams, we started with the simplest policy: AI can write code, humans must review all of it.

Within two months, we hit a wall:

Review queue grew faster than we could clear it
Senior engineers burned out on routine review work
AI-generated code volume increased 3x (more code = more review)
Review became the bottleneck, not code generation

We had made code generation faster but hadn’t adapted our verification systems. Classic optimization fallacy.

What We Built Instead

We designed a multi-layered verification approach that catches different types of issues at different stages, reducing the burden on human reviewers.

Layer 1: Pre-Commit Automated Checks

Before code even reaches a reviewer:

Static Analysis:

Linting for code style and common errors
Security scanning (OWASP dependency check, secrets detection)
Complexity metrics (cyclomatic complexity, maintainability index)

AI-Specific Checks:

Pattern matching for known AI failure modes (we’ve cataloged 23 common mistakes our AI tools make)
Diff analysis - flag unusually large changes or rewrites
Test coverage verification - AI code must include tests

Result: Catches ~40% of issues automatically, before human review

Layer 2: Automated Testing Gate

Code cannot be reviewed until:

All unit tests pass (100% of existing tests, new tests for new code)
Integration tests pass for affected systems
Performance benchmarks meet thresholds

Result: Catches ~30% of remaining issues, proves functionality

Layer 3: Risk-Based Human Review

Only after passing automated checks, code goes to human review. But review depth varies by risk:

High-Risk (Tier 1): Money movement, auth, customer data

Two senior engineers review architecture and business logic
Security team reviews for vulnerabilities
Compliance spot-checks audit trail

Medium-Risk (Tier 2): Business features, integrations

One peer review focused on design and maintainability
Automated checks must pass, reviewer focuses on “does this solve the right problem?”

Low-Risk (Tier 3): Internal tools, scripts, documentation

Self-review with automated verification
Spot-checks by team lead (10% sampling)

Result: Human reviewers focus on problems machines can’t catch

Layer 4: Post-Deployment Monitoring

Even after review and deployment:

Automated monitoring for errors and performance degradation
Feature flags for gradual rollout
Incident tracking tagged by “AI-generated” vs “human-written”

Result: We learn AI failure patterns and feed them back to Layer 1

The Data After 6 Months

We’ve been running this system for six months. Here’s what changed:

Before (manual review only):

Average review time: 4.2 hours per PR
Review queue: 22 PRs average
Senior engineer time on review: 35% of week
Defects caught in review: 73%
Defects reaching production: 8 per month

After (layered verification):

Average review time: 1.8 hours per PR (human review only)
Review queue: 6 PRs average
Senior engineer time on review: 12% of week
Defects caught in automation: 68%
Defects caught in review: 28%
Defects reaching production: 6 per month (25% reduction)

Key insight: We’re catching MORE defects overall with LESS human review time. Automation catches the routine stuff, humans catch the subtle stuff.

Implementation Lessons

Start Small, Prove Value:
We piloted with one team (internal tools - low risk) for one month. Proved the system worked before rolling out to critical systems.

Catalog AI Failure Patterns:
We tracked every defect in AI-generated code for three months. Common patterns emerged:

Incorrect error handling (AI often forgets edge cases)
Over-optimization (AI writes complex code when simple would work)
Context misunderstanding (AI doesn’t know business rules it wasn’t told)

These patterns became automated checks.

Invest in Infrastructure:
This isn’t free. We spent 6 weeks building:

CI/CD pipeline enhancements
Custom linting rules for AI patterns
Dashboards for tracking defects by source (AI vs human)

But the ROI is clear: senior engineering time freed up for higher-value work.

The Remaining Challenges

This system isn’t perfect:

Junior engineers still need mentorship: Automated checks don’t teach judgment
Novel problems slip through: AI sometimes fails in creative ways automation doesn’t catch
Maintenance overhead: Verification rules need updating as AI tools evolve
Cultural resistance: Some engineers believe “manual review is the only real review”

We’re still iterating.

My Question for This Community

What verification automation are you running? Are you just using standard linting, or have you built AI-specific checks?

And for those still doing mostly manual review: What would it take to trust automation more? Is it lack of tooling, or lack of confidence in what automation can catch?

I’m convinced that as AI code generation accelerates, verification must accelerate too - and humans can’t review fast enough. Automation isn’t optional anymore.

maya_builds · March 15, 2026, 6:02am

Luis, this is EXACTLY what we’ve been doing in design systems, and I’m fascinated by how parallel the problems are.

Design System Verification Layers

We have almost the same approach for design component review:

Layer 1: Automated Design Linting

Figma plugins check spacing, typography, color tokens
Accessibility checks (contrast ratios, touch targets)
Component naming conventions
Icon usage verification

Layer 2: Visual Regression Testing

Automated screenshots on every component change
Pixel-diff comparison against baseline
Catches unintended visual side effects

Layer 3: Human Design Review (Risk-Based)

New patterns: Full design team review
Variant additions: Peer review
Token updates: Self-review + automated checks
Documentation: Spot-check sampling

Layer 4: Usage Monitoring

Track which components are actually being used
Identify abandoned or problematic patterns
Feed insights back into design system roadmap

The parallel is striking: Automated tools catch the mechanical stuff, humans focus on intent and judgment.

The Question You Raise

Your cataloging of AI failure patterns is brilliant. We did the same thing with design inconsistencies:

Common designer mistakes when using the design system:

Wrong component selection (using Button when they need Link)
Misapplying responsive patterns
Ignoring accessibility guidelines
Creating one-off variations instead of extending the system

We turned these into automated nudges in Figma: “Are you sure you want to create a custom button? We have 12 button variants available.”

Can This Work for Code Review?

Your Layer 1 (AI-specific checks for known failure modes) is the game-changer. Most teams I talk to are just using generic linting.

What I’m curious about: How did you build the catalog of AI failure patterns? Did you:

Manually review 3 months of AI code and tag defects?
Use automated analysis to find patterns?
Interview engineers about what they commonly fix?

And how do you keep it updated as AI tools improve? I imagine AI failure patterns evolve as the models get better.

The Cultural Resistance Point

You mentioned “some engineers believe manual review is the only real review.” We had the EXACT same problem with designers who felt automated checks were “missing the craft.”

What worked for us:

Show the data (defects caught by automation vs. missed)
Let skeptics opt-in first (don’t force it)
Celebrate time saved, not just defects caught (“You reviewed 5 components this week instead of 15, and focused on the actually interesting problems”)

The mindset shift: Automation doesn’t replace review, it elevates what review focuses on.

Your senior engineers aren’t reviewing less rigorously - they’re reviewing more valuable problems. That reframe helped our team.

My Question

For hybrid creative-technical work (design systems, AI code), is there a common framework for deciding what to automate vs. what requires human judgment?

I keep seeing the same pattern: Automate the mechanical, elevate the creative. But defining that boundary is hard.

product_david · March 15, 2026, 6:02am

Luis, this is incredibly helpful context for the conversation we were having in the other thread about velocity vs. quality.

The Product Perspective

What I find most compelling about your layered approach: You’re not just moving faster, you’re building a system that scales.

From a product standpoint, here’s what matters:

Velocity That Compounds:
Your senior engineers went from 35% of their time on review to 12%. That’s not just time savings - that’s unlocking capacity for higher-leverage work like architecture, mentoring, and technical strategy.

If I extrapolate: 3 senior engineers spending 23% less time on routine review = basically gaining an extra senior engineer’s worth of strategic capacity.

That’s a hiring/scaling win, not just a productivity win.

Predictability:
One of my biggest frustrations with “mandate manual review” policies: unpredictable review times. Sometimes a PR gets reviewed in an hour, sometimes it sits for days.

Your system with automated gates and risk-based review creates predictability. Product teams can better estimate when features will ship.

Predictability matters more than raw speed for roadmap planning and stakeholder management.

The ROI Question

You mentioned 6 weeks to build the verification infrastructure. That feels expensive, but let me do some rough math:

Investment:

6 weeks × 2-3 engineers = ~12-18 engineer-weeks
Ongoing maintenance: ~1-2 days/month

Return:

3 senior engineers save 23% time each = ~0.7 FTE freed up
Over 6 months: 0.7 × 26 weeks = ~18 engineer-weeks gained
After 6 months: Pure profit on the time investment

Plus:

25% reduction in production defects = fewer customer issues, less support burden
Faster review cycles = faster feature delivery = faster customer feedback

From a product ROI perspective, this is a no-brainer. The payback period is about 6 months, then it’s compounding value.

Questions for Luis

Sprint Planning: How do you handle features that span multiple risk tiers? Do you break them into separate stories or treat the whole feature as high-risk?
Product Input: Does Product have any input into risk tier classification? Sometimes what engineering sees as low-risk is high-visibility to customers (and vice versa).
Communication: How do you communicate review timelines to product/stakeholders? Do you say “High-risk features take 2x longer to review” upfront?

What I’m Taking Back to My Team

I’m going to propose we invest in building our own Layer 1 verification automation. Even if it’s just cataloging failure patterns and creating custom linting rules.

The question isn’t “Can we afford to build this?” It’s “Can we afford NOT to build this as AI code generation accelerates?”

Thanks for sharing the playbook, Luis. This is exactly the kind of practical implementation detail that helps product and engineering get aligned.

vp_eng_keisha · March 15, 2026, 6:03am

Luis, I love the systematic approach here, but I want to add the people/culture dimension that I think is critical for this to actually work.

The Change Management Challenge

You built a great system, but you also mentioned “cultural resistance” from engineers who believe manual review is the only real review. In my experience scaling teams, that resistance will kill your system if you don’t address it head-on.

Here’s what I’ve learned rolling out process changes across 80+ engineers:

Start With Opt-In, Not Mandate

When we introduced new development practices (including our AI coding assistant adoption), we:

Pilot with volunteers (1-2 teams): Find the engineers who are excited about process improvement, not resistant
Prove value with data: Track metrics that matter to engineers (time saved, defects caught, time to ship)
Share success stories: Let pilot team members evangelize to their peers
Expand gradually: Opt-in becomes opt-out becomes standard after 6-9 months

Why this matters: Engineers who choose to adopt a system defend it. Engineers who have a system imposed on them resist it.

Make the Why Crystal Clear

Your Layer 1-4 explanation is great for this forum, but have you communicated WHY this system exists to your team?

The framing I’ve found that works:

“We’re automating review to save time” (engineers hear: “management thinks review isn’t important”)
“We’re automating routine checks so YOU can focus on architecture and design decisions” (engineers hear: “your expertise is valued for harder problems”)

Reframe automation as elevation, not replacement.

Address the Real Fear

When engineers resist automation, they’re often not resisting efficiency - they’re afraid of:

Loss of craftsmanship: “Will I become just a code generator operator?”
Deskilling: “Will junior engineers learn without manual review mentorship?”
Job security: “If automation does review, why do we need senior engineers?”

These fears are real, even if they’re not explicitly stated. Your system actually addresses them (senior engineers do more strategic work), but you need to make that explicit.

The Junior Engineer Problem

David mentioned sprint planning, but I want to flag another concern: How do junior engineers learn in this system?

Manual code review is often where junior engineers learn:

How to write clean code (by seeing what gets flagged)
What good architecture looks like (by feedback on their design choices)
Domain knowledge (by reviewers explaining business context)

Your automated Layer 1 catches mechanical issues, but it doesn’t teach judgment. Your Layer 3 risk-based review might mean junior engineers get LESS human feedback because their work is often low-risk.

My question: How are you ensuring junior engineers still get the mentorship and learning they need? Are you tracking review feedback quality, not just review volume?

What’s Working for Us

We run similar layered verification, but we added a learning layer:

For junior engineers (<2 years):

All PRs get human review regardless of risk tier
Reviewers use a “teaching template” that explains WHY not just WHAT
Monthly 1-on-1s include review of their code review feedback patterns
Pair programming time with seniors (separate from review process)

For engineers showing strong judgment:

Graduated to risk-based review
Trusted for self-review on low-risk code
Become reviewers for others to reinforce their learning

Result: Automation saves senior time, but junior engineers still get mentorship.

My Advice for Rollout

If you’re implementing Luis’s system (and you should!):

Communicate the why relentlessly - don’t assume engineers understand the strategic goals
Show, don’t tell - pilot first, prove value with data, then scale
Celebrate the freed-up time - highlight what senior engineers are doing with their reclaimed 23%
Preserve learning pathways - make sure junior engineers still get mentorship
Iterate publicly - admit what’s not working and adjust

Process changes without culture change just create compliance theater. You need both.

cto_michelle · March 15, 2026, 6:03am

Let me be the contrarian here and ask the uncomfortable question: Are we building the right verification system, or are we optimizing for yesterday’s problems?

Luis, your layered approach is well-designed and clearly effective based on your metrics. But I want to challenge the underlying assumption.

The Platform Engineering Lens

What you’ve built is essentially verification infrastructure - a platform for ensuring code quality at scale. That’s exactly right, and most companies don’t invest enough in this.

But here’s my question: Should you build this, or should you buy/adopt it?

You spent 6 weeks building custom CI/CD enhancements, custom linting rules, and dashboards. David calculated a 6-month payback period. That’s good ROI.

But consider:

Maintenance cost: Those custom rules need updates as AI tools evolve
Opportunity cost: What could your senior engineers have built in those 6 weeks instead?
Scale limits: If you 10x your engineering team, does this system scale or need a rewrite?

Build vs. Buy Decision Framework

I’m seeing a pattern across companies:

What to Build (Company-Specific):

Risk tier classification for YOUR domain
Business logic verification for YOUR products
Failure pattern catalogs for YOUR AI usage patterns
Monitoring and feedback loops for YOUR systems

What to Buy/Adopt (Commodity):

Standard linting and security scanning (Sonar, Snyk, etc.)
CI/CD platforms (GitHub Actions, CircleCI, etc.)
AI code quality tools (emerging category - Qodo, etc.)
Test coverage and performance monitoring

The question: Are you building verification infrastructure that’s unique to financial services, or are you rebuilding generic capabilities that exist as products?

The Strategic Investment Question

As a CTO, I’m constantly making build-vs-buy decisions. My framework:

Build when:

It’s a competitive differentiator
No suitable vendor solution exists
You have spare platform engineering capacity
The solution needs deep integration with proprietary systems

Buy when:

It’s table-stakes functionality
Vendor solutions are mature
Your engineering capacity is constrained
Maintenance burden is high

Verification infrastructure feels like it’s transitioning from “build” to “buy” territory as the AI coding assistant market matures.

The Uncomfortable Prediction

Luis, you mentioned 23 cataloged AI failure patterns. I’d bet that:

15-18 of those are generic (every AI tool makes these mistakes)
5-8 are specific to your domain/architecture

The generic patterns should become a vendor’s problem. You should focus your engineering time on the domain-specific patterns that are actually differentiating.

My Challenge to This Discussion

Everyone’s talking about “how to build verification systems.” I want to flip it: How do we make sure we’re investing platform engineering capacity in the highest-leverage problems?

Questions to ask:

Is verification infrastructure our competitive moat? (For fintech: maybe. For most companies: probably not.)
Could we adopt 80% of this from vendors and build only the 20% that’s unique? (Probably yes.)
What could our best engineers build if they weren’t building verification infrastructure? (For Luis: probably core banking innovations that actually differentiate.)

What I’m Doing

At my company, we’re taking a hybrid approach:

Buy: Standard CI/CD, linting, security scanning, test coverage tools
Integrate: AI code quality platforms (Qodo, Codacy AI extensions)
Build: Domain-specific verification (our business rules, our failure patterns, our risk model)

We spent 2 weeks integrating vendor tools and 2 weeks building our custom layer. Net result: 4 weeks instead of 6, and we outsource maintenance of the commodity layers.

The Question Back to Luis

Love your system and your metrics. But I’m curious: If you could wave a magic wand and buy 50% of what you built, which 50% would you outsource and which 50% is worth keeping in-house?

That distinction matters for everyone reading this thread and thinking “should we build this too?”