Our AI Generates PRs Faster Than Security Scans Can Run—Are We Creating Vulnerability Windows We Don't Even See?

General
1

Our AI Generates PRs Faster Than Security Scans Can Run—Are We Creating Vulnerability Windows We Don’t Even See?

Okay, I need to share something that’s been keeping me up at night. :crescent_moon:

Three months ago, our engineering team adopted GitHub Copilot and Cursor across the board. The productivity gains were immediate—our PR volume tripled in the first month. Developers were shipping features faster than ever, and leadership was thrilled.

But here’s what nobody anticipated: our security scans couldn’t keep up.

The Problem We Didn’t See Coming

Our SAST security scans typically took 5-8 minutes to run. When we had ~10 PRs per day, that was fine. Now we’re averaging 30+ PRs daily, and we’ve created a queue. PRs are waiting 20-30 minutes just to get scan results back.

And here’s the scary part: developers started merging before scans completed because “it’s just a small change” or “I need to unblock the team.”

Last week, we discovered 2 vulnerabilities that shipped to staging—both would have been caught by our security scans if anyone had waited for them to finish. One was a SQL injection risk in a “quick fix” that an AI assistant generated. The other was missing input validation in an API endpoint.

The AI Speed vs Security Dilemma

This feels like a fundamental mismatch. AI coding assistants work at machine speed—one of our engineers told me his AI pair programmed through 15 function variations in the time it used to take him to write 2. But our security infrastructure still runs at “human developer” speed.

I’ve been researching this, and the data is alarming:

  • A Stanford/MIT 2026 study found that 14.3% of AI-generated code contains security vulnerabilities vs 9.1% for human-written code
  • DryRun Security reported that 87% of AI-generated PRs contained at least one vulnerability
  • We’re generating more code faster, but it’s objectively less secure

So What Do We Do?

I see a few options, and honestly none of them are perfect:

Option 1: Slow down AI adoption to match our security tooling capacity

  • Pro: No vulnerability windows
  • Con: Feels like we’re fighting progress, and developers will resist

Option 2: Upgrade to real-time security validation

  • Pro: Keeps pace with AI code generation
  • Con: Expensive (~$40/dev/month), and I’m not sure it catches everything

Option 3: Accept brief vulnerability windows as the new normal

  • Pro: Maintain development velocity
  • Con: This feels incredibly irresponsible, especially for production code

Option 4: Hybrid approach - real-time for critical paths, async for everything else

  • Pro: Balances speed and security
  • Con: Complex to implement, requires risk classification

The Design Leader’s Take

From my perspective, this is a classic system design problem—we’ve optimized one part of the system (code generation) without considering the downstream effects (security validation). It’s like making a component library with beautiful UI but forgetting about accessibility—the speed doesn’t matter if the foundation is broken.

But I also know that telling engineers “slow down and wait for security scans” won’t work. We need to design systems that make the secure path the easy path, not the friction-filled path.

What I’m Asking This Community

How are you handling this mismatch between AI code generation speed and security scanning speed?

  • Have you implemented real-time security validation? What works?
  • Are you accepting some level of vulnerability windows? How do you manage that risk?
  • Have you found ways to make security scanning faster without compromising thoroughness?
  • Is anyone successfully using risk-based approaches (different security levels for different code)?

I’ve looked at tools like Snyk, Checkmarx, and GitGuardian MCP, and they all promise real-time validation. But I want to hear from people actually running this in production—not just vendor marketing.

The irony isn’t lost on me that we’re using AI to code faster, which creates security risks, which we then try to solve with more AI-powered security tools. :robot::locked:

Is this sustainable? Or are we creating a vulnerability debt we’ll regret?

Would love to hear how other teams are navigating this. Especially if you’ve found approaches that don’t require choosing between speed and security.

:victory_hand:
Maya

2

Maya, this hits close to home. In financial services, we face this exact challenge with a critical constraint: we cannot accept ANY vulnerability window, period. Regulatory compliance doesn’t give us the luxury of “brief exposure.”

Our Solution: Pre-Commit Security Gates

We implemented pre-commit hooks with fast SAST scanning that completes in under 30 seconds. If code fails security checks, it doesn’t commit—no exceptions. This was painful to implement and definitely created some developer friction initially, but it’s the only way to ensure zero vulnerability windows.

Here’s what we learned:

1. Speed is non-negotiable for pre-commit scanning

  • We tested 5 different security tools and rejected 3 that took >45 seconds
  • Settled on a combination of Snyk Code (for quick pattern matching) + custom rules for our specific compliance requirements
  • Developers will bypass any check that takes longer than a coffee break

2. AI-enhanced security review runs parallel to human review

  • We use AI-powered analysis that looks for patterns specific to AI-generated code
  • Things like: hallucinated APIs, deprecated security practices, race conditions in auth logic
  • This catches ~40% more issues than our traditional SAST alone

3. Developer education is critical

  • We ran training sessions specifically on “AI code security blind spots”
  • Showed real examples from our codebase of vulnerabilities that AI assistants created
  • Changed the narrative from “security is slowing us down” to “security is protecting our customers”

The Trade-Off: Friction vs Discipline

Does this kill innovation? Honestly, it creates friction. Developers complain about pre-commit hooks failing on “minor issues.” Some have tried to disable them (we log these attempts and have…conversations).

But here’s the thing: after 6 months, we’ve had zero security vulnerabilities ship to production from AI-generated code. Before this system, we were averaging 2-3 per month.

From a leadership perspective, I frame it this way: Would you rather have developers frustrated by a 30-second security check, or have our CISO explaining a breach to regulators?

The Question This Raises

Your fourth option (hybrid approach with risk classification) might work for companies that can accept some vulnerability windows. For us, the regulatory environment makes that impossible.

But I’m genuinely curious: Does blanket security enforcement stifle the innovation that AI coding promises? Are we being too cautious, or is this the new baseline that everyone will need to adopt?

I’d love to hear from teams in less regulated industries—do you have the flexibility to accept brief vulnerability windows, and if so, how do you manage that risk?

— Luis

3

Both of you are circling around what I think is the real issue: We’re treating symptoms, not the disease.

The problem isn’t that security scans are too slow. The problem is that 87% of AI-generated PRs contain vulnerabilities in the first place. That’s from the DryRun Security research, and it’s frankly unacceptable.

Why AI Code Is Inherently Less Secure

The root cause is that AI models are trained on historical code—including vulnerable historical code. Our AI assistants are literally learning from Stack Overflow answers from 2015 and repeating decade-old security mistakes.

Example from our recent audit: Every AI-generated WebSocket implementation in our codebase was missing authentication. Why? Because the training data likely included tutorial code that skipped auth for simplicity. The AI doesn’t understand why authentication matters; it just patterns-matches on what it’s seen before.

The Real Solution: Context-Rich Validation Across the Lifecycle

Luis, I respect the pre-commit approach, but I think it’s a band-aid. You’re catching vulnerabilities after they’re generated. What if we prevent them from being generated in the first place?

Here’s what we’re moving toward:

1. Secure coding policies integrated into AI prompts

  • We’ve fine-tuned our AI assistants with secure coding guidelines
  • Explicit guardrails: “Always use parameterized queries, never string concatenation for SQL”
  • “Always implement authentication before authorization”

2. Context-aware validation, not just pattern matching

  • Tools that understand data flow, not just syntax
  • Example: One of our payment API flaws was created by AI-assisted changes that bypassed every static check but surfaced instantly when validated against real execution paths

3. Defense in depth

  • Pre-commit checks (yes, like Luis suggested)
  • Pre-deployment validation (YAML configs, cloud infrastructure)
  • Runtime controls (RASP, WAF, anomaly detection)
  • Red teaming and penetration testing

The goal isn’t faster scanning—it’s generating more secure code in the first place.

The Strategic Question

Maya, you asked if this is sustainable. My answer: Not with current AI coding assistants. We need a fundamental shift in how these tools work.

The vendors need to prioritize security in model training, not just code completion accuracy. Until then, we’re in an arms race—AI generates vulnerable code faster, and we scramble to scan it faster.

I’d rather slow down and build it right than sprint toward a security incident.

— Michelle

4

This conversation is highlighting something I see across our organization: We have a people problem disguised as a tooling problem.

Maya, you mentioned that developers started merging before scans completed because “it’s just a small change.” That’s the real issue right there. The fact that your team even considers merging before security validation completes reveals broken incentives and culture.

The Culture Shift We Had to Make

Last quarter, we discovered that 25% of our security vulnerabilities came from “small changes” that bypassed review. Developers thought they were being efficient. They were actually creating risk.

Here’s what we did:

1. Hard requirement: Security scan approval before merge

  • No exceptions, no overrides, even for “urgent” fixes
  • This was unpopular. Engineers pushed back hard.
  • Leadership backed it anyway because security is non-negotiable

2. Made the pain visible

  • Dashboard showing average scan wait time by team
  • This surfaced that some teams had 45-minute waits (tooling problem)
  • Other teams had 5-minute waits but still merged early (culture problem)
  • Different solutions needed for each

3. Education on AI-specific risks

  • Training session: “Why AI Code Needs Different Security Attention”
  • Showed real examples of AI hallucinations, deprecated patterns, logic errors
  • Shifted mindset from “AI makes me faster” to “AI needs my oversight”

4. Empowered teams to own security

  • Designated “security champions” in each squad
  • Their job: Understand scan results, educate team, escalate complex issues
  • Removed it from being a compliance checkbox to being a team competency

The Metrics That Changed Behavior

We started tracking:

  • Scan wait time (how long PRs wait for security results) - tooling metric
  • Bypass rate (how often PRs merge before scan completes) - culture metric
  • Vulnerability escape rate (how many vulns make it to production) - outcome metric

The bypass rate was the game-changer. When teams saw their own data showing they were merging before scans, it created accountability. No one wants to be the team that ships vulnerabilities because they couldn’t wait 5 minutes.

This Intersects with Luis and Michelle’s Points

Luis, I love your pre-commit approach. That architecturally prevents the bypass behavior. We considered it but didn’t want to block developers during feature work. Instead, we made merge-time checks mandatory.

Michelle, you’re absolutely right that we need better AI models. But while we wait for vendors to fix their training data, we still have to ship code safely today. Culture and process are the bridges.

The Leadership Question

Here’s what keeps me up at night: How do you scale security culture across a fast-growing engineering org?

We’re hiring 15 engineers this quarter. Each new hire comes with different security awareness. Some worked at security-conscious companies (Google, Microsoft). Others worked at move-fast startups where security was an afterthought.

You can’t rely on tooling alone. You need engineers who understand why security matters and make good judgment calls when the tools aren’t enough.

Question for the group: How do you instill security-first culture without making it feel like compliance theater?

The goal is engineers who want to write secure code, not engineers who have to pass security scans.

— Keisha

5

Great perspectives from everyone. Let me add the business and product lens here, because I think we’re missing a critical question:

What’s the Actual Business Risk of Brief Vulnerability Windows?

Not all vulnerabilities are created equal. A SQL injection in a production API endpoint that handles payment data is completely different from a XSS vulnerability in an internal admin tool that requires authentication and is used by 3 people.

The severity and exploitability matter enormously for risk calculation.

Risk-Based Security Framework:

Code Type Exposure Acceptable Window Validation Approach
Production API (customer data) Public internet 0 seconds Real-time pre-commit
Internal tools (auth required) Private network 24 hours Async scanning
Dev/staging only No production access 1 week Periodic audits

This isn’t about accepting vulnerabilities—it’s about prioritizing security investment where business risk is highest.

The Cost-Benefit Analysis

Luis mentioned spending ~$40/dev/month for real-time security tooling. For a 100-person engineering team, that’s $48K annually.

Compare that to:

  • Average cost of a data breach: $4.5M (IBM 2025 Cost of Data Breach Report)
  • Cost of security incident response: $500K-$2M
  • Regulatory fines for compliance failures: $1M-$50M depending on industry

From a pure ROI perspective, even expensive security tooling is a bargain.

But here’s the nuance: Not all code paths carry the same breach risk. Spending equally on securing internal build scripts and customer-facing APIs doesn’t optimize for business outcomes.

The Product Question No One’s Asking

Maya asked if this is sustainable. I think the real question is: At what percentage of AI-generated code does the security risk outweigh the velocity benefit?

If AI code has a 14.3% vulnerability rate vs 9.1% for human code, you’re accepting 57% more security risk per line of code. At what volume does that become unacceptable?

Let’s do the math:

  • Team writes 10,000 lines/month human code = ~910 vulnerabilities
  • Same team with AI writes 30,000 lines/month AI code = ~4,290 vulnerabilities
  • That’s a 4.7x increase in absolute vulnerability count

Even if you ship features 3x faster, you’re creating more security debt than you’re burning down.

The Customer Perspective

Here’s something that hasn’t come up yet: Some of our enterprise customers are starting to ask in security reviews what percentage of our code is AI-generated.

They’re concerned about the vulnerability rates Michelle cited. If we can’t demonstrate that we have appropriate security controls around AI-generated code, we may lose deals.

Conversely, if we can show: “We use AI for 3x productivity, AND we have real-time security validation that catches 99% of AI-specific vulnerabilities,” that becomes a competitive differentiator.

Questions for the Group

  1. How are you calculating ROI on security tooling investment? Is it based on prevented breaches, or compliance requirements, or something else?

  2. For teams using risk-based security approaches: How do you classify code into risk tiers? Is it manual or automated?

  3. Has anyone tracked customer response to “AI-generated code” in security questionnaires? Is it a liability or neutral?

I’m particularly curious about Michelle’s point on training better AI models. If vendors offered “security-first AI coding assistants” that had lower vulnerability rates but maybe slightly slower code generation, would teams adopt them? Or is speed the dominant factor?

— David