We reduced on-call alerts by 70% with AI-powered correlation - here's what worked

General
1

After six months of watching my team struggle with alert fatigue, we finally found a solution that actually works—and it wasn’t just tweaking thresholds or adding more runbooks.

The Problem We Faced

Leading a team of 40+ engineers across three products, our on-call rotation had become unsustainable. Engineers were averaging 15-20 alerts per shift, with most turning out to be noise, duplicates, or symptoms of the same underlying issue. During our quarterly retrospectives, multiple engineers cited on-call stress as their top concern. One senior engineer told me bluntly: “I’m getting alerts about alerts about alerts.”

The metrics told the story: our mean-time-to-resolution was climbing despite having experienced engineers on-call. Why? Because they were spending more time sorting through alert noise than actually solving problems.

What We Implemented

We brought in an AI-powered alert correlation platform that learns from our historical incident data. The key insight: instead of just grouping alerts by time proximity, it uses machine learning to understand causal relationships between different system components.

Here’s what changed:

  • Alert grouping: The AI automatically clusters related alerts. When a database query timeout triggers cascading failures across three microservices, we now see one intelligent alert instead of fifteen separate ones.

  • Root cause prediction: Based on similar past incidents, the platform suggests the most likely root cause within the alert itself. It’s right about 75% of the time.

  • Context augmentation: Each alert now includes relevant metrics, recent deployments, and similar historical incidents—all auto-generated.

The Results (3 Months In)

The numbers speak for themselves:

  • 70% reduction in alert volume (from ~17/shift to ~5/shift)
  • 40% faster mean-time-to-resolution
  • Zero critical incidents missed (our biggest fear)
  • 85% of engineers report improved work-life balance

More importantly, the qualitative feedback has been incredible. Engineers tell me they’re sleeping better, can focus on proactive work during business hours, and actually enjoy their on-call shifts now.

The Challenges

I’ll be honest—the first two weeks were rough. The AI needed time to learn our specific patterns, and we saw some false positives where it grouped unrelated alerts. We also had to resist the urge to second-guess it constantly. Building trust in the system took deliberate effort from the team.

The other challenge was cost. The platform isn’t cheap, but when I calculated the cost of engineer turnover plus lost productivity, the ROI was clear within the first month.

Looking Forward

This experience taught me that sometimes the solution isn’t better processes or more training—it’s fundamentally rethinking our approach. We’re now exploring how to extend this to other parts of our observability stack.

For teams struggling with similar issues, my advice: start with measuring the actual cost of alert fatigue (time, turnover, morale), then make the business case. The technology is ready; the bigger challenge is organizational buy-in.

Would love to hear from others who’ve tackled alert fatigue. What worked for you? What didn’t?

2

Luis, this is exactly the kind of strategic thinking we need more of in engineering leadership. Thank you for sharing the numbers—they’re compelling.

I’m currently rolling out a similar initiative across our EdTech platform, and your experience validates what we’re seeing in the early stages. The burnout prevention angle is especially relevant for us. We lost two senior SREs last quarter, and in both exit interviews they cited on-call stress as a primary factor.

A few questions from the leadership perspective:

  1. ROI beyond alert reduction: You mentioned calculating turnover costs, but what about the opportunity cost? Are you seeing engineers use the freed-up mental space to take on more strategic projects? We’re trying to measure this at my company but it’s tricky to quantify.

  2. Engineer buy-in: How did you handle skeptics on your team? When we piloted an AI correlation tool at my previous company, some senior engineers were resistant—they felt like we were introducing another black box they couldn’t trust. It took about 6 months before they saw real value.

  3. False negative rate: The 70% reduction is impressive, but I’m curious about the other side—what’s your false negative rate? Missing a critical alert because the AI misclassified it would be worse than the original alert fatigue problem. How did you establish confidence that nothing was being dropped?

Broader organizational impact

This aligns perfectly with our company-wide initiative around engineering wellness. I’ve been advocating that sustainable on-call practices aren’t just “nice to have”—they’re a competitive advantage in talent retention. When I share stories like yours with our executive team, it helps make the business case.

One thing we’ve added: we now track “on-call wellness metrics” alongside our traditional SLAs. Things like:

  • Average sleep-hour interruptions per engineer per month (target: <3)
  • Distribution of on-call load (variance should be <20% across team)
  • Post-on-call recovery time utilized

The data has been eye-opening. It’s one thing to know people are stressed; it’s another to see that one engineer is getting 8 sleep interruptions while another gets 1.

Building trust in AI recommendations

The biggest challenge I see with AI-powered incident management is trust. Engineers need to believe the system won’t miss something critical. Did you run the AI in “shadow mode” first, where it makes suggestions but doesn’t actually suppress alerts? We’re considering that approach.

Thanks again for sharing this. These kinds of honest, metrics-driven discussions about engineering leadership are exactly why I value this community. Would love to hear how this evolves over the next quarter.

3

This sounds great in theory, but I have some concerns based on my experience building AI infrastructure at Google Cloud and now at a startup.

AI correlation is powerful, but what about edge cases?

I’ve seen firsthand how ML models can be incredibly good at pattern matching—until they encounter something novel. My main worry: what happens when you have a failure mode the AI has never seen before?

At our AI startup, we run hundreds of LLM inference servers. We had an incident last month where a specific combination of GPU memory pressure + a rare CUDA driver bug caused silent data corruption. The symptoms looked similar to a network latency issue we’d seen before, but the root cause was completely different. An AI trained on our historical data would have sent us down the wrong path.

The hallucination problem

Working with LLMs every day, I see how confidently they can be wrong. They’ll give you a perfectly formatted, authoritative-sounding answer that’s complete nonsense. I’m concerned that alert correlation AI might exhibit similar behavior—clustering unrelated alerts together because it finds spurious patterns in the noise.

What about the AI itself?

Here’s my favorite irony: what happens when your AI-powered observability system goes down? Do you have alerts for your alerting system? We’re talking about adding another complex dependency to your critical path.

At our scale (we’re processing millions of inference requests per day), any additional latency or failure point in the observability stack is a real concern.

Some specific questions:

  1. Which platform are you using? Is it open source or a vendor solution? We’re evaluating options and I’m curious about real-world experiences.

  2. How does it handle dynamic systems? Our infrastructure autoscales constantly—new GPU pods spinning up, old ones shutting down. Traditional correlation breaks down when the system topology is constantly changing.

  3. Can you override it? When an engineer’s gut says “something’s different this time,” can they easily bypass the AI grouping and see raw alerts?

Don’t get me wrong—I think AI has a huge role to play in observability. I just want to make sure we’re not creating new problems while solving old ones. Over-reliance on AI without understanding its limitations can be dangerous, especially in production systems where mistakes are expensive.

Would love to hear how you’re thinking about these edge cases.

4

This is fascinating from a mobile observability perspective—our alert noise problem is even worse than what you’re describing.

The mobile multiplier effect

When you’re dealing with millions of devices across hundreds of network conditions, alert fatigue takes on a whole new dimension. At Uber, we’re tracking metrics from apps running in São Paulo on 3G networks, San Francisco on 5G, and everything in between. A “slow API response” alert could be:

  • Actual backend degradation
  • Regional network issue in Brazil
  • A specific Android manufacturer’s battery optimization killing background requests
  • Or just someone riding the subway through a tunnel

Our mobile team was getting absolutely hammered with alerts that turned out to be client-side environmental issues, not actual problems we could fix.

The cross-platform correlation challenge

Your AI correlation approach sounds promising, but I’m curious: does it work across client-side and backend signals? That’s where we struggle most.

For example, if we see a spike in mobile app crashes AND a spike in backend 500 errors, is that:

  1. Backend issue causing app crashes (causal)
  2. App crashes causing more retries, overloading backend (reverse causal)
  3. Coincidental and unrelated

Traditional time-based correlation gets this wrong constantly. We’ve been exploring whether ML could understand the actual request flow to determine directionality.

Regional diversity adds complexity

The other challenge for global mobile products: what’s “normal” varies wildly by region. Network latency that would be an incident in San Francisco is Tuesday afternoon in Jakarta. We need alert correlation that understands geographic context—does your AI platform handle multi-region complexity?

Mobile-specific metrics the AI needs to understand

I’m curious if AI correlation platforms are thinking about mobile-specific signals:

  • Battery drain patterns
  • Crash-free session rates
  • App start time by device tier
  • Background task success rates on different OS versions

These metrics interact in complex ways. High battery drain might predict higher crash rates 2-3 days later as users force-stop the app. Can AI learn these longer-term patterns?

Our current workaround

Right now we do manual tiering: critical alerts go to on-call immediately, everything else goes to a daily digest that gets triaged during business hours. It’s better than nothing, but we’re still missing the intelligent correlation you’re describing.

If you extend this to mobile observability, I’d love to be a case study. The pain is real, and I think the mobile space is even more ready for AI-powered solutions than backend infrastructure.

Great post—this gives me some concrete ideas to bring to our platform team.

5

As someone currently on the receiving end of on-call alerts, this sounds like an absolute dream.

Last month I was on-call for six microservices, and I had a night where I got woken up at 2AM for a database connection pool alert, then again at 2:15 for a downstream service timeout alert, then at 2:30 for a cache miss rate spike. All three turned out to be symptoms of the same issue: a deployment that introduced a slow query.

By the time I figured out the root cause, I was wide awake and couldn’t fall back asleep. The next day I was useless.

Practical questions from someone who would actually use this:

  1. Implementation timeline: How long did it take from decision to having it actually running in production? I want to bring this to my team lead, but if it’s a 6-month project, that’s a hard sell.

  2. Smaller team approach: We’re not a 40-person engineering org. Is there a lighter-weight version of this for teams our size? Or do you need a certain volume of alerts for the AI to have enough data to learn from?

  3. OpenTelemetry integration: We’re in the middle of migrating to OTel for our observability stack. Does the AI platform integrate with OpenTelemetry data, or does it require proprietary instrumentation?

The human side

Reading through the thread, I appreciate the leadership perspective from @vp_eng_keisha and the technical skepticism from @alex_infrastructure—both are important. But from an individual contributor standpoint, what I care about most is: will this let me actually sleep through the night?

The current situation is unsustainable. I’m watching several senior engineers on my team start to disengage. One told me privately they’re updating their resume because they can’t handle the on-call stress anymore.

A tangent on alert hygiene

One thing I’ve noticed: a lot of our alert noise comes from alerts that were set up 2-3 years ago and never revisited. Someone left the company, nobody owns the alert, but it keeps firing. Do you do any kind of alert cleanup as part of this process? Like, “this alert hasn’t led to any action in 6 months, maybe we should delete it”?

Anyway, this gives me hope. Going to share this thread with my manager and see if we can at least start evaluating options. Thanks for writing this up, @eng_director_luis.