Too many alerts: can AI cut the noise?

Every on-call engineer knows the feeling. Your phone buzzes. You check it: CPU at 85% on a node that's been at 85% for a year. Buzz again: a disk warning that resolves itself in two minutes. Buzz: ten alerts for the same database problem, one from each service that uses it.

After a few weeks of this, people stop reacting. The one alert that really matters then gets ignored too. That's alert fatigue, and it's dangerous.

AIOps tools promise to fix it with machine learning. Do they?

Why static thresholds fail

Most alerts are built like this: "if metric X is above Y for Z minutes, page someone." Simple and predictable. Also frequently wrong:

• Normal changes over time. Traffic at 3 p.m. on Monday looks nothing like 3 a.m. on Sunday. One threshold can't fit both.
• Thresholds are guesses. Someone picked 80% years ago, and nobody remembers why.
• Causes, not symptoms. High CPU isn't a problem if users are happy. Slow checkout is a problem even if CPU is fine.
• One failure, many alerts. A database outage triggers alerts in every dependent service.

What AIOps tools promise

"AIOps" is a broad marketing term, but it usually covers a few real features:

• Anomaly detection. Instead of fixed thresholds, the tool learns what's normal for each metric, including daily and weekly patterns, and alerts on unusual behaviour.
• Correlation and grouping. Related alerts from different services are grouped into one incident, ideally with a guess at the root cause.
• Deduplication and suppression. Flapping alerts and duplicates are merged or muted.
• Summaries. With LLMs, many tools now write a plain-language explanation of what's going on.

Datadog, Dynatrace, New Relic, Grafana, PagerDuty, BigPanda and others all offer some version of these features.

What they actually deliver

In my experience, honestly: some real value, and less magic than the sales deck.

Grouping and correlation is where I see the most benefit. Turning forty alerts into one incident with a timeline is a big improvement for whoever's on call.

Anomaly detection is useful for metrics with strong patterns, like traffic and business numbers, and noisy for everything else. "Anomalous" doesn't mean "bad". A marketing campaign is an anomaly too.

AI summaries help get oriented faster, with the usual caveat: check the evidence.

AI can sort the noise, but it can't decide what you should care about. That part is still your job.

Fixing the basics first

Before buying anything, most teams can remove half their alert noise with discipline alone:

• Alert on symptoms users feel: error rate, latency, availability. Ideally based on SLOs and burn rates.
• Every page must need a human action now. If not, make it a ticket or a dashboard, not a page.
• Every alert has an owner and a runbook.
• Group and inhibit in your alerting tool. Alertmanager can suppress downstream alerts when the upstream one is firing.
• Review alerts monthly. Which ones fired? Which were useful? Delete or fix the rest.

AIOps on top of a messy alert setup just gives you organised mess.

Measuring if an alert was worth it

The metric I like most is actionability rate: of all pages last month, how many led to someone actually doing something? If it's below about half, you have a noise problem.

Other useful numbers: pages per on-call shift, pages outside working hours, and time to acknowledge. If those trend in the right direction, whatever you changed is working, with or without AI.

So, can AI cut the noise?

Partly. It's genuinely good at grouping and summarising. But the biggest wins still come from deciding, as a team, what deserves to wake someone up. Do that first, then let AI help with the rest.

My take: the quietest pager comes from good alert design. AI is a helpful filter, not a substitute for deciding what matters.

— Desly