Can AI help at 3 a.m.? AI in incident response

In Will AI take my DevOps job? I said two things about incidents. AI is very good at reading logs. And humans still have to own the decision. This post goes deeper into what that looks like in practice, step by step through an incident.

A typical incident, with AI in the loop

Let's walk through a realistic scenario. Error rates on the checkout service jump at 03:12. The on-call engineer gets paged.

Minute 0 to 5: understanding what's happening. Without AI, this means opening dashboards, checking recent deployments, grepping logs. With an AI assistant connected to observability tools (through MCP or a vendor integration), you can ask: "What changed in the checkout service in the last hour, and what do the error logs say?"

A good assistant will come back with something like: a deployment at 03:05, a new error appearing at 03:11 (connection timeouts to the payments database), and a spike in database connection count. In a few seconds, it did what used to take ten minutes of clicking.

Minute 5 to 15: forming a hypothesis. The AI suggests the new release changed the connection pool settings. It points to the diff. That's useful, and it might be right.

Minute 15: the decision. Roll back the release, or increase the database connection limit? Wake up the database team? Tell customers? This is where the human takes over. The AI can list options and their risks. It can't take responsibility for choosing.

AI shortens the time to understand an incident. It doesn't shorten the time it takes to be accountable for the fix.

What AI is genuinely good at during incidents

• Summarising huge amounts of data. Thousands of log lines, dozens of alerts, multiple dashboards, condensed into "here's what's unusual".
• Correlating with changes. Linking errors to recent deployments, config changes or feature flag flips.
• Searching past incidents. "Have we seen this error before?" across old postmortems and tickets.
• Keeping the timeline. Recording what happened, who did what, and when, while people focus on fixing.
• Drafting status updates. A clear, calm customer-facing message, ready for a human to approve.

What it gets wrong

I've seen AI make the same kinds of mistakes in incident scenarios:

• Blaming the most recent change even when it's a coincidence. Correlation is not cause.
• Missing the real cause outside its view. If it can only see application logs, it won't notice the cloud provider's network issue.
• Hallucinating details: a config value or a metric that doesn't exist.
• Sounding certain when the evidence is thin.

So the rule for on-call engineers: treat AI conclusions as hypotheses to verify, not answers to act on. Ask it to show the evidence: the log line, the metric, the diff.

Checking AI's conclusions

A simple routine that works well:

1. Ask the AI what it thinks happened.
2. Ask it for the specific evidence behind each claim.
3. Check that evidence yourself in the original tool.
4. Ask it what else could explain the same symptoms.

That fourth question is underrated. It helps you avoid locking onto the first theory, which is a classic human mistake during incidents too.

Writing the postmortem together

After the incident, AI is a big help with the part nobody enjoys: writing it up.

Give it the incident channel, the timeline and the relevant graphs, and ask for a first draft of a blameless postmortem: summary, impact, timeline, root cause, contributing factors, action items. Then edit it properly. The AI gives you structure, but the insights about why the system allowed this to happen need human thinking.

Tools worth a look

Many incident and observability platforms now include AI features: incident.io, Rootly, PagerDuty, Datadog, Grafana, New Relic and others. There's also a new wave of "AI SRE" products that investigate alerts on their own. Try them on past incidents first: replay an old incident and see whether the AI would have found the cause.

So, can AI help at 3 a.m.?

Yes, a lot. It makes the first fifteen minutes of an incident much faster and less stressful. But the person holding the pager still makes the calls, and that's how it should be.

My take: let AI do the reading, and keep the deciding for yourself. Especially at 3 a.m.

— Desly