When things break: incidents and blameless postmortems

Every system breaks eventually. What separates good teams from struggling ones isn't whether they have incidents. It's how they handle them: calmly or chaotically, and whether they learn something afterwards or just move on and wait for the next one.

Here's the incident process I prefer, and how to write postmortems that actually improve things.

Before the incident: decide how you'll work

Incidents are stressful. Stress makes people bad at improvising. So decide the basics in advance:

• Severity levels, with clear examples.
• Who declares an incident, and how (a chat command, a button, a phone call).
• Where people gather: a dedicated chat channel per incident, plus a call if needed.
• Roles, so everyone knows who does what.
• Communication templates, so updates go out quickly and consistently.

A simple severity scale:

Severity	Meaning	Example
SEV1	Major impact on many users or core business	Checkout down for everyone
SEV2	Significant impact, workaround exists or partial outage	Payments failing for one provider
SEV3	Minor impact, limited users	A non-critical feature broken

When in doubt, declare higher. It's easy to downgrade.

Who does what in an incident

The most important role is the Incident Commander (IC). The IC doesn't fix the problem. They coordinate: make sure the right people are involved, decisions are made, and nothing falls through the cracks.

Other common roles:

• Operations / subject matter experts: the people actually investigating and fixing.
• Communications lead: writes updates for stakeholders and customers, so the engineers can focus.
• Scribe: keeps a timeline of what happened and what was decided. (AI tools increasingly help here.)

In a small team, one person might take on multiple roles. That's fine, as long as someone is clearly in charge.

In an incident, the most dangerous situation is five smart people all fixing things at once, without anyone knowing what the others changed.

During the incident

A few habits that make a big difference:

• Mitigate first, investigate later. Roll back, fail over, scale up, turn off the feature flag. Find the root cause once users are okay.
• Say what you're doing before you do it: "I'm going to restart the payment workers now." It avoids conflicting actions.
• Write things down in the channel: observations, hypotheses, actions, results.
• Regular updates, even when there's nothing new: "Still investigating, next update in 30 minutes."
• Know when to escalate. Pulling in another team early is better than struggling alone for an hour.

A simple status update template:

[SEV2] Payment failures, investigating. Since 14:05 CET, around 10% of card payments fail. Other payment methods work. We're investigating a possible issue with our payment provider. Next update by 14:45.

After the incident: blameless postmortems

A postmortem is a written analysis of what happened and how to prevent it. Blameless means the focus is on the system, not on who made a mistake.

Why blameless? Because people make mistakes all the time, and systems should be designed to handle that. If the takeaway from an incident is "Alex should have been more careful", nothing changes, except that people start hiding mistakes. If the takeaway is "our deploy process let an untested config reach production without a check", you can fix the process.

Questions to ask:

• What made this mistake easy to make?
• Why didn't our tests, reviews or monitoring catch it?
• What made it hard to detect, diagnose or fix?
• What went well? (Always include this.)

My postmortem template

1. Summary: two or three sentences a manager could understand.
2. Impact: who was affected, how, for how long. Numbers if possible.
3. Timeline: key events with timestamps, from the trigger to full resolution.
4. Root cause and contributing factors: usually several things combined, not one single cause.
5. Detection: how did we find out? How could we have found out sooner?
6. Response: what helped, what slowed us down.
7. What went well.
8. Action items: specific, with an owner and a due date.
9. Lessons learned.

Making sure action items happen

The most common failure of postmortems: great analysis, a list of action items, and then nothing happens. Fixes compete with feature work and lose.

What helps:

• Few, important action items instead of a long wish list.
• Tracked in the normal backlog, not a separate document.
• Reviewed regularly, for example in a monthly reliability meeting.
• Leadership support, so fixing the cause of an incident is considered real work.

Tools that help

incident.io, Rootly, FireHydrant, PagerDuty, Jira Service Management and others automate the process: creating channels, assigning roles, keeping timelines, posting status updates and generating postmortem drafts. Useful, but the process and culture matter more than the tool.

So, where do I start?

Agree on three severity levels, the incident commander role, and a postmortem template. Run a short practice incident (a "game day") so people know the process before a real one. Then, after the next real incident, write a blameless postmortem and follow through on its actions.

My take: incidents will happen. Handle them calmly, learn from them honestly, and fix the system instead of blaming the person.

— Desly