← All posts

Running an incident - from the first chat message to a postmortem that files work

A practical way to run an incident on a small team, from the moment someone says "is anything broken?" to a postmortem whose follow-ups are tracked issues. Declare early, keep an append-only timeline, ask what changed first, and end with work rather than a document.

Oct 3, 2026 · 9 min read · Kash Gohil · LinkedIn · X

It starts in a channel, nearly always. Somebody types "is CI slow for anyone else?" at 14:02. Two people react with the eyes emoji. A third says jobs are failing at the first secret read. By 14:20 four people are debugging in three threads and one call, and nobody has written anything down.

The fix arrives at 14:31. The postmortem gets written a week later, from memory, by whoever lost the argument. It lists six action items. Two of them get done.

None of this is about skill. The people in that channel are good at their jobs. The incident failed because it had no shape. This post is a shape. It works on any tool, and the last section shows how we built it into Rezee.

Why incidents go badly

Three failures repeat, and each one makes the next worse.

The record is rebuilt afterwards. During the incident everybody is busy, so nobody writes. Afterwards the timeline is put together from chat scrollback and guesses. The guesses are where the wrong conclusions come from.

Nobody asks what changed. Most production problems follow a change: a deploy, a config edit, a flag. Teams still start by reading logs, because logs feel like work. The first minutes go into a question that "what did we ship in the last hour?" answers at once.

The postmortem ends as a document. It has a good narrative and a list of action items. The items live in the document and nowhere else. Nobody owns them, nothing tracks them, and the same failure comes back in a quarter.

1. Declare early, and declare where it was reported

Declaring an incident should be cheap. The test is not "are we sure this is serious?" The test is "is something broken for users right now?" If yes, declare it. A false alarm costs one line. A real incident with no declaration costs the whole record.

A declaration needs three things:

• A title that says what users see, not what you think the cause is. "Jobs fail at the first secret read" is good. "Key store bug" is a guess.
• A severity. Use a fixed scale with four levels and write down what each one means for your product. Do not debate it for more than a minute. You can change it later, and the change is part of the record.
• Where it is happening. Which environment, and which service if you know.

Declare it from the message that reported it. The thread that already exists is where people are talking, so make that thread the incident's discussion. Do not open a new channel and ask everyone to move. Half of them will not see the message, and the first ten minutes of evidence stay behind in the old thread.

2. Keep one timeline, and never edit it

The timeline is the most valuable thing an incident produces. Treat it like a log, not like a document.

• Write what you saw, with the time. "14:09 - p95 claim time is 40s, was 2s" is useful. "Things look bad" is not.
• Write what you did. "14:11 - rolled runner production back to v0.9.3."
• Write what you decided, and who decided it.
• Never edit an entry. If an entry is wrong, add a new one that corrects it. The wrong belief is evidence too. It shows what the team thought at 14:09, which is exactly what the postmortem has to explain.

On a team of more than three, give one person the job of writing it down. That person does not debug. It feels like a waste of an engineer. It is the opposite: it is the reason the next hour of work produces a record.

3. Ask what changed, before you debug

The first technical question in any incident is: what changed just before it started? Look at deploys first, then config, then traffic.

If a deploy went out in the window, name it on the incident. Then consider rolling it back before you understand the bug. A rollback is a guess you can test in minutes. Debugging a live system under pressure is a guess you test in hours. If the rollback fixes it, you have mitigated the incident and found your suspect at the same time.

Keep every deploy during an incident visible. Every deploy into the affected environment, including the rollback and including somebody's unrelated hotfix, belongs on the timeline. A deploy that nobody mentioned is the most common way a second problem hides inside the first.

4. One alert, one incident

If your monitoring opens incidents for you, make sure the same alert cannot open two. An alert that flaps should add to the incident it already opened, not start a new one each time it fires. When the monitor says the alert resolved, that is an entry on the timeline, not a separate event somebody has to match up by hand.

The rule is simple. An alert has an identity. While an incident for that identity is open, everything that alert says goes onto that incident.

5. Mitigate, then resolve, and keep both times

These are different moments, and you should record them separately.

Mitigated means users stopped being hurt. Resolved means it is over and you do not expect it back. The gap between them is often hours, or days. Rolling back mitigates. Finding and fixing the bug that made the deploy bad resolves.

Keep the first mitigation time even if the incident gets worse again later. That is when the bleeding first stopped. If the incident reopens, the next resolution is the one recovery is measured to.

6. Write the postmortem from the timeline, the same week

Start the postmortem from the record, not from a blank page. If your timeline is good, half the document already exists. Copy it in, then write the parts only people can write:

• Summary. What happened and why, in two or three sentences.
• Impact. Severity, where, and when it opened, was mitigated and was resolved.
• Timeline. From the record, untouched.
• What went well, and what did not. Blameless. Ask what made the wrong action look right at the time, not who took it.
• Follow-ups. Each one a separate piece of work, in a form that can be tracked.

That last part is the one teams get wrong. A follow-up written as a bullet in a document is a wish. Make each follow-up something your tracker can hold: a title, a description, an owner, and a link back to the incident. Then have somebody other than the author approve the postmortem before the work is filed. The second reader is what catches "improve monitoring" and turns it into something a person can actually start.

7. Close the loop in your delivery numbers

If you track change failure rate, an incident is a reason a change failed. Connect them. Six weeks later, a failed change should open to the incident it caused. From there you should reach the postmortem and the issues that came out of it. A failure rate with no stories behind it is a number people learn to ignore.

What to drop

• The war-room call with no scribe. Talking is faster than typing, and nothing said on a call survives it.
• Root cause, singular. Incidents have contributing causes. Asking for one root cause ends the analysis at the first plausible answer.
• Action items with no owner. If it has no owner, it is not an action item.
• The postmortem meeting with no draft. Read a draft built from the timeline. Do not reconstruct the timeline in the meeting.

How Rezee does it

We built Rezee's incidents around the steps above. Every mechanism here is in the product today.

Declaring. An incident has a title, a severity from sev1 to sev4 (default sev3) and an environment. It belongs to the workspace, not to one repository, because outages cross repositories. You can declare one from any message in a chat room. That message becomes its first entry, and its thread becomes where the incident is discussed. The thread is also told in plain words that an incident now exists, with a link. You can also declare one from a failed deployment, or straight after a rollback. Then the dialog is prefilled and the incident names that deployment and takes its environment.

The timeline. Incidents live under Ship. Each note, status change, severity change and link is its own entry. A database trigger refuses any update to an entry. So the record reads the same in the review as it did while the incident was open. Mitigate, Resolve and Reopen sit beside the note that explains them, and pressing Resolve twice does not write two resolutions.

What changed. An incident can name the deployment it followed, at declare time or later. Every deployment into the incident's environment, and every deployment the incident names, is written onto its timeline as it starts and as it ends. An environment's history then shows which incident a deployment is in.

Alerts. An outside monitor can open, add to and close incidents through one endpoint. It accepts our own shape, or an Alertmanager or Grafana webhook as sent. Each incident remembers the monitor and the alert's key, and a unique index allows one live incident per alert. A repeat firing adds to the open one. A resolve from the monitor resolves it here. Scripts can do the same with rezee incident report.

The postmortem. One click, or rezee incident postmortem 12 from the CLI, starts a private draft page titled "Postmortem:" plus the incident's title. Its impact and timeline sections are already filled from the record, along with what the incident links to. The summary and the review are headings with a prompt under each, because only people can write them. Each level-two heading on that page becomes a follow-up issue once the page has a project and somebody other than its author approves it. The issues link back to the incident. This is the same rule we use for design reviews: a decided page files the work, and a draft does not.

The delivery numbers. Ship's stability figure lists each change that failed or was rolled back. Each one names the incident behind it, or the incident naming the deployment that fixed it. Where there is none, it offers to declare one.

Agents. An agent with the On-call kit can list, read, declare and update incidents through MCP, using the same tools a person's session uses. Its entries go on the same append-only timeline, under its own name.

One thing Rezee does not do is page anyone. It records. Getting a person out of bed at 3am is the job of your monitor or your paging tool, and Rezee takes the alert from there.

The test

Six weeks after an incident, ask three questions. What happened? What changed just before it? Did the follow-ups ship?

If the answers are a timeline nobody edited, a named deployment, and closed issues that point back at the incident, the process worked. If the answers come from somebody's memory, you will run the same incident again, and you will not know it is the same one.