# How we draft incident postmortems

When an incident resolves, an agent pulls the timeline from the incident channel, correlates deploys and log spikes, and drafts a structured postmortem as a doc PR for the team to review and edit.

Canonical page: https://openchatly.com/use-cases/incident-postmortem

A postmortem is worth writing and easy to skip. Once an incident is resolved and
the pressure is off, someone has to reconstruct the timeline from a scrolling
channel, line it up against the deploys and log spikes, work out the root cause,
and write it all up. It's an hour of careful work at the moment everyone most
wants to move on, so it often gets a thin summary or nothing.

We run an agent on Kortix that drafts the first version. When an incident
resolves, it reads the incident channel, correlates the deploys and log spikes,
and opens a structured postmortem as a doc PR. It drafts; humans review and
finalize. This is how we write our own postmortems.

- **Team:** Kortix
- **Runs on:** Incident resolved, the agent drafts
- **Connected systems:** Incident channel · Logs · GitHub
- **Mode:** Drafts a doc PR, humans finalize

## The problem

The facts of an incident are scattered across places that don't line up on their
own: the back-and-forth in the incident channel, the deploy history, and the log
spikes. Turning them into a timeline means reading the channel top to bottom,
matching each moment against what shipped and what the logs did, and inferring
where the root cause sits.

The common workarounds are thin. A blank template gets filled in from memory, so
the timeline drifts and details get rounded off. A resolved incident with no
writeup means the same failure mode can recur with nothing to point back to. The
information to do it properly exists; assembling it by hand is exactly the work
no one wants right after an incident.

## What we built

On Kortix, resolving an incident triggers an agent. It reads the incident channel
for the timeline, pulls the deploy history and log spikes for the same window,
correlates them, and drafts a structured postmortem, timeline, root cause, and
action items, as a doc PR against the repo. The team reviews and edits the draft
the way they'd review any change. The agent drafts; it never publishes a final
postmortem on its own.

## How it works

### Copy the setup prompt into a new session

Open your Kortix workspace, start a new session, paste this prompt and send it.
The agent writes the agent file, the postmortem-draft skill and the
15-minute trigger, and opens a change request for you to review.
Prefer clicks? Choose **Use this template** instead.

**Setup prompt**

```
Set up an incident postmortem agent in this workspace.

1. Create the agent agents/incident-postmortem.md. Its job is to draft a postmortem for each resolved incident in #incidents and open it as a pull request in owner/repo. It may use the GitHub and Datadog connectors, the Slack channel and the GH_TOKEN secret.

2. Create the skill skills/postmortem-draft/SKILL.md with these rules:
   - Skip an incident that already has a postmortem pull request or branch. One pull request per incident.
   - Use the incident thread in #incidents as the timeline, from declaration to resolution.
   - Line up GitHub merges and Datadog error-rate and latency spikes against the same window on one clock.
   - Write under postmortems/ in the format of past postmortems: summary, timeline, proposed root cause, impact and candidate action items.
   - Mark the root cause as an inference. Never assign owners, close the incident or merge the pull request.

3. Add a cron trigger resolved-incident-sweep to kortix.yaml. It runs the incident-postmortem agent every 15 minutes with the prompt "Load the postmortem-draft skill and check #incidents for incidents marked resolved since the last sweep."

4. Tell me which of these connectors are not connected yet.

5. Open a change request with all of it. Do not merge it.
```

### Connect your tools

Open **Connectors** in your workspace and connect GitHub and Datadog, add a
GitHub token as the `GH_TOKEN` secret, and connect Slack under **Channels**.
Credentials are brokered server-side and never enter the agent's machine.

- **GitHub** (Required):
Reads deploys and merges and opens the postmortem pull request.
- **Datadog** (Required):
Reads log spikes over the incident window.
- **Slack** (Required):
Reads the incident channel timeline for resolved incidents.

### Run it once with this prompt

Merge the change request, then start a session with the **incident-postmortem** agent
and send this prompt. Pick a recently resolved incident and read the
draft before you let the schedule take over.

**Prompt**

```
Check #incidents for incidents marked resolved and draft their postmortems.

1. For each resolved incident, check owner/repo for an existing postmortem pull request or branch, and skip it if there is one.
2. Read the full incident thread as the timeline.
3. List the GitHub merges and the Datadog error-rate and latency spikes in the incident window.
4. Draft the postmortem under postmortems/ in our format, with the root cause marked as proposed.
5. Open one pull request per incident. Do not assign owners, close the incident or merge.
```

### Let the 15-minute schedule take over

The change request added this trigger to `kortix.yaml`. From now on the agent
runs every 15 minutes, each run in a new session on its own cloud computer.
To change the time, edit the cron line, or open **Triggers** in the workspace.

**kortix.yaml**

```yaml
triggers:
  - slug: resolved-incident-sweep
    name: Resolved incident sweep
    on:
      cron: "0 */15 * * * *"
    run:
      agent: incident-postmortem
      prompt: Load the postmortem-draft skill and check #incidents for incidents marked resolved since the last sweep.
```

### Propose the root cause, never finalize it

The draft states the root cause and the action items as proposals, with the
evidence listed. The action items have no owners. The channel, GitHub history
and Datadog are read-only sources, and the pull request is the one write. The
team corrects the draft, sets owners and merges the version they stand
behind.

### Read the postmortem draft

Each resolved incident ends with one pull request. This is what a run looks like in the session:

> **Resolved incident sweep · 16:15**
>
> Prompt: Load the postmortem-draft skill and check #incidents for incidents marked resolved since the last sweep.
>
> Found **1** newly resolved incident, open for **42** minutes. The error rate
> rose **3** minutes after a merge to the checkout service, so the draft
> proposes that merge as the root cause. Opened one pull request with **4**
> candidate action items and no owners.

> **The pattern**
> A resolved incident on the **channel** trigger spawns a session with read-only
> **connectors** into the channel, the logs, and GitHub. The format lives as
> **skills** and **memory**. The agent correlates the timeline and opens a doc
> PR; humans finalize.

## Make it do more

Once it works, paste any of these into the same session.

**Share the draft**

```
When you open a postmortem pull request, post its link and the one-paragraph summary in the incident thread.
```

**Find repeat causes**

```
Each month, read the merged postmortems and list root causes that appeared more than once, with links.
```

**Track action items**

```
After a postmortem pull request merges, file each action item as an unassigned GitHub issue linked to it.
```

## Guardrails

The agent reads an incident channel, logs, and deploy history and writes a
document others will act on, so the access is scoped and contained:

- **Isolation.** Every incident runs in its own isolated sandbox. The session
  reads the timeline, correlates the deploys and logs, and only the drafted doc
  PR is written back out.
- **Scoped secrets.** The channel, logs, and GitHub credentials are encrypted in
  the Secrets Manager and injected into the sandbox at runtime, scoped to the agents you grant them to.
- **PR-gated.** The postmortem lands as a doc PR, not a published document. A
  human reviews the timeline, edits the root cause and action items, and owns the
  merge.
- **Everything is code.** The postmortem format and the per-tool permissions are
  files in the repo, versioned and changed through a reviewed **change request**
  rather than a dashboard setting.

## The outcome

- **Every incident:** A first draft the moment it resolves
- **PR-gated:** The agent drafts, the team reviews and finalizes
- **3 systems:** Incident channel, logs, and GitHub in one agent

Resolved incidents now come with a drafted postmortem waiting as a doc PR, its
timeline already correlated against the deploys and log spikes. The team spends
its time judging the root cause and deciding the action items rather than
reconstructing what happened, and fewer incidents close with no writeup at all.
