How we handle GDPR data-subject requests on a deadline
Legal
The triage agent we run on Kortix — connected to Sentry, our logs, and GitHub. It works up a first-pass diagnosis on every alert and only pages a human when it can't resolve it.
An alert fires at 3am. Before anyone can act on it, someone has to wake up, pull the stack trace, check what deployed recently, grep the logs, and work out whether this is a real incident or noise. Most of that is mechanical, and most of it happens before the human has any context — which is the worst time to be doing it.
We run a triage agent on Kortix that does the first pass. When an alert fires, the agent gathers the stack trace, the recent deploys, and the correlated logs, posts a first-pass diagnosis to the incident channel, and only pages a human when it can't resolve the alert or the severity is high. This write-up covers how the setup works: the trigger, the session model, and the guardrails.
The first minutes of an incident are spent gathering context, not fixing anything. Which error is this, when did it start, what shipped just before, is it one user or all of them — the on-call engineer answers these by hand, half-awake, before they can even judge whether the page was warranted.
The common fixes are incomplete. Paging on every alert burns the on-call rotation on noise and trains people to ignore the pager. Tuning thresholds cuts the noise but also hides real regressions. A runbook helps, but someone still has to be awake to follow it. None of it does the gathering for you.
On Kortix, each alert triggers an agent. When Sentry fires, the alert spawns an isolated session (a cloud sandbox) with scoped, read-only access to Sentry, the logs, and GitHub. The agent pulls the stack trace, lists the deploys since the error first appeared, correlates the logs around the spike, and posts a first-pass diagnosis to the incident channel. It pages a human only when it can't resolve the alert or the severity is high.
Open your Kortix workspace, start a new session, paste this prompt and send it. The agent writes the agent file, the incident triage skill and the 5-minute trigger, and opens a change request for you to review. Prefer clicks? Choose Use this template instead.
Set up an on-call triage agent in this workspace.
Create the agent agents/oncall-triage.md. Its job is to triage new Sentry alerts and post a first-pass diagnosis before anyone is paged. It may use the Sentry, Datadog and GitHub connectors, read-only, and post to Slack.
Create the skill skills/incident-triage/SKILL.md with these rules:
Add a cron trigger alert-triage-sweep to kortix.yaml. It runs the oncall-triage agent every 5 minutes with the prompt "Load the incident-triage skill and triage every new or unresolved alert in the backend Sentry project."
Tell me which of these connectors are not connected yet.
Open a change request with all of it. Do not merge it.
Open Connectors in your workspace and connect your error tracker, your logs, your code host and Slack. Credentials are brokered server-side and never enter the agent's machine.
Pulls the stack trace and error spike for each alert.
Finds the deploy that shipped just before the spike.
Posts the first-pass diagnosis to the incident channel.
Merge the change request, then start a session with the oncall-triage agent and send this prompt. Check its suspect deploy against what you know before you let the schedule take over.
Triage every new or unresolved alert in the backend Sentry project.
The change request added this trigger to kortix.yaml. From now on the agent
runs every 5 minutes, each run in a new session on its own cloud computer.
To change the time, edit the cron line, or open Triggers in the workspace.
triggers:- slug: alert-triage-sweep name: Alert triage sweep on: cron: "0 */5 * * * *" run: agent: oncall-triage prompt: Load the incident-triage skill and triage every new or unresolved alert in the backend Sentry project.Anything at error severity or above pages the on-call rotation, however confident the diagnosis is. An inconclusive diagnosis pages too, with the partial evidence attached. The agent is read-only across Sentry, Datadog and GitHub. It never deploys, rolls back or changes a setting, and the person who is paged owns every fix.
Each alert ends with one diagnosis in the channel. This is what a run looks like in the session:
Triaged 2 alerts. A spike of 312 checkout timeouts started 4 minutes after a merged PR that changed the payment client, so it paged on-call with that PR as the suspect. The other alert matches a known third-party rate limit and closed with the reasoning attached.
Once it works, paste any of these into the same session.
When an alert pages on-call, keep a running timeline of what you found in the Slack thread so the postmortem starts from it.
When an engineer replies that a diagnosis was benign, propose a change to the incident-triage skill that records the pattern as a known flake.
Every weekday at 09:00, post the alerts you closed overnight, with the reasoning for each, to #incidents.
The agent reads production telemetry, so the access is scoped and contained:
The mechanical first minutes of an incident happen before anyone is paged, and when the agent does page, it pages with the trace, the suspect deploy, and the correlated logs attached. The on-call engineer spends their time deciding and fixing rather than gathering context half-awake.