Get started

How we triage on-call alerts before paging a human

The triage agent we run on Kortix — connected to Sentry, our logs, and GitHub. It works up a first-pass diagnosis on every alert and only pages a human when it can't resolve it.

×
GitHub
Written by
The OpenChatly Team
Published
February 25, 2026
Reading time
7 min read
Topics
SRE, Case Study, Enterprise

An alert fires at 3am. Before anyone can act on it, someone has to wake up, pull the stack trace, check what deployed recently, grep the logs, and work out whether this is a real incident or noise. Most of that is mechanical, and most of it happens before the human has any context — which is the worst time to be doing it.

We run a triage agent on Kortix that does the first pass. When an alert fires, the agent gathers the stack trace, the recent deploys, and the correlated logs, posts a first-pass diagnosis to the incident channel, and only pages a human when it can't resolve the alert or the severity is high. This write-up covers how the setup works: the trigger, the session model, and the guardrails.

Team
Kortix
Runs on
Every alert, as it fires
Connected systems
Sentry · Logs · GitHub
Mode
Trigger-driven · pages a human only when needed

The problem

The first minutes of an incident are spent gathering context, not fixing anything. Which error is this, when did it start, what shipped just before, is it one user or all of them — the on-call engineer answers these by hand, half-awake, before they can even judge whether the page was warranted.

The common fixes are incomplete. Paging on every alert burns the on-call rotation on noise and trains people to ignore the pager. Tuning thresholds cuts the noise but also hides real regressions. A runbook helps, but someone still has to be awake to follow it. None of it does the gathering for you.

What we built

On Kortix, each alert triggers an agent. When Sentry fires, the alert spawns an isolated session (a cloud sandbox) with scoped, read-only access to Sentry, the logs, and GitHub. The agent pulls the stack trace, lists the deploys since the error first appeared, correlates the logs around the spike, and posts a first-pass diagnosis to the incident channel. It pages a human only when it can't resolve the alert or the severity is high.

How it works

1

Copy the setup prompt into a new session

Open your Kortix workspace, start a new session, paste this prompt and send it. The agent writes the agent file, the incident triage skill and the 5-minute trigger, and opens a change request for you to review. Prefer clicks? Choose Use this template instead.

Setup prompt

Set up an on-call triage agent in this workspace.

  1. Create the agent agents/oncall-triage.md. Its job is to triage new Sentry alerts and post a first-pass diagnosis before anyone is paged. It may use the Sentry, Datadog and GitHub connectors, read-only, and post to Slack.

  2. Create the skill skills/incident-triage/SKILL.md with these rules:

    • For each new or unresolved alert, pull the stack trace, the frequency and the first-seen time from Sentry.
    • Pull the Datadog logs from 10 minutes before to 5 minutes after first-seen, for the services in the trace.
    • Check the PRs and commits merged to owner/repo in the 2 hours before first-seen. Name the closest match as the suspect deploy, not as a certainty.
    • Page the on-call rotation in #incidents for any alert at error severity or above, and for any alert you cannot resolve.
    • Post exactly one diagnosis per alert. Stay read-only: never comment in Sentry, commit, deploy or roll back.
  3. Add a cron trigger alert-triage-sweep to kortix.yaml. It runs the oncall-triage agent every 5 minutes with the prompt "Load the incident-triage skill and triage every new or unresolved alert in the backend Sentry project."

  4. Tell me which of these connectors are not connected yet.

  5. Open a change request with all of it. Do not merge it.

2

Connect your tools

Open Connectors in your workspace and connect your error tracker, your logs, your code host and Slack. Credentials are brokered server-side and never enter the agent's machine.

Sentry

Pulls the stack trace and error spike for each alert.

Or useDatadogNew Relic
Required
GitHub

Finds the deploy that shipped just before the spike.

Or useGitLab
Required
Slack

Posts the first-pass diagnosis to the incident channel.

Or useMicrosoft TeamsPagerDuty
Required
3

Run it once with this prompt

Merge the change request, then start a session with the oncall-triage agent and send this prompt. Check its suspect deploy against what you know before you let the schedule take over.

Prompt

Triage every new or unresolved alert in the backend Sentry project.

  1. For each alert, pull the stack trace, the frequency and the first-seen time from Sentry.
  2. Pull the Datadog logs around first-seen for the services in the trace.
  3. List the PRs merged to owner/repo in the 2 hours before first-seen and name the most likely suspect.
  4. Post one diagnosis per alert to #incidents: the error, the logs, the suspect deploy and the severity.
  5. Page the on-call rotation for anything at error severity or above, or anything you cannot resolve. Close benign alerts with the reasoning. Change nothing.
4

Let the 5-minute schedule take over

The change request added this trigger to kortix.yaml. From now on the agent runs every 5 minutes, each run in a new session on its own cloud computer. To change the time, edit the cron line, or open Triggers in the workspace.

kortix.yaml
triggers:- slug: alert-triage-sweep  name: Alert triage sweep  on:    cron: "0 */5 * * * *"  run:    agent: oncall-triage    prompt: Load the incident-triage skill and triage every new or unresolved alert in the backend Sentry project.
5

Page a person on high severity or doubt

Anything at error severity or above pages the on-call rotation, however confident the diagnosis is. An inconclusive diagnosis pages too, with the partial evidence attached. The agent is read-only across Sentry, Datadog and GitHub. It never deploys, rolls back or changes a setting, and the person who is paged owns every fix.

6

Read the diagnosis

Each alert ends with one diagnosis in the channel. This is what a run looks like in the session:

Alert triage sweep · 03:05
Load the incident-triage skill and triage every new or unresolved alert in the backend Sentry project.
Worked for 1m 34s
sentry · list issuessentry · get event × 2datadog · search logs × 2github · list pull requestsslack · post message → #incidents × 2

Triaged 2 alerts. A spike of 312 checkout timeouts started 4 minutes after a merged PR that changed the payment client, so it paged on-call with that PR as the suspect. The other alert matches a known third-party rate limit and closed with the reasoning attached.

Illustrative example. Your alerts depend on your services.

The pattern

A trigger on every alert spawns a session with scoped, read-only connectors into Sentry, the logs, and GitHub. The triage playbook is encoded as skills and memory. The agent diagnoses first and pages a human only when it can't resolve the alert or the severity is high.

Make it do more

Once it works, paste any of these into the same session.

Keep an incident timeline

When an alert pages on-call, keep a running timeline of what you found in the Slack thread so the postmortem starts from it.

Learn known flakes

When an engineer replies that a diagnosis was benign, propose a change to the incident-triage skill that records the pattern as a known flake.

Morning digest

Every weekday at 09:00, post the alerts you closed overnight, with the reasoning for each, to #incidents.

Guardrails

The agent reads production telemetry, so the access is scoped and contained:

  • Isolation. Every alert runs in its own isolated sandbox. The session can pull traces, logs, and deploy history to build the diagnosis; only the posted message and any page are written back out.
  • Scoped secrets. The Sentry, logging, and GitHub credentials are encrypted in the secrets manager and injected into the sandbox at runtime, scoped to the agents you grant them to.
  • Human approval gate. The agent never deploys or rolls back. High-severity and unresolved alerts page a human, who owns any action taken.
  • Everything is code. The agent's configuration, skills, and permissions are files in the repo, versioned and changed through a reviewed change request rather than a dashboard setting.

The outcome

Diagnosis in the channel before the pager fires
First-pass
Benign alerts closed with reasoning, not paged
Less noise
Sentry, logs, and deploy history in one agent
3 systems

The mechanical first minutes of an incident happen before anyone is paged, and when the agent does page, it pages with the trace, the suspect deploy, and the correlated logs attached. The on-call engineer spends their time deciding and fixing rather than gathering context half-awake.

On this page

  • The problem
  • What we built
  • How it works
  • Make it do more
  • Guardrails
  • The outcome

Run this yourself

Install this exact setup — agent, connectors, schedule, and guardrails — in a guided flow.

More use cases.

  • How we handle GDPR data-subject requests on a deadline

    Legal·July 11, 2026

  • OpenChatly

    How we keep our docs in sync with the code

    Documentation·July 5, 2026

  • OpenChatly

    How we QA every pull request automatically

    QA·July 5, 2026

Own your AI.Start today, in our cloud.

Start for free

OpenChatly is the AI workspace for everything. One person directing, agents executing, from goal to finished work.

©2026 OpenChatly AI.沪ICP备2025124785号-5

Resource guide

  • Product overview
  • Skills marketplace

Terms of service

  • User agreement
  • Privacy policy

Contact us