Get started

How we groom our production error backlog

The error-triage agent we run on Kortix — connected to Sentry and GitHub. Hourly, it groups new and spiking errors, dedupes against our existing GitHub issues, and drafts an issue with the stack trace and impact for the top offenders, then alerts Slack. It never resolves, ignores, or assigns an error itself.

×
GitHub
Written by
The OpenChatly Team
Published
March 28, 2026
Reading time
8 min read
Topics
Engineering, Case Study, Enterprise

Sentry catches every error. Turning that stream into work someone actually picks up is a different problem. New error types pile up next to ones that have been spiking for days, half of them already have a GitHub issue somewhere and half don't, and by the time an engineer has the hour to sit down and sort it, the backlog is long enough that sorting it feels like its own project. Most weeks, no one has that hour, so the backlog just grows.

We run an error-triage agent on Kortix that grooms that backlog every hour. It reads the current state of Sentry, groups new and spiking errors, checks which ones already have a tracked GitHub issue, and drafts an issue with the stack trace and impact for the ones that don't — then posts a summary to Slack. This is grooming, not paging: it doesn't wake anyone up and it doesn't decide anything is fixed.

Team
Kortix
Runs on
Hourly cron
Connected systems
Sentry · GitHub · Slack
Mode
Drafts issues + alerts only · never resolves or assigns

The problem

A Sentry project accumulates issues faster than anyone triages them. Some are brand new. Some have been sitting unresolved for weeks at a low, steady rate. Some just started spiking after today's deploy. Working out which of those deserve an engineer's attention — and whether that attention already exists as an open GitHub issue — means reading the whole backlog by hand, which is exactly the task that gets deferred every time something more urgent comes up.

The common fixes don't groom the backlog, they just move it. Auto-creating a GitHub issue for every Sentry error floods the tracker with duplicates and one-offs no one will ever action. Leaving Sentry as the system of record means the backlog is only as current as the last time someone opened the dashboard. And on-call triage tools are built for the opposite end of the problem — one loud alert right now — not the quiet pile of errors that never paged anyone but still need a decision.

What we built

On Kortix, an hourly cron triggers an agent. It spawns a fresh session with read-only access to Sentry, pulls the errors that are new or whose event frequency has spiked over their trailing baseline, and groups them by fingerprint. For each one, it checks our GitHub repo for an existing issue before doing anything else. For the top offenders that aren't already tracked, it drafts a GitHub issue with the stack trace and the impact — event count, affected users, first seen — and posts a summary of what's new, what's spiking, and what got drafted to Slack. It never touches Sentry's issue state and never assigns anything.

How it works

1

Copy the setup prompt into a new session

Open your Kortix workspace, start a new session, paste this prompt and send it. The agent writes the agent file, the triage skill and the hourly trigger, and opens a change request for you to review. Prefer clicks? Choose Use this template instead.

Setup prompt

Set up an hourly error triage agent in this workspace.

  1. Create the agent agents/error-triage.md. Its job is to groom the Sentry backlog into a short list of GitHub issues. It may use the Sentry and Slack connectors and a GH_TOKEN secret for GitHub.

  2. Create the skill skills/error-backlog-triage/SKILL.md with these rules:

    • Read the web project in Sentry. Pick issues first seen in the last hour, and issues at 3x or more of their trailing hourly baseline.
    • Group issues by fingerprint. Rank the groups by event count times affected users.
    • Before you draft, search owner/repo for an issue that already has the Sentry issue ID. Skip anything already tracked.
    • Draft at most 5 GitHub issues per sweep, each with the stack trace and the impact. Never assign them.
    • Never resolve, ignore, mute or merge anything in Sentry.
    • Post one summary to #eng-errors: new, spiking, drafted and already tracked. Post nothing when the sweep finds nothing.
  3. Add a cron trigger hourly-error-backlog-sweep to kortix.yaml. It runs the error-triage agent every hour, on the hour, with the prompt "Load the error-backlog-triage skill and run one complete sweep of the web project."

  4. Tell me which of these connectors are not connected yet.

  5. Open a change request with all of it. Do not merge it.

2

Connect your tools

Open Connectors in your workspace and connect your error tracker and Slack. Credentials are brokered server-side and never enter the agent's machine. Add a GitHub token as the GH_TOKEN secret so the agent can draft issues.

Sentry

Reads new and spiking errors, read-only.

Or useDatadogRollbar
Required
GitHub

Skips tracked errors and drafts issues for the rest.

Or useLinearJira
Required
Slack

Posts the sweep summary.

Or useMicrosoft Teams
Required
3

Run it once with this prompt

Merge the change request, then start a session with the error-triage agent and send this prompt. Read the drafted issues before you let the schedule take over.

Prompt

Run one error-backlog sweep of the web project in Sentry.

  1. Pull issues first seen in the last hour, and issues at 3x or more of their trailing hourly baseline.
  2. Group them by fingerprint and rank the groups by event count times affected users.
  3. Search owner/repo for an issue that already has each Sentry issue ID, and skip those.
  4. Draft a GitHub issue for each of the top 5 that are not tracked, with the stack trace and impact. Do not assign them.
  5. Post one summary to #eng-errors. Change nothing in Sentry.
4

Let the hourly schedule take over

The change request added this trigger to kortix.yaml. From now on the agent runs every hour, on the hour, each run in a new session on its own cloud computer. To change the time, edit the cron line, or open Triggers in the workspace.

kortix.yaml
triggers:- slug: hourly-error-backlog-sweep  name: Hourly error-backlog sweep  on:    cron: "0 0 * * * *"  run:    agent: error-triage    prompt: Load the error-backlog-triage skill and run one complete sweep of the web project.
5

Never resolve, mute or assign

Sentry access is read-only. The agent never resolves, ignores or mutes an error, and a drafted GitHub issue has no assignee. It drafts at most 5 issues per sweep. The team decides priority and ownership.

6

Read the sweep summary

A sweep that finds new or spiking errors posts one summary, and a quiet sweep posts nothing. This is what a run looks like in the session:

Hourly error-backlog sweep · 15:00
Load the error-backlog-triage skill and run one complete sweep of the web project.
Worked for 2m 03s
sentry · list issuessentry · get issue events × 7github · search issues × 7github · create issue × 3slack · post message → #eng-errors

Found 4 new and 3 spiking errors. 4 were already tracked, so the agent drafted 3 issues. The top one, a TypeError in checkout, had 1,240 events from 312 users this hour.

Illustrative example. Your errors and traffic differ.

The pattern

An hourly cron spawns a session with a read-only connector into Sentry and a scoped GitHub token for drafts. The grooming rules live as skills and memory. The agent dedupes against existing issues, drafts the top offenders, and alerts Slack — it never resolves, ignores, or assigns an error.

Make it do more

Once it works, paste any of these into the same session.

Link the suspect commit

For each drafted issue, find the latest commit that touched the top frame of the stack trace and link it in the issue body.

Morning error digest

Every weekday at 09:00, post to #eng-errors the 5 errors with the most affected users in the last 24 hours and their issue links.

Watch each release

When a new release appears in Sentry, run a sweep right away and compare its error rate with the previous release.

Guardrails

The agent reads production error telemetry and writes issues others will act on, so its access is scoped and one-directional:

  • Isolation. Every run happens in its own isolated sandbox. The session can read Sentry and search GitHub; only the drafted issue and the Slack post are written back out.
  • Scoped secrets. The Sentry credential is brokered through the connector and the GitHub token is encrypted in the Secrets Manager and injected into the sandbox at runtime, scoped to the agents you grant them to.
  • Never resolves, ignores, or mutes. The agent has no path to changing an error's state in Sentry. It reports; it doesn't manage the backlog's status.
  • Never assigns. A drafted GitHub issue has no assignee. Triage priority and ownership are decided by the team, not the agent.
  • Everything is code. The grooming rules, thresholds, and per-system permissions are files in the repo, versioned and changed through a reviewed change request rather than a dashboard setting.

The outcome

The backlog re-read from Sentry's current state
Every hour
Checked against existing GitHub issues before drafting
Deduped
Issues and a Slack alert — never a resolve, ignore, or assign
Draft-only

The error backlog that used to need a dedicated hour to sort now arrives pre-groomed every hour: new errors and real spikes ranked by impact, checked against what's already tracked, and the top offenders already drafted as issues with the trace attached. The team spends its time deciding what to fix, not rediscovering what's broken.

On this page

  • The problem
  • What we built
  • How it works
  • Make it do more
  • Guardrails
  • The outcome

Run this yourself

Install this exact setup — agent, connectors, schedule, and guardrails — in a guided flow.

More use cases.

  • How we handle GDPR data-subject requests on a deadline

    Legal·July 11, 2026

  • OpenChatly

    How we keep our docs in sync with the code

    Documentation·July 5, 2026

  • OpenChatly

    How we QA every pull request automatically

    QA·July 5, 2026

Own your AI.Start today, in our cloud.

Start for free

OpenChatly is the AI workspace for everything. One person directing, agents executing, from goal to finished work.

©2026 OpenChatly AI.沪ICP备2025124785号-5

Resource guide

  • Product overview
  • Skills marketplace

Terms of service

  • User agreement
  • Privacy policy

Contact us