Get started

How we detect and quarantine flaky tests

The flaky-test agent we run on Kortix — connected to GitHub CI history and Slack. It scores every test's non-determinism, opens a quarantine PR for the worst offenders, and files a tracking issue for a human to review.

Written by
The OpenChatly Team
Published
February 20, 2026
Reading time
7 min read
Topics
Testing, Case Study, Engineering

Flaky tests erode trust in CI fast. A test fails for no reason anyone can pin down, gets rerun, goes green, and CI moves on — until the day a real regression hides behind the same "oh that one's just flaky" shrug. Nobody schedules time to fix the true flakes because nobody has a ranked list of which ones are actually costing the team reruns.

We run a flaky-test-triage agent on Kortix that reads the CI run history every day, scores every test on how often it flips outcome on unchanged code, and opens a quarantine PR for the worst offenders with a tracking issue attached. It only skips tests, never deletes one, and it never merges its own PR — a human reviews the quarantine and eventually retires it once the test is fixed.

Team
Kortix
Runs on
Daily cron, one persistent session
Connected systems
GitHub · CI run history · Slack
Mode
Quarantine PR + tracking issue only — never merges, never deletes

The problem

Flakiness hides in plain sight. A test fails, the job reruns, it passes, and CI goes green — so the failure never becomes a signal anyone tracks. Spread across a few hundred tests and a few months, a handful of tests are quietly eating a rerun every week, and genuinely broken tests get the same "just rerun it" treatment as the flaky ones.

The common responses don't fix this. Rerunning failed jobs until green hides the problem instead of measuring it. A channel where someone occasionally asks "is this one flaky again?" depends on a person noticing and remembering. Deleting a flaky test outright throws away whatever real coverage it had, and doing any of this by hand means first digging through weeks of CI logs to find which tests are actually the worst offenders.

What we built

On Kortix, a daily cron re-prompts one persistent agent session. It resumes from a flakiness ledger, pulls the CI run history from GitHub since the last check, updates every test's non-determinism score, and once a test crosses the quarantine threshold, opens a PR that skips it with a reason and a link to the evidence — plus a running tracking issue listing every currently quarantined test. It never deletes a test and never merges its own PR.

How it works

1

Copy the setup prompt into a new session

Open your Kortix workspace, start a new session, paste this prompt and send it. The agent writes the agent file, the flaky-test-quarantine skill and the daily trigger, and opens a change request for you to review. Prefer clicks? Choose Use this template instead.

Setup prompt

Set up a daily flaky-test triage agent in this workspace.

  1. Create the agent agents/flaky-test-triage.md. Its job is to score CI tests in owner/repo for flakiness and quarantine the worst ones. It may use the GitHub connector, the Slack channel and the GH_TOKEN secret.

  2. Create the skill skills/flaky-test-quarantine/SKILL.md with these rules:

    • Read memory/flaky-test-ledger.md first, and process only CI runs newer than the last processed run.
    • Score each test by flips between pass and fail on unchanged code over its last 20 runs. A single failure is not flakiness.
    • Quarantine a test at 3 or more flips in the last 20 runs. Add a framework skip marker with a reason and a link to the evidence. Never delete a test, an assertion or a file.
    • Batch the day's new quarantines into one pull request, and keep one running issue titled Flaky test quarantine tracker.
    • Never edit CI config to force a pass, never push to the default branch, and never merge.
    • Post a summary to #ci-health: newly quarantined, still flaky, and stable enough to de-quarantine.
  3. Add a cron trigger daily-flaky-test-sweep to kortix.yaml. It runs the flaky-test-triage agent every day at 06:00 with the prompt "Load the flaky-test-quarantine skill and run one triage pass on owner/repo."

  4. Tell me which of these connectors are not connected yet.

  5. Open a change request with all of it. Do not merge it.

2

Connect your tools

Open Connectors in your workspace and connect GitHub, add a GitHub token as the GH_TOKEN secret, and connect Slack under Channels. Credentials are brokered server-side and never enter the agent's machine.

GitHub

Reads CI history and opens quarantine pull requests and issues.

Or useGitLabCircleCIBuildkite
Required
Slack

Posts the daily flaky-test summary.

Or useMicrosoft Teams
Required
3

Run it once with this prompt

Merge the change request, then start a session with the flaky-test-triage agent and send this prompt. Check the scores and the quarantine pull request before you let the schedule take over.

Prompt

Run one flaky-test triage pass on owner/repo.

  1. Read memory/flaky-test-ledger.md, the open quarantine pull request and the Flaky test quarantine tracker issue.
  2. Pull CI run history since the last processed run and update each test's flip count and failure rate.
  3. For each test at 3 or more flips in the last 20 runs that is not quarantined yet, add a skip marker with a reason and an evidence link.
  4. Put those changes in one pull request and update the tracker issue. Do not merge it.
  5. Post a summary to #ci-health and update the ledger.
4

Let the daily schedule take over

The change request added this trigger to kortix.yaml. From now on the agent runs every day at 06:00, each run in a new session on its own cloud computer. The ledger in memory/flaky-test-ledger.md carries the scores between runs. To change the time, edit the cron line, or open Triggers in the workspace.

kortix.yaml
triggers:- slug: daily-flaky-test-sweep  name: Daily flaky-test sweep  on:    cron: "0 0 6 * * *"  run:    agent: flaky-test-triage    prompt: Load the flaky-test-quarantine skill and run one triage pass on owner/repo.
5

Skip tests, never delete them

The only change the agent makes to a test file is a skip marker with a reason. It never removes a test, an assertion or a file, and it never edits CI config to hide a failure. Deleting the marker restores the test. A person reviews and merges the pull request, and decides when to de-quarantine.

6

Read the daily summary

Each run ends with one message in the channel. This is what a run looks like in the session:

Daily flaky-test sweep · 06:00
Load the flaky-test-quarantine skill and run one triage pass on owner/repo.
Worked for 5m 51s
github · list workflow runsgithub · download artifact × 9shell · git commit skip markersgithub · create pull requestgithub · update issueslack · post message → #ci-health

Scored 1,240 tests across 9 new runs. Quarantined 2 tests that flipped 4 and 5 times in their last 20 runs, in one pull request. 3 tests are still watched, and 1 is stable enough to de-quarantine.

Illustrative example. Your counts depend on your CI history.

The pattern

A daily cron re-prompts one persistent session so the flakiness ledger survives across runs. The agent reads CI history through GitHub, scores and ranks every test, and its only outputs are a quarantine PR, a tracking issue, and a Slack summary — never a merge, never a deletion.

Make it do more

Once it works, paste any of these into the same session.

Find the likely cause

For each newly quarantined test, add a comment to the pull request with the most common error message across its failed runs.

Weekly flakiness trend

Every Monday, post the number of quarantined tests and the CI pass rate for the last 4 weeks to #ci-health.

Propose de-quarantines

When a quarantined test has passed 20 runs in a row on a branch with its skip removed, open a pull request that removes the skip marker.

Guardrails

The agent changes what runs in CI, so its access is scoped and one-directional:

  • Isolation. Every run happens in the session's own isolated sandbox. Only the PR, the issue, and the Slack post leave it.
  • Scoped secrets. The GitHub connector and the GH_TOKEN used by the gh CLI are encrypted in the Secrets Manager and injected at runtime, scoped to the agents you grant them to.
  • Skip, never delete. The agent's only edit to a test file is a skip/quarantine marker with a reason; it never removes a test, its assertions, or its file.
  • PR-gated. The agent opens a PR and an issue and stops. It never merges and never pushes to the default branch; a human owns the merge and the eventual de-quarantine.
  • Everything is code. The agent's scoring rules, skills, and permissions are files in the repo, versioned and changed through a reviewed change request rather than a dashboard setting.

The outcome

Test flakiness re-scored against the latest CI history
Every day
Quarantine targets picked from evidence, not gut feel
Ranked, not guessed
The agent proposes the quarantine; the team decides
Human merge

The tests that used to eat a silent rerun every week now show up ranked, with a PR and a tracking issue attached to the worst of them. CI gets quieter without losing coverage nobody meant to drop, and the team spends its fixing time on the flakes that are actually costing the most reruns.

On this page

  • The problem
  • What we built
  • How it works
  • Make it do more
  • Guardrails
  • The outcome

Run this yourself

Install this exact setup — agent, connectors, schedule, and guardrails — in a guided flow.

More use cases.

  • How we handle GDPR data-subject requests on a deadline

    Legal·July 11, 2026

  • OpenChatly

    How we keep our docs in sync with the code

    Documentation·July 5, 2026

  • OpenChatly

    How we QA every pull request automatically

    QA·July 5, 2026

Own your AI.Start today, in our cloud.

Start for free

OpenChatly is the AI workspace for everything. One person directing, agents executing, from goal to finished work.

©2026 OpenChatly AI.沪ICP备2025124785号-5

Resource guide

  • Product overview
  • Skills marketplace

Terms of service

  • User agreement
  • Privacy policy

Contact us