How we handle GDPR data-subject requests on a deadline
Legal
The flaky-test agent we run on Kortix — connected to GitHub CI history and Slack. It scores every test's non-determinism, opens a quarantine PR for the worst offenders, and files a tracking issue for a human to review.
Flaky tests erode trust in CI fast. A test fails for no reason anyone can pin down, gets rerun, goes green, and CI moves on — until the day a real regression hides behind the same "oh that one's just flaky" shrug. Nobody schedules time to fix the true flakes because nobody has a ranked list of which ones are actually costing the team reruns.
We run a flaky-test-triage agent on Kortix that reads the CI run history every day, scores every test on how often it flips outcome on unchanged code, and opens a quarantine PR for the worst offenders with a tracking issue attached. It only skips tests, never deletes one, and it never merges its own PR — a human reviews the quarantine and eventually retires it once the test is fixed.
Flakiness hides in plain sight. A test fails, the job reruns, it passes, and CI goes green — so the failure never becomes a signal anyone tracks. Spread across a few hundred tests and a few months, a handful of tests are quietly eating a rerun every week, and genuinely broken tests get the same "just rerun it" treatment as the flaky ones.
The common responses don't fix this. Rerunning failed jobs until green hides the problem instead of measuring it. A channel where someone occasionally asks "is this one flaky again?" depends on a person noticing and remembering. Deleting a flaky test outright throws away whatever real coverage it had, and doing any of this by hand means first digging through weeks of CI logs to find which tests are actually the worst offenders.
On Kortix, a daily cron re-prompts one persistent agent session. It resumes from a flakiness ledger, pulls the CI run history from GitHub since the last check, updates every test's non-determinism score, and once a test crosses the quarantine threshold, opens a PR that skips it with a reason and a link to the evidence — plus a running tracking issue listing every currently quarantined test. It never deletes a test and never merges its own PR.
Open your Kortix workspace, start a new session, paste this prompt and send it. The agent writes the agent file, the flaky-test-quarantine skill and the daily trigger, and opens a change request for you to review. Prefer clicks? Choose Use this template instead.
Set up a daily flaky-test triage agent in this workspace.
Create the agent agents/flaky-test-triage.md. Its job is to score CI tests in owner/repo for flakiness and quarantine the worst ones. It may use the GitHub connector, the Slack channel and the GH_TOKEN secret.
Create the skill skills/flaky-test-quarantine/SKILL.md with these rules:
Add a cron trigger daily-flaky-test-sweep to kortix.yaml. It runs the flaky-test-triage agent every day at 06:00 with the prompt "Load the flaky-test-quarantine skill and run one triage pass on owner/repo."
Tell me which of these connectors are not connected yet.
Open a change request with all of it. Do not merge it.
Open Connectors in your workspace and connect GitHub, add a GitHub token as
the GH_TOKEN secret, and connect Slack under Channels. Credentials are
brokered server-side and never enter the agent's machine.
Reads CI history and opens quarantine pull requests and issues.
Posts the daily flaky-test summary.
Merge the change request, then start a session with the flaky-test-triage agent and send this prompt. Check the scores and the quarantine pull request before you let the schedule take over.
Run one flaky-test triage pass on owner/repo.
The change request added this trigger to kortix.yaml. From now on the agent
runs every day at 06:00, each run in a new session on its own cloud computer. The ledger in
memory/flaky-test-ledger.md carries the scores between runs.
To change the time, edit the cron line, or open Triggers in the workspace.
triggers:- slug: daily-flaky-test-sweep name: Daily flaky-test sweep on: cron: "0 0 6 * * *" run: agent: flaky-test-triage prompt: Load the flaky-test-quarantine skill and run one triage pass on owner/repo.The only change the agent makes to a test file is a skip marker with a reason. It never removes a test, an assertion or a file, and it never edits CI config to hide a failure. Deleting the marker restores the test. A person reviews and merges the pull request, and decides when to de-quarantine.
Each run ends with one message in the channel. This is what a run looks like in the session:
Scored 1,240 tests across 9 new runs. Quarantined 2 tests that flipped 4 and 5 times in their last 20 runs, in one pull request. 3 tests are still watched, and 1 is stable enough to de-quarantine.
Once it works, paste any of these into the same session.
For each newly quarantined test, add a comment to the pull request with the most common error message across its failed runs.
Every Monday, post the number of quarantined tests and the CI pass rate for the last 4 weeks to #ci-health.
When a quarantined test has passed 20 runs in a row on a branch with its skip removed, open a pull request that removes the skip marker.
The agent changes what runs in CI, so its access is scoped and one-directional:
GH_TOKEN used by the gh
CLI are encrypted in the Secrets Manager and injected at runtime, scoped to the agents you grant them to.The tests that used to eat a silent rerun every week now show up ranked, with a PR and a tracking issue attached to the worst of them. CI gets quieter without losing coverage nobody meant to drop, and the team spends its fixing time on the flakes that are actually costing the most reruns.