Get started

How we QA every pull request automatically

The QA agent we run on Kortix — connected to GitHub and our test environment. It checks out each PR, runs the suite, exercises the change, and posts the result.

OpenChatly
Written by
The OpenChatly Team
Published
July 5, 2026
Reading time
7 min read
Topics
QA, Case Study, Engineering

Code review catches problems a person can find by reading a diff. It misses the ones you only find by running the change: a test that passes locally but flakes in CI, a path that works in the happy case and 500s on an edge case, a migration that reads fine but locks a table under load. These tend to surface in staging or production, after the PR is approved.

We run a QA agent on Kortix that does this checking when the PR opens. This is how we QA our own changes, including the connections and guardrails involved.

Team
Kortix
Runs on
Every PR, on open and on push
Connected systems
GitHub · Test environment · Cloudflare
Mode
Trigger-driven · result posted on the PR

The problem

Some bugs don't show up in a diff. A reviewer reads the code and approves, but no one has checked out the branch, run the suite, hit the new endpoint, and watched what happens. CI runs the tests the author wrote; it doesn't cover the paths they missed.

The common fixes are incomplete. Green CI shows the existing tests pass, not that the change is correct. A manual QA pass is thorough but slow and lands after the review. A generic AI reviewer only sees the diff; it can't run the branch, so it misses failures that only appear at runtime.

What we built

On Kortix, each PR triggers an agent. On open and on every push, the PR spawns an isolated session (a cloud sandbox) with scoped access to the branch, the test suite, a deployable test environment, and the Cloudflare edge in front of it. The agent checks out the change, runs the suite, exercises the new behavior on a live deploy, and posts a pass/fail result on the PR.

How it works

1

Copy the setup prompt into a new session

Open your Kortix workspace, start a new session, paste this prompt and send it. The agent writes the agent file, the pr-qa skill and the 5-minute trigger, and opens a change request for you to review. Prefer clicks? Choose Use this template instead.

Setup prompt

Set up a PR QA agent in this workspace.

  1. Create the agent agents/qa-agent.md. Its job is to QA every new pull request revision on owner/repo against the test environment and post a pass or fail result. It uses the gh CLI with the GH_TOKEN secret, and deploys with the TEST_ENV_DEPLOY_TOKEN secret.

  2. Create the skill skills/pr-qa/SKILL.md with these rules:

    • Skip any PR whose current head commit already has a qa-agent check. QA each other PR in its own clean clone, and never let a failure on one PR block the others.
    • Run the unit, integration and end-to-end suites, and keep the full failure output.
    • Deploy the branch to the test environment and wait until it is healthy. Then exercise the changed endpoint or page directly, and again through the edge.
    • A test that fails and then passes on an isolated re-run is flaky. Report it with both logs. Never retry until it passes.
    • Post exactly one result per revision as a qa-agent status check and a PR comment: what you exercised on a pass, or the failing command, the logs and repro steps on a fail.
    • Test environment only: no production access, no production deploy, and never merge. Tear down each test deploy when you are done.
  3. Add a cron trigger pr-qa-sweep to kortix.yaml. It runs the qa-agent agent every 5 minutes with the prompt "Load the pr-qa skill and QA every PR on owner/repo whose current head has no QA result yet."

  4. Tell me which of these connectors are not connected yet. Tell me if the GH_TOKEN or TEST_ENV_DEPLOY_TOKEN secret is missing.

  5. Open a change request with all of it. Do not merge it.

2

Connect your tools

Add a GitHub token as the GH_TOKEN secret and your test-environment deploy token as TEST_ENV_DEPLOY_TOKEN. Both are injected at runtime and scoped to this agent. The edge check is a public HTTPS request and needs no credential.

GitHub

Checks out each new PR and posts a pass or fail result.

Or useGitLab
Required
Vercel

Deploys each PR branch to the test environment.

Or useNetlifyFly.ioRender
Required
Cloudflare

Serves the edge the agent re-checks each change through.

Or useFastly
Optional
3

Run it once with this prompt

Merge the change request, then start a session with the qa-agent agent and send this prompt. Replace owner/repo with your repository. Read the first results on the PRs before you let the schedule take over.

Prompt

QA the open PRs on owner/repo.

  1. List open PRs with gh pr list, and skip any whose current head commit already has a qa-agent check.
  2. For each other PR, clone it clean, install it, and run the unit, integration and end-to-end suites.
  3. Deploy the branch to the test environment, wait until it is healthy, and exercise the changed behavior directly and through the edge.
  4. Post one qa-agent status check and one comment per PR: what you exercised on a pass, or the failing command, the logs and repro steps on a fail.
  5. Tear down each test deploy. Do not touch production and do not merge anything.
4

Let the 5-minute schedule take over

The change request added this trigger to kortix.yaml. From now on the agent runs every 5 minutes, each run on its own cloud computer. It QAs only the PR revisions that have no result yet. To change the time, edit the cron line, or open Triggers in the workspace.

kortix.yaml
triggers:- slug: pr-qa-sweep  name: PR QA sweep  on:    cron: "0 */5 * * * *"  run:    agent: qa-agent    prompt: Load the pr-qa skill and QA every PR on owner/repo whose current head has no QA result yet.
5

Stay on the test environment

The agent has no production credential. It deploys only to the test environment, tears each deploy down, and never merges. Its output is a check and a comment on the PR. A person reads the result and merges.

6

Read the result on the PR

Each PR revision gets one check and one comment. This is what a sweep looks like in the session:

PR QA sweep · 14:05
Load the pr-qa skill and QA every PR on owner/repo whose current head has no QA result yet.
Worked for 6m 52s
gh · pr listgit · clone × 2shell · run test suites × 2shell · deploy to test × 2http · check changed path × 4gh · create status × 2gh · pr comment × 2

QA'd 2 new PR revisions. 1 passed: 412 tests green, and the new endpoint returned 200 direct and through the edge. 1 failed: a redirect works direct to the origin but returns 404 through the edge. Its comment has the failing request and the repro steps.

Illustrative example. Your results depend on your test suite and environment.

Summary

A trigger on every PR spawns a session with scoped connectors into the branch, the test suite, the test environment, and Cloudflare. The QA playbook is encoded as skills and memory. The agent stays on the test environment and a human owns the merge.

Make it do more

Once it works, paste any of these into the same session.

Record new edge cases

When a run finds a new edge case, add it to memory/qa-known-issues.md with the PR link, and open a change request for that file only.

Screenshot changed pages

For each PR that changes a page, add a screenshot of that page on the test deploy to the PR comment.

Daily failure list

Every day at 17:00, post to #eng a list of the PRs that failed QA that day, with the failing command for each one.

Guardrails

The agent has access to the test environment and the edge, so the access is scoped and contained:

  • Isolation. Every PR runs in its own isolated sandbox on its own branch. The session can install, deploy, and exercise the change to reproduce a failure; only the reported result is written back out.
  • Scoped secrets. The GitHub, test-environment, and Cloudflare credentials are encrypted in the Secrets Manager and injected into the sandbox at runtime, scoped to the agents you grant them to.
  • Test environment only. The agent's access stops at the test plane: no production access, no prod deploy, no merge. It reports; the team decides.
  • Everything is code. The agent's configuration, skills, and permissions are files in the repo, versioned and changed through a reviewed change request rather than a dashboard setting.

The outcome

Run, deployed, and exercised before human review
Every PR
Runtime failures caught before they reach staging
Pre-merge
Branch, suite, test environment, and edge in one agent
4 systems

Runtime failures that a diff can't show now surface on the PR when it opens, with the failing command, the logs, and the repro attached. Reviewers spend their time on design rather than checking out branches by hand, and the changes that reach staging have already been run against the test environment.

On this page

  • The problem
  • What we built
  • How it works
  • Make it do more
  • Guardrails
  • The outcome

Run this yourself

Install this exact setup — agent, connectors, schedule, and guardrails — in a guided flow.

More use cases.

  • How we handle GDPR data-subject requests on a deadline

    Legal·July 11, 2026

  • OpenChatly

    How we keep our docs in sync with the code

    Documentation·July 5, 2026

  • Slack
    GitHub
    Linear
    OpenChatly

    How we run operations from Slack

    Operations·July 5, 2026

Own your AI.Start today, in our cloud.

Start for free

OpenChatly is the AI workspace for everything. One person directing, agents executing, from goal to finished work.

©2026 OpenChatly AI.沪ICP备2025124785号-5

Resource guide

  • Product overview
  • Skills marketplace

Terms of service

  • User agreement
  • Privacy policy

Contact us