# How we QA every pull request automatically

The QA agent we run on Kortix — connected to GitHub and our test environment. It checks out each PR, runs the suite, exercises the change, and posts the result.

Canonical page: https://openchatly.com/use-cases/qa-agent

Code review catches problems a person can find by reading a diff. It misses the
ones you only find by running the change: a test that passes locally but flakes
in CI, a path that works in the happy case and 500s on an edge case, a migration
that reads fine but locks a table under load. These tend to surface in staging or
production, after the PR is approved.

We run a QA agent on Kortix that does this checking when the PR opens. This is how
we QA our own changes, including the connections and guardrails involved.

- **Team:** Kortix
- **Runs on:** Every PR, on open and on push
- **Connected systems:** GitHub · Test environment · Cloudflare
- **Mode:** Trigger-driven · result posted on the PR

## The problem

Some bugs don't show up in a diff. A reviewer reads the code and approves, but no
one has checked out the branch, run the suite, hit the new endpoint, and watched
what happens. CI runs the tests the author wrote; it doesn't cover the paths they
missed.

The common fixes are incomplete. Green CI shows the existing tests pass, not that
the change is correct. A manual QA pass is thorough but slow and lands after the
review. A generic AI reviewer only sees the diff; it can't run the branch, so it
misses failures that only appear at runtime.

## What we built

On Kortix, each PR triggers an agent. On open and on every push, the PR spawns an
isolated session (a cloud sandbox) with scoped access to the branch, the test
suite, a deployable test environment, and the Cloudflare edge in front of it. The
agent checks out the change, runs the suite, exercises the new behavior on a live
deploy, and posts a pass/fail result on the PR.

## How it works

### Copy the setup prompt into a new session

Open your Kortix workspace, start a new session, paste this prompt and send it.
The agent writes the agent file, the pr-qa skill and the 5-minute trigger, and
opens a change request for you to review. Prefer clicks? Choose **Use this
template** instead.

**Setup prompt**

```
Set up a PR QA agent in this workspace.

1. Create the agent agents/qa-agent.md. Its job is to QA every new pull request revision on owner/repo against the test environment and post a pass or fail result. It uses the gh CLI with the GH_TOKEN secret, and deploys with the TEST_ENV_DEPLOY_TOKEN secret.

2. Create the skill skills/pr-qa/SKILL.md with these rules:
   - Skip any PR whose current head commit already has a qa-agent check. QA each other PR in its own clean clone, and never let a failure on one PR block the others.
   - Run the unit, integration and end-to-end suites, and keep the full failure output.
   - Deploy the branch to the test environment and wait until it is healthy. Then exercise the changed endpoint or page directly, and again through the edge.
   - A test that fails and then passes on an isolated re-run is flaky. Report it with both logs. Never retry until it passes.
   - Post exactly one result per revision as a qa-agent status check and a PR comment: what you exercised on a pass, or the failing command, the logs and repro steps on a fail.
   - Test environment only: no production access, no production deploy, and never merge. Tear down each test deploy when you are done.

3. Add a cron trigger pr-qa-sweep to kortix.yaml. It runs the qa-agent agent every 5 minutes with the prompt "Load the pr-qa skill and QA every PR on owner/repo whose current head has no QA result yet."

4. Tell me which of these connectors are not connected yet. Tell me if the GH_TOKEN or TEST_ENV_DEPLOY_TOKEN secret is missing.

5. Open a change request with all of it. Do not merge it.
```

### Connect your tools

Add a GitHub token as the `GH_TOKEN` secret and your test-environment deploy
token as `TEST_ENV_DEPLOY_TOKEN`. Both are injected at runtime and scoped to
this agent. The edge check is a public HTTPS request and needs no credential.

- **GitHub** (Required):
Checks out each new PR and posts a pass or fail result.
- **Vercel** (Required):
Deploys each PR branch to the test environment.
- **Cloudflare** (Optional):
Serves the edge the agent re-checks each change through.

### Run it once with this prompt

Merge the change request, then start a session with the **qa-agent** agent and
send this prompt. Replace owner/repo with your repository. Read the first
results on the PRs before you let the schedule take over.

**Prompt**

```
QA the open PRs on owner/repo.

1. List open PRs with gh pr list, and skip any whose current head commit already has a qa-agent check.
2. For each other PR, clone it clean, install it, and run the unit, integration and end-to-end suites.
3. Deploy the branch to the test environment, wait until it is healthy, and exercise the changed behavior directly and through the edge.
4. Post one qa-agent status check and one comment per PR: what you exercised on a pass, or the failing command, the logs and repro steps on a fail.
5. Tear down each test deploy. Do not touch production and do not merge anything.
```

### Let the 5-minute schedule take over

The change request added this trigger to `kortix.yaml`. From now on the agent
runs every 5 minutes, each run on its own cloud computer. It QAs only the PR
revisions that have no result yet. To change the time, edit the cron line, or
open **Triggers** in the workspace.

**kortix.yaml**

```yaml
triggers:
  - slug: pr-qa-sweep
    name: PR QA sweep
    on:
      cron: "0 */5 * * * *"
    run:
      agent: qa-agent
      prompt: Load the pr-qa skill and QA every PR on owner/repo whose current head has no QA result yet.
```

### Stay on the test environment

The agent has no production credential. It deploys only to the test
environment, tears each deploy down, and never merges. Its output is a check
and a comment on the PR. A person reads the result and merges.

### Read the result on the PR

Each PR revision gets one check and one comment. This is what a sweep looks
like in the session:

> **PR QA sweep · 14:05**
>
> Prompt: Load the pr-qa skill and QA every PR on owner/repo whose current head has no QA result yet.
>
> QA'd **2** new PR revisions. **1** passed: **412** tests green, and the new
> endpoint returned 200 direct and through the edge. **1** failed: a redirect
> works direct to the origin but returns **404** through the edge. Its comment
> has the failing request and the repro steps.

> **Summary**
> A **trigger** on every PR spawns a session with scoped **connectors** into the
> branch, the test suite, the test environment, and Cloudflare. The QA playbook is
> encoded as **skills** and **memory**. The agent stays on the test environment
> and a human owns the merge.

## Make it do more

Once it works, paste any of these into the same session.

**Record new edge cases**

```
When a run finds a new edge case, add it to memory/qa-known-issues.md with the PR link, and open a change request for that file only.
```

**Screenshot changed pages**

```
For each PR that changes a page, add a screenshot of that page on the test deploy to the PR comment.
```

**Daily failure list**

```
Every day at 17:00, post to #eng a list of the PRs that failed QA that day, with the failing command for each one.
```

## Guardrails

The agent has access to the test environment and the edge, so the access is
scoped and contained:

- **Isolation.** Every PR runs in its own isolated sandbox on its own branch. The
  session can install, deploy, and exercise the change to reproduce a failure;
  only the reported result is written back out.
- **Scoped secrets.** The GitHub, test-environment, and Cloudflare credentials are
  encrypted in the Secrets Manager and injected into the sandbox at runtime, scoped to the agents you grant them to.
- **Test environment only.** The agent's access stops at the test plane: no
  production access, no prod deploy, no merge. It reports; the team decides.
- **Everything is code.** The agent's configuration, skills, and permissions are
  files in the repo, versioned and changed through a reviewed **change request**
  rather than a dashboard setting.

## The outcome

- **Every PR:** Run, deployed, and exercised before human review
- **Pre-merge:** Runtime failures caught before they reach staging
- **4 systems:** Branch, suite, test environment, and edge in one agent

Runtime failures that a diff can't show now surface on the PR when it opens, with
the failing command, the logs, and the repro attached. Reviewers spend their time
on design rather than checking out branches by hand, and the changes that reach
staging have already been run against the test environment.
