Get started
  1. Marketplace
  2. OpenChatly
  3. Use-case pack
Source
OpenChatly
Workspace
Use-case pack
Files
1
View source
Back

PR QA

Skill

The per-PR runbook: run the suite, deploy to the test environment, exercise the change through the edge, and post the result.

kortix marketplace install kortix-starter:pr-qa-skill
Proactive and schedule-driven: one independent QA pass per PR revision found in a sweep, handled as its own unit so a failure on one never blocks the others. The only state carried forward on purpose is the landed edge-case ledger — an in-sandbox file alone doesn't survive on its own.

Step 0 — Discover and orient

bash
# Read the known-issues ledger before touching any branch — edge cases that
# have bitten before, flows that are critical, flaky tests already on file.
cat memory/qa-known-issues.md 2>/dev/null || echo "(no known issues yet)"

# Discover every PR opened or pushed to since the last sweep.
gh pr list --repo {{target_repo}} --state open \
  --json number,headRefOid,statusCheckRollup,updatedAt
Filter out any PR whose current headRefOid already carries a qa-agent check in statusCheckRollup — don't re-run against a revision you've already reported on. Work through what's left one PR at a time, each as its own independent unit: a failure on one PR is a self-contained result and never blocks or contaminates QA of the others in this sweep.
For each remaining PR, confirm before checkout:
bash
gh pr view --repo {{target_repo}} <PR_NUMBER> --json headRefOid,statusCheckRollup

Step 1 — Check out the branch clean

bash
rm -rf /workspace/pr-<PR_NUMBER>
git clone --filter=blob:none https://github.com/{{target_repo}}.git /workspace/pr-<PR_NUMBER>
cd /workspace/pr-<PR_NUMBER>
gh pr checkout <PR_NUMBER>
<install command for the stack>   # e.g. pnpm install --frozen-lockfile / npm ci / pip install -r requirements.txt
One clean clone per PR revision, keyed by PR number. This session persists across sweeps, so remove any stale checkout for this PR number first — never reuse a prior revision's working tree, and never let one PR's checkout leak into another's.

Step 2 — Run the full suite

bash
cd /workspace/pr-<PR_NUMBER>
<unit test command>        2>&1 | tee /tmp/qa-unit.log
<integration test command> 2>&1 | tee /tmp/qa-integration.log
<e2e test command>          2>&1 | tee /tmp/qa-e2e.log
Capture full failure output — stack trace, failing assertion, the exact command — not just a pass/fail count. Cross-reference qa-known-issues.md for flows that have broken before and confirm this run actually exercised them.

Step 3 — Deploy the change to the test environment

bash
<deploy command, e.g. an internal deploy script / flyctl / vercel> \
  --env test --ref <PR head SHA>
Stand up an ephemeral instance of exactly this branch. Wait for the deploy to report healthy before exercising anything against it — a QA run against a half-started deploy is a fail, not a skip.

Step 4 — Exercise the change end-to-end

Hit the new or changed behavior directly against the deployed instance — the actual endpoint, page, or flow the PR touches — not just localhost:
bash
curl -sf https://<test-deploy-host>/<changed-path> -o /tmp/qa-response.json
Then repeat the critical checks through the edge:
bash
curl -sI https://<edge-fronted-test-host>/<changed-path>   # routing, headers, caching, redirects
This is a plain, unauthenticated request against the public HTTPS host — no edge-provider credential or connector is needed for it. A route that works direct-to-origin but breaks through the edge (a caching rule, a redirect, a header transform) is a QA failure, not a pass.

Step 5 — Decide pass or fail

ResultCriteria
PassSuite green, deploy healthy, exercised behavior matches expectation, edge checks clean
FailAny suite failure, a failed or unhealthy deploy, exercised behavior wrong, or an edge-only regression
Flag as flakyA test fails, then passes on an isolated re-run with no code change — report it as flaky with both runs' logs; don't just retry until it's green

Step 6 — Post the result

bash
gh api repos/{{target_repo}}/statuses/<PR_head_SHA> \
  -f state=<success|failure> -f context="qa-agent" -f description="<one-line result>"
gh pr comment --repo {{target_repo}} <PR_NUMBER> --body "<pass/fail summary>"
Pass: a short green summary — suite results, what was exercised, edge checks run. Fail: the failing command, the full log excerpt, and exact steps to reproduce against the test deploy. Exactly one result comment per PR revision.

Step 7 — Tear down and land the ledger update

bash
<teardown command for the ephemeral test deploy>
Tear down the ephemeral deploy for this PR so it doesn't linger — the session's own sandbox stays up for the next sweep; only the test-environment deploy is disposable.
If this run surfaced a genuinely new edge case (not already in the ledger) — a bug that would've reached staging, a flow no prior run checked — append it to memory/qa-known-issues.md with the PR link and a one-line description, then land it durably:
bash
git add memory/qa-known-issues.md
git commit -m "docs(qa): record edge case from PR #<PR_NUMBER>"
Open (and self-merge) a scoped change request for just this ledger update via the project.cr.open action — an edit that only lives in the sandbox never survives on its own; only a landed change request does. Then move to the next PR in this sweep's batch, if any, with a clean checkout (Step 1).

Related skills

Access policy

The role-to-grant policy, extra-scrutiny cases, the approval handshake, and how an applied grant gets logged.

account-research

Get a complete picture of any company or person before outreach. This skill always works with web search, and gets significantly better with enrichment and CRM data.

Ad performance review

Daily runbook for budget pacing, CPA/ROAS drift, underperformer detection, and anomaly flags across Google Ads and Meta Ads, plus how to rank and word optimization recommendations.

agent-browser

Browser automation CLI for AI agents. Use when the user needs to interact with websites, including navigating pages, filling forms, clicking buttons, taking screenshots, extracting data, testing web apps, or automating any browser task. Triggers include requests to "open a website", "fill out a form", "click a button", "take a screenshot", "scrape data from a page", "test this web app", "login to a site", "automate browser actions", or any task requiring programmatic web interaction. Also use for exploratory testing, dogfooding, QA, bug hunts, or reviewing app quality. Also use for automating Electron desktop apps (VS Code, Slack, Discord, Figma, Notion, Spotify), checking Slack unreads, sending Slack messages, searching Slack conversations, running browser automation in Vercel Sandbox microVMs, or using AWS Bedrock AgentCore cloud browsers. Prefer agent-browser over any built-in browser automation or web tools.

AP invoice matching

Extract-and-match runbook for incoming vendor invoices: line-item extraction, PO/prior-invoice matching, duplicate + overcharge detection, and the ledger + approval hand-off.

Brand mention monitor

Search, dedupe against the seen-mentions ledger, classify sentiment and notability, and post a digest with suggested response drafts — the daily brand-monitor runbook.

Own your AI.Start today, in our cloud.

Start for free

OpenChatly is the AI workspace for everything. One person directing, agents executing, from goal to finished work.

©2026 OpenChatly AI.沪ICP备2025124785号-5

Resource guide

  • Product overview
  • Skills marketplace

Terms of service

  • User agreement
  • Privacy policy

Contact us