Evidence-first feature testing for Claude Code. A verdict you can open, not a claim.
QA-Pilot runs feature tests against a deployed environment and publishes a verdict only after a script has checked its evidence. Every executed case leaves a committed Playwright spec, a trace, and the deploy SHA read from the running environment, so QA reviews evidence instead of re-running the tests. ClickUp holds the record, or plain local files when you run without a tracker.
Documentation: sandeeptakasi.github.io/qa-pilot
claude plugin marketplace add https://github.com/SandeepTakasi/qa-pilot
claude plugin install qa-pilotRequires Node 20 or newer and, in the repo you test, Playwright 1.51 or newer. Then run
/qa-pilot:qa-init in that repo to write its host profile:
Setting up a project.
"Claude clicked through it and it looked fine" is a sentence, not evidence. It names no build,
holds no recording, and gives a second person nothing to open. QA-Pilot replaces it with three
things that can be checked: a committed spec that produced the verdict and regresses the feature
in CI later, one trace.zip per case (video, console, DOM snapshots and network log in one file),
and the deployed build the run was against, read before and after the run.
Six commands, with QA approving between design and execution:
/qa-pilot:qa-init write the project's host profile (once)
/qa-pilot:setup-profiles save a login per account (once per developer)
/qa-pilot:generate-tests author the feature's cases from its task and code
QA approves the scenarios, the write policy and the cases
/qa-pilot:run-tests run committed specs against a deployed build
/qa-pilot:publish-results validate the run, then publish verdicts
/qa-pilot:qa-review approve, reject with a reason, or retest
The publish step is a script, not a judgement call. It rejects:
- a verdict without evidence: every executed case needs a trace and a deploy SHA;
- a verdict from a sandbox: mock backends always succeed, so they can stabilise specs but never carry a verdict;
- a pass that was really a retry: pass-on-retry is
flaky, always; - a verdict for a case QA never approved: checked against the case's recorded status at run time;
- a run that lost its ground truth: if the deployed build changes mid-run, every case is
blocked, and past 10% blocked the run halts; - a run that broke its write policy: each feature declares whether its tests may write, and the write guard records every write it saw.
An environment declares its kind instead of the tool guessing from its hostname. A production
environment must also name its test_account, the account the runs sign in as and how it is
restricted. On production, traces stay on the executor's machine (not attached to the tracker,
uploaded by CI or committed), every case keeps a full trace, and a feature must be read-only or
scoped-write under a live write guard that judges clicks in the page and requests at the network
layer. The tracker record of a production run carries no application data.
Production has the details, and says
plainly what the gates cannot verify, including that test_account is an attestation and the guard
a second layer, not the boundary.
Off production, traces also stay on the machine by default; attaching them to the tracker is an opt-in per environment (see the host profile).
qa-pilot/ the plugin: skills, zero-dependency scripts, schemas, the ClickUp guard hook,
and templates (the write guard, a CI workflow, the spec lint config
eslint.qa-pilot.config.mjs) to copy into a host repo
site/ the documentation site, built from the plugin's own markdown (site/DESIGN.md)
docs/ architecture notes and decision records
fixtures/ a fake host repo the tests and the browser proof of the write guard run against
node --test qa-pilot/scripts/tests/*.test.mjs
claude plugin validate ./qa-pilotCLAUDE.md lists the checks to run before every commit, including the write guard's real-browser proof.
MIT. The one bundled dependency, a copy of yaml, is ISC licensed; see THIRD_PARTY_NOTICES.md.