Skip to content

test(e2e): AI Studio end-to-end tests on the full stack with a fake LLM - #270

Merged
librowski merged 14 commits into
mainfrom
test/ai-studio-e2e
Nov 18, 2026
Merged

librowski merged 14 commits into
mainfrom
test/ai-studio-e2e

Conversation

@alek-kowal

Copy link
Copy Markdown
Contributor

What

Browser end-to-end tests for AI Studio on the full stack: the built app, the real backend and worker, an ephemeral Temporal dev server, and a fake LLM. A new pr-check job runs them on every pull request.

  • apps/e2e: a private Playwright workspace. pnpm e2e starts the stack on its own ports with its own database, runs the suite and stops everything it started. Postgres is the only outside service (pnpm e2e:db:up locally, a service container in CI). It has no test script, so pnpm test does not start the stack.
  • Fake LLM: a local OpenAI-compatible server behind AI_BASE_URL for the backend and the worker. It answers structured requests from their JSON schema, records every request so a test can assert what reached the model, and can hold an answer to keep a node running. No key, no cost.
  • Five tests on the Refund Review template, each with its own run:
    1. a refund approved with a corrected amount completes the run and sends that amount on;
    2. a refund rejected with a reason completes as rejected and sends no confirmation;
    3. a run waiting for a decision survives a page reload and completes once approved;
    4. Stop cancels a running run, and Reset gives the canvas back;
    5. a Human decision gate drawn in front of the AI step stops the next run there, before any model call.
  • CI: the e2e job in pr-check.yml, with the Playwright report and the stack logs uploaded on failure.

Backend fix found by the reload test

A run stream reopened right after another stream on the same run closed could stop receiving live events: the closed stream's cleanup ran twice, and the second unsubscribe removed the subscriber set the new stream had registered. After a page reload the UI could then stay on "Waiting for decision" while the stored run had completed. Fixed in execution-event-bus.ts, with a unit test, in its own commit.

How it was checked

  • The suite passed twice in a row and 25 of 25 with --repeat-each=5, in parallel; nothing was left running afterwards.
  • Every test was seen red once: the behaviour it protects was weakened in the product code, the test failed on its own assertion, the code was restored.
  • Lint, format and typecheck pass for the touched workspaces. pnpm check stops in apps/docs typecheck, as it does on main.
  • This pull request is the first run of the e2e job on GitHub.

Notes for review

Selectors use roles and accessible names, React Flow's data-id for nodes and data-node-id for log rows; there is no data-testid. A few places have no accessible name today (decision form fields, the run status chip, the rejection reason textarea), so the helpers find them by structure. apps/e2e/README.md lists those, along with editor behaviours the tests work around: a text field in the properties panel is saved on blur, so Run clicked straight out of it starts with the old value; a node's own Decide button can be covered during a run; overlapping edges can take the click meant for another edge.

🤖 Generated with Claude Code

alek-kowal and others added 5 commits November 13, 2026 00:16
… seed test

Playwright drives the built AI Studio against the real backend, worker and
an ephemeral Temporal dev server; a local OpenAI-compatible server stands in
for the model. A new pr-check job runs the suite on every pull request.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…old one closes

A stream's cleanup runs on abort and again from its completion check. The
second unsubscribe removed the subscriber set a newer stream on the same run
had registered in between, so a page reloaded during a run could stop
getting live events.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…s the run

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
alek-kowal and others added 3 commits November 13, 2026 00:35
CI forces colour, so Vite prints escape codes inside its "Local:" line and
the runner never saw the preview come up.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The runner starts and stops each process as a POSIX process group, which
native Windows does not have. The README now says macOS and Linux only
and points Windows users at WSL2.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
… streams close

Co-Authored-By: Claude <noreply@anthropic.com>
alek-kowal and others added 6 commits November 18, 2026 13:07
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Playwright handles only SIGINT, and the Temporal runtime that TestWorkflowEnvironment starts
listens for SIGTERM without exiting. SIGHUP therefore killed the runner without an exit event and
left the backend, the worker, vite preview, the Temporal dev server and the database behind;
SIGTERM did not stop the run at all. Global setup now runs the teardown on both and exits with
128 + the signal number. stop() also kills the process group when its pnpm leader has already
exited.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The upload step runs on failure(), which a job cancelled by its timeout never reaches. A step
timeout under the job's limit fails the step instead.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… prompt

The old check filtered by a phrase from the send step's prompt and expected nothing, so rewording
that prompt would have made it pass whatever happened.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… ran first

The waiting-state checks only sample one moment, so a draft running alongside the gate could slip
past them. The worker passes every finished step's output to the model, so the gate's entry in the
draft's prompt shows the order from the data.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A mutation run on the suite left three product faults undetected: an approved
run stored with the rejected outcome, a Reset that kept the old log, and swapped
ports on the senior decision.

The approve and reload tests now assert the stored outcome, and the Stop test
asserts the log is gone after Reset. Three new tests walk the escalation path,
Stop on a parked run and a second run after Reset.
@librowski
librowski merged commit 90361b0 into main Nov 18, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants