Repository navigation
test(e2e): AI Studio end-to-end tests on the full stack with a fake LLM - #270
Merged
Merged
Conversation
… seed test Playwright drives the built AI Studio against the real backend, worker and an ephemeral Temporal dev server; a local OpenAI-compatible server stands in for the model. A new pr-check job runs the suite on every pull request. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…old one closes A stream's cleanup runs on abort and again from its completion check. The second unsubscribe removed the subscriber set a newer stream on the same run had registered in between, so a page reloaded during a run could stop getting live events. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…s the run Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
alek-kowal
requested review from
Jacek-Synergy,
Kuba-Skibinski,
librowski,
lukasz-jazwa,
piotrblaszczyk and
szymon-t-sc
as code owners
November 12, 2026 23:30
CI forces colour, so Vite prints escape codes inside its "Local:" line and the runner never saw the preview come up. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The runner starts and stops each process as a POSIX process group, which native Windows does not have. The README now says macOS and Linux only and points Windows users at WSL2. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
… streams close Co-Authored-By: Claude <noreply@anthropic.com>
librowski
approved these changes
Nov 16, 2026
librowski
approved these changes
Nov 16, 2026
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Playwright handles only SIGINT, and the Temporal runtime that TestWorkflowEnvironment starts listens for SIGTERM without exiting. SIGHUP therefore killed the runner without an exit event and left the backend, the worker, vite preview, the Temporal dev server and the database behind; SIGTERM did not stop the run at all. Global setup now runs the teardown on both and exits with 128 + the signal number. stop() also kills the process group when its pnpm leader has already exited. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The upload step runs on failure(), which a job cancelled by its timeout never reaches. A step timeout under the job's limit fails the step instead. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… prompt The old check filtered by a phrase from the send step's prompt and expected nothing, so rewording that prompt would have made it pass whatever happened. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… ran first The waiting-state checks only sample one moment, so a draft running alongside the gate could slip past them. The worker passes every finished step's output to the model, so the gate's entry in the draft's prompt shows the order from the data. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A mutation run on the suite left three product faults undetected: an approved run stored with the rejected outcome, a Reset that kept the old log, and swapped ports on the senior decision. The approve and reload tests now assert the stored outcome, and the Stop test asserts the log is gone after Reset. Three new tests walk the escalation path, Stop on a parked run and a second run after Reset.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Browser end-to-end tests for AI Studio on the full stack: the built app, the real backend and worker, an ephemeral Temporal dev server, and a fake LLM. A new
pr-checkjob runs them on every pull request.apps/e2e: a private Playwright workspace.pnpm e2estarts the stack on its own ports with its own database, runs the suite and stops everything it started. Postgres is the only outside service (pnpm e2e:db:uplocally, a service container in CI). It has notestscript, sopnpm testdoes not start the stack.AI_BASE_URLfor the backend and the worker. It answers structured requests from their JSON schema, records every request so a test can assert what reached the model, and can hold an answer to keep a node running. No key, no cost.e2ejob inpr-check.yml, with the Playwright report and the stack logs uploaded on failure.Backend fix found by the reload test
A run stream reopened right after another stream on the same run closed could stop receiving live events: the closed stream's cleanup ran twice, and the second unsubscribe removed the subscriber set the new stream had registered. After a page reload the UI could then stay on "Waiting for decision" while the stored run had completed. Fixed in
execution-event-bus.ts, with a unit test, in its own commit.How it was checked
--repeat-each=5, in parallel; nothing was left running afterwards.pnpm checkstops inapps/docstypecheck, as it does onmain.e2ejob on GitHub.Notes for review
Selectors use roles and accessible names, React Flow's
data-idfor nodes anddata-node-idfor log rows; there is nodata-testid. A few places have no accessible name today (decision form fields, the run status chip, the rejection reason textarea), so the helpers find them by structure.apps/e2e/README.mdlists those, along with editor behaviours the tests work around: a text field in the properties panel is saved on blur, so Run clicked straight out of it starts with the old value; a node's own Decide button can be covered during a run; overlapping edges can take the click meant for another edge.🤖 Generated with Claude Code