Skip to content

feat(remote-react-components): cross-version backwards-compatibility smoke tests (#2600)#2661

Merged
mfal merged 58 commits into
mainfrom
claude/issue-2600-f1c7ef
Jul 21, 2026
Merged

feat(remote-react-components): cross-version backwards-compatibility smoke tests (#2600)#2661
mfal merged 58 commits into
mainfrom
claude/issue-2600-f1c7ef

Conversation

@mfal

@mfal mfal commented Jul 16, 2026

Copy link
Copy Markdown
Member

Closes #2600.

What

Adds a cross-version backwards-compatibility smoke-test harness for the remote side of Flow. It renders old published versions of @mittwald/flow-remote-react-components through the current host renderer and asserts the host-rendered output still matches — guarding the contract that mStudio (running the current version) keeps rendering extensions built against older published versions.

Why

Extensions ship against a published remote version but render inside a host on the current version. Nothing tested that the versioned remote↔host contract stays backwards-compatible. This suite catches prop/serialization/structure drift before it breaks real extensions.

How it works

  • Real production path (e2e iframe): each old version runs as its real published bundle inside a hidden iframe (its own realm), connected to the current host over the versioned @quilted/threads connection — exactly how mStudio loads extensions. (An earlier in-process attempt was abandoned: two versions can't share one custom-element registry in a single realm, which silently dropped all props. The iframe gives each side its own realm.)
  • HTML-output comparison, ephemeral reference: the same scenario is rendered through the current version (reference) and the old version (candidate); their normalized host HTML must match. Nothing is committed as a baseline — the reference is produced live each run. normalizeHtml strips volatile bits (generated ids → stable placeholders, the hidden connection iframe, data-flr-*, spinner animation offsets) while keeping semantic attributes, text, and label↔control wiring.
  • Reused visual scenarios: scenarios are authored once as (components) => JSX in src/tests/visual/*.scenarios.tsx and drive both the existing in-process Local/Remote screenshot tests and the cross-version HTML comparison. Discovery is generic (glob) — no per-test bridges.
  • Version selection: selectTargetVersions picks old versions by semver category with an alpha-offset fallback; a checked-in cross-version.exclude.json skips known-broken releases (alpha.889–895, which shipped async serializers before threads was bundled). prepare installs the targets in isolation and writes a git-ignored manifest; un-installable versions (npm publish gaps) are dropped with a warning.
  • Legitimate evolution: when a component's remote output legitimately changed across versions, it's recorded positively in scenarioVersionSupport.ts (minVersion = "comparable from version X onward", plus skipVersions), so newer in-range versions still get real coverage.
  • CI: a scheduled-only job (test-visual-scheduled.yml) — not on PRs. Dynamic OS-assigned server ports, server.warmup for cold-cache safety, and failure reporting via a $GITHUB_STEP_SUMMARY table + ::error:: annotations naming the version+scenario + an enriched Slack alert.

Reviewer notes

  • Scope of the guarantee: this is HTML/DOM-output comparison, not pixel/screenshot. It validates prop/attribute serialization + rendered structure across versions, not visual fidelity.
  • Corpus: only Button is migrated to a *.scenarios.tsx so far. The generic discovery + dual-mode pattern are in place and proven; migrating the remaining src/tests/visual/* is a follow-up (mechanical, batchable) that grows the cross-version corpus toward the full suite.
  • Everything is currently one 0.2.0-alpha.* prerelease line, so version selection uses the alpha-offset fallback (offset-10/100/200); the semver-category path is exercised by unit tests.
  • Commit history is exploratory (in-process approach → pivot to iframe → reuse → hardening). Recommend a squash-merge.

Verification

  • Cross-version suite green across the installed targets (alpha.883 / .791 / .686), including a cold Vite cache (CI's condition).
  • test:e2e and test:visual unaffected; selectTargetVersions + normalizeHtml unit tests pass; tsc / eslint / prettier clean.

mfal and others added 30 commits July 15, 2026 15:26
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…2600)

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Adds crossVersionEnvironmentsPlugin.ts, which writes a generated
testEnvironments module (one entry per cross-version manifest target,
pairing each OLD RemoteRoot/components with the CURRENT host renderer)
to a gitignored file and aliases @/tests/lib/environments to it, so
ordinary subset test files pick up the injected envs unmodified.

JSX lives in a real renderRemoteVersion.tsx helper (virtual modules
aren't picked up by @vitejs/plugin-react's transform). A generated
real file is used instead of a Vite virtual module for easier
debugging, though a cold-cache dependency-scanner limitation
(documented in the plugin's top comment) means CI wiring must budget
for a throwaway warm-up run.
…fest

A manifest that parses but resolved zero targets would silently emit
`export const testEnvironments = []`, making Task 6's
`test.each(testEnvironments)` run zero assertions and let CI go green
while validating nothing. Guard writeCrossVersionEnvironmentsModule to
throw a clear error instead.
Wire the cross-version harness to a curated 4-file subset (BigNumber,
CopyButton, LabeledValue, Tooltip) instead of the __spike__ glob, and
delete the spike test now that it's superseded.

The subset is far short of the ~10 files hoped for. Empirical curation
across ~30 candidates (all "stable primitive" components, plus every
visual test avoiding a top-level Flex wrapper) found a systemic pattern:
any prop other than `children` (color/size/boolean variants, Flex
direction/gap, or a data-testid used by a later .click()/.hover())
fails to reach the current host renderer when paired with an OLD
remote-react-components version, identically across all 3 resolved
versions. Only children composition survives; the 4 kept files pass
because none of their pixels depend on a prop. See subset.ts's top
comment and the Task 6 report for the full evidence trail and leading
hypothesis (vitest.config.ts's sharedRemoteElementsAlias forcing old
callers onto the current remote-elements package).

Also attempts optimizeDeps.entries for the documented cold-cache
flakiness (Task 5): it reduces the required cold-cache warm-up from 3
runs to 2 but does not eliminate it, so kept alongside the existing
warm-up note.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…rness

Superseded by the e2e iframe harness. Keeps the resolver, prepare script,
and exclude list; removes the same-realm vitest config, environments
generator, render helper, and subset that could not test prop-level
backwards compatibility.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…-version tests

Versions alpha.889-895 shipped async serializers (#2596) while
@quilted/threads was still externalized (not bundled until #2620/alpha.896),
so they DataCloneError on connect without a pnpm-patched threads. They are
genuinely broken published releases and must never be tested against.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…#2600)

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…hots later) (#2600)

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Adds a dedicated remote test server (e2e/cross-version/) that builds its
remote document against an installed OLD version of
@mittwald/flow-remote-react-components, selected via FLOW_CROSS_VERSION and
resolved from cross-version.manifest.json (defaults to the first target).

The server aliases the bare @mittwald/flow-remote-react-components (+ /RemoteRoot)
specifiers to the old install's dist, plus react/react-dom to the old install's
copies so the remote document uses a single React instance. It runs in its own
document/realm loaded into an iframe by RemoteRenderer, so no
@mittwald/flow-remote-elements or @quilted/threads special-casing is needed.

Port lives in a single shared constant (crossVersionServerPort) reused by the
server and the future host test. A temporary Smoke remote entry proves R1;
R2 replaces it with the real screenshot entries.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…on + references

Builds on the R1 cross-version server. For a set of representative remote
entries (Button states, Checkbox, Switch, Badge, CounterBadge, Text, Heading,
Link, TextField), the host renders each through the current RemoteRenderer over
the real iframe connection, normalizes the resulting HTML, and compares it to a
single version-independent reference per entry.

- normalizeHtml: pure, unit-tested transform that strips volatile bits
  (react-aria/React generated ids + id-referencing attrs, data-flr-version /
  data-flr-initialized, the hidden connection iframe, insignificant
  whitespace) while keeping semantic attributes and text.
- entries.browser.test.remote.tsx: deterministic render entries imported from
  the package specifier (version alias applies).
- entries.browser.test.tsx: host suite; toMatchFileSnapshot against
  __html__/<entry>.html (shared by all versions + the current reference).
- FLOW_CROSS_VERSION=current aliases the package specifier to the workspace src
  to generate the references; installed old versions are compared against them.

All 12 entries match across current, 0.2.0-alpha.883 and 0.2.0-alpha.686 — the
old versions serialize identical props/structure over the real connection. No
entries dropped. normalizeHtml unit test runs in the unit-dev project; __html__
references are prettier-ignored (compared byte-for-byte).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… instead of deleting

The cross-version HTML normalizer previously DELETED id/for/aria-* attributes
whose value was a generated id, making "attribute present, pointing at a
generated id" indistinguishable from "attribute absent" — masking real
label/control association regressions (e.g. the textField for/id wiring was
erased entirely).

- Rewrite generated id VALUES to stable, position-based placeholders
  (GENERATED_ID_1, _2, … by first appearance) and rewrite every reference to
  the same id to the same token, preserving attribute presence + cross-refs.
- Scope hidden-iframe removal to RemoteRenderer's connection iframe only
  (matched by its `visibility: hidden` inline style), so a real iframe
  component is not silently erased.
- Anchor the generated-id match to the full attribute-value token, so an
  author-set id merely containing such a substring is not dropped.
- Note the block-level assumption of the whitespace-collapse step.

Regenerated the __html__ references from current; all 12 entries still match
across current, 0.2.0-alpha.883 and 0.2.0-alpha.686 with the stricter
normalizer (the label/input associations are identical across versions), so no
entries were dropped.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
dev/cross-version/run.ts reads cross-version.manifest.json and runs the
HTML-output comparison suite once per installed old version (FLOW_CROSS_VERSION
set per target), aggregating a PASS/FAIL summary and exiting non-zero if any
version fails. Looping in node (not a shell loop) keeps it cross-platform; the
vitest CLI is spawned with the current node binary to avoid nested
pnpm/corepack spawns.

Scripts:
- test:cross-version — loop installed old versions vs. __html__ references
- test:cross-version:update — regenerate references from current (--update)
- test:cross-version:dev — vitest dev against the suite config

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add nx targetDefaults for the cross-version targets:
- test:cross-version:prepare — dependsOn ^build, cache false (network installs)
- test:cross-version — dependsOn ^build + test:cross-version:prepare, cache
  true, with inputs covering e2e/cross-version sources, dev/cross-version
  sources, the manifest, and the exclude list
- test:cross-version:update / :dev — dependsOn ^build + prepare, cache false

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add a scheduled-only `cross-version` job to test-visual-scheduled.yml (not part
of affected:test:browser on PRs). It installs the playwright shell, installs
the target old published versions (test:cross-version:prepare, a separate step
so a publish-gap install failure is visibly its own red step), then runs
test:cross-version through nx (^build served from cache). Reuses the visual
job's 3-attempt retry loop and Slack-alert-on-failure pattern.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add a "Cross-version smoke tests" section to CONTRIBUTE.md: the honest scope
(full old remote stack — old flow-remote-react-components + flow-remote-core —
in a real iframe realm, connected to the current host over the versioned
@quilted/threads connection, HTML/DOM-output comparison, not screenshots), how
to run and update references locally and when, the version-selection resolver
(semver categories + alpha-offset fallback), the broken-window exclude list
(alpha.889–895, #2596#2620) and how to exclude a version, and the publish-gap
drop behavior.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ic specifier

Rewrites the e2e remote test entries and the test server to import from
@mittwald/flow-remote-react-components (public API) instead of workspace
src paths, with the dev server aliasing the specifier back to src. This
lets the same entries be reused by the cross-version harness, where the
specifier is aliased to an old installed version instead.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ml comparison

Replaces the dedicated cross-version entries + committed HTML fixtures with:
- reuse of the existing e2e remote entries as the corpus (enumerated from
  their source, so no per-entry maintenance)
- an ephemeral reference rendered from the CURRENT version at run time (the
  existing e2e server) compared against the OLD version (cross-version
  server) — no committed baselines
- graceful skip when an old version lacks a component (predates it)
- a documented per-entry exclude list for components whose DOM legitimately
  evolved across versions (Modal#inList / List empty-view)
- run.ts fails on a zero-target manifest

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…exclusions

- excludedEntries carries a maxAffectedVersion; entries are skipped only for
  old versions at/below it, so newer in-range versions keep real coverage
  (Modal#inList now runs + passes on alpha.883, skipped on <=alpha.791)
- expose the tested version to the host bundle via define
- drop the obsolete test:cross-version:update (no committed references) and
  simplify run.ts; mark test:cross-version non-cacheable
- update CONTRIBUTE.md for the reuse + ephemeral-reference model

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Invoke the package test script directly instead of the nx target, so the
retry loop doesn't re-trigger the network prepare dependency; prepare runs
once as its own step.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…as minVersion

Replace the 'excluded up to version Y' model with a positive 'comparable from
version X onward' (minVersion) rule plus skipVersions for one-off exceptions —
easier to maintain (state when a test starts working). Behaviour unchanged:
Modal#inList runs on alpha.883, skipped on <=alpha.791.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
mfal and others added 6 commits July 17, 2026 10:02
…rness

Add a second, in-process cross-version harness alongside the high-fidelity
iframe one (hybrid). It reuses the existing src/tests/visual/*.browser.test.tsx
corpus UNMODIFIED by injecting a CrossVersion environment (old RemoteRoot +
current RemoteRenderer via a shared RemoteReceiver; first-wins customElements
patch so the old flr-* registrations win).

Comparison is STRUCTURE-ONLY: structuralHtml strips all attributes, so only the
element tree (tags + nesting + text) is compared. This is the right guarantee
for a one-realm harness — attribute-level details (prop serialization form,
class modifiers, inline styles, icon geometry) drift across versions without
being a backwards-compat break, and asserting them in-process produces false
positives (verified against the real-connection iframe harness). Structure-only
gives broad, low-noise coverage of the one thing this harness asserts reliably:
does the current host still build the same DOM shape from an old remote
version's output? All six curated tests (Checkbox, Switch, Separator, Badge,
Text, Heading) pass across the target versions with no per-diff handling.

The two harnesses cover deliberately different things — iframe = attribute
fidelity over the real connection for curated scenarios; in-process = DOM shape
across the broad reused corpus — documented in CONTRIBUTE.md.

- structuralHtml + unit tests
- run-inprocess.ts orchestration (current reference pass, per-version compare,
  GitHub ::error::/step-summary/output reporting), mirroring run.ts
- testVersionSupport for scoping genuine structural divergences
- screenshotFailures disabled (HTML comparison; avoids polluting the reused
  tests' src/**/__screenshots__ baselines)
- scripts, nx target, .gitignore, and CONTRIBUTE.md docs (explicit
  covers/does-not-cover for both harnesses)

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ss-version job

Fix the cross-version job resolving @mittwald/flow-react-components/internal and
@mittwald/flow-remote-react-renderer: the runs invoke the package scripts
directly (not the nx target), so nx's ^build never fired and the host packages
were never built from source in CI. Add an explicit
`pnpm nx run remote-react-components:build` step (its ^build closure covers
components + remote-react-renderer) before both cross-version runs.

Also wire the in-process harness into the same job (shares the one prepare step)
and extend the Slack alert to report failures from either harness.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…rom Button.scenarios

Restore Button.browser.test.tsx to its standalone, inline form instead of
importing and looping over Button.scenarios. The iframe cross-version harness
keeps Button.scenarios.tsx as its own corpus, so the Button JSX is now
duplicated across the two — a deliberate trade: a bit of duplication over
coupling a normal visual test to the cross-version scenario format.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…-process harness

Broaden the in-process cross-version harness from 6 curated tests to the whole
src/tests/visual/*.browser.test.tsx suite (136 tests green across the target
versions), keeping the structure-only comparison.

- Wire dev/vitest/setupBrowser.ts into the harness setupFiles (after the
  first-wins customElements patch) so the reused tests get their normal helpers
  (page.getByLocator, all.css).
- Restructure reusedVisualTests.ts around subtraction: REUSED_VISUAL_TESTS is
  the whole-suite glob; EXCLUDED_VISUAL_TESTS drops whole files that can't RUN
  (interaction tests whose overlay never opens in the one-realm setup — a split
  react-aria overlay context; and files that render-error on a component
  undefined in an old bundle); VERSION_SCOPED_TESTS lists individual tests whose
  element tree genuinely changed, with the fromVersion they became comparable
  from (Checkbox edge cases pinned to alpha.884 by bisect).
- testVersionSupport now gates the comparison from VERSION_SCOPED_TESTS.
- testScreenshot polls the structure until it settles (readStableStructure)
  before comparing, so async rendering can't cause a spurious diff.
- CONTRIBUTE.md documents the two subtraction lists and the overlay limitation.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…lusion reason

The excluded interaction files were annotated as "overlay never opens /
react-aria context split" — that's wrong. Overlays do render (ComboBox
interaction passes; manual clicks open the Select). The real cause: a
click-to-open control's open doesn't fire under a PROGRAMMATIC click in the
one-realm setup (the press→open round-trip to the remote side doesn't trigger;
keyboard-driven overlays and manual clicks work), so the popover never opens and
getByTestId("option") times out. Also confirmed data-testid forwards fine and
the HTML normalization doesn't touch the live-DOM selectors.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ia test.skipIf

Replace the in-process harness's two subtraction lists (file-level
EXCLUDED_VISUAL_TESTS + comparison-level VERSION_SCOPED_TESTS, plus the
render error boundary and render-gating) with a single per-test skip
predicate: test.skipIf(crossVersion({ below, exclude })). A version-bound
reused test now gates itself and is skipped whole — render, interaction and
comparison — on versions where its component didn't exist yet or its element
tree genuinely differs. In the normal visual suite the predicate is a no-op,
so every test still runs there.

Also:
- strip the hidden remote-DOM (<flr-*>) subtree in structuralHtml so the
  comparison is purely the host output the renderer builds (the flr-* mirror
  is a remote-side implementation detail that drifts freely between versions);
- split the ContextMenu visual test into a cross-version-safe basic case and
  a full (sections/Kbd) case scoped from alpha.791;
- update CONTRIBUTE.md to describe the single skip mechanism and flr-stripping;
- delete testVersionSupport.ts (no longer needed).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@mfal mfal added the update-screenshots Label a PR to update the screenshots used for visual regression testing label Jul 17, 2026
@github-actions github-actions Bot removed the update-screenshots Label a PR to update the screenshots used for visual regression testing label Jul 17, 2026
github-actions Bot and others added 4 commits July 17, 2026 16:37
Co-authored-by: mfal <4696750+mfal@users.noreply.github.com>
…for split

The ContextMenu visual test was split into a basic and a full case; regenerate
the darwin baselines (updated "ContextMenu - opened", new "ContextMenu sections
and shortcuts"). Linux baselines are handled by the update-screenshots CI job.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Behavior-preserving readability pass on the in-process cross-version harness:
rename the structuralHtml accumulator, clarify helper names in
crossVersionEnvironment, drop a redundant type annotation, and fix a
long-standing TestEnvironment typo. No functional change.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…es" cross-version

The "Checkbox edge cases" test name exists in both Checkbox.browser.test.tsx and
CheckboxButton.browser.test.tsx. The former was already scoped to alpha.884; the
latter was missed, so it compared on alpha.883/791/686 and failed (CheckboxButton
dropped its wrapper <div>s in alpha.884). Apply the same
test.skipIf(crossVersion({ below: "0.2.0-alpha.884" })) gate.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@mfal
mfal marked this pull request as ready for review July 17, 2026 19:14
mfal and others added 3 commits July 17, 2026 21:15
The skip predicate already carries the version; the comment only needs the
human-readable "comparable/available from alpha.X" note.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
# Conflicts:
#	packages/remote-react-components/src/tests/lib/environments.tsx
# Conflicts:
#	packages/remote-react-components/src/tests/visual/__screenshots__/ContextMenu.browser.test.tsx/ContextMenu-opened-firefox-darwin.png
#	packages/remote-react-components/src/tests/visual/__screenshots__/ContextMenu.browser.test.tsx/ContextMenu-opened-firefox-linux.png
#	packages/remote-react-components/src/tests/visual/__screenshots__/ContextMenu.browser.test.tsx/ContextMenu-opened-webkit-darwin.png
#	packages/remote-react-components/src/tests/visual/__screenshots__/ContextMenu.browser.test.tsx/ContextMenu-opened-webkit-linux.png
@mfal mfal added the update-screenshots Label a PR to update the screenshots used for visual regression testing label Jul 20, 2026
@github-actions github-actions Bot removed the update-screenshots Label a PR to update the screenshots used for visual regression testing label Jul 20, 2026
github-actions Bot and others added 9 commits July 20, 2026 07:53
Co-authored-by: mfal <4696750+mfal@users.noreply.github.com>
…pare step

The standalone "Build workspace packages" step only existed because the
cross-version steps ran the package scripts directly, bypassing nx's `^build`.
That direct-script choice was originally to stop a retry loop from re-running
the network prepare — but the retry loop was later removed, so the workaround
was vestigial. `test:cross-version:prepare` already declares `^build` as an nx
dependency, so running prepare via `pnpm nx run` builds the host packages
(nx-cached) and installs the old versions in one step; the standalone build
step is dropped. The two suites stay direct-script steps so each keeps its own
`failures` output for the Slack alert and the in-process suite still runs if the
iframe one fails.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ary to alpha.926

CheckboxButton's edge-case wrapper structure was flattened again in the
alpha.926 release (nothing was published between alpha.921 and alpha.926), so
the reused "Checkbox edge cases" test diverged from current on every target
>= the old alpha.884 boundary. Raise the skipIf boundary to alpha.926 — the
first published version with the current structure. Older targets skip until a
newer version ships to compare against. Only the raw CheckboxButton test is
affected; the <Checkbox> edge case stays comparable from alpha.884.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… npm not pnpm

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The selection tests only spanned two minor lines within one major, so the
cross-major behaviour of `latestOfPreviousLine` was never asserted. Add a case
with two majors where the current version is the first minor line of the new
major, verifying latestOfPreviousLine resolves to the latest published version
of the previous major.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…connection flakes

The real-iframe suite intermittently fails with "Could not establish remote
connection: Timeout reached" — a transient @quilted/threads handshake flake
under CI, not an HTML diff (~2 of 3 recent scheduled runs hit it on varying
versions). The prior "no retry" comment assumed this suite doesn't flake, which
is wrong. Wrap it in a 3x retry loop (same pattern as the visual job's Firefox
hang retry): a real divergence fails deterministically on every attempt, while a
connection flake clears on a fresh run. Kept as a direct package-script call so a
retry doesn't re-trigger the non-cached network prepare. The in-process suite
keeps its per-test vitest retry (no real connection to time out).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
run.ts (iframe) and run-inprocess.ts (in-process) were ~90% identical: manifest
reading, failed-test parsing, GitHub error/summary/output reporting, vitest-bin
resolution, and the per-version loop. Extract all of it into crossVersionRunner.ts;
the two entry scripts now just pass what differs — vitest config, temp prefix,
report labels, and (in-process only) the optional --update reference pass. No
behavioural change; the vitest invocations are identical.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…own file

Move the ScenarioVersionRule interface and the scenarioVersionRules data array
to scenarioVersionRules.ts; scenarioVersionSupport.ts keeps only the matching
logic (isScenarioComparable) and imports the data. Separates the version-scoping
data from the logic that consumes it.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…against truncation transient

readStableStructure returned on the first single pair of matching samples, which
could capture a ResizeObserver-driven text-truncation transient (a measuring DOM
copy briefly duplicated alongside the displayed one) before it settled — making
the long-label "edge cases" tests flake intermittently in the structure-only
comparison. Wait for a sustained quiet window instead (5 consecutive identical
samples, up to a ~4s cap).

This cleanly separated the two edge-case failures: <Checkbox> now compares
reliably (it was pure capture flake), while <CheckboxButton> is confirmed a real
deterministic divergence at <= alpha.921 vs current — so its alpha.926 boundary
stays; comment clarified.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@mfal
mfal merged commit 1f3efa0 into main Jul 21, 2026
9 of 11 checks passed
@mfal
mfal deleted the claude/issue-2600-f1c7ef branch July 21, 2026 06:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Visual regression tests for different remote versions

2 participants