Skip to content

fix(remote-react-components): scope cross-version refs per test file - #2688

Merged
mfal merged 2 commits into
mainfrom
claude/cross-version-tests-failing-63de8d
Jul 23, 2026
Merged

fix(remote-react-components): scope cross-version refs per test file#2688
mfal merged 2 commits into
mainfrom
claude/cross-version-tests-failing-63de8d

Conversation

@mfal

@mfal mfal commented Jul 23, 2026

Copy link
Copy Markdown
Member

Problem

The scheduled Run scheduled visual tests workflow's cross-version job has failed on every run since it was introduced (2026-07-21) — specifically the in-process harness, with a Checkbox edge cases(CrossVersion) structural-snapshot mismatch against the previous published version.

It is not a version incompatibility. The in-process harness keyed each ephemeral reference HTML file only on testName + description:

`${REF_DIR}/${slugify(`${testName}-${description}`)}.html`

Two reused visual tests share the exact name "Checkbox edge cases":

  • Checkbox.browser.test.tsx → renders one <Checkbox> (7-tag tree)
  • CheckboxButton.browser.test.tsx → renders a Flex with two <CheckboxButton>s (18-tag tree) — the test is mis-named "Checkbox edge cases"

Both therefore read and write the same .refs/*.html file. Whichever test writes the reference last makes the other's comparison mismatch. This is deterministic (it failed on the job's very first scheduled run and every one since), and unrelated to the actual old version — current and the compared version render identically once the tests are keyed apart.

The normal visual screenshot suite does not hit this because vitest scopes screenshots per test file (__screenshots__/<file>/…); only the cross-version harness used a flat, non-file-scoped key.

Fix

Include the source-file basename in the reference key, mirroring vitest's own per-file screenshot scoping. Small, self-contained change to crossVersionEnvironment.tsx. This resolves the collision and prevents any future same-name collision.

Validation

  • Installed @mittwald/flow-remote-react-components@0.2.0-alpha.932 locally and reproduced the failure via the full 83-file corpus.
  • After the fix: full-corpus comparison vs alpha.932 → 167/167 pass, run twice, deterministically green; the reference pass now writes two distinct edge-case files instead of one.
  • test:compile (tsc), ESLint, and Prettier all clean.
  • No impact on the visual suite or committed screenshots — the .refs are wiped and regenerated on every run.

Notes for reviewers

  • Supersedes the approach in Stabilize cross-version CI by skipping incompatible CheckboxButton edge-case snapshots before alpha.933 #2683, which bumps the cross-version skip watermark. That treats this harness bug as a version break (and targets alpha.933, a docs-only release that renders identically to 932), so it would not durably fix the failure.
  • Optional follow-up: rename the CheckboxButton test from "Checkbox edge cases" to "CheckboxButton edge cases" for clarity. Left out here because the description also names that test's committed visual screenshot baselines (CheckboxButton.browser.test.tsx/Checkbox-edge-cases-*.png), so renaming would require regenerating those (including the Linux baseline via the update-screenshots label). The file-scoping fix makes the duplicate name harmless regardless.
  • Separate, pre-existing flake (out of scope): the iframe step's Button scenarios intermittently hit ~180–240s @quilted/threads connection-timeout hangs; the workflow's retry loop usually absorbs it.

🤖 Generated with Claude Code

The in-process cross-version harness keyed each ephemeral reference HTML
file only on the test name + description. Two reused visual tests share a
name — `Checkbox.browser.test.tsx` and `CheckboxButton.browser.test.tsx`
both have a `Checkbox edge cases` test — so both read and wrote the SAME
`.refs/*.html` file. They render structurally different trees (a lone
`<Checkbox>` vs a `Flex` with two `<CheckboxButton>`s), so whichever test
wrote the reference last made the other's version comparison mismatch.

Since the cross-version job was added (2026-07-21) it failed on its first
scheduled run and every one since: deterministic, not a flake, and not a
real version incompatibility (current and the compared old version render
identically once keyed apart).

Include the source-file basename in the reference key, mirroring how
vitest scopes browser screenshots per test file. This resolves the
collision and prevents any future same-name collision. No impact on the
visual suite or committed screenshots; the `.refs` are regenerated each run.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

Coverage Report for ./packages/components/

Status Category Percentage Covered / Total
🔵 Lines 78.09% 328 / 420
🔵 Statements 78.08% 335 / 429
🔵 Functions 81.72% 76 / 93
🔵 Branches 66.81% 151 / 226
File CoverageNo changed files found.
Generated in workflow #5727 for commit 0f9003b by the Vitest Coverage Report Action

… edge cases"

The test was mis-named identically to Checkbox's own "Checkbox edge cases"
test. The preceding commit made the cross-version reference key file-scoped
so the shared name is no longer harmful, but the name is still confusing.
Rename it for clarity; the render is unchanged, so the committed visual
baselines are moved (git mv) rather than regenerated.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@mfal mfal added the update-screenshots Label a PR to update the screenshots used for visual regression testing label Jul 23, 2026
@github-actions github-actions Bot removed the update-screenshots Label a PR to update the screenshots used for visual regression testing label Jul 23, 2026
@github-actions

Copy link
Copy Markdown
Contributor

🚀 Preview Deployment

Preview environments are ready:

Type URL
docs pr-2688.docs.review.flow-components.de
storybook pr-2688.storybook.review.flow-components.de

Images:

  • docs: ghcr.io/mittwald/flow/docs:pr-2688
  • storybook: ghcr.io/mittwald/flow/storybook:pr-2688

@mfal
mfal enabled auto-merge (squash) July 23, 2026 10:21
@mfal mfal added the update-screenshots Label a PR to update the screenshots used for visual regression testing label Jul 23, 2026
@github-actions github-actions Bot removed the update-screenshots Label a PR to update the screenshots used for visual regression testing label Jul 23, 2026
@mfal mfal added the run-visual-tests Runs the full visual regression suite against the existing baselines and fails the check on mismatch label Jul 23, 2026
@github-actions github-actions Bot removed the run-visual-tests Runs the full visual regression suite against the existing baselines and fails the check on mismatch label Jul 23, 2026
@mfal
mfal merged commit 65c3cc5 into main Jul 23, 2026
10 of 11 checks passed
@mfal
mfal deleted the claude/cross-version-tests-failing-63de8d branch July 23, 2026 10:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants