Skip to content

ci(e2e): bound the playwright apt step so a stalled mirror can't eat the job - #6239

Merged
pedrofrxncx merged 1 commit into
mainfrom
ci/bound-playwright-install-deps
Aug 19, 2026
Merged

pedrofrxncx merged 1 commit into
mainfrom
ci/bound-playwright-install-deps

Conversation

@pedrofrxncx

@pedrofrxncx pedrofrxncx commented Aug 19, 2026 •

Copy link
Copy Markdown
Collaborator

What happened

Run 32264701513 shard 2, attempt 1, step by step:

step shard 1 shard 2
npx playwright install-deps chromium 25s 759s
Run e2e tests 87s never started

The log shows apt printing its mirrorlist at 14:35:57 and then nothing at all until the job was cancelled at 14:46:56 — 11 silent minutes on an Azure apt mirror. The runner then reaped npm exec playwright install-deps chromium as an orphan. The tests themselves take 87 seconds.

With no bound on the step, a stalled mirror is free to consume the job's whole 35-minute budget, and the only signal is a check that sits there looking busy.

Change

timeout-minutes: 5 on both Playwright install steps, in e2e.yml (e2e-shard) and test.yml (web-component-tests) — all four have the same apt exposure. GitHub kills the step's whole process tree, so a stall becomes a fast, obvious red that re-runs in ~4 minutes.

Headroom is generous: 25s observed for install-deps on a cache hit, and the cache-miss path (install --with-deps, which also downloads the browser) is well under 5 minutes.

Not in scope

  • Retrying the step. timeout on the shell command only signals npx; apt survives as an orphan holding the dpkg lock, so attempt 2 fails on lock contention. Worth revisiting if mirror stalls turn out to be frequent rather than rare — right now this is n=1.
  • Shard rebalancing. Measured over the last ~60 PR runs, e2e-shard (2) exec p50 is 4.5m / p90 9.4m against shard 1's 3.9m / 6.0m. Real, but a separate problem from this one.
  • Self-hosted runners (ci: opt-in self-hosted runners for checks, build and release #5828). Queue time on this repo right now measures p50 2s / p90 5s across test.yml, and effectively zero for e2e.yml. Whatever this run cost, it was not queue.

Testing

YAML parses and the four timeouts land on the intended steps. Otherwise this is CI config — the e2e run on this PR exercises the cache-hit path.


Summary by cubic

Bounds the Playwright install steps in CI to 5 minutes so a stalled apt mirror can’t consume the whole job. Previously these steps were unbounded and could hang silently; now they time out in 5 minutes and GitHub kills the process tree, failing fast and making reruns quick.

  • Adds timeout-minutes: 5 to four steps across .github/workflows/e2e.yml and .github/workflows/test.yml for both install chromium --with-deps and install-deps chromium.
  • Headroom is safe: cache hit ~25s; cache miss is typically under 5m. If a slow cache miss exceeds 5m, re-run the job.
  • No retry logic added; this change only bounds the step duration.

Written for commit 55c583b. Summary will update on new commits.

Review in cubic

@pedrofrxncx
pedrofrxncx merged commit 8a05fc6 into main Aug 19, 2026
29 checks passed
@pedrofrxncx
pedrofrxncx deleted the ci/bound-playwright-install-deps branch August 19, 2026 15:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant