ci: opt-in self-hosted runners for checks, build and release - #5828
Draft
nicacioliveira wants to merge 2 commits into
Draft
ci: opt-in self-hosted runners for checks, build and release#5828nicacioliveira wants to merge 2 commits into
nicacioliveira wants to merge 2 commits into
Conversation
format, lint, typecheck and test now resolve `runs-on` from the CI_RUNNER_LINUX repository variable, defaulting to ubuntu-latest when it is unset. This commit is a no-op until someone sets the variable. These four run 0.5-2 min but have waited up to ~90 min at the p99 — GitHub's hosted concurrency ceiling saturating at ~915 runs/day, not slow execution (queue p50 is 6s). They also need nothing beyond checkout + setup-bun, so they run on the stock ARC runner image with no sudo, no docker and no privileged sidecar. changes, build and web-component-tests stay hosted. Driving it from a variable rather than a hardcoded label matters because these are required checks: a job pointed at an unavailable scale set sits queued until GitHub's 24h timeout instead of failing fast, and nothing merges. Rollback is `gh variable delete CI_RUNNER_LINUX`, effective on the next run, with no PR. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
test.yml: `build` joins format/lint/typecheck/test on CI_RUNNER_LINUX. It
needs nothing beyond checkout + setup-bun + node, so it runs on the stock
ARC image. `changes` stays hosted so a scale-set outage still produces a
run that reports; `web-component-tests` stays hosted because
`playwright install --with-deps` shells out to sudo apt-get, which the
stock image has no root for.
release-studio.yaml: three separate variables, all defaulting to the
hosted runners used today, so unset is a no-op.
CI_RUNNER_RELEASE build-dist, publish-npm, release,
bump-deco-apps-cd, notify-docs-agent
CI_RUNNER_RELEASE_DIND prepare + docker jobs, linux/amd64
CI_RUNNER_RELEASE_DIND_ARM64 docker jobs, linux/arm64
Kept separate from CI_RUNNER_LINUX because the risk is asymmetric: a stuck
PR check annoys one author, a stuck release blocks shipping.
The arm64 variable is not cosmetic — this matrix builds arm64 natively on
ubuntu-24.04-arm, and pointing that leg at an amd64 runner would silently
regress it to QEMU emulation.
`prepare` is grouped with the docker jobs because docker/login-action and
`docker manifest inspect` need the docker CLI, which the stock runner image
only has wired up in dind mode.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Draft on purpose — do not merge until
decocms/terraform-eks-cluster#94is applied anddecocms/infra_applications#241reports the scale sets Ready.Makes runner selection resolve from repository variables, all defaulting to the hosted runners used today. Unset = no-op, so merging this changes nothing on its own.
Mapping
test.yml— one variable:format,lint,typecheck,test,build${{ vars.CI_RUNNER_LINUX || 'ubuntu-latest' }}changesweb-component-testsplaywright install --with-depsshells out tosudo apt-get, no root on the stock ARC imagerelease-studio.yaml— three variables, kept separate from the above:build-dist,publish-npm,release,bump-deco-apps-cd,notify-docs-agentCI_RUNNER_RELEASEubuntu-latestprepare,build-docker(amd64),build-nginx-docker(amd64),merge-docker,merge-nginx-dockerCI_RUNNER_RELEASE_DINDubuntu-latestbuild-docker(arm64),build-nginx-docker(arm64)CI_RUNNER_RELEASE_DIND_ARM64ubuntu-24.04-armWhy the checks and the release path use different variables
The risk is asymmetric. A stuck PR check annoys one author; a stuck release blocks shipping. These are required checks and a job pointed at an unavailable scale set does not fail fast — it sits queued until GitHub's 24h timeout. Separate variables mean the checks can be proven for a week before the release path is touched at all.
Why arm64 has its own variable
build-dockerandbuild-nginx-dockerbuildlinux/arm64natively onubuntu-24.04-arm, not under QEMU. Pointing that leg at an amd64 runner would silently regress it to emulation and roughly triple the build. The variable must resolve to a genuinely arm64 scale set (deco-linux-arm64-dind) or stay hosted.Why
prepareis grouped with the docker jobsIt runs
docker/login-actionanddocker manifest inspect, which need the docker CLI — wired up on the stock runner image only in dind mode.Why these jobs at all
Measured across 146 runs / 2.824 jobs on this repo:
formatlinttypechecktestbuildQueue p50 across the repo is 6s and p90 is 48s — the median is healthy. But 15% of runs exceed 20 minutes and, in those, ~59% of the wall clock is queue, with p99 reaching ~90 min. That is the hosted concurrency ceiling saturating at ~915 runs/day, not slow execution.
Not a cost change
This repo is public, so its hosted minutes are already free. The pilot adds spot EC2 spend. It buys the p99, not the invoice.
Toggling
Effective on the next run. No PR, no merge queue.
Known trade-off while any variable is on
The runner pool is spot-only. A reclaim fails the job outright — GitHub does not reschedule a runner that vanished — so expect occasional re-runs. Exposure is small for the short check jobs and larger for image builds, which is part of why the release path is gated behind its own variables.
Rollout
terraform-eks-cluster#94, createhub/arc/github-app, mergeinfra_applications#241.CI_RUNNER_LINUX. Watch one PR, then a week; re-measure queue p99.CI_RUNNER_RELEASE*.🤖 Generated with Claude Code