Skip to content

ci: opt-in self-hosted runners for checks, build and release - #5828

Draft
nicacioliveira wants to merge 2 commits into
mainfrom
ci/arc-selfhosted-runners-pilot
Draft

ci: opt-in self-hosted runners for checks, build and release#5828
nicacioliveira wants to merge 2 commits into
mainfrom
ci/arc-selfhosted-runners-pilot

Conversation

@nicacioliveira

@nicacioliveira nicacioliveira commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

Draft on purpose — do not merge until decocms/terraform-eks-cluster#94 is applied and decocms/infra_applications#241 reports the scale sets Ready.

Makes runner selection resolve from repository variables, all defaulting to the hosted runners used today. Unset = no-op, so merging this changes nothing on its own.

Mapping

test.yml — one variable:

job runner
format, lint, typecheck, test, build ${{ vars.CI_RUNNER_LINUX || 'ubuntu-latest' }}
changes stays hosted — it gates the rest, so a scale-set outage still produces a run that reports
web-component-tests stays hosted — playwright install --with-deps shells out to sudo apt-get, no root on the stock ARC image

release-studio.yaml — three variables, kept separate from the above:

jobs variable default
build-dist, publish-npm, release, bump-deco-apps-cd, notify-docs-agent CI_RUNNER_RELEASE ubuntu-latest
prepare, build-docker (amd64), build-nginx-docker (amd64), merge-docker, merge-nginx-docker CI_RUNNER_RELEASE_DIND ubuntu-latest
build-docker (arm64), build-nginx-docker (arm64) CI_RUNNER_RELEASE_DIND_ARM64 ubuntu-24.04-arm

Why the checks and the release path use different variables

The risk is asymmetric. A stuck PR check annoys one author; a stuck release blocks shipping. These are required checks and a job pointed at an unavailable scale set does not fail fast — it sits queued until GitHub's 24h timeout. Separate variables mean the checks can be proven for a week before the release path is touched at all.

Why arm64 has its own variable

build-docker and build-nginx-docker build linux/arm64 natively on ubuntu-24.04-arm, not under QEMU. Pointing that leg at an amd64 runner would silently regress it to emulation and roughly triple the build. The variable must resolve to a genuinely arm64 scale set (deco-linux-arm64-dind) or stay hosted.

Why prepare is grouped with the docker jobs

It runs docker/login-action and docker manifest inspect, which need the docker CLI — wired up on the stock runner image only in dind mode.

Why these jobs at all

Measured across 146 runs / 2.824 jobs on this repo:

job exec (median) queue (p90)
format 0,4 min 4,9 min
lint 0,5 min 6,5 min
typecheck 0,9 min 4,7 min
test 1,1 min 6,3 min
build 1,8 min 5,2 min

Queue p50 across the repo is 6s and p90 is 48s — the median is healthy. But 15% of runs exceed 20 minutes and, in those, ~59% of the wall clock is queue, with p99 reaching ~90 min. That is the hosted concurrency ceiling saturating at ~915 runs/day, not slow execution.

Not a cost change

This repo is public, so its hosted minutes are already free. The pilot adds spot EC2 spend. It buys the p99, not the invoice.

Toggling

on:  gh variable set CI_RUNNER_LINUX --body deco-linux-amd64 --repo decocms/studio
off: gh variable delete CI_RUNNER_LINUX --repo decocms/studio

Effective on the next run. No PR, no merge queue.

Known trade-off while any variable is on

The runner pool is spot-only. A reclaim fails the job outright — GitHub does not reschedule a runner that vanished — so expect occasional re-runs. Exposure is small for the short check jobs and larger for image builds, which is part of why the release path is gated behind its own variables.

Rollout

  1. Apply terraform-eks-cluster#94, create hub/arc/github-app, merge infra_applications#241.
  2. Merge this (still a no-op).
  3. Set CI_RUNNER_LINUX. Watch one PR, then a week; re-measure queue p99.
  4. Only then consider CI_RUNNER_RELEASE*.

🤖 Generated with Claude Code

nicacioliveira and others added 2 commits August 6, 2026 17:28
format, lint, typecheck and test now resolve `runs-on` from the
CI_RUNNER_LINUX repository variable, defaulting to ubuntu-latest when it
is unset. This commit is a no-op until someone sets the variable.

These four run 0.5-2 min but have waited up to ~90 min at the p99 —
GitHub's hosted concurrency ceiling saturating at ~915 runs/day, not slow
execution (queue p50 is 6s). They also need nothing beyond checkout +
setup-bun, so they run on the stock ARC runner image with no sudo, no
docker and no privileged sidecar. changes, build and web-component-tests
stay hosted.

Driving it from a variable rather than a hardcoded label matters because
these are required checks: a job pointed at an unavailable scale set sits
queued until GitHub's 24h timeout instead of failing fast, and nothing
merges. Rollback is `gh variable delete CI_RUNNER_LINUX`, effective on
the next run, with no PR.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
test.yml: `build` joins format/lint/typecheck/test on CI_RUNNER_LINUX. It
needs nothing beyond checkout + setup-bun + node, so it runs on the stock
ARC image. `changes` stays hosted so a scale-set outage still produces a
run that reports; `web-component-tests` stays hosted because
`playwright install --with-deps` shells out to sudo apt-get, which the
stock image has no root for.

release-studio.yaml: three separate variables, all defaulting to the
hosted runners used today, so unset is a no-op.

  CI_RUNNER_RELEASE             build-dist, publish-npm, release,
                                bump-deco-apps-cd, notify-docs-agent
  CI_RUNNER_RELEASE_DIND        prepare + docker jobs, linux/amd64
  CI_RUNNER_RELEASE_DIND_ARM64  docker jobs, linux/arm64

Kept separate from CI_RUNNER_LINUX because the risk is asymmetric: a stuck
PR check annoys one author, a stuck release blocks shipping.

The arm64 variable is not cosmetic — this matrix builds arm64 natively on
ubuntu-24.04-arm, and pointing that leg at an amd64 runner would silently
regress it to QEMU emulation.

`prepare` is grouped with the docker jobs because docker/login-action and
`docker manifest inspect` need the docker CLI, which the stock runner image
only has wired up in dind mode.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@nicacioliveira nicacioliveira changed the title ci: allow the four cheap check jobs to run on self-hosted runners ci: opt-in self-hosted runners for checks, build and release Aug 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant