ci(repo): scope integration test inputs to packages the e2e apps use

.changeset/scope-integration-test-inputs.md

@@ -0,0 +1,2 @@
+---
+---

.claude/skills/clerk-monorepo/SKILL.md

@@ -138,6 +138,60 @@ If the editor or a build reports stale types from `@clerk/shared`, rebuild the f
 Integration-test variants (`pnpm test:integration:*`) and canary/snapshot releases are the long tail:
 see [`references/setup-and-footguns.md`](references/setup-and-footguns.md) and `docs/CONTRIBUTING.md`.
 
+## Integration CI caching
+
+Each Playwright suite is a root turbo task, `//#test:integration:<suite>`, run by the `integration-tests`
+matrix in `.github/workflows/ci.yml`. Turbo caches a suite's result, so a PR that cannot change the
+outcome skips the suite entirely: no build, no local registry publish, no browser install.
+
+How the cache key is built:
+
+- `dependsOn` lists `<pkg>#build:inputs` for each package the suite uses: always `@clerk/clerk-js` and
+  `@clerk/ui`, the packages the harness imports (the `devDependencies` of `integration/package.json`),
+  plus the framework packages that suite's apps install from the local registry. `build:inputs` is a transit node: no package defines the script, so it never
+  runs or restores anything, but it hashes the same inputs as `build` and depends on `^build:inputs`,
+  so transitive packages such as `@clerk/shared` are covered without being listed. A cache hit
+  therefore downloads no build outputs; on a miss the runner script builds.
+- `inputs` covers the harness, the test directories the suite's tagged tests live in (and import
+  from), the templates its apps copy, and any package source file a test imports directly, such as a
+  `packages/*/src/test` fixture the package build excludes. Other suites' directories are not inputs. Tests live under
+  `integration/tests/<suite>/`; a test tagged for several suites lives in `generic/`, and tests no
+  suite runs live in `unscheduled/`.
+- `scripts/integration-deps.test.mjs` derives both lists from `integration/presets` and the tagged
+  tests and fails when `turbo.json` drifts. Its failure message prints the block to paste.
+
+Rules that keep this correct:
+
+- The integration test step must never pass `--only`. It drops dependency hashes from the task hash,
+  so package changes would stop invalidating suites.
+- Keep `build:inputs` identical to `build`'s inputs, in the root `turbo.json` and in every package
+  `turbo.json` that overrides `build.inputs`; `integration-deps.test.mjs` enforces both.
+- The harness's workspace dependencies live in `integration/package.json`, never the root
+  `package.json`. Turbo folds every file of a root workspace dependency, tests and READMEs included,
+  into the global hash, so any edit to it would rerun every task in the repo.
+- The merge queue sets `TURBO_FORCE=true` on the test step and nightly passes `--force`, so every suite
+  runs for real before a commit lands on `main`. Forcing reaches only the suite and its transit nodes;
+  the runner script's own `turbo build` resets `TURBO_FORCE` so package builds still restore from cache. A `dependsOn` gap is a wasted-cache bug on PRs, not a
+  correctness hole on `main`.
+- Per-job setup lives in `integration/scripts/run-integration.mjs`, which the `test:integration:base`
+  script wraps. When `E2E_CI_SETUP` names the suite it builds, publishes to pkglab, installs clerk-js
+  and ui into temp dirs, installs Chromium and, for electron, Xvfb, then runs Playwright. Locally
+  `E2E_CI_SETUP` is unset and it only runs Playwright. It drops `CLERK_*` from the build's
+  environment: `CLERK_*` is in `globalEnv`, so the test step's secrets would otherwise change every
+  build hash and miss the Build Packages job's cache. A cache hit runs none of it, including the
+  user cleanup step, which is gated on the `tests-started` output the script writes.
+
+Adding a suite:
+
+1. Add `test:integration:<suite>` to the root `package.json`, tagging tests with `@<suite>` and
+   pointing `E2E_APP_ID` at the long-running apps it needs.
+2. Add `//#test:integration:<suite>` to `turbo.json`. Run `pnpm vitest run --root scripts
+integration-deps` and paste the `dependsOn` and `inputs` the failure prints.
+3. Add the suite to the `integration-tests` matrix in `ci.yml`.
+
+To see what a change invalidates, run `node scripts/integration-hash-matrix.mjs` on a clean tree. It
+touches a few representative files, dry-runs turbo, and prints which suites would rerun for each.
+
 ## The hard rules
 
 Each rule below restates `AGENTS.md`; the parenthetical is how it is enforced.

.github/actions/init-blacksmith/action.yml

@@ -9,6 +9,10 @@ inputs:
     description: 'Enable Playwright?'
     required: false
     default: 'false'
+  playwright-install-enabled:
+    description: 'Install Playwright browsers after restoring the cache?'
+    required: false
+    default: 'true'
   turbo-summarize:
     description: 'The token to use for Turbo task summaries'
     required: false
@@ -146,7 +150,7 @@ runs:
         key: ${{ runner.os }}-node-${{ env.NODE_VERSION }}-playwright-${{ steps.playwright-version.outputs.VERSION }}-v2
 
     - name: Install Playwright Browsers
-      if: inputs.playwright-enabled == 'true' && steps.playwright-cache.outputs.cache-hit != 'true'
+      if: inputs.playwright-enabled == 'true' && inputs.playwright-install-enabled == 'true' && steps.playwright-cache.outputs.cache-hit != 'true'
       shell: bash
       run: |
         pnpm playwright install chromium

.github/workflows/ci.yml

@@ -379,9 +379,9 @@ jobs:
         env:
           NODE_VERSION: ${{ matrix.node-version }}
 
-      - name: Run verify script tests
+      - name: Run CI script tests
         if: matrix.node-version == '24.15.0'
-        run: pnpm vitest run --root scripts verify-attach verify-flaky
+        run: pnpm vitest run --root scripts verify-attach verify-flaky run-integration integration-deps
 
       - name: Upload Turbo Summary
         uses: actions/upload-artifact@v4.6.2
@@ -582,81 +582,12 @@ jobs:
           turbo-team: ${{ vars.TURBO_TEAM }}
           turbo-token: ${{ secrets.TURBO_TOKEN }}
           playwright-enabled: true
-
-      - name: Verify jq is installed
-        shell: bash
-        run: |
-          if ! command -v jq &> /dev/null; then
-            echo "jq not found, installing..."
-            sudo apt-get update && sudo apt-get install -y jq
-          fi
-          jq --version
+          playwright-install-enabled: false
 
       - name: Validate turbo task
-        env:
-          E2E_APP_CLERK_JS_DIR: ${{runner.temp}}
-          E2E_APP_CLERK_UI_DIR: ${{runner.temp}}
-          E2E_CLERK_JS_VERSION: 'latest'
-          E2E_CLERK_UI_VERSION: 'latest'
-          E2E_NEXTJS_VERSION: ${{ matrix.next-version }}
-          E2E_PROJECT: ${{ matrix.test-project }}
-          INTEGRATION_INSTANCE_KEYS: ${{ secrets.INTEGRATION_INSTANCE_KEYS }}
         run: |
-          # Validate the turbo task exists. Turbo's --affected flag is a no-op for
-          # root-level tasks (//#) — they are always returned as affected. The previous
-          # --affected check only served to silently skip tests when the task was missing
-          # from turbo.json, hiding real configuration errors.
           TASK_NAME="test:integration:${{ matrix.test-name }}"
-          TURBO_STDERR=$(mktemp)
-          if ! TURBO_RAW=$(pnpm turbo run "$TASK_NAME" --dry=json 2>"$TURBO_STDERR"); then
-            echo "::error::Turbo task '$TASK_NAME' failed validation"
-            cat "$TURBO_STDERR"
-            exit 1
-          fi
-
-          # pnpm can prepend non-JSON lines (e.g. `WARN  Unsupported engine: ...`)
-          # to stdout, which would break jq. Drop everything before the first `{`.
-          TURBO_JSON=$(printf '%s\n' "$TURBO_RAW" | awk '/^\{/{found=1} found')
-
-          if ! TASK_COUNT=$(jq -er '.tasks | length' <<< "$TURBO_JSON"); then
-            echo "::error::Turbo task '$TASK_NAME' returned invalid JSON or missing .tasks"
-            printf '%s\n' "$TURBO_JSON"
-            exit 1
-          fi
-
-          if [ "$TASK_COUNT" -eq 0 ]; then
-            echo "::error::Turbo task '$TASK_NAME' returned 0 tasks"
-            exit 1
-          fi
-
-          echo "Task '$TASK_NAME' validated ($TASK_COUNT tasks in graph)"
-
-      - name: Build packages
-        run: pnpm turbo build $TURBO_ARGS --only
-
-      - name: Publish to local registry
-        run: pkglab pub --force
-
-      - name: Edit .npmrc [link-workspace-packages=false]
-        run: sed -i -E 's/link-workspace-packages=(deep|true)/link-workspace-packages=false/' .npmrc
-
-      - name: Install @clerk/clerk-js in os temp
-        working-directory: ${{runner.temp}}
-        run: |
-          mkdir clerk-js && cd clerk-js
-          pnpm init
-          pkglab add @clerk/clerk-js
-
-      - name: Install @clerk/ui in os temp
-        working-directory: ${{runner.temp}}
-        run: |
-          mkdir clerk-ui && cd clerk-ui
-          pnpm init
-          pkglab add @clerk/ui
-
-      - name: Copy components @clerk/astro
-        if: ${{ matrix.test-name == 'astro' }}
-        run: cd packages/astro && pnpm copy:components
+          node -e 'const pkg = require("./package.json"); const turbo = require("./turbo.json"); const name = process.argv[1]; if (!pkg.scripts[name] || !turbo.tasks["//#" + name]) process.exit(1)' "$TASK_NAME"
 
       - name: Write all ENV certificates to files in integration/certs
         uses: actions/github-script@v7.1.0
@@ -675,18 +606,6 @@ jobs:
                 fs.writeFileSync(path.join(process.env.GITHUB_WORKSPACE, 'integration/certs', name), cert);
             }
 
-      - name: LS certs
-        working-directory: ./integration/certs
-        run: ls -la && pwd
-
-      - name: Ensure Xvfb is installed
-        if: ${{ matrix.test-name == 'electron' }}
-        run: |
-          if ! command -v xvfb-run &> /dev/null; then
-            sudo apt-get update
-            sudo apt-get install -y xvfb
-          fi
-
       - name: Configure test user cleanup
         run: echo "INTEGRATION_TEST_RUN_KEY=${RUN_ID}-${RUN_ATTEMPT}-${TEST_NAME}-${TEST_PROJECT}-${NEXT_VERSION:-default}" >> "$GITHUB_ENV"
         env:
@@ -699,13 +618,10 @@ jobs:
       - name: Run Integration Tests
         id: integration-tests
         timeout-minutes: 25
-        run: |
-          if [ "${{ matrix.test-name }}" = "electron" ]; then
-            xvfb-run -a pnpm turbo test:integration:${{ matrix.test-name }} $TURBO_ARGS
-          else
-            pnpm turbo test:integration:${{ matrix.test-name }} $TURBO_ARGS
-          fi
+        run: pnpm turbo test:integration:${{ matrix.test-name }} $TURBO_ARGS
         env:
+          TURBO_FORCE: ${{ github.event_name == 'merge_group' }}
+          E2E_CI_SETUP: ${{ matrix.test-name }}
           CLERK_E2E_OAUTH_PROVIDER_CLIENT_SECRET: ${{ secrets.CLERK_E2E_OAUTH_PROVIDER_CLIENT_SECRET }}
           CLERK_PLATFORM_API_KEY: ${{ secrets.CLERK_PLATFORM_API_KEY }}
           E2E_DEBUG: '1'
@@ -737,7 +653,7 @@ jobs:
           retention-days: 1
 
       - name: Delete integration-test users
-        if: ${{ always() && steps.integration-tests.outcome != 'skipped' }}
+        if: ${{ always() && steps.integration-tests.outputs.tests-started == 'true' }}
         timeout-minutes: 10
         run: pnpm test:integration:cleanup
         env:

.github/workflows/e2e-staging.yml

@@ -290,7 +290,7 @@ jobs:
         run: pnpm turbo test:integration:${{ matrix.test-name }} $TURBO_ARGS
         env:
           # Staging e2e validates the LIVE staging deploy, which is not a turbo input
-          # (the cache key is integration/** + env). Force re-execution so a cached green
+          # (the cache key is the suite's inputs, package sources and env). Force re-execution so a cached green
           # pass can't stand in for an untested deploy, and so the Playwright JSON report
           # is regenerated every run instead of being skipped on a `>>> FULL TURBO` hit.
           TURBO_FORCE: 'true'

.github/workflows/nightly-checks.yml

@@ -74,7 +74,7 @@ jobs:
           if [ "${{ matrix.test-name }}" == "nextjs" ]; then
             E2E_DEBUG=1 E2E_APP_ID=quickstart.next.appRouter pnpm test:integration:base --grep @quickstart 2>&1 | tee "$OUTPUT_FILE"
           else
-            E2E_DEBUG=1 pnpm turbo test:integration:${{ matrix.test-name }} $TURBO_ARGS --only 2>&1 | tee "$OUTPUT_FILE"
+            E2E_DEBUG=1 pnpm turbo test:integration:${{ matrix.test-name }} $TURBO_ARGS --only --force 2>&1 | tee "$OUTPUT_FILE"
           fi
           echo "exit_code=${PIPESTATUS[0]}" >> $GITHUB_OUTPUT
         env:

integration/models/applicationConfig.ts

@@ -153,6 +153,9 @@ export const applicationConfig = () => {
     get scripts() {
       return scripts;
     },
+    get template() {
+      return template;
+    },
     get pkglabDependencies() {
       return [...dependencies.entries()].filter(([, version]) => version === PKGLAB).map(([name]) => name);
     },

integration/package.json

@@ -1,4 +1,11 @@
 {
+  "name": "@clerk/integration",
+  "version": "0.0.0",
   "private": true,
-  "type": "module"
+  "type": "module",
+  "devDependencies": {
+    "@clerk/backend": "workspace:*",
+    "@clerk/shared": "workspace:*",
+    "@clerk/testing": "workspace:*"
+  }
 }

integration/scripts/run-integration.mjs

@@ -0,0 +1,72 @@
+import { spawnSync } from 'node:child_process';
+import { appendFileSync, mkdirSync } from 'node:fs';
+import { dirname, join, resolve } from 'node:path';
+import { fileURLToPath } from 'node:url';
+
+const root = resolve(dirname(fileURLToPath(import.meta.url)), '../..');
+const [command, ...args] = process.argv.slice(2);
+const suite = process.env.E2E_CI_SETUP;
+
+if (!command) {
+  fail('usage: node integration/scripts/run-integration.mjs <command> [args...]');
+}
+
+if (suite) {
+  prepareCiRunner(suite);
+}
+
+if (suite === 'electron') {
+  run('xvfb-run', ['-a', command, ...args]);
+} else {
+  run(command, args);
+}
+
+function prepareCiRunner(suite) {
+  const turboArgs = required('TURBO_ARGS').split(/\s+/).filter(Boolean);
+  const installs = [
+    ['@clerk/clerk-js', join(required('E2E_APP_CLERK_JS_DIR'), 'clerk-js')],
+    ['@clerk/ui', join(required('E2E_APP_CLERK_UI_DIR'), 'clerk-ui')],
+  ];
+  const githubOutput = required('GITHUB_OUTPUT');
+
+  const buildEnv = Object.fromEntries(Object.entries(process.env).filter(([name]) => !name.startsWith('CLERK_')));
+  run('pnpm', ['turbo', 'build', ...turboArgs, '--only'], { env: { ...buildEnv, TURBO_FORCE: 'false' } });
+  run('pkglab', ['pub', '--force']);
+  for (const [packageName, dir] of installs) {
+    mkdirSync(dir, { recursive: true });
+    run('pnpm', ['init'], { cwd: dir });
+    run('pkglab', ['add', packageName], { cwd: dir });
+  }
+  if (suite === 'astro') {
+    run('pnpm', ['copy:components'], { cwd: join(root, 'packages/astro') });
+  }
+  run('pnpm', ['playwright', 'install', 'chromium']);
+  if (suite === 'electron' && spawnSync('sh', ['-c', 'command -v xvfb-run'], { stdio: 'ignore' }).status !== 0) {
+    run('sudo', ['apt-get', 'update']);
+    run('sudo', ['apt-get', 'install', '-y', 'xvfb']);
+  }
+  appendFileSync(githubOutput, 'tests-started=true\n');
+}
+
+function required(name) {
+  const value = process.env[name];
+  if (!value) {
+    fail(`${name} must be set when E2E_CI_SETUP is set`);
+  }
+  return value;
+}
+
+function run(command, args, options = {}) {
+  const result = spawnSync(command, args, { cwd: root, stdio: 'inherit', ...options });
+  if (result.error) {
+    fail(result.error.message);
+  }
+  if (result.status !== 0) {
+    process.exit(result.status ?? 1);
+  }
+}
+
+function fail(message) {
+  console.error(`run-integration: ${message}`);
+  process.exit(1);
+}

integration/tests/billing-hooks.test.ts → integration/tests/billing/billing-hooks.test.ts

@@ -1,7 +1,7 @@
 import { expect, test } from '@playwright/test';
 
-import type { FakeUser } from '../testUtils';
-import { createTestUtils, testAgainstRunningApps } from '../testUtils';
+import type { FakeUser } from '../../testUtils';
+import { createTestUtils, testAgainstRunningApps } from '../../testUtils';
 
 testAgainstRunningApps({})('billing hooks @billing', ({ app }) => {
   test.describe.configure({ mode: 'parallel' });

integration/tests/per-seat-pricing.test.ts → integration/tests/billing/per-seat-pricing.test.ts

@@ -1,7 +1,7 @@
 import { expect, test } from '@playwright/test';
 
-import type { FakeOrganization, FakeUser } from '../testUtils';
-import { createTestUtils, testAgainstRunningApps } from '../testUtils';
+import type { FakeOrganization, FakeUser } from '../../testUtils';
+import { createTestUtils, testAgainstRunningApps } from '../../testUtils';
 
 const INVITEE_EMAIL = 'one+clerk_test@clerk.dev';
 

integration/tests/pricing-table.test.ts → integration/tests/billing/pricing-table.test.ts

@@ -1,8 +1,8 @@
 import type { Locator } from '@playwright/test';
 import { expect, test } from '@playwright/test';
 
-import type { FakeUser } from '../testUtils';
-import { createTestUtils, testAgainstRunningApps } from '../testUtils';
+import type { FakeUser } from '../../testUtils';
+import { createTestUtils, testAgainstRunningApps } from '../../testUtils';
 
 testAgainstRunningApps({})('pricing table @billing', ({ app }) => {
   test.describe.configure({ mode: 'parallel' });

integration/tests/cache-components.test.ts → integration/tests/cache-components/cache-components.test.ts

@@ -1,8 +1,8 @@
 import { expect, test } from '@playwright/test';
 
-import { appConfigs } from '../presets';
-import type { FakeUser } from '../testUtils';
-import { createTestUtils, testAgainstRunningApps } from '../testUtils';
+import { appConfigs } from '../../presets';
+import type { FakeUser } from '../../testUtils';
+import { createTestUtils, testAgainstRunningApps } from '../../testUtils';
 
 testAgainstRunningApps({ withEnv: [appConfigs.envs.withEmailCodes], withPattern: ['next.cacheComponents'] })(
   'Next.js Cache Components @cache-components',

integration/tests/client-trust.test.ts → integration/tests/generic/client-trust.test.ts

@@ -1,8 +1,8 @@
 import { expect, test } from '@playwright/test';
 
-import { appConfigs } from '../presets';
-import type { FakeUser } from '../testUtils';
-import { createTestUtils, testAgainstRunningApps } from '../testUtils';
+import { appConfigs } from '../../presets';
+import type { FakeUser } from '../../testUtils';
+import { createTestUtils, testAgainstRunningApps } from '../../testUtils';
 
 testAgainstRunningApps({ withEnv: [appConfigs.envs.withNeedsClientTrust] })(
   'client trust flow @generic @nextjs',