Skip to content

ci: validate and test Claude Code mods, and auto-bump the pinned CLI #1564

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
potiuk merged 1 commit into from
Oct 9, 2026
Merged

ci: validate and test Claude Code mods, and auto-bump the pinned CLI #1564

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
235 changes: 235 additions & 0 deletions .github/workflows/bump-hook-npm-pins.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,235 @@
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
#
# Bumps the npm packages pinned in `.pre-commit-config.yaml` hooks'
# `additional_dependencies`, which Dependabot's `pre-commit` ecosystem never
# touches. Today that is the Claude Code CLI behind the `check-claude-mods`
# hook. `tools/dev/bump-hook-npm-pins.py` picks the newest stable release that
# has cleared the package's cooldown — 12 hours for Claude Code, 7 days by
# default. Claude Code is a dev-only tool here: CI runs it to check the mods,
# and nothing from it ships.
#
# Three times a week (Monday, Wednesday, Friday), so a pin trails the newest
# eligible release by a few days at most without a PR every day. Also runnable
# from the Actions tab (*Run workflow*).
#
# ONE PULL REQUEST, UPDATED IN PLACE. Every run targets the same branch,
# `chore/bump-hook-npm-pins`. While a bump PR is open, a later run with a newer
# target replaces that branch's commit with a fresh one on top of current
# `main`, and the same PR now carries the newer bump; bumps never pile up as
# separate PRs. A run whose result matches what the open PR already carries
# changes nothing.
#
# The replacement is built on a throwaway branch first and the PR branch is
# moved onto it only once the signed commit exists, so a failed run never
# leaves the PR pointing at an empty or half-made branch.
#
# SHAPE, as in `bump-dev-version.yml`, whose header explains each choice:
#
# prepare -> signed candidate commit, then the PR branch moved onto it
# open-pr -> open the draft PR, or refresh the open one and return it to draft
# cleanup -> delete the throwaway branch, whatever happened
#
# The PR is a draft because GitHub raises no `pull_request` events for what
# `GITHUB_TOKEN` does, so its checks start only when a maintainer presses
# *Ready for review*. An update puts an already-ready PR back into draft for
# the same reason: the new commit has no checks until it is marked ready
# again. Those checks include the `check-claude-mods` step of `prek`, which
# re-checks every mod against the new pin — the point of the bump.
---
name: bump hook npm pins

on: # yamllint disable-line rule:truthy
schedule:
- cron: "17 4 * * 1,3,5"
workflow_dispatch:

permissions: {}

env:
BRANCH: chore/bump-hook-npm-pins

jobs:
prepare:
name: Build the candidate commit
runs-on: ubuntu-slim
timeout-minutes: 10
permissions:
# Creating the throwaway branch, committing onto it, moving the PR branch.
contents: write
outputs:
updated: ${{ steps.candidate.outputs.updated }}
tmp_branch: ${{ steps.candidate.outputs.tmp_branch }}
bumps: ${{ steps.bump.outputs.bumps }}
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
# The commit is created through GitHub's API rather than `git push`,
# so the job never needs the checkout's credentials on disk.
persist-credentials: false
fetch-depth: 1

- uses: astral-sh/setup-uv@c18668ad3cf93ea998bef934396af7bb5c839dc7 # v10.2.0
with:
enable-cache: true
# A distinct cache slot per job; see `bump-dev-version.yml`.
cache-suffix: "bump-hook-npm-pins"

- name: Bump the pins
id: bump
run: |
set -euo pipefail
uv run --project tools/dev python tools/dev/bump-hook-npm-pins.py \
| tee "$RUNNER_TEMP/bumps.txt"
{
echo "bumps<<BUMPS_EOF"
cat "$RUNNER_TEMP/bumps.txt"
echo "BUMPS_EOF"
} >> "$GITHUB_OUTPUT"

- name: Commit the candidate and move the PR branch onto it
id: candidate
env:
GH_TOKEN: ${{ github.token }}
REPO: ${{ github.repository }}
HEAD_SHA: ${{ github.sha }}
RUN_ID: ${{ github.run_id }}
run: |
set -euo pipefail
config=.pre-commit-config.yaml

# Compare against what the bump branch carries, when there is one,
# not against `main`: while a bump PR is open, `main` still has the
# old pins, but the PR may already have exactly these.
if gh api "repos/$REPO/git/ref/heads/$BRANCH" >/dev/null 2>&1; then
gh api -H "Accept: application/vnd.github.raw" \
"repos/$REPO/contents/$config?ref=$BRANCH" > "$RUNNER_TEMP/current.yaml"
else
git show "HEAD:$config" > "$RUNNER_TEMP/current.yaml"
fi
if cmp -s "$config" "$RUNNER_TEMP/current.yaml"; then
echo "Pins already current on \`main\` or in the open bump PR; nothing to do." \
>> "$GITHUB_STEP_SUMMARY"
echo "updated=false" >> "$GITHUB_OUTPUT"
exit 0
fi

{
echo "chore: bump npm pins in prek hooks"
echo
cat "$RUNNER_TEMP/bumps.txt"
echo
echo "Generated by .github/workflows/bump-hook-npm-pins.yml with"
echo "tools/dev/bump-hook-npm-pins.py: the newest stable release past the"
echo "package's cooldown (12 hours for Claude Code, 7 days otherwise)."
} > "$RUNNER_TEMP/bump-message.txt"

tmp="chore/bump-hook-npm-pins-tmp-$RUN_ID"
gh api "repos/$REPO/git/refs" \
-f "ref=refs/heads/$tmp" -f "sha=$HEAD_SHA" >/dev/null
# From here on the throwaway branch exists; `cleanup` deletes it.
echo "tmp_branch=$tmp" >> "$GITHUB_OUTPUT"

# Committed through GitHub's API, not `git commit`: the commit comes
# back signed by GitHub and shows as Verified, with no key material
# in CI.
uv run --project tools/dev python tools/dev/gh-signed-commit.py \
--branch "$tmp" \
--expected-head-oid "$HEAD_SHA" \
--message-file "$RUNNER_TEMP/bump-message.txt" \
--repo "$REPO"
commit=$(gh api "repos/$REPO/git/ref/heads/$tmp" --jq .object.sha)

# Point the PR branch at the new commit: create it on the first bump,
# force-move it while a bump PR is open. A force move is right here —
# the branch holds nothing but the previous bump, which this one
# replaces.
if gh api "repos/$REPO/git/ref/heads/$BRANCH" >/dev/null 2>&1; then
gh api -X PATCH "repos/$REPO/git/refs/heads/$BRANCH" \
-f "sha=$commit" -F force=true >/dev/null
else
gh api "repos/$REPO/git/refs" \
-f "ref=refs/heads/$BRANCH" -f "sha=$commit" >/dev/null
fi
echo "updated=true" >> "$GITHUB_OUTPUT"

open-pr:
name: Open or refresh the bump pull request
needs: prepare
if: needs.prepare.outputs.updated == 'true'
runs-on: ubuntu-slim
timeout-minutes: 5
permissions:
contents: read
pull-requests: write
steps:
- name: Open the pull request, or refresh the open one
env:
GH_TOKEN: ${{ github.token }}
REPO: ${{ github.repository }}
BUMPS: ${{ needs.prepare.outputs.bumps }}
run: |
set -euo pipefail
{
echo "Bumps the npm packages pinned in \`.pre-commit-config.yaml\` hooks:"
echo
echo '```text'
printf '%s\n' "$BUMPS"
echo '```'
echo
echo "Each is the newest stable release past its cooldown (12 hours for Claude"
echo "Code, 7 days otherwise) — see \`tools/dev/bump-hook-npm-pins.py\`. Later runs"
echo "update this PR in place until it is merged."
echo
echo "The commit is signed by GitHub. GitHub raises no events for anything"
echo "\`GITHUB_TOKEN\` does, so the required checks start out unstarted:"
echo "**press *Ready for review* to start them.** The \`prek\` run includes"
echo "\`check-claude-mods\`, which re-checks every mod against the new pin."
echo
echo "Opened by [\`bump-hook-npm-pins.yml\`](.github/workflows/bump-hook-npm-pins.yml)."
} > "$RUNNER_TEMP/body.md"

number=$(gh pr list --repo "$REPO" --head "$BRANCH" --state open \
--json number --jq '.[0].number // empty')
if [[ -n "$number" ]]; then
gh pr edit "$number" --repo "$REPO" --body-file "$RUNNER_TEMP/body.md"
# The new commit has no checks yet; back to draft so marking it
# ready starts them, as for a fresh bump PR.
gh pr ready "$number" --repo "$REPO" --undo || true
echo "Updated #$number with the new bump." >> "$GITHUB_STEP_SUMMARY"
else
url=$(gh pr create --repo "$REPO" --base main --head "$BRANCH" --draft \
--title "chore: bump npm pins in prek hooks" \
--body-file "$RUNNER_TEMP/body.md")
echo "Opened $url" >> "$GITHUB_STEP_SUMMARY"
fi

cleanup:
name: Drop the throwaway branch
needs: [prepare, open-pr]
if: always() && needs.prepare.outputs.tmp_branch != ''
runs-on: ubuntu-slim
timeout-minutes: 5
permissions:
contents: write
steps:
- name: Delete the throwaway branch
env:
GH_TOKEN: ${{ github.token }}
REPO: ${{ github.repository }}
TMP_BRANCH: ${{ needs.prepare.outputs.tmp_branch }}
run: gh api "repos/$REPO/git/refs/heads/$TMP_BRANCH" -X DELETE || true
19 changes: 18 additions & 1 deletion .github/workflows/pre-commit.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -223,13 +223,17 @@ jobs:
# matches no hook at all. It warns rather than failing, so the
# comma form looks like it worked while skipping nothing.
#
# `--skip check-claude-mods-local`: the contributor-side twin of
# `check-claude-mods`, using a local Claude Code the runner does not
# have. The pinned hook runs in its own step below.
#
# `--skip lychee`: the link check is whole-repo on every event,
# in the dedicated step below. See its comment for why it cannot
# ride along with `PREK_SCOPE`.
run: >-
prek run --show-diff-on-failure --color=always $PREK_SCOPE
--skip workspace-pytest --skip identity --skip skill-token-count
--skip lychee
--skip lychee --skip check-claude-mods-local
# The link check runs over the whole repository on every event,
# `main` and pull request alike — `PREK_SCOPE` does not apply.
#
Expand Down Expand Up @@ -266,3 +270,16 @@ jobs:
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: prek run lychee --color=always --all-files
# Claude Code mods: `claude plugin validate --strict` and `claude plugin
# test` on every plugin whose `hooks/hooks.json` declares `modules`. The
# hook is `stages: [manual]` so contributors' commits never fetch the
# Claude Code binary it pins, which means the `Run prek` step above never
# runs it — this step is the only place it runs on its own.
#
# Whole repo on both events, like lychee and for a similar reason: the
# pin bump in `.pre-commit-config.yaml` touches no mod file, yet it is
# exactly the change that has to re-check every mod against the new
# version. A no-op (no binary fetched) while the tree has no mod.
- name: Run prek (Claude Code mods — whole repo, both events)
if: ${{ !cancelled() }}
run: prek run check-claude-mods --hook-stage manual --color=always --all-files
55 changes: 55 additions & 0 deletions .pre-commit-config.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -335,6 +335,61 @@ repos:
entry: python3 tools/dev/check-family-plugins.py --fix
files: ^(skills/.*/SKILL\.md|plugins/.*|\.claude-plugin/(marketplace|plugin)\.json|\.codex-plugin/plugin\.json|\.agents/plugins/marketplace\.json|(plugin|marketplace)\.json|gemini-extension\.json|apm\.yml|pyproject\.toml)$
pass_filenames: false
# Claude Code mods — `claude plugin validate --strict` and `claude plugin
# test` on every plugin whose `hooks/hooks.json` declares `modules`
# (docs/when-to-use-mods.md). Claude Code loads a hooks module only after
# static analysis passes; a module that fails it never runs, and nothing
# else in the tree notices. A no-op until the tree has a mod.
#
# `language: node` with the CLI pinned in `additional_dependencies`, so
# prek installs it into a cached hook env, the way `lychee` gets its
# toolchain: no local Claude Code needed, and CI needs no install step.
#
# `stages: [manual]`: the hook env is a ~200 MB native binary, which is
# not worth fetching on every contributor's first commit for a check that
# concerns only the few who write mods. CI runs it on every event (its own
# step in `.github/workflows/pre-commit.yml`); locally it is
# `prek run check-claude-mods --hook-stage manual --all-files`.
#
# Claude Code is a dev-only tool here: CI runs it to check the mods, it is
# not a dependency of anything Magpie ships, and nothing from it goes into a
# release or a plugin.
#
# The pin is moved by `.github/workflows/bump-hook-npm-pins.yml` (Dependabot's
# `pre-commit` ecosystem never touches `additional_dependencies`), with a
# 12-hour cooldown rather than the 7 days the rest of the tree uses. A
# cooldown buys time for a bad or compromised release to be withdrawn
# before CI installs it. Claude Code ships several times a day and replaces
# a bad release within hours, so 12 hours already covers that window; and
# a security problem in it is fixed by the next release, so a week-long wait
# would keep CI on the known-bad version instead of protecting it. The mods
# API is also still moving, and adopters auto-update, so the check has to
# track what they actually run. Being dev-only (above), the worst a bad
# version can do is fail or wrongly pass this check. See
# `tools/dev/bump-hook-npm-pins.py`.
- repo: local
hooks:
- id: check-claude-mods
name: check-claude-mods (claude plugin validate --strict / test)
language: node
entry: tools/dev/check-claude-mods.sh
additional_dependencies: ["@anthropic-ai/claude-code@2.1.295"]
files: ^(.*/hooks/.*|.*\.claude-plugin/plugin\.json|tools/dev/check-claude-mods\.sh)$
pass_filenames: false
stages: [manual]
# The same check on ordinary commits, with the contributor's own Claude
# Code: `language: system`, so nothing is downloaded. It skips itself when
# `claude` is not installed, and when the installed one is older than the
# pin above (an older CLI may not know events a mod uses). A newer one
# runs. CI's pinned run above stays the authoritative result.
- repo: local
hooks:
- id: check-claude-mods-local
name: check-claude-mods-local (local Claude Code, skipped if absent)
language: system
entry: tools/dev/check-claude-mods.sh --local
files: ^(.*/hooks/.*|.*\.claude-plugin/plugin\.json|tools/dev/check-claude-mods\.sh)$
pass_filenames: false
# Every shared prose block a skill carries — the auto-inserted setup
# pre-flight, plus any number of declared blocks. It cannot be a hook: on
# most harnesses *no code runs at all* when a plugin is installed or
Expand Down
13 changes: 11 additions & 2 deletions docs/when-to-use-mods.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -177,11 +177,20 @@ The hooks manifest at `hooks/hooks.json` lists the mod modules to load:
```

The module (`hooks/register.ts`) exports a `register(on, options)` function that registers event listeners with the harness.
It imports only its own files, by relative path, and `claude-code`.
Node built-ins such as `node:fs` are not available: reach files, the environment and processes through the mods API (`$.fs`, `$.env`, `$.process`), so the `calls:` line `claude plugin validate` prints is the module's whole surface.
`hooks/hooks.json` can also carry the plugin's settings hooks under `hooks`, next to `modules`; a mod that takes over from a settings hook keeps that hook wired there for sessions where mods do not load.

Every mod must include unit tests and pass strict validation before landing:
1. **Static Validation**: Mod modules must pass `claude plugin validate <dir> --strict` to verify hooked events and API calls.
2. **Automated Unit Tests**: Hook logic must be tested using `claude plugin test` with corresponding `*.test.ts` test suites.
3. **CI Integration**: Plugin validation and test suites must run alongside standard pre-commit hooks and Python test runners.
Claude Code runs the same analysis when it loads a mod and skips a module that fails it, so a mod that fails here does nothing at all on an adopter's machine.
2. **Automated Unit Tests**: Hook logic must be tested using `claude plugin test` with corresponding `*.test.ts` test suites written against `claude-code/testing`.
A mod with no tests fails the check.
3. **CI Integration**: the `check-claude-mods` prek hook ([`tools/dev/check-claude-mods.sh`](../tools/dev/check-claude-mods.sh)) runs both commands on every plugin whose `hooks/hooks.json` declares `modules`, on every pull request and push to `main`.
prek installs a pinned Claude Code into the hook environment, so neither a local install nor a login is needed.
The hook is in the `manual` stage, so ordinary commits skip it; run it before pushing a mod with `prek run check-claude-mods --hook-stage manual --all-files`.
The pin is bumped three times a week, with a 12-hour cooldown, by [`bump-hook-npm-pins.yml`](../.github/workflows/bump-hook-npm-pins.yml).
Claude Code is a dev-only tool here: CI runs it to check the mods, it is not a dependency of anything Magpie ships, and nothing from it goes into a release or a plugin.

## See also

Expand Down
Loading