Add a workflow for testing parser changes against a sandbox repo

[potiuk]

Why

Our unit tests show that a feature parses its YAML and calls the API it means to
call. They don't show that GitHub accepts the call, or that the result is the one
the contributor had in mind. Today a reviewer who wants to know that has to point
the CLI at some repository by hand, an author has no way to check at all before
asking for review, and there's no shared account of how any of it is done.

What this adds

A workflow_dispatch workflow (Test a change against the sandbox repo) that
applies a given version of the parser to a sandbox repo. Inputs: pr (or ref),
sandbox_branch, sandbox_repo, and noop (defaulting to a dry run).

It runs in two places, with no fork-specific code path:

• In this repo, a committer tests a PR before merging. A PR number resolves to
  refs/pull/<n>/head — a ref on the base repo — so PRs opened from forks can be
  tested without pushing a mirror branch anywhere.
• In a contributor's own fork, the author tests their own branch before asking
  anyone to look. The sandbox defaults to the sandbox repo of whoever owns the
  repository the workflow is running in, so a fork's run acts on the fork owner's
  sandbox with the fork owner's token — and cannot reach the Apache sandbox
  through it. The contributor needs four things once (Actions enabled on the
  fork, their own sandbox repo, a sandbox environment, a token scoped to that
  repo); the docs walk through it.

The run summary records the resolved ref, the commit SHA, the PR author, and the
.asf.yaml that was read, so a reviewer can confirm the run executed the commit
they actually read.

--branch on asfyaml-run and asfyaml-validate. Both hardcoded
refs/heads/main, so nothing branch-sensitive (whoami, website staging and
publishing, ghp_branch) could be exercised, and the workflow needs to name the
sandbox branch carrying the config under test. main and refs/heads/main are
both accepted; tag refs are rejected.

docs/testing-a-change.md — both paths end to end: read the diff → push a
test/pr-NNN branch to the sandbox → dry run → apply → verify (with a
per-directive table of where to look in Settings) → reset the sandbox → report
the result on the PR. Plus the fork setup, the local-CLI equivalent, and the things
that catch people out: why the workflow dropdown stays on main, why github:
metadata applies regardless of which branch it was read from while publish:
doesn't, and that noop still reads from the API.

AGENTS.md, with CLAUDE.md and GEMINI.md as symlinks to it. The repo had no
contributor-facing instructions at all, so things that are only discoverable by
reading configuration — that tests are not named test_*.py (python_files = "*.py"), that a feature is invisible unless it's imported in
asfyaml/feature/__init__.py, that a schema change isn't real until README.md
describes it — had to be rediscovered each time. It leads with the mistake that
costs most: asfyaml-run derives the repository it acts on from the directory
name it is given, so a run pointed at a directory called airflow reconfigures
apache/airflow. Then setup, conventions, how to test in your own fork (pointing
here for the full process), and how to open a PR. Symlinks rather than copies so
the three files cannot drift apart.

Security model — worth a careful look

Dispatching in this repo executes code from the named ref with a token that has
admin rights on the Apache sandbox, and for a fork PR that's code from outside
the project. workflow_dispatch limits triggering to people with write access, and
that person is the control: read the diff, then check the SHA in the run summary
matches what you read. (A run in one's own fork spends nobody else's trust, which
is the other argument for self-testing first.)

Hardening in the workflow itself: every input goes through env: rather than being
interpolated into run:; a PR number is validated as ^[0-9]+$ before it reaches a
ref; refs are held to characters a ref can legitimately contain; a sandbox repo must
be a well-formed owner/name; the workflow definition always comes from main, so
a PR can't rewrite the runner.

tests/sandbox_workflow.py extracts that step's script from the YAML and runs it
under bash with hostile inputs (42; whoami, $(whoami), owner/repo; whoami,
embedded newlines that would inject a second $GITHUB_OUTPUT line). Each guard was
deleted in turn to confirm the tests fail without it.

If committer-only dispatch is judged too weak, adding required reviewers to the
sandbox environment tightens it with no change to this workflow.

Needs infra before it can run here

The workflow fails with an explicit error rather than silently no-opping if these
are missing. Contributors testing in their own forks need none of it — they set up
their own equivalents.

1. apache/infrastructure-asfyaml-sandbox, public, with a baseline .asf.yaml on
   main — the reset step re-applies it.
2. A sandbox environment on this repo.
3. ASFYAML_SANDBOX_TOKEN on that environment: a fine-grained PAT whose repository
   access is only the sandbox repo. A GITHUB_TOKEN can't change repository
   settings, so a PAT is unavoidable here.

Testing

38 new tests, all passing (136 total). The CLI tests were written first and watched
to fail. The workflow tests were written against an existing YAML, so each guard was
removed in turn to confirm the tests catch its absence.

AGENTS.md adds no tests; its factual claims were checked against pyproject.toml,
.pre-commit-config.yaml and asfyaml/feature/__init__.py rather than asserted.

🤖 Generated with Claude Code

https://claude.ai/code/session_018yUbPm1PL7tsKZJhXFsfj7

[erisu]

I was not sure if it was intentional to use older versions of the following actions but left a couple of potential changes.

The defined versions of actions/checkout and actions/setup-python has node20 runtime which is being deprecated/removed from the GitHub Action runners on September 23rd, 2026.

• actions/checkout updated its runtime to node24 in version 5.
• actions/setup-python updated its runtime to node24 in version 6.

https://github.blog/changelog/2025-09-19-deprecation-of-node-20-on-github-actions-runners/

[potiuk]

I was not sure if it was intentional to use older versions of the following actions but left a couple of potential changes.

Dependabot PR is coming :)

[potiuk]

:D ?

[potiuk]

The dependabot is there #129

And I hope we can get that one quickly to unblock testing for .asfyaml.

[potiuk]

I think a number of people would love to be able to move faster with .asfyaml - and testing seems to be bottleneck here - we have many PRs open now. @Humbedooh -> is there anything blocking us here?