test: the version gate could not fail, so it is gone (C-341, C-346, #425) - #436
Conversation
) Closes #363. "Delete it" was not one deletion. TestF1VersionBumped asserted "the current version is not already tagged". Version here is bumped only at release time, so from the moment a release lands until the next bump the version IS a tag that exists — the entire inter-release period. Measured in both states rather than argued: version tagged (steady state) XFAIL exit 0 version untagged (just bumped) XPASS exit 0 Green either way. It asked "have you bumped yet?", which repo state cannot answer, because "about to release" is not knowable from the repo — only from the tag that triggers a release. Deleted, and replaced by the two halves that ARE answerable: TestVersionMatchesItsTag (if HEAD is on a tag, the version must equal it) and an unskippable guard in publish_package.yml comparing github.ref_name to the version before the build. The new test was drilled to a genuine rc=1 failure in an isolated worktree at v1.11.0; its predecessor could not fail in any state. C-341's last residue is gone, so C-341 resolves. Grepping for the thing being deleted found FOUR MORE copies, in three other deploy suites. They use a conditional xfail that reads as more rigorous and is worse: it runs only when the version is untagged, then asserts the version is untagged — asserting the condition that selected it. Registered as C-346 and left in place; removing four classes plus the meta-test that ENFORCES the marker is its own change, and the replacement now exists. The publish guard had a real bug the drill caught: it read the version with bare `python3 -c "import tomllib"`. tomllib needs >= 3.11, local python3 is 3.10, and the runner's is whatever the image ships. Pinned to `uv run --no-project --python 3.12`, matching the guard beside it. That one would have surfaced at a release, against an immutable tag. Two of my own drills were wrong before they were right, the same way both times. `git checkout v1.11.0` was silently refused because of uncommitted changes, so the "want FAIL" case ran off-tag, skipped, and returned rc=0. Then pytest was invoked from the main repo while the test shells out to `git describe` in the CURRENT directory — still the wrong tree. Fourth and fifth instances this week of a drill whose setup did not happen while the exit code looked fine. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
/code-review medium on this PR found a defect in the test written to close one — and in the only direction that matters. _head_tag() used `git describe --tags --exact-match HEAD`. That returns EXACTLY ONE tag, the lexicographically smallest, and a commit can carry several. Reproduced in a throwaway repo: tags on HEAD: checkpoint v1.11.0 git describe --tags --exact-match HEAD -> checkpoint _head_tag() -> None -> the test SKIPS So a stray non-v tag on a release commit hides the release tag, and the test goes quiet precisely when a real mismatch is sitting on that commit. Switched to `git tag --points-at`, which lists all of them, and drilled the multi-tag case that had no drill before: A. single tag, version matches rc=0 B. multi-tag (checkpoint + v1.11.0), match rc=0 C. multi-tag, MISMATCH rc=1, names the v* tag Where it was NOT a problem is the useful part: publish_package.yml compares github.ref_name directly and never shells out to git describe, so the unskippable half was immune. The two halves failing differently is the argument for having both — which I would have asserted without evidence if the review had not produced it. Also verified by the review and left alone: `set -euo pipefail` does abort the guard when `uv run` fails, so it never compares against an empty string; the guard runs after setup-uv; and the deploy_v130 meta-test still runs non-vacuously after the deletion — it greps and checks each surviving sibling for real, confirmed by running it in a state where its early-return does not short-circuit. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Ritual complete —
|
…kipped (C-340, #426) (#437) Two mechanisms, two different fixes, because they fail differently. MECHANISM 2 gets a git hook. #416 merged the instant CI went green; a follow-up commit was pushed to that branch, which by then had no open PR, and two pieces of work were simply not on development. `git push` reported success. scripts/git-hooks/pre-push refuses that push, names the PR and when it merged, and says what to do instead. It is a hook rather than a checklist item deliberately. Everything else in this epic runs whether or not anyone remembers; a step someone has to recall would be the same class of defect the epic exists to remove. It ALLOWS the push whenever it cannot answer — gh absent, unauthenticated, offline — and says why. A hook that blocks work when it does not know gets uninstalled within a day and then guards nothing. Same idiom as the deploy gates (C-320): skip where the environment cannot answer. MECHANISM 1 gets a helper that reads the value back. `gh pr merge --auto --<method>` on an already-armed PR silently refuses to change the method — prints nothing, exits 0. scripts/arm_automerge.sh uses the GraphQL disable/enable pair, which does honour it, then reads auto_merge.merge_method back and exits non-zero on mismatch. Reading back is the point; arming is the easy part. Seven drills, all against the real repository: merged PR (#436) REFUSE exit 1, names the PR and merge time branch with no PR ALLOW exit 0 development ALLOW exit 0 gh absent from PATH ALLOW exit 0 with the reason gh unauthenticated ALLOW exit 0 with the reason arm on a MERGED pr exit 1, refuses to mutate arm with bad arguments exit 2 The gh-absent drill was wrong first: PATH=/usr/bin:/bin still found gh at ~/.local/bin, so it tested nothing. Redone with a sandbox PATH containing git and bash but not gh, with the absence verified before the run. tests/test_git_hooks.py guards the three things that fail silently: the executable bit (git ignores a non-executable hook with no warning), the fail-open paths, and the documented install step. Drilled by breaking each — three mutations, three catches. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Closes #425 and #363. Part of #421, Story 4.
The instruction was "delete F1". It turned out not to be one deletion, and the investigation changed the shape twice.
The gate could not fail, and that was structural
TestF1VersionBumpedasserted "the current version is not already tagged". Version here is bumped only at release time, so from the moment a release lands until the next bump the version is a tag that exists — the entire inter-release period. Hence its unconditionalxfail: not a temporary concession, the only way it could coexist with this workflow.Measured in both states rather than argued:
It asked "have you bumped yet?" — unanswerable from repo state, because "about to release" is not knowable from the repo, only from the tag that triggers a release.
Replaced by the two halves that are answerable
TestVersionMatchesItsTag— if HEAD is on av*tag, the version must equal it. Skips with a reason otherwise (C-320 idiom: do not guess). Drilled to a genuine failure in an isolated worktree checked out at v1.11.0:A guard in
publish_package.ymlcomparinggithub.ref_nameto the version, before the build. Unlike the test it cannot be skipped — the triggering tag is known there and nowhere else. Drilled with the body extracted from the workflow, not retyped:C-341's last residue was exactly this gate, so C-341 resolves.
Grepping for the deletion found four more
test_version_not_already_taggedalso lives intest_falsification_{deploy_v130,ghspop_deploy_v2,ghsbuilts_deploy_v2}andtest_falsification_vdem_deploy. They use a conditionalxfailthat reads as more rigorous and is worse:It runs only when the version is untagged, then asserts the version is untagged — asserting the condition that selected it. Green in every reachable state.
Registered as C-346 (Tier 4) and deliberately left in place: removing four classes across unrelated suites, plus the meta-test in
deploy_v130that enforces the marker, is its own change — and the replacement now exists, so this is cleanup rather than a gap. The entry notes the v1.2.29 post-mortem item that whoever removes them should close.A real bug the drill caught
The publish guard first read the version with bare
python3 -c "import tomllib".tomllibneeds ≥ 3.11; localpython3is 3.10 and the runner's is whatever the image ships. Pinned touv run --no-project --python 3.12, matching the guard beside it.That one would have surfaced at a release, against an immutable tag — the single worst moment for a guard to fail.
Two of my own drills were wrong first, the same way both times
git checkout v1.11.0was silently refused (uncommitted changes), so the "want FAIL" case ran off-tag, skipped, and returnedrc=0— which I nearly recorded as a result.git describein the current directory — still answering about the wrong tree.Fourth and fifth instances this week of a drill whose setup did not happen while the exit code looked fine. Both redone with the setup verified (
HEAD tag=v1.11.0,version=1.11.0) before the assertion ran.Verification
ruff check .→ All checks passedpytest→ PYTEST_EXIT=0,FAILED count: 0(unpiped, two readers)docs/validate_docs.sh→ PASSEDpublish_package.ymlchecked semantically: every step has exactly one ofrun/uses; the tag guard precedes the buildTestF1VersionBumpedconfirmed absent as a class (the one remaining mention is the replacement's docstring); 6 tests collect in that file🤖 Generated with Claude Code