fix: the heartbeat URL leaves the command line (C-331, #423) - #432
Conversation
HEARTBEAT_URL is a capability: whoever holds it can forge a success ping and silence the dead-man alert permanently. It was an argv element at three curl sites, and /proc/<pid>/cmdline is world-readable while four accounts hold shells on the box. All three now pass it on stdin as a curl config file. C-331's own prescribed fix was the bug. The entry said `printf 'url=%s\n'` — unquoted. Measured before shipping it, with a value carrying a stray space: url=%s -> parses http://h/uuid, DROPS the /fail, sends anyway url = "%s" -> exit 3, nothing sent A trailing space or CR would have turned the failure ping into a success ping, silently. The register's own remedy for a fails-green concern carried a fails-green defect, unexamined since July because a prescribed fix reads like a settled thing. Shipped form is quoted; the reason is in the script so it survives a rewrite. The /proc claim is drilled with a negative control, since a clean scan otherwise only proves the scanner is broken. Control leaked the canary from curl's argv; the fix showed `curl -fsS --max-time 20 -K -` in flight with nothing anywhere carrying it, three times. A local listener confirmed all three paths arrive byte-exact — re-run against the three lines extracted verbatim from the committed file, not retyped. The guard was wrong first. It asserted "no line contains both curl and HEARTBEAT_URL", which fails against the FIXED script, because `printf ... "$HEARTBEAT_URL" | curl ...` puts both on one line. The property is "never AFTER the curl token". Drilling it is what found that; it now fails on the pre-change file and passes on this one. Three stale line citations fixed and the hole that hid them closed: ADR-018 had drifted ~55 lines, ADR-051 cited one number that was never right, and C-331's own Location read 93,163,290 against 112/182/309 — in an entry whose trigger was "next edit to this file". test_docs_citations could see none of them: its pattern was `.py` only. Widened to `.py|.sh`, drilled, and those three were the only offenders. Also: server_operations said "two signals" while listing three, and 24h grace where the live check is 48h; its two operator verification commands used the argv form, i.e. ran this exact exposure on the box. Not live on the server. #423 said no server change was needed; per C-343 the change lands one run after a deploy, so worst case it is in production two months from merge. Residual left open: HEARTBEAT_URL still sits in ~/.profile, and if that is not mode 600 it is readable by all four accounts permanently. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Chasing C-331's residual instead of writing it down as a caveat. The operator ran three commands on the host: stat -c '%a %n' /home/views-deploy/.profile 644 stat -c '%a %n' /home/views-deploy 751 test -r /home/views-deploy/.profile READABLE BY simmaa_prio 751 on the home lets any account traverse in; 644 on the file then lets it read. Confirmed by direct test from a second account, not inferred from permission bits. Exposed: UCDP_API_TOKEN, ACLED_USERNAME, ACLED_PASSWORD, GDL_API_TOKEN, HEARTBEAT_URL — to dylpin, simmaa_prio and sonja_prio, continuously, from deployment until today. Not a window; a standing condition. A Tier 4 story about a 10-second exposure surfaced a Tier 2 one about a permanent exposure. Fixed with chmod 600 and verified in BOTH directions: no longer readable by the other account, still readable by the owner with the pipeline path intact. A permission fix that also breaks the pipeline is not a fix. Swept the rest of the home rather than assuming. .netrc was already 600 — the data-server credentials were never exposed. .ssh 700, all history files 600. One hypothesis raised and killed: .local is 775 and refresh_pipeline.sh prepends $HOME/.local/bin to PATH, which would have been code execution as views-deploy rather than mere disclosure — but `getent group views-deploy` returns no other members. Both setup guides caused this: each showed `>> ~/.profile` with no chmod, so the mode was whatever umask gave it. Both now carry the chmod and a verification line. Left open on purpose: whether the four credentials need rotating. The readers are trusted colleagues, so this is a policy judgement rather than a breach — but C-322's GDL token was rotated on weaker evidence. The operator's call, deliberately unmade. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… basis The operator's call: no rotation. The three accounts belong to known colleagues and there is no indication any read the file. Recorded with the basis, because the basis is what can expire. This rests on WHO HOLDS THE ACCOUNTS, not on evidence of non-access — no audit record exists that could establish the latter and none was consulted. Reasonable on a single-team research host; it would not survive the accounts being held outside the team. Revisit trigger named: a new shell account (C-88). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
/code-review medium and /review-diff on this PR. The shell change came
back clean; the guard did not.
The reviewer found a false negative: route the secret through an
intermediate variable —
FAIL_URL="$HEARTBEAT_URL/fail"; curl -fsS "$FAIL_URL"
— and C-331 is fully reintroduced with the test still green. Not
contrived: that is exactly what collapsing the three WET ping lines into
a helper would produce, and the file's own docstring anticipates a
future rewrite.
The fix for that did not work either. Assignment-tracking was added; the
drill returned rc=0 where it had to return rc=1, because the regex
anchored at ^ with no allowance for indentation and every assignment in
that script is indented. Only running it against a state it was not
written for exposed that.
Three versions, three claims narrower than the property:
v1 "line contains both curl and HEARTBEAT_URL" -> failed on the
FIXED script (printf ... | curl ... has both on one line)
v2 "literal HEARTBEAT_URL after curl" -> passed a two-line
reintroduction of the defect; caught by review, not the author
v3 assignment-tracking to a fixpoint -> matched no
indented assignment; caught by drill, not by reading
v4 drills clean four ways: clean tree passes, direct argv fails, one
indirection fails, two indirections fail.
Also from /review-diff, both mine: the docstring still described v2's
property, and bare substring matching would over-match a one-character
tainted name (the two-level drill produces `A`). Word boundaries now.
/register-risk: 0 new, 1 merged. This is C-336's mechanism a third time
— a claim narrower than the property, failing green — so it is a second
addendum there rather than a new ID. The generalisation: a guard is a
claim about a property, and only running it against states it was not
written for tells the two apart.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Ritual complete —
|
| version | claimed | what it actually did |
|---|---|---|
| v1 | line contains both curl and HEARTBEAT_URL |
failed on the FIXED script — printf … | curl … has both on one line |
| v2 | literal HEARTBEAT_URL after curl |
passed a two-line reintroduction of the defect — caught by review, not the author |
| v3 | assignment-tracking to a fixpoint | matched no indented assignment — caught by drill, not by reading |
v4, drilled four ways:
rc=0 clean tree (want 0)
rc=1 direct argv form (want 1)
rc=1 one level indirection (want 1)
rc=1 two levels indirection (want 1)
/review-diff — 2 warnings, both mine, both fixed
The test's docstring still described v2's property after the logic moved to a taint set — doc/code mismatch. And bare substring matching would over-match a one-character tainted name (the two-level drill produces A); word boundaries now, which costs nothing and weakens nothing.
/register-risk — 0 new, 1 merged
This is C-336's mechanism a third time — a claim narrower than the property it names, failing green — so it became a second addendum on C-336 rather than a new ID. The generalisation worth keeping:
A guard is a claim about a property, and the claim is usually narrower than the property. Only running it against states it was not written for tells them apart.
Three gaps here, all three failing green.
Verification
bash -n scripts/refresh_pipeline.sh→ OKruff check .→ All checks passedmypy src/→ Success, 89 source filespytest→ PYTEST_EXIT=0, zeroFAILED(read from the output file, not the task notification — a trailingechomasked a real failure earlier this session)docs/validate_docs.sh→ PASSED- Register: 344 IDs, 302 resolved, 39 open (0 T1, 3 T2, 11 T3, 19 T4, 6 deferred), 118 struck, tier sum == open count
/code-review medium fact-checked the entry against the repository rather than against memory, and found two claims that do not survive it. 1. "up to two months in production", attributed to C-343. C-343 says ONE month and never derives the larger figure — the two-month version requires an unstated compounding worst case. Present in TWO places: C-345, and inside C-331 where it had already shipped in #432. Both corrected, in the register and the changelog, with a visible correction note rather than a silent edit. 2. "two recorded false-readiness incidents that each cost a full day." A grep of the register, the changelog and every post-mortem returns only the sentence making the claim. The incidents are real and known to the operator, but THIS REPOSITORY RECORDS NEITHER. Citing evidence a reader cannot find is how a register stops being checkable, which is C-336's subject. Removed rather than softened, and the fact that they went unrecorded is now flagged for the next post-mortem. C-345's tier now rests on the one basis that is verifiable in the repo: it fired twice in one session. That is stated as deliberately the only evidence cited. Also from the review, verified and left alone: the shell claim is exactly right (pipefail changes it, the mitigation works), the header arithmetic is correct and guarded, the cluster section's counts match its eleven rows and nine open members, and every cross-reference points at an entry in the state implied. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…hat was red (#433) * docs(register): C-345 — the instrument that detects this cluster is a member of it Tier 2. Twice in one session a failing suite was reported as passing. uv run pytest -q | tail -2; echo "EXIT=$?" A pipeline's exit status is its LAST element's, so tail returned 0 while pytest had exited 1. Then a backgrounded run's task notification said "exit code 0" for the same reason — the command ended in an echo. The second was caught only by reading the output file, after the user had already been told the suite was running and would be folded in. One step from reporting green on red. Registered rather than remembered because this project has two recorded false-readiness incidents that each cost a full day; because the suite is the gate on every story in this epic; and because C-343 means a defect that slips through can sit in production for two months. C-339 is the precedent for registering a workflow hazard rather than a code one. The shape is the cluster's own, which is the uncomfortable part. C-330 was a nightly no-op exiting 0. C-337 was a lockfile frozen with no error. C-343 was a deploy that deployed nothing. This is the same defect in the instrument used to find all three. Mitigation adopted and explicitly NOT a control: redirect to a file, capture $? unpiped, grep ^FAILED as a second independent reader. A habit is not machinery. The instrument — refusing to report a result not obtained unpiped — is proposed for #424. Also carries the changelog note on what was NOT registered: the unbounded poll loop (one-line fix, no consequence) and the group-writable dotfiles (group has no other members, already inside C-344 as a killed hypothesis). The guard-narrower-than-the-property pattern went to C-336 as a second addendum during #432. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs(register): two false claims in C-345, both mine (#433) /code-review medium fact-checked the entry against the repository rather than against memory, and found two claims that do not survive it. 1. "up to two months in production", attributed to C-343. C-343 says ONE month and never derives the larger figure — the two-month version requires an unstated compounding worst case. Present in TWO places: C-345, and inside C-331 where it had already shipped in #432. Both corrected, in the register and the changelog, with a visible correction note rather than a silent edit. 2. "two recorded false-readiness incidents that each cost a full day." A grep of the register, the changelog and every post-mortem returns only the sentence making the claim. The incidents are real and known to the operator, but THIS REPOSITORY RECORDS NEITHER. Citing evidence a reader cannot find is how a register stops being checkable, which is C-336's subject. Removed rather than softened, and the fact that they went unrecorded is now flagged for the next post-mortem. C-345's tier now rests on the one basis that is verifiable in the repo: it fired twice in one session. That is stated as deliberately the only evidence cited. Also from the review, verified and left alone: the shell claim is exactly right (pipefail changes it, the mitigation works), the header arithmetic is correct and guarded, the cluster section's counts match its eleven rows and nine open members, and every cross-reference points at an entry in the state implied. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Closes #423 (by hand —
Closes #Ndoes not fire on PRs to a non-default branch). Part of #421, Story 2.HEARTBEAT_URLis a capability: whoever holds the string can send a success ping, mark the monthly pipeline healthy, and silence the dead-man alert permanently. It was an argv element at threecurlsites, and/proc/<pid>/cmdlineis world-readable (-r--r--r--) while four accounts hold shells on the box. All three now pass it on stdin as a curl config file.C-331's own prescribed fix was the bug
The entry did not merely describe the problem — it prescribed the remedy, in its body:
printf 'url=%s\n'. Unquoted. Measured against curl 7.81.0 before shipping it, with a value carrying a stray space:url=%shttp://h/uuid, drops the/fail, and sends it anywayurl = "%s"A trailing space or CR — a CRLF-edited
.profile, a copy-paste — would have turned the failure ping into a success ping. Silently. The register's own remedy for a fails-green concern carried a fails-green defect, unexamined since 2026-07-31 because a prescribed fix reads like a settled thing.Shipped form is quoted, and the reason lives in the script's comment rather than only in the register, so it survives the next rewrite.
Drills — control first
A clean scan proves only that the scanner is broken. Canary is a literal
not-a-secretstring against an unrouted RFC1918 address, so the request hangs until--max-timeand stays in flight while/procis walked. Nothing here is sensitive.printfis a bash builtin, so no helper process carries the URL in its own argv either — which is what Drill B establishes, rather than the weaker "stdin implies safety". Drill C was deliberately re-run against text pulled out of the file withgrep, not retyped: that proves the file is right, not just the form.The guard was wrong first
tests/test_heartbeat_secret.pyinitially asserted "no line contains bothcurlandHEARTBEAT_URL" — which fails against the fixed script, becauseprintf … "$HEARTBEAT_URL" | curl …legitimately puts both on one line. The property is "HEARTBEAT_URLnever appears after thecurltoken". Drilling the guard is what found that. It now fails on the pre-change file (naming all three offending lines) and passes on this one.A second test asserts all three pings still exist, so the first cannot be satisfied by deleting them. Neither asserts the
printftext,-K -,-fsS, line numbers or indentation — a curl ≥ 8.3--variable/--expand-urlmigration should not redden this.Three stale citations, and the hole that hid them
refresh_pipeline.shthat had drifted ~55 lines93,163,290against real lines 112/182/309 — in an entry whose trigger was "next edit to this file"tests/test_docs_citations.pycould see none of them: its pattern was\.py:\d+—.pyonly. Widened to\.(?:py|sh):\d+and drilled (reintroduce a.sh:NNN→ red). Those three were the only offenders, so it starts green.Also corrected
docs/guides/server_operations.mdsaid "two signals" while listing three, and a 24-hour grace where the live check is 48 h (observed on the dashboard 2026-08-10). Its two operator verification commands used the argv form — telling the operator to run, on the box, the exact exposure this story removes. Replaced with the stdin form, quoting verified against a local listener before being written down.Not live on the server
#423's note said "no server change is required — the script is deployed by tag." Per C-343 that is false: bash buffers the script, so this lands one run after a deploy, and deploying is three steps. Worst case it is in production two months from merge, and the argv exposure continues until then. The issue has been corrected.
Residual, left open deliberately
HEARTBEAT_URLstill lives in~/.profile. If that file is not mode 600 it is readable by all four accounts permanently, which would dominate the ≤10 s × 3 window closed here. Its mode is not in evidence; the register entry carries a pending marker rather than an assumption, and the check isstat -c %a /home/views-deploy/.profile.Also not closed: the process environment (
/proc/<pid>/environ, mode-r--------, owner and root only), and root.Verification
bash -n scripts/refresh_pipeline.sh→ OKruff check src/ tests/ scripts/→ All checks passedmypy src/→ Success, 89 source filespytest→ PYTEST_EXIT=0, zero FAILED (exit captured unpiped)docs/validate_docs.sh→ PASSED🤖 Generated with Claude Code