Skip to content

fix: the heartbeat URL leaves the command line (C-331, #423) - #432

Merged
Polichinel merged 4 commits into
developmentfrom
fix/heartbeat-url-off-argv
Aug 10, 2026
Merged

fix: the heartbeat URL leaves the command line (C-331, #423)#432
Polichinel merged 4 commits into
developmentfrom
fix/heartbeat-url-off-argv

Conversation

@Polichinel

Copy link
Copy Markdown
Contributor

Closes #423 (by hand — Closes #N does not fire on PRs to a non-default branch). Part of #421, Story 2.

HEARTBEAT_URL is a capability: whoever holds the string can send a success ping, mark the monthly pipeline healthy, and silence the dead-man alert permanently. It was an argv element at three curl sites, and /proc/<pid>/cmdline is world-readable (-r--r--r--) while four accounts hold shells on the box. All three now pass it on stdin as a curl config file.

C-331's own prescribed fix was the bug

The entry did not merely describe the problem — it prescribed the remedy, in its body: printf 'url=%s\n'. Unquoted. Measured against curl 7.81.0 before shipping it, with a value carrying a stray space:

form result
url=%s parses http://h/uuid, drops the /fail, and sends it anyway
url = "%s" exit 3, nothing sent

A trailing space or CR — a CRLF-edited .profile, a copy-paste — would have turned the failure ping into a success ping. Silently. The register's own remedy for a fails-green concern carried a fails-green defect, unexamined since 2026-07-31 because a prescribed fix reads like a settled thing.

Shipped form is quoted, and the reason lives in the script's comment rather than only in the register, so it survives the next rewrite.

Drills — control first

A clean scan proves only that the scanner is broken. Canary is a literal not-a-secret string against an unrouted RFC1918 address, so the request hangs until --max-time and stays in flight while /proc is walked. Nothing here is sensitive.

DRILL A — NEGATIVE CONTROL (must LEAK)
  LEAK /proc/1037255 :: curl -fsS --max-time 20 http://10.255.255.1:8099/CANARY-7f3a1e9c-not-a-secret/fail

DRILL B — THE FIX (must be CLEAN), once per ping
  in flight: 1038214 curl -fsS --max-time 20 -K -
  (clean — no process carries the canary in argv)
  in flight: 1039268 curl -fsS --max-time 20 -K -
  (clean — no process carries the canary in argv)
  in flight: 1040235 curl -fsS --max-time 20 -K -
  (clean — no process carries the canary in argv)

DRILL C — DELIVERY, re-run against the three lines extracted VERBATIM from the committed file
  GET /shipped-verbatim/fail
  GET /shipped-verbatim/start
  GET /shipped-verbatim

DRILL D — BEHAVIOUR under set -euo pipefail
  REACHED, rc=0                                   <- fire-and-forget still fires and forgets
  TRAP ran, original exit_code=1 (must be 1)      <- ERR-trap site does not clobber $?
  script exit=1

printf is a bash builtin, so no helper process carries the URL in its own argv either — which is what Drill B establishes, rather than the weaker "stdin implies safety". Drill C was deliberately re-run against text pulled out of the file with grep, not retyped: that proves the file is right, not just the form.

The guard was wrong first

tests/test_heartbeat_secret.py initially asserted "no line contains both curl and HEARTBEAT_URL" — which fails against the fixed script, because printf … "$HEARTBEAT_URL" | curl … legitimately puts both on one line. The property is "HEARTBEAT_URL never appears after the curl token". Drilling the guard is what found that. It now fails on the pre-change file (naming all three offending lines) and passes on this one.

A second test asserts all three pings still exist, so the first cannot be satisfied by deleting them. Neither asserts the printf text, -K -, -fsS, line numbers or indentation — a curl ≥ 8.3 --variable/--expand-url migration should not redden this.

Three stale citations, and the hole that hid them

  • ADR-018 cited a line range in refresh_pipeline.sh that had drifted ~55 lines
  • ADR-051 cited three line numbers, one of which had never been right
  • C-331's own Location field read 93,163,290 against real lines 112/182/309 — in an entry whose trigger was "next edit to this file"

tests/test_docs_citations.py could see none of them: its pattern was \.py:\d+.py only. Widened to \.(?:py|sh):\d+ and drilled (reintroduce a .sh:NNN → red). Those three were the only offenders, so it starts green.

Also corrected

docs/guides/server_operations.md said "two signals" while listing three, and a 24-hour grace where the live check is 48 h (observed on the dashboard 2026-08-10). Its two operator verification commands used the argv form — telling the operator to run, on the box, the exact exposure this story removes. Replaced with the stdin form, quoting verified against a local listener before being written down.

Not live on the server

#423's note said "no server change is required — the script is deployed by tag." Per C-343 that is false: bash buffers the script, so this lands one run after a deploy, and deploying is three steps. Worst case it is in production two months from merge, and the argv exposure continues until then. The issue has been corrected.

Residual, left open deliberately

HEARTBEAT_URL still lives in ~/.profile. If that file is not mode 600 it is readable by all four accounts permanently, which would dominate the ≤10 s × 3 window closed here. Its mode is not in evidence; the register entry carries a pending marker rather than an assumption, and the check is stat -c %a /home/views-deploy/.profile.

Also not closed: the process environment (/proc/<pid>/environ, mode -r--------, owner and root only), and root.

Verification

  • bash -n scripts/refresh_pipeline.sh → OK
  • ruff check src/ tests/ scripts/ → All checks passed
  • mypy src/ → Success, 89 source files
  • pytestPYTEST_EXIT=0, zero FAILED (exit captured unpiped)
  • docs/validate_docs.sh → PASSED
  • Register: 343 IDs, 301 resolved, 39 open (0 T1, 3 T2, 11 T3, 19 T4, 6 deferred), 117 struck, header 1912 ≤ 3500, tier sum == open count, cluster eight open of ten rows

🤖 Generated with Claude Code

Polichinel and others added 4 commits August 10, 2026 10:55
HEARTBEAT_URL is a capability: whoever holds it can forge a success ping
and silence the dead-man alert permanently. It was an argv element at
three curl sites, and /proc/<pid>/cmdline is world-readable while four
accounts hold shells on the box. All three now pass it on stdin as a
curl config file.

C-331's own prescribed fix was the bug. The entry said
`printf 'url=%s\n'` — unquoted. Measured before shipping it, with a
value carrying a stray space:

  url=%s        -> parses http://h/uuid, DROPS the /fail, sends anyway
  url = "%s"    -> exit 3, nothing sent

A trailing space or CR would have turned the failure ping into a success
ping, silently. The register's own remedy for a fails-green concern
carried a fails-green defect, unexamined since July because a prescribed
fix reads like a settled thing. Shipped form is quoted; the reason is in
the script so it survives a rewrite.

The /proc claim is drilled with a negative control, since a clean scan
otherwise only proves the scanner is broken. Control leaked the canary
from curl's argv; the fix showed `curl -fsS --max-time 20 -K -` in
flight with nothing anywhere carrying it, three times. A local listener
confirmed all three paths arrive byte-exact — re-run against the three
lines extracted verbatim from the committed file, not retyped.

The guard was wrong first. It asserted "no line contains both curl and
HEARTBEAT_URL", which fails against the FIXED script, because
`printf ... "$HEARTBEAT_URL" | curl ...` puts both on one line. The
property is "never AFTER the curl token". Drilling it is what found
that; it now fails on the pre-change file and passes on this one.

Three stale line citations fixed and the hole that hid them closed:
ADR-018 had drifted ~55 lines, ADR-051 cited one number that was never
right, and C-331's own Location read 93,163,290 against 112/182/309 — in
an entry whose trigger was "next edit to this file". test_docs_citations
could see none of them: its pattern was `.py` only. Widened to
`.py|.sh`, drilled, and those three were the only offenders.

Also: server_operations said "two signals" while listing three, and 24h
grace where the live check is 48h; its two operator verification
commands used the argv form, i.e. ran this exact exposure on the box.

Not live on the server. #423 said no server change was needed; per C-343
the change lands one run after a deploy, so worst case it is in
production two months from merge. Residual left open: HEARTBEAT_URL
still sits in ~/.profile, and if that is not mode 600 it is readable by
all four accounts permanently.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Chasing C-331's residual instead of writing it down as a caveat. The
operator ran three commands on the host:

  stat -c '%a %n' /home/views-deploy/.profile   644
  stat -c '%a %n' /home/views-deploy            751
  test -r /home/views-deploy/.profile           READABLE BY simmaa_prio

751 on the home lets any account traverse in; 644 on the file then lets
it read. Confirmed by direct test from a second account, not inferred
from permission bits.

Exposed: UCDP_API_TOKEN, ACLED_USERNAME, ACLED_PASSWORD, GDL_API_TOKEN,
HEARTBEAT_URL — to dylpin, simmaa_prio and sonja_prio, continuously,
from deployment until today. Not a window; a standing condition. A Tier
4 story about a 10-second exposure surfaced a Tier 2 one about a
permanent exposure.

Fixed with chmod 600 and verified in BOTH directions: no longer readable
by the other account, still readable by the owner with the pipeline path
intact. A permission fix that also breaks the pipeline is not a fix.

Swept the rest of the home rather than assuming. .netrc was already 600
— the data-server credentials were never exposed. .ssh 700, all history
files 600. One hypothesis raised and killed: .local is 775 and
refresh_pipeline.sh prepends $HOME/.local/bin to PATH, which would have
been code execution as views-deploy rather than mere disclosure — but
`getent group views-deploy` returns no other members.

Both setup guides caused this: each showed `>> ~/.profile` with no
chmod, so the mode was whatever umask gave it. Both now carry the chmod
and a verification line.

Left open on purpose: whether the four credentials need rotating. The
readers are trusted colleagues, so this is a policy judgement rather
than a breach — but C-322's GDL token was rotated on weaker evidence.
The operator's call, deliberately unmade.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… basis

The operator's call: no rotation. The three accounts belong to known
colleagues and there is no indication any read the file.

Recorded with the basis, because the basis is what can expire. This
rests on WHO HOLDS THE ACCOUNTS, not on evidence of non-access — no
audit record exists that could establish the latter and none was
consulted. Reasonable on a single-team research host; it would not
survive the accounts being held outside the team. Revisit trigger
named: a new shell account (C-88).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
/code-review medium and /review-diff on this PR. The shell change came
back clean; the guard did not.

The reviewer found a false negative: route the secret through an
intermediate variable —

  FAIL_URL="$HEARTBEAT_URL/fail"; curl -fsS "$FAIL_URL"

— and C-331 is fully reintroduced with the test still green. Not
contrived: that is exactly what collapsing the three WET ping lines into
a helper would produce, and the file's own docstring anticipates a
future rewrite.

The fix for that did not work either. Assignment-tracking was added; the
drill returned rc=0 where it had to return rc=1, because the regex
anchored at ^ with no allowance for indentation and every assignment in
that script is indented. Only running it against a state it was not
written for exposed that.

Three versions, three claims narrower than the property:
  v1  "line contains both curl and HEARTBEAT_URL"  -> failed on the
      FIXED script (printf ... | curl ... has both on one line)
  v2  "literal HEARTBEAT_URL after curl"           -> passed a two-line
      reintroduction of the defect; caught by review, not the author
  v3  assignment-tracking to a fixpoint            -> matched no
      indented assignment; caught by drill, not by reading

v4 drills clean four ways: clean tree passes, direct argv fails, one
indirection fails, two indirections fail.

Also from /review-diff, both mine: the docstring still described v2's
property, and bare substring matching would over-match a one-character
tainted name (the two-level drill produces `A`). Word boundaries now.

/register-risk: 0 new, 1 merged. This is C-336's mechanism a third time
— a claim narrower than the property, failing green — so it is a second
addendum there rather than a new ID. The generalisation: a guard is a
claim about a property, and only running it against states it was not
written for tells the two apart.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@Polichinel

Copy link
Copy Markdown
Contributor Author

Ritual complete — /code-review medium/register-risk/review-diff/register-risk

The shell change came back clean. The guard did not, three times.

/code-review medium — one confirmed bug, in the guard

The reviewer independently verified the three curl sites: || true binds to the whole pipeline under pipefail, printf 'url = "%s"\n' "$ARG" never reinterprets % or \ inside the substituted argument (only the fixed format string is escape-processed), local exit_code=$? runs before the heartbeat block so nothing clobbers it, and missing errtrace is a non-issue since every site is || true-guarded anyway. No shell bug.

But tests/test_heartbeat_secret.py had a false negative:

FAIL_URL="$HEARTBEAT_URL/fail"
curl -fsS --max-time 10 "$FAIL_URL"

C-331 fully reintroduced — secret back on argv, readable via /proc/<pid>/cmdline — with the test still green. Not contrived: that is precisely what collapsing the three WET ping lines into a helper would produce, and this file's own docstring anticipates a future rewrite.

And the fix for it didn't work either

Assignment-tracking was added. The drill returned rc=0 where it had to return rc=1: the regex anchored at ^ with no allowance for indentation, and every assignment in that script is indented. The hardening did nothing until it was drilled.

version claimed what it actually did
v1 line contains both curl and HEARTBEAT_URL failed on the FIXED scriptprintf … | curl … has both on one line
v2 literal HEARTBEAT_URL after curl passed a two-line reintroduction of the defect — caught by review, not the author
v3 assignment-tracking to a fixpoint matched no indented assignment — caught by drill, not by reading

v4, drilled four ways:

rc=0  clean tree              (want 0)
rc=1  direct argv form        (want 1)
rc=1  one level indirection   (want 1)
rc=1  two levels indirection  (want 1)

/review-diff — 2 warnings, both mine, both fixed

The test's docstring still described v2's property after the logic moved to a taint set — doc/code mismatch. And bare substring matching would over-match a one-character tainted name (the two-level drill produces A); word boundaries now, which costs nothing and weakens nothing.

/register-risk — 0 new, 1 merged

This is C-336's mechanism a third time — a claim narrower than the property it names, failing green — so it became a second addendum on C-336 rather than a new ID. The generalisation worth keeping:

A guard is a claim about a property, and the claim is usually narrower than the property. Only running it against states it was not written for tells them apart.

Three gaps here, all three failing green.

Verification

  • bash -n scripts/refresh_pipeline.sh → OK
  • ruff check . → All checks passed
  • mypy src/ → Success, 89 source files
  • pytestPYTEST_EXIT=0, zero FAILED (read from the output file, not the task notification — a trailing echo masked a real failure earlier this session)
  • docs/validate_docs.sh → PASSED
  • Register: 344 IDs, 302 resolved, 39 open (0 T1, 3 T2, 11 T3, 19 T4, 6 deferred), 118 struck, tier sum == open count

@Polichinel
Polichinel enabled auto-merge (squash) August 10, 2026 22:06
@Polichinel
Polichinel merged commit 085ef2a into development Aug 10, 2026
5 checks passed
@Polichinel
Polichinel deleted the fix/heartbeat-url-off-argv branch August 10, 2026 22:20
Polichinel added a commit that referenced this pull request Aug 11, 2026
/code-review medium fact-checked the entry against the repository rather
than against memory, and found two claims that do not survive it.

1. "up to two months in production", attributed to C-343. C-343 says ONE
   month and never derives the larger figure — the two-month version
   requires an unstated compounding worst case. Present in TWO places:
   C-345, and inside C-331 where it had already shipped in #432. Both
   corrected, in the register and the changelog, with a visible
   correction note rather than a silent edit.

2. "two recorded false-readiness incidents that each cost a full day."
   A grep of the register, the changelog and every post-mortem returns
   only the sentence making the claim. The incidents are real and known
   to the operator, but THIS REPOSITORY RECORDS NEITHER. Citing evidence
   a reader cannot find is how a register stops being checkable, which
   is C-336's subject. Removed rather than softened, and the fact that
   they went unrecorded is now flagged for the next post-mortem.

C-345's tier now rests on the one basis that is verifiable in the repo:
it fired twice in one session. That is stated as deliberately the only
evidence cited.

Also from the review, verified and left alone: the shell claim is exactly
right (pipefail changes it, the mitigation works), the header arithmetic
is correct and guarded, the cluster section's counts match its eleven
rows and nine open members, and every cross-reference points at an entry
in the state implied.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Polichinel added a commit that referenced this pull request Aug 11, 2026
…hat was red (#433)

* docs(register): C-345 — the instrument that detects this cluster is a member of it

Tier 2. Twice in one session a failing suite was reported as passing.

  uv run pytest -q | tail -2; echo "EXIT=$?"

A pipeline's exit status is its LAST element's, so tail returned 0 while
pytest had exited 1. Then a backgrounded run's task notification said
"exit code 0" for the same reason — the command ended in an echo. The
second was caught only by reading the output file, after the user had
already been told the suite was running and would be folded in. One step
from reporting green on red.

Registered rather than remembered because this project has two recorded
false-readiness incidents that each cost a full day; because the suite is
the gate on every story in this epic; and because C-343 means a defect
that slips through can sit in production for two months. C-339 is the
precedent for registering a workflow hazard rather than a code one.

The shape is the cluster's own, which is the uncomfortable part. C-330
was a nightly no-op exiting 0. C-337 was a lockfile frozen with no error.
C-343 was a deploy that deployed nothing. This is the same defect in the
instrument used to find all three.

Mitigation adopted and explicitly NOT a control: redirect to a file,
capture $? unpiped, grep ^FAILED as a second independent reader. A habit
is not machinery. The instrument — refusing to report a result not
obtained unpiped — is proposed for #424.

Also carries the changelog note on what was NOT registered: the unbounded
poll loop (one-line fix, no consequence) and the group-writable dotfiles
(group has no other members, already inside C-344 as a killed
hypothesis). The guard-narrower-than-the-property pattern went to C-336
as a second addendum during #432.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* docs(register): two false claims in C-345, both mine (#433)

/code-review medium fact-checked the entry against the repository rather
than against memory, and found two claims that do not survive it.

1. "up to two months in production", attributed to C-343. C-343 says ONE
   month and never derives the larger figure — the two-month version
   requires an unstated compounding worst case. Present in TWO places:
   C-345, and inside C-331 where it had already shipped in #432. Both
   corrected, in the register and the changelog, with a visible
   correction note rather than a silent edit.

2. "two recorded false-readiness incidents that each cost a full day."
   A grep of the register, the changelog and every post-mortem returns
   only the sentence making the claim. The incidents are real and known
   to the operator, but THIS REPOSITORY RECORDS NEITHER. Citing evidence
   a reader cannot find is how a register stops being checkable, which
   is C-336's subject. Removed rather than softened, and the fact that
   they went unrecorded is now flagged for the next post-mortem.

C-345's tier now rests on the one basis that is verifiable in the repo:
it fired twice in one session. That is stated as deliberately the only
evidence cited.

Also from the review, verified and left alone: the shell claim is exactly
right (pipefail changes it, the mitigation works), the header arithmetic
is correct and guarded, the cluster section's counts match its eleven
rows and nine open members, and every cross-reference points at an entry
in the state implied.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant