diff --git a/reports/register_changelog.md b/reports/register_changelog.md index fa59f4c..e3302a7 100644 --- a/reports/register_changelog.md +++ b/reports/register_changelog.md @@ -17,6 +17,51 @@ entry could. --- +## C-345 — the instrument that detects this cluster is a member of it (2026-08-11) + +`/register-risk` after #432. Tier 2, and the only entry so far found by watching **my own +verification** rather than the system. + +**Twice in one session a failing suite was reported as passing.** First +`uv run pytest -q | tail -2; echo "EXIT=$?"` — a pipeline's status is its *last* element's, so `tail` +returned 0 while pytest had exited 1. Then a backgrounded run's task notification said *"exit code +0"* for the same reason: the command ended in an `echo`. The second was caught only by reading the +output file, and by then the user had already been told the suite was running and would be folded +in. One step from reporting green on red. + +**Why it is registered rather than remembered.** The suite is the gate on every story in this epic; +a false green means a defect merges, and per C-343 a `refresh_pipeline.sh` change reaches production +only at the next cron run after a deploy — a further month later still if the deploy is tag-file-only. +The precedent for registering a workflow hazard rather than a code one is C-339, an assistant-authored +command that destroyed a production log. + +**A claim was cut from this entry during review, and the cut is worth recording.** The first draft +justified the tier partly on *"two recorded false-readiness incidents that each cost a full day"*. +Those incidents are real and known to the operator — but **nothing in this repository records them**. +A grep of the register, the changelog and every post-mortem returns only the sentence making the +claim. It also attributed "two months in production" to C-343, which says *one* month and never +derives the larger figure. Both were caught by fact-checking the entry against the repo rather than +against memory. Citing evidence a reader cannot find is precisely how a register stops being +checkable, and the same overreach had already shipped inside C-331 in #432 — corrected here too. + +**The shape is the cluster's own, which is the uncomfortable part.** C-330 was a nightly no-op +exiting 0. C-337 was a lockfile frozen with no error. C-343 was a deploy that deployed nothing. This +is the same defect in the instrument used to find all three. An epic about mechanisms that report +success while doing nothing spent a week using one. + +**Mitigation adopted, and explicitly not a control:** redirect to a file, capture `$?` unpiped, grep +`^FAILED` as a second independent reader. A habit is not machinery — nothing stops the next pipeline +masking a status the same way. The instrument would refuse to report a result it did not obtain +unpiped; proposed for #424. + +**Two findings from the same session deliberately not registered.** The guard-narrower-than-the- +property pattern (three versions of `test_heartbeat_secret.py`, each failing green) went to C-336 as +a second addendum — same mechanism, new location. And an unbounded `until … sleep 30` poll that died +on a transient DNS failure was fixed by bounding the loop; a one-line workflow correction with no +consequence, below the register's bar. + +--- + ## C-331 RESOLVED — and the entry's own prescribed fix was the bug (2026-08-10) Epic #421 Story 2 (#423). The heartbeat URL now reaches `curl` on stdin via `-K -` at all three @@ -64,7 +109,10 @@ form — running, on the box, the exact exposure this story removes. Replaced wi quoting verified against a local listener before being written down. **Not live on the server.** #423's note said no server change was required. Per C-343 that is false: -the change lands one run after a deploy, so worst case it is in production two months from merge. +the change reaches production at the first cron run after a deploy, and a further month later if the +deploy is tag-file-only — weeks to months depending on when the next release happens. (This read "two +months from merge" when written; corrected in #433 — C-343 says one month and never derives the +larger figure.) Recorded rather than quietly assumed, and the issue has been corrected. **The residual was checked, and it was worse than the concern.** C-331's entry flagged that diff --git a/reports/technical_risk_register.md b/reports/technical_risk_register.md index 0ba8e4b..9bd9881 100644 --- a/reports/technical_risk_register.md +++ b/reports/technical_risk_register.md @@ -1,9 +1,9 @@ # Technical Risk Register **Date:** 2026-03-17 (updated 2026-07-27) -**Last update:** 2026-08-10 — C-331 RESOLVED (#423, heartbeat URL now on stdin, drilled with a negative control) and C-344 registered-and-resolved: chasing C-331's residual found `views-deploy`'s `~/.profile` at mode 644, i.e. every harvest credential readable by all four accounts continuously — fixed, with the rotation question left open for the operator. Full narrative history, including corrections and retractions, is in [`register_changelog.md`](register_changelog.md). Keep this line to one sentence: the header is an index, and the search-window guard (`test_falsification_merge_readiness.py`, 8000 chars) is what it protects. New narrative goes in the changelog, never here (#404). +**Last update:** 2026-08-11 — C-345 registered (Tier 2): the verification tooling reported a green suite that was red, twice in one session, so the instrument used to detect this whole cluster is itself a member of it. Full narrative history, including corrections and retractions, is in [`register_changelog.md`](register_changelog.md). Keep this line to one sentence: the header is an index, and the search-window guard (`test_falsification_merge_readiness.py`, 8000 chars) is what it protects. New narrative goes in the changelog, never here (#404). **Source:** 71 audits, reviews, and incidents — multi-expert engineering review, repo assimilation, falsification audits, test reviews, security sweeps, and production incidents. Full list in [`register_changelog.md`](register_changelog.md#where-the-findings-came-from). Add new sources there, not here (#404). -**Status:** 344 concern IDs assigned (C-28 merged into C-31, C-107 merged into C-60, C-183 merged into C-44, C-44 merged into C-164, C-03 merged into C-176): 302 resolved-or-demoted, 39 open concerns (0 Tier 1, 3 Tier 2, 11 Tier 3, 19 Tier 4, 6 deferred by design; 4 with fired trigger); 5 demoted to tech-debt backlog 2026-08-04, 8 open disagreements. 167 resolved concerns as full entries + 19 early-archive reference rows + 118 struck-through in active register (297 unique after dedup — 5 appear in both archive and active) + 32 resolved disagreements in archive. 42 disagreement IDs total: 34 resolved, 8 open. +**Status:** 345 concern IDs assigned (C-28 merged into C-31, C-107 merged into C-60, C-183 merged into C-44, C-44 merged into C-164, C-03 merged into C-176): 302 resolved-or-demoted, 40 open concerns (0 Tier 1, 4 Tier 2, 11 Tier 3, 19 Tier 4, 6 deferred by design; 4 with fired trigger); 5 demoted to tech-debt backlog 2026-08-04, 8 open disagreements. 167 resolved concerns as full entries + 19 early-archive reference rows + 118 struck-through in active register (297 unique after dedup — 5 appear in both archive and active) + 32 resolved disagreements in archive. 42 disagreement IDs total: 34 resolved, 8 open. **Archive:** Resolved concerns and disagreements are in `archive/technical_risk_register_resolved.md`. **Ranking criteria:** Impact if wrong x likelihood x detectability. Items marked **[DEFER]** are accepted risks or wait for a specific trigger condition. See ADR-020 for governance rationale. @@ -161,6 +161,7 @@ | C-328 | 4 | HEAD re-publishes the admin username the 2026-07-27 go-public redaction removed, plus two colleagues' shell accounts | Next edit to `technical_risk_register.md` — placeholders + a pre-commit guard so the policy is enforced, not remembered | Server hardening | | C-329 | 3 | The PyPI-publishing job runs unpinned third-party actions while holding OIDC publish rights — a poisoned wheel needs no secret to leak | Before the next release — pin `publish_package.yml` actions to full commit SHAs | Supply chain | | ~~C-330~~ | ~~4~~ | ~~Rotation undocumented; file mode inferred, not observed~~ | Resolved 2026-08-03 on the server: the config pointed at `/root/...`, a path the pipeline left months ago, and `missingok` made it exit successfully every night. Path fixed, `monthly`, `create 0640`, `su views-deploy`; verified by dry run. Mode observed: was 644, now 640 | Server hardening | +| C-345 | 2 | Verification tooling reported a green suite that was red, twice in one session — a piped `pytest \| tail; echo $?` yields the pipe's status, and a task notification reported "exit code 0" for a run that exited 1 | **When capturing a long-running check's result** — piping it, backgrounding it, or reading a notification instead of an unpiped `$?`. Redirect to a file, capture `$?` unpiped, and grep `^FAILED` as a second reader | Test infra | | ~~C-344~~ | ~~2~~ | ~~`views-deploy`'s `~/.profile` was mode 644 inside a 751 home — every harvest credential (`UCDP_API_TOKEN`, `ACLED_*`, `GDL_API_TOKEN`, `HEARTBEAT_URL`) readable by all four accounts, continuously~~ | Registered and resolved 2026-08-10 (#432): `chmod 600`, verified unreadable from a second account and still readable by the owner. Rotation considered and **declined** by the operator 2026-08-10 — a judgement about who holds the three accounts, not evidence of non-access; revisit if a new shell account appears (C-88) | Credential hygiene | | ~~C-331~~ | ~~4~~ | ~~`HEARTBEAT_URL` capability URL passed on the curl command line — readable via `/proc`~~ | Resolved 2026-08-10 (#423): all three pings take the URL on stdin via `-K -`; drilled with a canary and a negative control. The entry's own suggested unquoted form was superseded — it truncates at whitespace and sends anyway | Operational monitoring | | C-332 | 3 | Credential redaction incomplete — `_redact_url` ignores URL userinfo, `zarr_path` interpolated raw into 7 messages, netrc exceptions log contents, `BasicAuth`/`_TokenState` reprs | **Before interpolating any URL, path, or credential-bearing value into a log line or exception message** — that is the act that creates the exposure, not editing these files | Credential hygiene | @@ -192,18 +193,19 @@ *Added by `/review-rr strategic`, 2026-08-04.* -Eight open entries are symptoms of one root cause, and read very differently together than apart. +Nine open entries are symptoms of one root cause, and read very differently together than apart. Individually each is small. Together they say that **this project's characteristic failure is silence, not error** — a thing reports success while not doing what it claims. *C-342 was added to the cluster 2026-08-08, found while building the guard for C-337 — which is the cluster's own rule working: the drill found a defect adjacent to the one it was aimed at. C-343 was added 2026-08-08 from the production host, and C-317 was **closed by drill** 2026-08-10 — struck in -the table below rather than removed, so the table has ten rows. C-331 was closed 2026-08-10, leaving eight open members.* +the table below rather than removed, so the table has ten rows. C-331 was closed 2026-08-10 and C-345 added 2026-08-11, leaving nine open members of eleven rows.* | ID | What reported success while being wrong | |---|---| | ~~**C-317**~~ | ~~`SIGKILL` bypasses the `ERR`/`EXIT` traps, so a killed run sends no failure ping~~ — **closed by drill 2026-08-10, the first member closed by observation** | +| **C-345** | A piped exit status and a task notification each reported a failing suite as passing | | ~~**C-331**~~ | ~~`HEARTBEAT_URL` on the curl command line — leaks via `/proc` with nothing to notice~~ — **closed 2026-08-10, drilled with a negative control** | | **C-336** | Governance docs true when written, false later, nothing failing in between | | **C-337** | `views-frames>=1.0` let `uv.lock` freeze at 1.0.0 for six weeks; no error, ever | @@ -224,7 +226,7 @@ looking at the production host. **The counts in this paragraph used to be exact twice** — they said "eight" while the table held nine and then ten. That is C-336 happening inside the cluster section about it, so the tally is now qualitative on purpose. -**Cluster-level action, cheaper than eight separate ones:** when adding any guard, workflow, or +**Cluster-level action, cheaper than nine separate ones:** when adding any guard, workflow, or operational step, drill it by breaking it. Every guard drilled in the 2026-08 window found a real defect — including one in its own author's work, one day after writing it. @@ -2548,8 +2550,10 @@ different blast radius. **Not live on the server yet**, and the issue said otherwise. #423 stated "no server change is required — the script is deployed by tag." C-343 established that bash buffers the script, so a change to `refresh_pipeline.sh` lands one run *after* a deploy — a month on a monthly cron — and -deploying is three steps, not one. Worst case this fix is in production two months from merge, and -the argv exposure continues until then. +deploying is three steps, not one. The fix reaches production at the first cron run after a deploy, +and a further month later if the deploy is tag-file-only — so weeks to months depending on when the +next release happens, with the argv exposure continuing until then. (Corrected 2026-08-11: this +read "two months from merge", a figure C-343 does not derive — #433.) Cross-ref: C-131 (the alert this would silence), ~~C-317~~ (the SIGKILL gap the failure ping closes, resolved by drill the same week), C-343 (why this is not yet live), C-336 (why this entry's own @@ -3124,6 +3128,42 @@ Cross-ref: ~~C-331~~ (the residual this was found chasing), C-322/~~C-324~~ (GDL --- +### C-345: The verification tooling reported a green suite that was red — twice in one session + +**Source:** observed while executing epic #421 Story 2 (2026-08-10/11). Not analysis — it happened, twice, and was caught by accident the second time. + +**Trigger:** **When capturing the result of a long-running check** — piping it, backgrounding it, or reading its outcome from a task notification rather than its exit status. Concretely: any `pytest`/`ruff`/`mypy` invocation whose status is read from anything other than an unpiped `$?`. + +**Location:** the verification step of every story in this epic. Not a repo file — a workflow defect, registered on the precedent of C-339. + +**What happened.** Twice, a full-suite run was reported as passing when it had failed. + +1. `uv run pytest -q | tail -2; echo "EXIT=$?"` — a pipeline's exit status is that of its **last** element. `tail` succeeds whatever pytest did, so `EXIT` was `0` while pytest had exited `1`. +2. A backgrounded run's harness task-notification reported *"exit code 0"* for a command whose pytest had exited `1`, because the command ended in an `echo`. + +The second was caught only by reading the output file instead of the notification — and by then the user had already been told the suite was running and would be folded in. **One step from reporting a green suite that was red.** + +**Tier 2, and the justification is required rather than assumed.** Not Tier 3: this is not a maintainability cost. The suite is the primary gate on every story in this epic, and a false green on it means a defect merges. And per C-343 the defect is not merely merged but slow to surface: a `refresh_pipeline.sh` change reaches production at the next cron run after a deploy, and **one run later still** — a further month — if the deploy is tag-file-only. Not Tier 1: no data was corrupted and no model output was wrong; the failure is in *knowing whether* work is sound, not in the work itself. + +**The trigger is not hypothetical — it fired twice in one session.** That is the whole evidentiary basis, and it is deliberately the only one cited. An earlier draft justified the tier by appealing to "two prior false-readiness incidents that each cost a full day". Those incidents are real and known to the operator, but **this repository records neither of them** — not in the register, the changelog, or any post-mortem. Citing history a reader cannot find is how a register stops being checkable, which is C-336's whole subject. The claim is removed rather than softened; that the incidents went unrecorded is itself a gap, and belongs to whoever writes the next post-mortem. + +**The shape is the cluster's, exactly.** A mechanism that reports success while establishing nothing. It is the same defect as C-330 (a nightly no-op exiting 0), C-337 (a lockfile frozen with no error) and C-343 (a deploy that deployed nothing) — this time in the instrument used to detect all of them, which is why it belongs registered rather than remembered. + +**Mitigation adopted mid-session, not yet enforced by anything:** redirect to a file, capture `$?` on the **unpiped** command, then grep the output for `^FAILED` as a second independent reader: + +```bash +uv run pytest -q > out.txt 2>&1; echo "PYTEST_EXIT=$?" +grep -cE '^FAILED' out.txt +``` + +Two readers, because one reader that can be wrong is what this entry is about. + +**Open, because a habit is not a control.** Nothing prevents the next pipeline from masking a status the same way. The instrument would be a wrapper that refuses to report a result it did not obtain unpiped — proposed for **#424**, which is the story about giving checks somewhere to run other than one operator's discretion. + +Cross-ref: C-339 (the other assistant-workflow hazard, and the precedent for registering one), C-330/C-337/C-343 (same shape, different mechanisms), C-341 (gates that assure only whoever ran them — this is the failure mode *of* running them). Part of the **mechanisms that fail green** cluster. GitHub: #432. + +--- + ## Deferred by Design ### C-10: Ontology vocabulary overhead