docs(register): C-343 registered (Tier 2) and C-317 RESOLVED by live drill - #431
Conversation
Observed on the production host, not read off the repo. The tag file
said v1.11.0, the working tree was at v1.10.0, views-frames was still
the frozen 1.0.0, and `git fetch --tags` reported v1.11.0 as a NEW tag.
Three sources of truth disagreeing for five days with no error anywhere
— and the release in question had been cut for the sole purpose of
raising that views-frames floor.
Two reasons the in-script `git checkout` does not rescue this, tested in
a throwaway repo at 31 KB rather than reasoned about:
1. bash buffers the script and never re-reads it, so a change to
refresh_pipeline.sh itself lands one run later — a month, on a
monthly cron.
2. `uv sync` is never invoked by the pipeline at all, so a dependency
change never lands however often it runs.
Tier 2 with the justification stated: not maintainability, but the
server running code nobody believes it is running, with the status page
green and the heartbeat pinging. It does not corrupt data; it silently
prevents fixes from landing. The trigger fires at the next release, and
#423 ships a refresh_pipeline.sh change — exactly the class that lags.
Blast radius on this occasion was small and the entry says so: no src/
file and no pipeline script differs between the two tags, production
imports no views_frames estimator, and the PyPI wheel carried the right
floor throughout. The damage was to what we could claim.
Remediated live: uv sync moved views-frames 1.0.0 -> 1.10.2 and
views-datafactory 1.10.0 -> 1.11.0, and FeatureFrame was imported on the
host afterwards. Entry stays OPEN — nothing prevents recurrence. The
instrument (pre-flight tag-vs-HEAD check, fail-loud per ADR-011) goes to
#424, not here.
Also fixes the contradiction that caused it: the script header said
deploying was one step. It now says it is three, and why.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… alert (#427) Epic #421 Story 6. The first entry in the fails-green cluster closed by observation rather than by argument. PR #359 added a /start ping on the theory that healthchecks.io would flag a run that began and never finished — the OOM-kill case, where SIGKILL bypasses both the ERR and EXIT traps so neither the success nor the /fail ping ever fires. Nobody had watched that happen; the entry has read "OPEN pending live grace-timeout drill" since July. Closing it because the ping is sent would have been the cluster's own mistake: the ping firing was never in doubt. Drill: throwaway check at 5 min period / 1 min grace so the timeout was observable in a minute instead of 30 days + 48 hours. Production HEARTBEAT_URL and the production check untouched. One /start, then nothing. Prediction recorded before the ping. Observed: "is DOWN (success signal did not arrive on time, grace time passed)", Last Ping Type: Started, 03:35:56 +0200. Detection latency for an OOM kill drops from ~32 days to the grace window. Throwaway deleted. Two things found that nobody was looking for. healthchecks.io's schedule dialog documents the mechanism outright — "Grace Time: when a check is late, OR has received a 'start' signal" — so it was vendor-documented all along and simply never read. And the sample check reused for the drill had sat grey, never red, for two months while permanently overdue: a check that has never been pinged does not alert, so a monitor created and never wired up is indistinguishable from a healthy one. Not closed by this: the status page is still not regenerated on SIGKILL (C-338's territory), and detection is not prevention (C-173). Three header counts were wrong across three edits — resolved total, summary-vs-entry 1:1, and struck-through. All three were caught by the register guards, not by me. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Polichinel
changed the title
Second commit added: C-317 RESOLVED (#427, epic #421 Story 6)Kept on this branch rather than a separate PR because both changes rewrite the same header counts; two overlapping PRs on those lines would be worse to review than one. The drill ran, and the mitigation worksPR #359 added a Method. Throwaway check, 5 min period / 1 min grace, so the timeout was observable in a minute rather than 30 days + 48 h. Production Prediction, recorded before the ping: red within ~1 minute, e-mail arrives. Observed:
Two things found that nobody was looking forhealthchecks.io documented the mechanism all along. Its own schedule dialog reads "Grace Time — when a check is late, or has received a 'start' signal, how long to wait to send an alert." Weeks of "unverified" was resolvable by reading a tooltip. A check that has never been pinged does not alert. The sample check reused for the drill had sat grey — never red — for two months while permanently overdue. A monitor created and never wired up is indistinguishable on the dashboard from a healthy one. That is Not closed by thisThe status page is still not regenerated on The guards caught three of my errorsResolving one entry needs four counts updated. I got three wrong and the register guards caught every one: the resolved total (299 → 300), the summary-row-to-detail-heading 1:1 match (I struck the row and not the heading), and the struck-through count (115 → 116). None was caught by me. One more worth recording: the first full-suite run after this edit failed, and the background task notification reported "exit code 0" — because the command ended in Verification
|
…#431) /code-review medium and /review-diff on this PR, all findings fixed here. Two stale counts, both already stale before this PR and made worse by it: the fails-green cluster prose said "three of the eight" and "cheaper than eight separate ones" while the table held nine, then ten. That is C-336 happening inside the cluster section about C-336. The tally is now qualitative on purpose — a number no guard protects will go stale again — and C-336 gains the location, rather than a new concern ID. One false claim of my own: C-343 said `git diff v1.10.0..v1.11.0` touches "only pyproject.toml, uv.lock and three GitHub workflows". It spans 31 files. I had filtered out docs/reports/tests and then reported the filtered result as the whole. The load-bearing part — no src/ file, no pipeline script — is true; the framing was not. Corrected in both the register and the changelog, in a paragraph whose subject is checking before believing. One self-invalidating anchor: the entry cited refresh_pipeline.sh at 11,592 bytes, which this same PR grows to 12,385. Now anchored to v1.11.0 explicitly, so the number cannot rot. And two from /review-diff on the shell comment. It claimed "the server always runs a specific, tested version" eight lines above explaining that this is the belief C-343 disproves — now scoped to the Python, with a pointer forward. And it restated the three deploy commands that already live in server_quickref.md, which is a second copy of the procedure whose drift IS C-343. The restatement is gone; precedence is stated instead: if the two ever disagree, the quickref wins. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Ritual complete —
|
1 participant
Part of #421. Not a story — a production finding surfaced while sizing #423's blast radius.
What was observed
Four commands on the production host, output pasted by the operator:
Three sources of truth disagreeing, for five days, with no error anywhere. The status page was green, the heartbeat was pinging, and the tag file read correctly. v1.11.0 was cut on 2026-08-03 for the sole purpose of raising the views-frames floor — and the floor on the box was still 1.0.0.
Why the in-script checkout does not rescue it — tested, not reasoned
refresh_pipeline.shrunsgit checkout "$DEPLOY_TAG"while bash is executing that same file. A throwaway repo was built to find out what bash actually does, at 31 KB — nearly triple the real script's 11,592 bytes:Bash buffers the script and does not re-read it. Two consequences:
refresh_pipeline.shitself takes effect one run later — a month, on a monthly cron.uv syncis never invoked by the pipeline, so a dependency change never lands, however often it runs.The cause
Two documents describe deploying, and they contradict:
refresh_pipeline.shheader~/.views-deploy-tagon the server."server_quickref.md§Deploygit fetch --tags && git checkout <tag>plusuv syncThe wrong one is the one a person reads while inside the file they are editing. Fixed here — the header now says deploying is three steps, and gives both reasons the checkout below cannot substitute.
Tier 2, justified
Not Tier 3: this is not a maintainability cost, it is the server running code nobody believes it is running, with no error at any point. Not Tier 1: it does not itself corrupt data. What it does is silently prevent fixes from landing, converting every future correctness fix into one that may or may not be in effect. The trigger is not hypothetical — it fires at the next release, and #423 ships a
refresh_pipeline.shchange, precisely the class that lags.Blast radius, stated honestly
The first framing of this was "the first fails-green instance with production consequences." That was wrong, and checking before believing it showed why:
No
src/file. No pipeline script. Production imports four non-estimator symbols from views_frames (feature_frame.py:10-15). No number was ever wrong, and PyPI consumers were never exposed — the published wheel carried the correct floor throughout. The damage was to what could be claimed, not to what was produced. The corrected framing is what the entry says.Remediated, and still open
The three documented steps ran live:
uv syncmoved views-frames 1.0.0 → 1.10.2 and views-datafactory 1.10.0 → 1.11.0, andFeatureFramewas then imported on the host — a ten-minor-version jump nobody has watched import is a belief, not an observation.The entry stays OPEN. The fix was a human running commands; nothing prevents recurrence. The instrument — a pre-flight check comparing
$DEPLOY_TAGagainst the HEAD the script started from, refusing to run on disagreement (fail-loud, ADR-011) — is proposed for #424 and deliberately not built here.Verification
bash -n scripts/refresh_pipeline.sh→ OKruff check .→ All checks passeddocs/validate_docs.sh→ PASSEDpytest→ exit 0🤖 Generated with Claude Code