Status reconciled: skills and ticket 20 match what merged

.scratch/mastering-to-done/issues/20-bookkeeping-without-legacy-or-custom-code.md

@@ -101,7 +101,7 @@ Each test file 20a deletes holds assertions a later slice must prove again on th
   - **Noted, not done:** `retry` in the envelope is the claim's contention backoff; capping a unit's work attempts is open (20c, where the first retried work arrives).
   - Local after the fixes: 107 passed (Bookkeeping, generated work, Journal, the wheel, architecture and contract tests).
 - [x] 20b: PR #802; CI green (all six checks); merged on the operator's word "merge", 2026-10-02 22:42 ET
-- [ ] 20c: SEC Company through the engine (expanded below by the operator's next request). On 2026-10-09 21:05 ET the four Company-named steps were still declared in `rules/sources/sec.submissions.company/source.yaml`; removed 2026-10-11 (below). Partial: Company preparation still runs outside Bookkeeping; it waits for Codex's Rust join port (below)
+- [x] 20c: SEC Company through the engine (expanded below by the operator's next request). On 2026-10-09 21:05 ET the four Company-named steps were still declared in `rules/sources/sec.submissions.company/source.yaml`; removed 2026-10-11 (below). Partial: Company preparation still runs outside Bookkeeping; it waits for Codex's Rust join port (below) Done 2026-10-13 08:50 ET (status reconciled after Codex's go-live review): Company preparation is the `mastering` Bookkeeping target (#928, #929), its installed proof masters 6,414 Companies on the installed workers (#930), and a batch that changes nothing queues nothing (#931). Left open by design: acquisition, `--limit`, planning and attempt caps moved to L8 (parked until the AWS rollout, #938); the speed-ups deferred by the operator; changed data across runs moved to ticket 22
   - [x] Company reading is the engine only: since #880 and #905, `mdm prepare-clean-company` reads the raw capture through `source.read` (`census-main.yaml`, `census.yaml`, `catalog.yaml`) and `source.combine` (`combine-census.yaml`), each unit run then verified and kept in the bundle; `company_source.py` and `loaders/` are deleted; the `mdm` target merges the bundles (installed proof, 6,414 Companies, #905 and #907) 2026-10-11 14:10 ET
   - [x] GoF consult (in this session): delete the dead target; the registry check lives in tests (control must not import workers) 2026-10-11 14:10 ET
   - [x] Company-named steps removed: the 12-step `company` target (`company.expand`, `company.silver`, `company.prepare`, `company.publish_expand`, `source.evidence`, `provider.capture`) was never validated or run (control validates only a run's selected target; tests selected `capture`); deleted, header comment and `docs/company-only-acquisition.md` rewritten to today's route; new test `test_every_declared_target_names_registered_workers` fails on the old file (`provider.capture`) and passes now 2026-10-11 14:10 ET
@@ -111,7 +111,7 @@ Each test file 20a deletes holds assertions a later slice must prove again on th
   - [ ] ~~A cap on a unit's work attempts (20b's note)~~ deferred to L8: the first retried work is a provider request
   - [x] The `acquisition:` block stays: `resolve_feed`, the catalog and the acquisition authority read it, and Person shares the feed; its `required_producers` name the steps L8's capture target must have 2026-10-11 14:35 ET
   - [ ] ~~Company preparation as Bookkeeping steps, not a Company-named Python module: `company_prepare.py` runs `source.read` and `source.combine` in-process (no claim, report or admission) and edits rules settings in Python (the census keys narrowed to the bundle, `max_input_bytes`, the `forms` and `tickers` group sources)~~ waits for Codex's Rust join port (operator, 2026-10-11 16:40 ET: "Wait for Codex's join port"). Measured on main e4be3019 with the cm27 census (43,245 names, 5.7 MB): a whole-census reading is 17.0 MB (of combine's 32 MB input limit) and costs 0.68 s to load per combine call against 0.008 s narrowed to a bundle's 960 names; combine runs about 480 times per 960-Company bundle (`census-main.yaml` `max_artifacts: 2`), run then verified, so about 11 more minutes per bundle (today about 9). Narrowing stays, and it can be configuration: `source.read`'s derived lookup sets (`_derived_sets`, 1..256 producer readings) narrow by the main readings. The gap is in Bookkeeping: `from` passes one prerequisite's receipt and expansion makes children from one parent, while the census read needs every main reading of a bundle and each combine three readings, so a fan-in input is needed in either design 2026-10-11 16:45 ET
-  - [ ] Company preparation as Bookkeeping steps, resumed: Codex's handoff (`.planning/workstreams/rust-configured-joins/HANDOFF-TO-CLAUDE.md` on `codex/configured-engine-claude-handoff`, 2026-10-11) removes the wait: #920 and #921 are on main; review fixes 2026-10-12 09:00 ET: the Bookkeeping spec documents composed inputs (`skills/bookkeeping/SPEC.md`), stored and checked references share `Artifacts.address`, the unused `Bookkeeping.item` removed, the CLI help and a 513-document test state the 512 cap; operator, 2026-10-12: "Yes" (start 20c next). Branch `claude/issue20-20c-fan-in`
+  - [x] Company preparation as Bookkeeping steps, resumed: Codex's handoff (`.planning/workstreams/rust-configured-joins/HANDOFF-TO-CLAUDE.md` on `codex/configured-engine-claude-handoff`, 2026-10-11) removes the wait: #920 and #921 are on main; review fixes 2026-10-12 09:00 ET: the Bookkeeping spec documents composed inputs (`skills/bookkeeping/SPEC.md`), stored and checked references share `Artifacts.address`, the unused `Bookkeeping.item` removed, the CLI help and a 513-document test state the 512 cap; operator, 2026-10-12: "Yes" (start 20c next). Branch `claude/issue20-20c-fan-in` Done: #928 (fan-in, census narrowing), #929 (the `mastering` target), #930 (installed proof); recorded 2026-10-13 08:50 ET
     - [x] Generic Bookkeeping fan-in, a composed input: `{"compose": {"document", "slots"}}` in a version-2 manifest; a template value that is exactly `{"slot": <name>}` is filled with that slot's verified receipt, or an ordered list of up to 256, or a nested composed document's reference (one level); every slot used and declared, every unit an existing unit of a declared prerequisite, 1 MiB. The database already holds a step until every prerequisite unit is verified (`bookkeeping.claim`), so no migration. Control fills without side effects on every check and stores the filled document (beside the unit's output, named by its sha256) only when handing work to a worker; the engine now refuses an input form it does not know. Workers unchanged. GoF consult: no new type, plain branches; fail closed on unknown forms; store only in `envelope`; a slot is a whole value. Tests: contract (a composed input frozen as its slots; eleven invalid forms blocked), Postgres (a gathering step waits for every prerequisite unit, reads a single slot, a list and a nested document, and is not re-run on resume; a corrupt slot blocks the run without executing it); Bookkeeping Postgres suite 48 passed, contract and architecture 55 passed, change journal, approval, worker artifact and artifact context suites 141 passed; the installed-bundle engine tests need `cargo`, not on this machine (CI runs them) 2026-10-12 08:27 ET
     - [x] Census narrowing as configuration: `census.yaml` declares two lookup sets, `names` and `ciks`, and keeps a census entry only when its key is a member (CIK keys compared without leading zeros); its 256 MiB input bound moves from Python into the file. The sets are derived (`source.read` version-2 lookups) from a new tiny reading, `census-keys.yaml` (each main document's name key, the bundled `name-key.yaml` recipe, and its CIK without leading zeros; one row a document), not from the main readings: a derived set counts every row of every producer table, at most 100,000 rows and 64 MiB, and a main reading carries each filer's whole filings table. Found: `source.read` takes at most 2 documents a reading and a derived set at most 256 readings, so a census read narrows at most 512 documents; a Company bundle now names at most 512 (it was 1,000; cm27 chunks were about 960), and each document is parsed twice (main and keys). No Python edits of the census contract remain; Company preparation sends the census a version-3 request whose lookups document names the key readings. Digests for approval: `census.yaml` `6b8616f7…` (was `319d58b7…`), new `census-keys.yaml` `7943a4a6…`; `census-main.yaml` and `combine-census.yaml` unchanged. Tests: census narrowed to a bundle's names and CIKs (a name and a CIK outside the sets dropped); the census-keys recipe equals `name-key.yaml`'s and `census-main.yaml`'s; Company preparation and census suites 37 passed; engine, rules, Mapping Document and CLI suites 1,035 passed plus the 13F Postgres worker test; the 15 architecture failures (secrets manifest, bootstrap scripts) are the same on main 2026-10-12 08:39 ET; operator, 2026-10-12 08:56 ET: "Approved" (both digests, after the plain explanation of each: the census reading kept to the bundle's names and CIKs with its input bound in the file, and the new census-keys reading; no Rules Database login here, so the approval is recorded on this ticket); known limit (Standards review): the census is now read whole and filtered by membership, where the old literal key list was projected first, so `source.read`'s 100,000-record cap for a materialized reading applies to the whole census (76,117 names over the whole-population capture, `.scratch/company-mastering/research/08-parity.json`; 24% headroom); past it the census needs stream framing (up to 10 million records), as GLEIF's census reading has; parsing cost is unchanged (the key list also parsed the whole document)
     - [x] Part 3b design (found 2026-10-12 before code): one `mdm.prepare` and `mdm.merge` per combine unit would change MDM state, not only cost: about 38,000 consumers, manifests and MDM runs over the population against about 150 today; publication keys per two Companies instead of per bundle; and revision fixed at 1, while survivorship orders a subject's readings by revision and refuses two bodies under one revision (`survivorship.py:39-54`: "Source native revision has contradictory publications"), so a second capture with changed facts would fail. So one records step and one merge per bundle: the records unit's composed input lists the bundle's combine outputs (up to 256), and `mdm.prepare` gains, generically, a bounded list of readings and optional `publication_key` and `revision` keys; the manifest's `scope` stays in the caller's report (no merge code reads it). Part 4's "same as today" means the same MDM state after merging the same capture on both paths (records, assertion ids, bronze occurrences, publications), not the same bundle files
@@ -125,7 +125,7 @@ Each test file 20a deletes holds assertions a later slice must prove again on th
     - [x] Part 4, the installed empty-store proof on the `mastering` target, 2026-10-12 17:40 ET: `.scratch/mastering-to-done/proof/installed_mastering_proof.py` (the 2026-10-10 harness reused for counts, bindings and GLEIF; new: each bundle a `mastering` run whose manifest the installed `mdm prepare-clean-company` writes, workers and verifiers under separate logins, two bundles prepared at a time, merges one at a time in bundle order). 14 bundles, 6,726 documents: 6,414 Companies, SEC 6,414 staged and bound (312 wait as classification deferred), GLEIF 3,716 staged and 3,052 bound, decisions `company-cik` 6,414, `sec-gleif-name-jurisdiction` 2,861, `sec-gleif-name-postal` 191, open reviews 664 binding required: all equal to the checkout replay pin; bindings `9216cc18…` equal to it; the 39 GLEIF records the pin lists as set aside (`unsupported_identity_kind`) are open reviews here. Recovery: bundle 1-1's merge worker killed while holding the item (exit -9, nothing written), the item verified on attempt 2. Second pass (every bundle's verified records merged again under new batch ids and consumers, and the GLEIF reading prepared and merged again): nothing changed. Time 40.4 min after the bundle install (about 2.5 min, not counted): manifests 0.6, SEC 18.5, GLEIF 18.8, second pass 3.1 (the report's `timings_minutes` are cumulative). Not a doubling of the 2026-10-10 9 min: that was one chunk of about 960 Companies prepared in one process with no merge and no verifier (about 63 min for the 7 chunks); 18.5 min is all 7 chunks prepared, verified in separate processes and merged, about 2.6 min a chunk. Process time of the SEC pass (two bundles at a time): source.read 670 s work and 623 s verify, source.combine 264 s and 263 s, mdm.prepare 87 s and 86 s, mdm.merge 160 s and 81 s, manifests 132 s, status and finalize 129 s; by subtraction from the in-process timings the census read is about 26 s a bundle to work and as much to verify (it parses all 43,245 entries, then keeps the bundle's names) Report `.scratch/mastering-to-done/proof/report-20261012.json`
     - [x] Part 5 review, three axes, 2026-10-12 17:46 ET: GoF: leave it (the 1..2 bound now one constant; `work`/`verify` loops and `_slices` vs Name Frequency's slicing noted, not recommended). Standards: claiming one unit at a time (a) retried a failing unit up to the limit within one call and (b) stopped when the first unit in order was held by another attempt, since the claim query looked at `limit` units (now: a unit that comes back ends the call; claims and verifications look over up to 1,000 units and stop at the limit; tests: a claim of one passes a held unit, a returning unit ends the call); stale docs and comments on 2 documents a reading and 256 key readings fixed (`company_prepare.py`, `census.yaml` comment, digest unchanged `6b8616f7…`, READING.md, data-onboarding SKILL.md, COMBINING.md, local-operations.md, `source_readings.py`); the `source_read.MAX_ARTIFACTS` re-export removed. Spec: the harness now asserts every count against the replay pin (Companies, staged, bound, decisions; open reviews equal the pin's plus its 39 set aside) and the bindings strictly (the saved report passes them); the second pass re-merges only (re-preparing gives the same batch ids, which MDM skips), and recovery is shown on the merge step only; the 512-document bound applies to every source contract (each declares its own `max_artifacts`; the 13F contract keeps 2); a document over 8 MiB reads alone, bounded by its contract's 32 MiB `read.limits` and the workers' output budgets (fails closed)
     - [x] Part 5 PR: #930, CI green on `04989fd6` (six checks; the first run's Engine failure was a worker that, after a unit failed, claimed it again and exited holding it; workers now stop at their first failure, test); operator: "merge"; merged as `692e0215` 2026-10-12 19:41 ET
-    - [ ] Only changed data (operator, 2026-10-12 17:58 ET: "Bookkeeping is to start where we left off, change journal is process only changed data are we true to this"). Measured on 20 Companies, the same records merged a second time under a new batch and consumer (installed proof harness, `writes`): no new source readings (20), decisions (20), Company versions (20) or match proposals (1); but a new batch row (1 → 2), three outbox rows (3 → 6, so downstream publishers would deliver a batch with no change), a checkpoint (new consumer, expected) and a change journal event (7 → 8, the run's own). Within a run Bookkeeping resumes where it stopped (the killed merge's unit finished on attempt 2; verified units are never redone; a resume re-checks completed outputs' hashes, no work). Across runs it does not: a new run reads every document its manifest names, unchanged ones included. To do: (a) an MDM batch that changes nothing writes no outbox rows; (b) a bundle names only documents whose receipt differs from the last merged occurrence of that filer Part (a) done (#931); part (b) deferred (below).
+    - [x] Only changed data (operator, 2026-10-12 17:58 ET: "Bookkeeping is to start where we left off, change journal is process only changed data are we true to this"). Measured on 20 Companies, the same records merged a second time under a new batch and consumer (installed proof harness, `writes`): no new source readings (20), decisions (20), Company versions (20) or match proposals (1); but a new batch row (1 → 2), three outbox rows (3 → 6, so downstream publishers would deliver a batch with no change), a checkpoint (new consumer, expected) and a change journal event (7 → 8, the run's own). Within a run Bookkeeping resumes where it stopped (the killed merge's unit finished on attempt 2; verified units are never redone; a resume re-checks completed outputs' hashes, no work). Across runs it does not: a new run reads every document its manifest names, unchanged ones included. To do: (a) an MDM batch that changes nothing writes no outbox rows; (b) a bundle names only documents whose receipt differs from the last merged occurrence of that filer Part (a) done (#931); part (b) deferred (below). Done for one run's batches 2026-10-13 08:50 ET: (a) #931; (b), a new run naming only changed documents, moved to ticket 22 (the Change Journal beside Bookkeeping: file-level changed-or-identical for every step; operator, 2026-10-13)
     - [x] (a) An MDM batch that changes nothing queues nothing, 2026-10-12 20:33 ET: migration 017. `write_batch` notes whether the batch changed anything (a reading it inserted, a set-aside record not yet stored, an entity, a decision, a Company version or alias, a current record whose body differs); the batch row, its run and the consumer's checkpoint are written either way; outbox rows only for a changed batch; the result says `changed` (a repeat save: whether the batch has outbox rows, exact from 017 on). `record_company_version` now returns whether it wrote, so it alone decides when a Company is the same (GoF review). GoF consult: replace the function whole, as each migration does; only SQL knows what is stored. Operator: "add an index no table scan allowed": every lookup the save adds reads an index: `set_aside_record_pkey`, `current_record_pkey`, `outbox_pkey` (plan test, sequential scans off, 300 Companies and 300 reviews) and the Company lookups inside `record_company_version` (`company_current_entity`; the save test refuses a full read of `company` and `company_alias`). Installed 20-document proof, second pass: outbox 3 → 3, asserted (`.scratch/mastering-to-done/proof/report-20261012-unchanged-batch.json`); the one new change journal event is Bookkeeping's record that the merge unit was verified, not MDM data. Review (three axes): new deferred records were not counted (fixed, test); the Company equality rule was copied (fixed); a test rewinding a store to before 015 restores 014's `write_batch` and re-applies 015 and 017. MDM suites by name 904 + 8 passed. PR #931, CI green (six checks), merged 2026-10-12 20:50 ET as `15f0deb5`
     - [ ] ~~(b) A bundle names only documents whose receipt differs from the last merged occurrence of that filer~~ deferred: operator, 2026-10-12 21:37 ET: "mark them deferred and continue the profiling plan"; speed-ups only, results are already correct; no ticket open for them
     - [ ] ~~Follow-up: read the Name Census once per pass, not once per bundle (about 26 s work and 26 s verify a bundle, about 12 min of process time over 14 bundles; measure one census read alone first)~~ deferred: operator, 2026-10-12 21:37 ET: "mark them deferred and continue the profiling plan"; speed-ups only, results are already correct; no ticket open for them