Whole-source census: installed Company and GLEIF proof (6,414 Companies, 3,052 GLEIF bindings)

.planning/workstreams/whole-source-census/TICKET.md

@@ -289,3 +289,42 @@ CI on 54b8bcd0 passed; these reduction changes require their own full CI gate.
 
 - [x] Make PR886 ready at operator request and save self-contained Grok handoff, upstream/CI receipts and broader original-goal backlog; `gh pr view` verifies `isDraft=false`, receipt JSON and handoff checklist inspected. The original-source comparison is recorded on the 16:40 ET bullets above. Deferred acceptance and retirement remain incomplete. 2026-10-09 13:42 ET
 - [x] Write the handoff for Claude or Codex. `.planning/workstreams/whole-source-census/HANDOFF-20261010.md`, 2026-10-10 06:42 ET. The installed proof stays open.
+
+## Operator handoff to Claude — 2026-10-10
+
+Operator, 2026-10-10 07:08 ET: "merge and take over the goals from grok handover including the checklist". Claude owns this goal from here, on `claude/whole-source-census` (worktree `edgartools-platform-claude-census`). Source: `HANDOFF-20261010.md` (Grok, 06:42 ET). Its guards stay: no further full `source.read` execute; the checkout replay is not the installed proof; no live SEC request; `sec_client.py` stays until a live-SEC proof is allowed; `READING.md`/`COMBINING.md` edited only with the guard; new prose says Name Frequency, code names stay.
+
+- [x] Take over: handoff read; evidence present (2026-10-10 07:08 ET): saved reading, separate verify, reduction, checkout counts (`/private/tmp/codex-worker-replay-proof/`, `/private/tmp/counts-pin-20261009.json`), engine build `/private/tmp/codex-census-eof-native-site-v3`, seven bundles (`proving/cm27/bundles`, 6,726 records)
+- [x] Installed empty PostgreSQL 16 proof: 6,414 Companies, 3,052 CIK+LEI bindings, an unchanged second pass, and recovery; a separate worker and verifier outside the checkout (driver: `installed_proof.py`). Passed 2026-10-10 10:36 ET in 33 min 46 s; report `evidence/installed-company-proof-20261010.json` sha256 `fa85e0d2…`
+  - [x] Before the long run: at least 3 GB free for the engine build. 01d worktree removed after Trial A; 4.7 GB free, the engine build used under 1 GB. 2026-10-10 08:09 ET
+  - [x] Before the long run: the installed tool loads GLEIF `level1-json.yaml` (the "object names fields" refusal was removed by #857, `220b8a49`; the older installed build still had it). `stream_policy` on the installed bundle's own rules, 2026-10-10 08:22 ET
+  - [x] SEC half alone (`COMPANY_PROOF_ONLY=sec`): plumbing, recovery and the unchanged second pass. 6,414 Companies, 6,414 SEC records bound (all by `company-cik`), 312 classification reviews deferred, second pass unchanged (bindings `67749419…`). The first try killed a worker that had already finished (exit 0): the merge writes an item in one transaction, so rows appear only on commit. The kill now lands while the item is `running` and the proof fails otherwise (`8882f9d6`). Rerun: killed with SIGKILL holding `sec-first-chunk1` attempt 1, no rows committed, resumed after the lease, same counts; 7 min 13 s. 2026-10-10 08:29 ET
+  - [x] Full proof: SEC and GLEIF, counts equal to the checkout bindings. Bundle installed from this branch by `uv tool install`; every step by `edgar-warehouse workers work` and checked by `workers verify` under a separate login. First pass: 6,414 Companies; SEC 6,414 bound (`company-cik`); GLEIF Level 1 3,716 records, 3,052 bound (2,861 `sec-gleif-name-jurisdiction`, 191 `sec-gleif-name-postal`); open reviews 664 binding required, 312 classification deferred, 39 unsupported identity kind. Bindings `9216cc18…`, equal to the checkout replay. Second pass changed nothing. Recovery: SIGKILL while holding `sec-first-chunk1` attempt 1, no rows committed, finished after the lease. GLEIF `source.read` execute and its verifier: 20 min. 2026-10-10 10:36 ET
+  - [x] Two failed runs, fixed: (1) both GLEIF passes named one consumer, so the second pass's checkpoint 0→1 was refused as stale; each unit now names its own (`mdm_prepare.py`: "a unit's consumer is its own") (`b142560e`), 09:28 ET. (2) The disk guard stopped the rerun at 870 MB free while the second pass re-read the 13 GB archive; the second GLEIF pass now prepares and merges the first pass's verified reading again (`gleif-replay` target, `424e5b09`), 09:59 ET. Leftovers of the killed run (its PostgreSQL, 652 MB) removed
+  - [x] Review (Standards, Spec, GoF), fixes in `2e982a2b` and `34fe52dc`: bindings equal to the checkout replay are asserted; the SEC second pass gives each copied batch its own id and consumer, so it is mastered again, not skipped; the resumed item must be verified on its next attempt; `pg16.server()` yields only once the server answers `SELECT 1`. 92 integration tests pass on pgserver. SEC half on `2e982a2b`, 10:47 ET: killed after the item's MDM rows committed (919 rows) and before it was reported; resumed on attempt 2, verified; second pass re-mastered and unchanged (`evidence/installed-sec-recovery-after-commit-20261010.json`). 2026-10-10 10:47 ET
+  - [x] Full run on the final commit `fdc020f9` (operator, 2026-10-10: "Delete Codex /tmp caches"; `codex-calendar-target` and `codex-streaming-uv-cache` removed, unused by any process). Passed 13:57 ET in 36 min 50 s: 6,414 Companies; 3,052 GLEIF bound; bindings `9216cc18…` asserted equal to the checkout replay; the killed item verified on attempt 2; both sources' second passes mastered again under their own batches and consumers and changed nothing. Report `evidence/installed-company-proof-final-20261010.json` sha256 `04f108b8…`. 2026-10-10 13:57 ET
+- [ ] Replace the active Name Frequency command (`cli.py` still calls `write_name_census`). Only after the installed proof. Blueprint: `crates/source-contract/tests/qualification/whole_source_frequency.py` builds the document from the configured reading and `cascade.assign`. Done test: on the same pinned inputs, the new command writes a file byte-for-byte equal (same sha256) to the old command's
+  - [x] One function assembles the document: `name_census.document()`, called by the old `build()` (GoF consult: extract it so the old and new paths share it; no pattern). 68 census, Company source and matching tests pass. Branch `claude/name-frequency-assembly` `3a3c1210`. 2026-10-10 08:19 ET
+  - [x] GLEIF half from the saved configured reading (`census-complete-stream.yaml`, read back with `source_readings.iter_load`): legal holders and last update, other-name holders, matching-pass candidates merged per LEI by source position, and global address counts; then `document()`. `name_census.fold_reading`; two tests read one fixture through `build()` and through the configured stream, documents equal, names and seven-pass answers alike (`4b6c0ec1`). 2026-10-10 08:31 ET
+  - [x] SEC half from configured readings: `name_frequency.write_name_frequency` reads each capture through `census-landed-company.yaml`, `census-landed-former.yaml` and the business address of `landed-preparation.yaml` with the `source.read` worker and verifier. Matching-pass candidates still go through `company_source.cascade_filer` (old adapters): moving them is part of L4. Fixture: the same bytes as `write_name_census` over two captures; refuses another population's reading, a delta or short publication, a filer in two captures (`a3c7ed5d`). 2026-10-10 08:35 ET
+  - [x] `mdm name-census` takes `--gleif-reading` and `--gleif-reading-sha256` in place of `--gleif-archive` and `--gleif-sha256` (`604775b6`). Full population: 69 captures and the saved 3,428,477-record reading, 43,245 entries in 174 s, sha256 `84522f67…`, byte-for-byte the `census.json` the seven cm27 bundles pin (the old command re-reads the Golden Copy for about an hour). 2026-10-10 08:39 ET
+  - [ ] The seven-pass answers: the pinned census has none (the policy's passes are off); when switched on, compare with `complete-name-frequency.json` from the whole-source run
+- [ ] Delete the old construction code once no caller remains (`gleif_source.py`: `source_publications.py`, `native_consumption.py`, `company_source.py`, `gleif_publication.py`; `adapters.py`: `mdm_merge.py`, `name_census.py`, `cli.py`, `company_source.py`, `gleif_source.py`, `quality.py`, `store.py`)
+- [ ] Issue 20 L4: whole-capture Company equivalence, then delete `company_source.py` (14 original columns match on 1,000 pinned submissions; the current read also carries `last_sync_run_id`, `last_synced_at`)
+- [ ] Issue 20 L6: Golden Copy equivalence of the older GLEIF reader, then delete `gleif_source.py` (52 in-memory fixture tests passed 2026-10-09 20:37 ET)
+  - [x] Found (2026-10-10 10:55 ET): the older reader is reachable only through a manifest's `native_source` (`cli.execute_manifest` → `native_consumption.prepare_native` → `gleif_source.record_evidence`), and nothing in production or tests writes one. Its checks are not all in the configured GLEIF rules: the configured Level 1 reading selects by approved scope and maps through the contract, but the deletion-flag refusal, the relationship checks (one RELATIONSHIP_PERIOD, LEI endpoints, a valid interval, status against the end date) and the reporting-exception category and reason checks live only in `record_evidence` (24 tests, `tests/mdm/test_clean_gleif_source.py`). Out-of-scope records are dropped by the configured reading, where the older reader deferred them. The configured relationship and reporting-exception field readings map no fields (`relationships-fields.yaml`, `reporting-exceptions-fields.yaml`: `fields: {}`), so only `record_evidence` builds the `start`, `end`, `relationship_type`, `valid_from`, `valid_to`, `status` the relationship contract reads: GLEIF relationships have no configured path yet
+  - [ ] Express those checks in the GLEIF MDM contract or quality rules (ruling above), the relationship and reporting-exception readings projecting the fields the contracts read, with each of the 24 behaviours tested on the configured path
+    - [x] A contract's record checks set a record aside first (`source_mapping.record_refusal`, `adapters._refuse` at the top of `normalize` and `mapped_values`; GoF consult: one refusal step, not a side effect of reading fields). Branch `claude/gleif-reader-retirement` `027f6b41`, 5 tests, 252 affected pass. 2026-10-10 13:30 ET
+    - [x] Level 1: LEI shape and check digits, a filled deletion flag (blocking), a category outside GLEIF's six, in the older reader's order, each read through a literal-path column (`00580b18`). Contract digest 9305c467… awaiting the operator's approval; peeled, it is the approved f53c9c42…. 284 GLEIF, census and digest tests pass. 2026-10-10 13:37 ET
+    - [x] Relationships read by their contract alone: generic `find ... one`, `time_format: instant`, `interval` and `status` link rules (GoF: leave the structure, one `_refuse_link`); the record checks take the LEI ends, endpoint types and deletion flag; the record's own time and whole record kept. 14 cases equal to the older reader under the approved contract b7493a71…; digest 78cb4dd5… awaiting approval (`a01d5b5c`). 2026-10-10 14:31 ET
+    - [x] Level 1 per-record fidelity (found 2026-10-10 14:05 ET, one fixture both ways; fixed `4da44047`: a contract may name `effective_at`, a provenance path `.` is the whole record; the Level 1 contract uses both, digest fc104ef9… approved by the operator 2026-10-10 14:13 ET: "Approved", on the 20,000-record evidence with the whole-file comparison running; peeled f53c9c42…): the configured path stores no effective time (the older reader stamps each record with its `Registration.LastUpdateDate`) and no native record in provenance (`native_record: None`; the older reader keeps the raw GLEIF record). Bindings are unaffected (the proof matched `9216cc18…`), but assertion ids differ. Needs the contract to name the record's effective time and keep the whole record as its native evidence
+    - [x] Reporting exceptions read by their contract alone: record checks for the LEI, deletion flag and category; generic `every_in` (every reason in GLEIF's list) and `retained_as` (`reported_parent_exception`). 13 cases equal to the older reader under the approved contract 77bb3a8d…; digest 8ec18c6d… awaiting approval (`8fa2f8f2`). 2026-10-10 14:40 ET
+    - [ ] Whole-file equivalence for relationships (487,721 records) and reporting exceptions (6,351,397), as for Level 1; then ask approval of 78cb4dd5… and 8ec18c6d…
+  - [x] Decision (operator, 2026-10-10 13:20 ET: "MDM contract (Recommended)"): where a GLEIF record-level refusal lives on the configured path. The engine's record checks (`lei`, `count`, `absent`, `before`, `required`, `in_set`, `in_lookup`, each with a `reason`) can express every `record_evidence` check, but a record a reading check sets aside stays in the reading (`mdm_prepare.py`) and never reaches MDM, so no steward review is opened and a blocking reason (a filled deletion flag) no longer blocks the run. Options: (a) checks in the readings, with set-aside records reported for review by `mdm.prepare`; (b) checks in the MDM contract or the GLEIF quality rules, so they defer in the Stage as today. Ruled (b): the checks go in the GLEIF MDM contract or its quality rules; the readings only select and project
+  - [ ] Golden Copy equivalence (`crates/source-contract/tests/qualification/gleif_level1_equivalence.py`, every record as if in scope; first 20,000 all equal, 2026-10-10 14:20 ET; whole file running in 3 shards, about 2 hours): each in-scope record reaches the same outcome (assertion digest, or deferral and its reason) both ways, all three members
+  - [ ] Delete `record_evidence`, `native_consumption.py` and the `native_source` branch; move `dataset_contract`, `validate_metadata`, `FORMATS`, `release_sequence` to where they belong; delete `gleif_source.py`
+- [ ] Issue 20 L8: a configured `provider.capture` worker, then delete `sec_client.py` (needs the operator's permission for a live SEC proof)
+- [ ] Issue 20 20c: Company through the engine only (the four Company-named steps renamed to registered profiles, or removed)
+- [ ] Issue 20 20d: Person through the engine only (blocked by 20c)
+- [ ] Skill audit remainder: installed separate-process qualification, full CI, independent review, a reviewable PR for any skill edit (`READING.md` and `COMBINING.md` still use the old wording)
+

.planning/workstreams/whole-source-census/evidence/installed-company-proof-20261010.json

@@ -0,0 +1,46 @@
+{
+ "after_first_pass": {
+  "companies_open": 6414,
+  "decisions_by_rule": {
+   "company-cik": 6414,
+   "sec-gleif-name-jurisdiction": 2861,
+   "sec-gleif-name-postal": 191
+  },
+  "open_reviews": {
+   "binding_required": 664,
+   "classification_deferred": 312,
+   "unsupported_identity_kind": 39
+  },
+  "stage_bound": {
+   "gleif.level1.v1": 3052,
+   "sec.submissions.company.v1": 6414
+  },
+  "stage_records": {
+   "gleif.level1.v1:company": 3716,
+   "sec.submissions.company.v1:company": 6414
+  }
+ },
+ "bindings": "9216cc18292f1c683f4767dfb86ae89c14a72f7df050df08e3800ce1e305b85b",
+ "bindings_equal_checkout_replay": true,
+ "gleif_scope_leis": 3755,
+ "goal": "whole-source census: installed empty-store Company proof",
+ "minutes": 30.8,
+ "only": null,
+ "policy": "34f548776040bb44451f0ac17e496ae30e1ce20568ad199bfa03616a6645b2e2",
+ "recovery": {
+  "item_at_kill": {
+   "attempt": "b9985863-9240-4cf4-8a6c-463bff57b4da",
+   "attempts": 1,
+   "checks": {},
+   "error": null,
+   "ordinal": 0,
+   "receipt": null,
+   "state": "running",
+   "step": "merge",
+   "unit_key": "sec-first-chunk1"
+  },
+  "killed_worker_exit": -9,
+  "rows_at_kill": {}
+ },
+ "second_pass_changed_nothing": true
+}

.planning/workstreams/whole-source-census/evidence/installed-company-proof-20261010.log

@@ -0,0 +1,12 @@
+{"at": "10:05:03", "stage": "provisioned", "policy": "34f548776040bb44451f0ac17e496ae30e1ce20568ad199bfa03616a6645b2e2"}
+{"at": "10:05:17", "stage": "worker killed", "killed_worker_exit": -9, "item_at_kill": {"attempt": "b9985863-9240-4cf4-8a6c-463bff57b4da", "attempts": 1, "checks": {}, "error": null, "ordinal": 0, "receipt": null, "state": "running", "step": "merge", "unit_key": "sec-first-chunk1"}, "rows_at_kill": {}}
+{"at": "10:11:27", "run": "5a9445b5-d6c9-4492-b6ca-d2b9cceb2851", "step": "mdm.merge", "verified": true, "minutes": 6.4}
+{"at": "10:31:33", "run": "b9a79d46-6b83-42b7-a1e6-c1e690f97c23", "step": "source.read", "verified": true, "minutes": 26.5}
+{"at": "10:31:38", "run": "b9a79d46-6b83-42b7-a1e6-c1e690f97c23", "step": "mdm.prepare", "verified": true, "minutes": 26.6}
+{"at": "10:33:38", "run": "b9a79d46-6b83-42b7-a1e6-c1e690f97c23", "step": "mdm.merge", "verified": true, "minutes": 28.6}
+{"at": "10:33:39", "stage": "first pass", "companies_open": 6414, "stage_records": {"gleif.level1.v1:company": 3716, "sec.submissions.company.v1:company": 6414}, "stage_bound": {"gleif.level1.v1": 3052, "sec.submissions.company.v1": 6414}, "open_reviews": {"binding_required": 664, "classification_deferred": 312, "unsupported_identity_kind": 39}, "decisions_by_rule": {"company-cik": 6414, "sec-gleif-name-postal": 191, "sec-gleif-name-jurisdiction": 2861}}
+{"at": "10:35:00", "run": "614b7219-da85-4869-9c07-38bdfb264abf", "step": "mdm.merge", "verified": true, "minutes": 30.0}
+{"at": "10:35:07", "run": "171d5f4e-5063-4efb-b816-bcaaa55b988c", "step": "mdm.prepare", "verified": true, "minutes": 30.1}
+{"at": "10:35:50", "run": "171d5f4e-5063-4efb-b816-bcaaa55b988c", "step": "mdm.merge", "verified": true, "minutes": 30.8}
+{"at": "10:35:52", "stage": "report", "goal": "whole-source census: installed empty-store Company proof", "only": null, "policy": "34f548776040bb44451f0ac17e496ae30e1ce20568ad199bfa03616a6645b2e2", "gleif_scope_leis": 3755, "after_first_pass": {"companies_open": 6414, "stage_records": {"gleif.level1.v1:company": 3716, "sec.submissions.company.v1:company": 6414}, "stage_bound": {"gleif.level1.v1": 3052, "sec.submissions.company.v1": 6414}, "open_reviews": {"binding_required": 664, "classification_deferred": 312, "unsupported_identity_kind": 39}, "decisions_by_rule": {"company-cik": 6414, "sec-gleif-name-postal": 191, "sec-gleif-name-jurisdiction": 2861}}, "bindings": "9216cc18292f1c683f4767dfb86ae89c14a72f7df050df08e3800ce1e305b85b", "second_pass_changed_nothing": true, "recovery": {"killed_worker_exit": -9, "item_at_kill": {"attempt": "b9985863-9240-4cf4-8a6c-463bff57b4da", "attempts": 1, "checks": {}, "error": null, "ordinal": 0, "receipt": null, "state": "running", "step": "merge", "unit_key": "sec-first-chunk1"}, "rows_at_kill": {}}, "bindings_equal_checkout_replay": true, "minutes": 30.8}
+1 passed in 2026.17s (0:33:46)

.planning/workstreams/whole-source-census/evidence/installed-company-proof-final-20261010.json

@@ -0,0 +1,47 @@
+{
+ "after_first_pass": {
+  "companies_open": 6414,
+  "decisions_by_rule": {
+   "company-cik": 6414,
+   "sec-gleif-name-jurisdiction": 2861,
+   "sec-gleif-name-postal": 191
+  },
+  "open_reviews": {
+   "binding_required": 664,
+   "classification_deferred": 312,
+   "unsupported_identity_kind": 39
+  },
+  "stage_bound": {
+   "gleif.level1.v1": 3052,
+   "sec.submissions.company.v1": 6414
+  },
+  "stage_records": {
+   "gleif.level1.v1:company": 3716,
+   "sec.submissions.company.v1:company": 6414
+  }
+ },
+ "bindings": "9216cc18292f1c683f4767dfb86ae89c14a72f7df050df08e3800ce1e305b85b",
+ "bindings_equal_checkout_replay": true,
+ "gleif_scope_leis": 3755,
+ "goal": "whole-source census: installed empty-store Company proof",
+ "minutes": 33.9,
+ "only": null,
+ "policy": "34f548776040bb44451f0ac17e496ae30e1ce20568ad199bfa03616a6645b2e2",
+ "recovery": {
+  "item_at_kill": {
+   "attempt": "7049568c-80ae-43cc-997c-320e3f76b433",
+   "attempts": 1,
+   "checks": {},
+   "error": null,
+   "ordinal": 0,
+   "receipt": null,
+   "state": "running",
+   "step": "merge",
+   "unit_key": "sec-first-chunk1"
+  },
+  "killed_worker_exit": -9,
+  "resumed_attempts": 2,
+  "rows_at_kill": {}
+ },
+ "second_pass_changed_nothing": true
+}