Company and Name Frequency read the raw capture (Issue 20 L4)

[paulananth]

Company and Name Frequency read the raw capture; company_source.py retired (Issue 20 L4, draft)

Ticket (.planning/workstreams/whole-source-census/TICKET.md, Issue 20 L4).

What

• Name Frequency counts the SEC population from the receipt-bound raw capture through rules/sources/sec.submissions.company/census-raw.yaml (each main document's cik, name, formerNames[].name), not landed parquet. A switched-on matching cascade is refused until a configured cascade reading exists (today's policy has it off).
• Company preparation (prepare_company_bundle) reads the raw capture through the registered workers in-process: main documents by receipt key (census-main.yaml, 1–2 CIKs a unit), one census read narrowed to the bundle's name keys, one exchange-catalog read, combine-census.yaml per unit. Each record names its bronze document (_origin.bronze). verify_company_bundle rebuilds the records from every combine worker.
• Batch cut: the bundle is cut as mdm.prepare cuts since MDM batches never fail on size at the end of a run #906 (at most limits.BATCH_RECORDS records and, past the first, limits.BATCH_RECORD_BYTES), through one shared limits.batches. It writes records.{n}.jsonl, one consumer, batch ids company:{bundle}:{n}, checkpoints n/n+1, one publication key.
• Retired: edgar_warehouse/mdm/clean/company_source.py, the five landed contracts (landed-preparation.yaml, landed-address.yaml, census-landed-company.yaml, census-landed-former.yaml, combine-landed.yaml), the landed-only qualification scripts and their tests. The governance constants (SOURCE_CODE, CONTRACT, FIELDS, POLICY, …) live in mdm/clean/sec_company.py.
• CLI: mdm name-census and mdm prepare-clean-company take --capture, --documents, --population.
• Tests: tests/mdm/test_clean_company_preparation.py (15, on a raw capture fixture from tests/support/sec_capture.py); the installed-bundle caller test prepares from a raw capture and compares with the checkout.
• Docs and skills: docs/specs/clean-mdm/local-operations.md, docs/specs/mdm/policy-language.md, the data-onboarding skill (FIELDS's home).

Evidence

Check	Result
Company main reading, whole capture, vs the retained loader	76,230 documents, 76,230 Company rows, 8,065,661 filing rows, all equal
Name Frequency from raw vs cm27 census.json	43,338 filers, 43,245 entries, 0 differences
cm27 chunks 1–7 prepared from raw vs the landed bundles	6,726 rows equal, except _origin, last_synced_at and the census digest (by design)
Chunk 1 mastered, raw vs landed	919 Companies, 919 bindings, 47 reviews, golden fields equal
Affected tests	MDM, unit, engine and architecture via testmon: 2,575 passed; locally only, 15 tests need jq and one starts Postgres with docker (CI has both)

Approved by the operator: Company main reading 18903901…, census-main reading f2ee7a31…, Dataset Contract 472295c9… ("all approved", 2026-10-10 19:25 ET; the contract's approval holds if the full proof matches), census-raw reading 96688655… ("approved, yes", 2026-10-11).

Full proof (passed)

The installed empty-store proof on the 7 raw-capture bundles with the batch cut, 12:13–12:42 ET 2026-10-11 (26.5 min): 6,414 Companies; GLEIF 3,716 records, 3,052 bound (2,861 by name and jurisdiction, 191 by name and postal code); bindings 9216cc18…, equal to the checkout replay; a killed worker's item recovered on attempt 2; the second pass changed nothing. Report .planning/workstreams/whole-source-census/evidence/installed-company-proof-raw-20261011.json.

Three-axis review (Standards, Spec, GoF) done; fixes in 7c9cb363 change no bundle byte (six real Companies prepared before and after give the same bundle).

Deferred (on the ticket): retained worker inputs (about 400 MB per ~960 Companies) before the first whole-capture run (20c); the landed-era provenance names in the approved contract go with the next SEC mapping.

🤖 Generated with Claude Code