Skip to content

Name Frequency from configured readings; retire the Python builder - #901

Merged
paulananth merged 6 commits into
mainfrom
claude/name-frequency-assembly
Oct 10, 2026
Merged

Name Frequency from configured readings; retire the Python builder#901
paulananth merged 6 commits into
mainfrom
claude/name-frequency-assembly

Conversation

@paulananth

@paulananth paulananth commented Oct 10, 2026 •
edited
Loading

Copy link
Copy Markdown
Owner

What changes

mdm name-census no longer re-reads the GLEIF Golden Copy in Python. It now counts from configured readings:

  • each SEC capture through census-landed-company.yaml, census-landed-former.yaml and the business address of landed-preparation.yaml, by the source.read worker and its verifier;
  • the Golden Copy as one verified source.read reading of census-complete-stream.yaml, passed as --gleif-reading / --gleif-reading-sha256 (in place of --gleif-archive / --gleif-sha256). The command checks the reading was made by that recipe, that its wanted lookup equals the names these captures hold, that its record count equals the publication's, and the publication metadata itself (full file, never a delta).

name_census.census() folds the reading and writes the document; production, the test helper and the installed-bundle test all call it. write_name_census, name_census.build, cascade_entity and _gleif_other_keys are deleted. tests/support/retired_name_census.py stays as the independent oracle every census test compares against.

Evidence

  • Full population, 2026-10-10 09:04 ET: 69 SEC captures and the saved 3,428,477-record Golden Copy reading give 43,245 entries in 135 s, sha256 84522f67…, byte for byte the census.json the seven cm27 bundles pin. The old command needed about an hour.
  • Fixture: two captures (other names, a later duplicate LEI, a branch) equal to the retired oracle, including an independently computed SEC population block; refusals tested for another population's reading, another recipe, a delta or short publication, bad metadata, a filer in two captures, and a cascade over several captures.
  • 97 affected tests pass; the installed-bundle census code, run alone, equals the oracle.

Kept as before, on purpose

  • A cascade over several captures is refused, as before (ticket 20 extends the cascade's addresses).
  • Matching-pass candidates still go through company_source.cascade_filer (the old adapters): moving them belongs to issue 20 L4.

Review

Three axes (Standards, Spec, GoF); fixes in c0ff1fa9.

🤖 Generated with Claude Code

https://claude.ai/code/session_01FFKdv4nqKXKM9d5Cf2hEFk

@paulananth
paulananth changed the base branch from claude/whole-source-census to main October 10, 2026 19:01
Aneena Ananth and others added 6 commits October 10, 2026 15:01
Aneena Ananth @claude Aneena Ananth
Name Frequency: one function assembles the document
2dd28d9 
`name_census.document()` builds the Name Frequency document from what was
counted; `build()` calls it after its Golden Copy scan. The configured
assembly that replaces `write_name_census` will call the same function, so
both write the same bytes by construction (GoF consult, 2026-10-10).

68 census, Company source and matching tests pass.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FFKdv4nqKXKM9d5Cf2hEFk
Aneena Ananth @claude Aneena Ananth
Name Frequency: fold the configured Golden Copy reading
6a2356f 
`name_census.fold_reading()` counts what the configured complete reading
(`census-complete-stream.yaml`) holds: legal holders with their last
update, other-name holders, matching-pass candidates by LEI (a later
record replaces, rows of one record add names) and global address counts.
`document()` then writes the file. Two tests read one fixture both ways,
through `build()` and through the configured stream and the fold; the
documents are equal, names and seven-pass answers alike.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FFKdv4nqKXKM9d5Cf2hEFk
Aneena Ananth @claude Aneena Ananth
Name Frequency from configured readings: write_name_frequency
4b2337a 
Each SEC capture is read by the `source.read` worker and its verifier
through `census-landed-company.yaml`, `census-landed-former.yaml` and the
business address of `landed-preparation.yaml`. The Golden Copy is one
verified configured reading (`census-complete-stream.yaml`); its archive
and record count must equal the full publication's metadata, and its
`wanted` lookup must equal the name keys counted here. `fold_reading` and
`document` write the file. No Golden Copy re-read in Python.

On a two-capture fixture (other names, a later duplicate LEI, a branch)
it writes the bytes `write_name_census` writes, and refuses another
population's reading, a delta or short publication, and a filer in two
captures. Matching-pass candidates still go through
`company_source.cascade_filer` until issue 20 L4.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FFKdv4nqKXKM9d5Cf2hEFk
Aneena Ananth @claude Aneena Ananth
mdm name-census counts from configured readings
37793bc 
The command now calls `write_name_frequency`: `--gleif-reading` and
`--gleif-reading-sha256` name the verified `source.read` output of
`census-complete-stream.yaml`, in place of `--gleif-archive` and
`--gleif-sha256`. `write_name_census` has no caller left.

Full population, 2026-10-10 08:39 ET: 69 SEC captures and the saved
3,428,477-record Golden Copy reading give 43,245 entries in 174 s,
sha256 84522f67…, byte-for-byte the census the seven cm27 bundles pin.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FFKdv4nqKXKM9d5Cf2hEFk
Aneena Ananth @claude Aneena Ananth
Retire the Python Name Frequency builder
3ac5d4d 
`mdm name-census` counts from configured readings, so nothing calls the
old construction: deleted `company_source.write_name_census`,
`name_census.build`, `cascade_entity`, `_gleif_other_keys` and the
per-record GLEIF recipes it loaded (the recipe files stay: the complete
stream is generated from them).

Tests now build every census through the configured complete reading,
`fold_reading` and `document`, still compared with the independent oracle
`tests/support/retired_name_census.py` (its `cascade_entity` is the bundle
test's oracle now). The installed-bundle test folds the installed
complete stream; its census code, run alone, equals the oracle. 180
affected tests pass. The two qualification scripts read the recipes
directly and build through the configured path.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FFKdv4nqKXKM9d5Cf2hEFk
Aneena Ananth @claude Aneena Ananth
Name Frequency: review fixes (Standards, Spec, GoF)
5ae2a20 
- Spec: the GLEIF reading must be made by `census-complete-stream.yaml`
  (its contract is compared with the committed recipe); the publication
  metadata goes through `validate_metadata` as before (CDF version, format,
  count, content time); a cascade over several captures is refused as
  before (ticket 20); a repeated CIK is refused only across captures, as
  before. The fixture test checks the `sec` block against the landed
  files, not against the output.
- Standards: bounded reads of the metadata and an existing output; two
  blank lines between functions.
- GoF: `name_census.census()` owns "is the cascade on", the address
  counts, the fold and the document; production, the test helper and the
  installed-bundle test call it, so the three cannot drift.

97 affected tests pass; the installed-bundle census code, run alone,
equals the retired oracle.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FFKdv4nqKXKM9d5Cf2hEFk
@paulananth
paulananth force-pushed the claude/name-frequency-assembly branch from c0ff1fa to 5ae2a20 Compare October 10, 2026 19:01
@paulananth
paulananth merged commit 8fec6bf into main Oct 10, 2026
6 checks passed
paulananth pushed a commit that referenced this pull request Oct 10, 2026
Aneena Ananth @claude Aneena Ananth
Ticket: Name Frequency caller replaced, #901 merged
74ff9b7 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FFKdv4nqKXKM9d5Cf2hEFk
paulananth pushed a commit that referenced this pull request Oct 10, 2026
Aneena Ananth @claude Aneena Ananth
Ticket: Name Frequency caller replaced, #901 merged
3f3bdd0 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FFKdv4nqKXKM9d5Cf2hEFk
paulananth added a commit that referenced this pull request Oct 10, 2026
@paulananth Aneena Ananth @claude
GLEIF read by its contract alone; the older reader retired (Issue 20 …
44782b8 
…L6) (#904)

* A contract's record checks set a record aside, first

L6, step 1 (operator ruling 2026-10-10: GLEIF refusals live in the MDM
contract; readings only select and project). `source_mapping.
record_refusal` runs only the declared record checks of a contract's
reading and returns the reason one sets the record aside with.
`adapters._refuse` raises it as an `UnsupportedRecord` at the top of
`normalize` and `mapped_values`, before the kind, key or fields are read,
so the record defers in the Stage with that reason (blocking unless the
contract lists it as non-blocking), as `gleif_source.record_evidence`
does today. A reading without checks refuses nothing; no contract
declares any yet. GoF consult: one refusal step at the top, not a side
effect of reading `fields`.

5 new tests; 252 affected tests pass.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FFKdv4nqKXKM9d5Cf2hEFk

* GLEIF Level 1 refusals in the contract (L6, step 2)

The Level 1 contract's reading now declares, as record checks, what
`gleif_source.record_evidence` checked before mapping, in its order:
the LEI's shape and check digits (`invalid_lei`, `invalid_lei_checksum`),
a filled deletion flag (`gleif_deletion_flag`, blocking), and a category
outside GLEIF's six (`invalid_identity_kind`). Each reads its value
through a literal-path column, as the fields do, so a malformed parent
is refused like the older reader refuses it, never a reading error.
`project_record` leaves the checks to the refusal step.

The contract digest moves to 9305c467..., awaiting the operator's
approval; peeling the checks gives back the approved f53c9c42...
(`test_rules_config_digests.py`). 284 GLEIF, census and digest tests
pass, among them the older reader's own.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FFKdv4nqKXKM9d5Cf2hEFk

* GLEIF Level 1: the record's own time and the whole record, in the contract

Compared one record both ways, the configured path stored no effective
time and no native record, where the older reader stamped each record
with its `Registration.LastUpdateDate` and kept the raw record. A
contract may now name `effective_at` (a path each record states its own
time at; otherwise the publication's), and a provenance path `.` is the
whole record. The Level 1 contract uses both. On that fixture the older
reader under the approved contract (f53c9c42...) and `normalize` under
this one now write the same assertion id.

Contract digest fc104ef9..., awaiting the operator's approval; peeling
the refusals, the time and the native record gives back f53c9c42....
160 GLEIF, refusal and digest tests pass.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FFKdv4nqKXKM9d5Cf2hEFk

* L6: Level 1 Golden Copy equivalence script (test-only)

Every record read by the older reader under the approved contract and by
normalize under the new one; same assertion id or same deferral reason.
20,000 records: all equal (18,593 assertions, 1,404 unsupported identity
kind, 3 invalid LEI checksum). Shards for the whole file.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FFKdv4nqKXKM9d5Cf2hEFk

* L6: the operator approves the Level 1 contract digest fc104ef9...

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FFKdv4nqKXKM9d5Cf2hEFk

* GLEIF relationships read by their contract alone (L6, step 3)

The relationships contract read a row `gleif_source.record_evidence`
built; it now reads the GLEIF record as published. Generic additions to a
relationship mapping (GoF consult: leave the structure, one small
`_refuse_link`): `find ... one: <reason>` (anything but exactly one
match sets the record aside), `time_format: instant`, an `interval` rule
(a start, before any end) and a `status` rule (allowed values, and those
that need an end), each with its reason. The contract's record checks
take the LEI ends, both endpoint types and the deletion flag; the record
keeps its own time and the whole record.

14 cases, compared with the older reader under the approved contract
(b7493a71...): all equal; kept as `test_clean_gleif_relationship_contract.py`
with their outcomes. Contract digest 78cb4dd5..., awaiting the operator's
approval; peeled, b7493a71.... 838 MDM and GLEIF tests pass.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FFKdv4nqKXKM9d5Cf2hEFk

* GLEIF reporting exceptions read by their contract alone (L6, step 4)

The contract's record checks take the LEI, the deletion flag and the two
consolidation categories; two generic adapter rules take the rest:
`every_in` (every value of a list in a set, with a reason for none and
one for a stranger; the engine's checks test one value) and `retained_as`
(every remaining record waits with that reason: evidence, never an
identity or a parent link).

13 cases compared with the older reader under the approved contract
(77bb3a8d...): all equal; kept as `test_clean_gleif_exception_contract.py`.
Contract digest 8ec18c6d..., awaiting the operator's approval; peeled,
77bb3a8d.... 184 GLEIF and digest tests pass.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FFKdv4nqKXKM9d5Cf2hEFk

* Ticket: Name Frequency caller replaced, #901 merged

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FFKdv4nqKXKM9d5Cf2hEFk

* L6: the Golden Copy equivalence covers all three GLEIF members (test-only)

The approved contract is peeled by the digest test's own _without_refusals;
every LEI is in scope; archives pinned by sha256.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FFKdv4nqKXKM9d5Cf2hEFk

* L6: GLEIF behaviours tested through the contract; fixtures off the older reader

The older reader's tests with no configured counterpart now have one
(Level 1 fields, probable kinds, corrections, agent address, check digit,
successor links, null deletion fields). fresh_mastering and the bundle
test read GLEIF through normalize; the cohort's assertion ids equal main's.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FFKdv4nqKXKM9d5Cf2hEFk

* L6: whole Golden Copy equal both ways, all three GLEIF members

Level 1 3,428,477, relationships 487,721, reporting exceptions 6,351,397
records: every outcome equal between the older reader and the contract.
Evidence under the ticket.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FFKdv4nqKXKM9d5Cf2hEFk

* L6: a non-object GLEIF Extension fails closed; the rulings recorded

The operator, 2026-10-10 17:55 ET: "approve both digests, go with your
recommendations". Relationships 78cb4dd5... and reporting exceptions
8ec18c6d... approved on the whole-file evidence. Out-of-scope records:
dropped by the reading (policy-language spec updated). A non-object
Extension is read as the deletion flag itself, as the older reader did:
new digests 67fe1aaa..., b9b1fd11..., 643e8335..., each peeling to the
approved one, await approval. MAPPING.xlsx regenerated from the rules.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FFKdv4nqKXKM9d5Cf2hEFk

* L6: the operator approved the Extension-check digests

Level 1 67fe1aaa…, relationships b9b1fd11… and reporting exceptions 643e8335…,
approved 2026-10-10 18:06 ET ("approved").

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FFKdv4nqKXKM9d5Cf2hEFk

* L6: retire the old GLEIF reader and the native mastering path

GLEIF records are read by their contracts alone (whole Golden Copy equal,
all three members). Deleted: gleif_source.py (record_evidence and the
release checks), native_consumption.py, the native_source branch in the
mastering command (now refused before any work), the native plumbing in
run.py, and source_publications.py, whose only caller was that path.
VERSION (unchanged text), FORMATS, validate_metadata and dataset_contract
move into gleif_publication.py. The equivalence script last ran at 68cc124.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FFKdv4nqKXKM9d5Cf2hEFk

* L6 review: relationships Extension test, plain names, checklists current

Spec review: the non-object Extension check is now tested on the relationships
contract too; the ticket and Issue 20 record the deletion. Standards review: rules
comments say "retired in Issue 20 L6"; the error reads "Unknown GLEIF member".
GoF review: leave it; its follow-up is recorded on the ticket.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FFKdv4nqKXKM9d5Cf2hEFk

---------

Co-authored-by: Aneena Ananth <aneenaananth@MacBook-Air.local>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
paulananth added a commit that referenced this pull request Oct 11, 2026
@paulananth Aneena Ananth @claude
Skill audit: skills state today's Company and GLEIF status (#914)
c482c64 
* Skill audit: skills state today's Company and GLEIF status

data-onboarding, refining-rules, READING.md and COMBINING.md no longer call
the installed proof, whole-capture equivalence and the older readers' deletion
open (#901, #904, #905, #907); they name what is open (in-process Company
preparation, adapters.py, capture). Prose says Name Frequency, file names
stay. The handoff's finished lines are ticked with evidence.
data-platform/SKILL.md waits for Codex's responsibility-map branch.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Skill audit: review fixes; one open-work list

READING.md and COMBINING.md: remaining census prose says Name Frequency,
stale status passages state today, a dated 2026-10-11 qualification
section, the retired builder and reader named. The open-work list lives in
data-onboarding; refining-rules points to it (GoF finding).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Aneena Ananth <aneenaananth@MacBook-Air.local>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
@paulananth
paulananth deleted the claude/name-frequency-assembly branch October 15, 2026 01:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@paulananth