Skip to content

GLEIF read by its contract alone; the older reader retired (Issue 20 L6) - #904

Merged
paulananth merged 15 commits into
mainfrom
claude/gleif-reader-retirement
Oct 10, 2026
Merged

GLEIF read by its contract alone; the older reader retired (Issue 20 L6)#904
paulananth merged 15 commits into
mainfrom
claude/gleif-reader-retirement

Conversation

@paulananth

@paulananth paulananth commented Oct 10, 2026 •
edited
Loading

Copy link
Copy Markdown
Owner

GLEIF read by its contract alone, and the older reader retired (Issue 20 L6)

What

  • The GLEIF Level 1, relationships and reporting exceptions contracts make the older reader's checks themselves, as record checks with reasons (operator ruling, 2026-10-10: refusals live in the MDM contract). Each keeps the record's own time and the whole record.
  • Relationships read the GLEIF record as published, through the generic rules find … one, time_format: instant, interval and status. Reporting exceptions use every_in and retained_as.
  • A non-object Extension fails closed as a deletion flag, as the older reader did (operator ruling, 17:55 ET).
  • Out-of-scope records are dropped by the reading and never reach the Stage (operator ruling, 17:55 ET); docs/specs/mdm/policy-language.md says so.
  • Deleted: gleif_source.py (record_evidence and the release checks), native_consumption.py, the native branch of the mastering command, the native plumbing in run.py, and source_publications.py, whose only caller was that path (its integration test left with Fresh Change Journal, Rules acquisition authority and fenced control #738). A manifest still naming native_source is refused before any work.
  • VERSION (text unchanged: it is hashed into every attestation), FORMATS, validate_metadata and dataset_contract moved into gleif_publication.py.
  • The equivalence script is deleted; it last ran at 68cc1240.

Contract digests, all approved by the operator:

Member Digest Approved
Level 1 67fe1aaa… 2026-10-10 18:06 ET (peels to fc104ef9…, approved 14:13 ET)
Relationships b9b1fd11… 2026-10-10 18:06 ET (peels to 78cb4dd5…, approved 17:55 ET)
Reporting exceptions 643e8335… 2026-10-10 18:06 ET (peels to 8ec18c6d…, approved 17:55 ET)

Evidence

The whole 2026-09-11 Golden Copy was read both ways, by the older reader under the approved contract and by the configured contract. Every outcome was equal. No record has a non-object Extension, so the evidence holds for the final digests. Files: .planning/workstreams/whole-source-census/evidence/.

Member Records Differences
Level 1 3,428,477 0
Relationships 487,721 0
Reporting exceptions 6,351,397 0

Tests

  • Every older-reader behaviour has a configured-path test (test_clean_gleif_level1_contract.py, the relationship and exception contract tests, test_clean_contract_refusal.py). The 12 archive and attestation tests survive in test_clean_gleif_publication.py.
  • tests/support/fresh_mastering.py and the bundle test read GLEIF through normalize; the cohort's assertion ids equal main's.
  • Local: 1,322 unit and MDM tests; 193 Postgres integration and engine GLEIF tests (PostgreSQL 16); 112 rules, digest and contract tests after the review fixes.

Review

Three axes. Standards: no violation; two names made plainer. Spec: the relationships Extension test added, checklists current. GoF: leave it; one follow-up recorded on the ticket (move per-item and per-link checks into the engine when a second source needs them).

🤖 Generated with Claude Code

https://claude.ai/code/session_01FFKdv4nqKXKM9d5Cf2hEFk

Aneena Ananth and others added 15 commits October 10, 2026 17:52
Aneena Ananth @claude Aneena Ananth
A contract's record checks set a record aside, first
e3f0430 
L6, step 1 (operator ruling 2026-10-10: GLEIF refusals live in the MDM
contract; readings only select and project). `source_mapping.
record_refusal` runs only the declared record checks of a contract's
reading and returns the reason one sets the record aside with.
`adapters._refuse` raises it as an `UnsupportedRecord` at the top of
`normalize` and `mapped_values`, before the kind, key or fields are read,
so the record defers in the Stage with that reason (blocking unless the
contract lists it as non-blocking), as `gleif_source.record_evidence`
does today. A reading without checks refuses nothing; no contract
declares any yet. GoF consult: one refusal step at the top, not a side
effect of reading `fields`.

5 new tests; 252 affected tests pass.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FFKdv4nqKXKM9d5Cf2hEFk
Aneena Ananth @claude Aneena Ananth
GLEIF Level 1 refusals in the contract (L6, step 2)
86fde85 
The Level 1 contract's reading now declares, as record checks, what
`gleif_source.record_evidence` checked before mapping, in its order:
the LEI's shape and check digits (`invalid_lei`, `invalid_lei_checksum`),
a filled deletion flag (`gleif_deletion_flag`, blocking), and a category
outside GLEIF's six (`invalid_identity_kind`). Each reads its value
through a literal-path column, as the fields do, so a malformed parent
is refused like the older reader refuses it, never a reading error.
`project_record` leaves the checks to the refusal step.

The contract digest moves to 9305c467..., awaiting the operator's
approval; peeling the checks gives back the approved f53c9c42...
(`test_rules_config_digests.py`). 284 GLEIF, census and digest tests
pass, among them the older reader's own.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FFKdv4nqKXKM9d5Cf2hEFk
Aneena Ananth @claude Aneena Ananth
GLEIF Level 1: the record's own time and the whole record, in the con…
0866d4b 
…tract

Compared one record both ways, the configured path stored no effective
time and no native record, where the older reader stamped each record
with its `Registration.LastUpdateDate` and kept the raw record. A
contract may now name `effective_at` (a path each record states its own
time at; otherwise the publication's), and a provenance path `.` is the
whole record. The Level 1 contract uses both. On that fixture the older
reader under the approved contract (f53c9c42...) and `normalize` under
this one now write the same assertion id.

Contract digest fc104ef9..., awaiting the operator's approval; peeling
the refusals, the time and the native record gives back f53c9c42....
160 GLEIF, refusal and digest tests pass.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FFKdv4nqKXKM9d5Cf2hEFk
Aneena Ananth @claude Aneena Ananth
L6: Level 1 Golden Copy equivalence script (test-only)
0836904 
Every record read by the older reader under the approved contract and by
normalize under the new one; same assertion id or same deferral reason.
20,000 records: all equal (18,593 assertions, 1,404 unsupported identity
kind, 3 invalid LEI checksum). Shards for the whole file.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FFKdv4nqKXKM9d5Cf2hEFk
Aneena Ananth @claude Aneena Ananth
L6: the operator approves the Level 1 contract digest fc104ef9...
d5d77da 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FFKdv4nqKXKM9d5Cf2hEFk
Aneena Ananth @claude Aneena Ananth
GLEIF relationships read by their contract alone (L6, step 3)
5d7b062 
The relationships contract read a row `gleif_source.record_evidence`
built; it now reads the GLEIF record as published. Generic additions to a
relationship mapping (GoF consult: leave the structure, one small
`_refuse_link`): `find ... one: <reason>` (anything but exactly one
match sets the record aside), `time_format: instant`, an `interval` rule
(a start, before any end) and a `status` rule (allowed values, and those
that need an end), each with its reason. The contract's record checks
take the LEI ends, both endpoint types and the deletion flag; the record
keeps its own time and the whole record.

14 cases, compared with the older reader under the approved contract
(b7493a71...): all equal; kept as `test_clean_gleif_relationship_contract.py`
with their outcomes. Contract digest 78cb4dd5..., awaiting the operator's
approval; peeled, b7493a71.... 838 MDM and GLEIF tests pass.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FFKdv4nqKXKM9d5Cf2hEFk
Aneena Ananth @claude Aneena Ananth
GLEIF reporting exceptions read by their contract alone (L6, step 4)
f28cee1 
The contract's record checks take the LEI, the deletion flag and the two
consolidation categories; two generic adapter rules take the rest:
`every_in` (every value of a list in a set, with a reason for none and
one for a stranger; the engine's checks test one value) and `retained_as`
(every remaining record waits with that reason: evidence, never an
identity or a parent link).

13 cases compared with the older reader under the approved contract
(77bb3a8d...): all equal; kept as `test_clean_gleif_exception_contract.py`.
Contract digest 8ec18c6d..., awaiting the operator's approval; peeled,
77bb3a8d.... 184 GLEIF and digest tests pass.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FFKdv4nqKXKM9d5Cf2hEFk
Aneena Ananth @claude Aneena Ananth
Ticket: Name Frequency caller replaced, #901 merged
3f3bdd0 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FFKdv4nqKXKM9d5Cf2hEFk
Aneena Ananth @claude Aneena Ananth
L6: the Golden Copy equivalence covers all three GLEIF members (test-…
130e723 
…only)

The approved contract is peeled by the digest test's own _without_refusals;
every LEI is in scope; archives pinned by sha256.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FFKdv4nqKXKM9d5Cf2hEFk
Aneena Ananth @claude Aneena Ananth
L6: GLEIF behaviours tested through the contract; fixtures off the ol…
3051a97 
…der reader

The older reader's tests with no configured counterpart now have one
(Level 1 fields, probable kinds, corrections, agent address, check digit,
successor links, null deletion fields). fresh_mastering and the bundle
test read GLEIF through normalize; the cohort's assertion ids equal main's.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FFKdv4nqKXKM9d5Cf2hEFk
Aneena Ananth @claude Aneena Ananth
L6: whole Golden Copy equal both ways, all three GLEIF members
b7a8f95 
Level 1 3,428,477, relationships 487,721, reporting exceptions 6,351,397
records: every outcome equal between the older reader and the contract.
Evidence under the ticket.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FFKdv4nqKXKM9d5Cf2hEFk
Aneena Ananth @claude Aneena Ananth
L6: a non-object GLEIF Extension fails closed; the rulings recorded
5136b09 
The operator, 2026-10-10 17:55 ET: "approve both digests, go with your
recommendations". Relationships 78cb4dd5... and reporting exceptions
8ec18c6d... approved on the whole-file evidence. Out-of-scope records:
dropped by the reading (policy-language spec updated). A non-object
Extension is read as the deletion flag itself, as the older reader did:
new digests 67fe1aaa..., b9b1fd11..., 643e8335..., each peeling to the
approved one, await approval. MAPPING.xlsx regenerated from the rules.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FFKdv4nqKXKM9d5Cf2hEFk
Aneena Ananth @claude Aneena Ananth
L6: the operator approved the Extension-check digests
68cc124 
Level 1 67fe1aaa…, relationships b9b1fd11… and reporting exceptions 643e8335…,
approved 2026-10-10 18:06 ET ("approved").

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FFKdv4nqKXKM9d5Cf2hEFk
Aneena Ananth @claude Aneena Ananth
L6: retire the old GLEIF reader and the native mastering path
54c7ab9 
GLEIF records are read by their contracts alone (whole Golden Copy equal,
all three members). Deleted: gleif_source.py (record_evidence and the
release checks), native_consumption.py, the native_source branch in the
mastering command (now refused before any work), the native plumbing in
run.py, and source_publications.py, whose only caller was that path.
VERSION (unchanged text), FORMATS, validate_metadata and dataset_contract
move into gleif_publication.py. The equivalence script last ran at 68cc124.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FFKdv4nqKXKM9d5Cf2hEFk
Aneena Ananth @claude Aneena Ananth
L6 review: relationships Extension test, plain names, checklists current
a9a2d9c 
Spec review: the non-object Extension check is now tested on the relationships
contract too; the ticket and Issue 20 record the deletion. Standards review: rules
comments say "retired in Issue 20 L6"; the error reads "Unknown GLEIF member".
GoF review: leave it; its follow-up is recorded on the ticket.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FFKdv4nqKXKM9d5Cf2hEFk
@paulananth paulananth changed the title GLEIF read by its contract alone (Issue 20 L6) GLEIF read by its contract alone; the older reader retired (Issue 20 L6) Oct 10, 2026
@paulananth
paulananth marked this pull request as ready for review October 10, 2026 22:28
@paulananth
paulananth merged commit 44782b8 into main Oct 10, 2026
6 checks passed
paulananth pushed a commit that referenced this pull request Oct 10, 2026
Aneena Ananth @claude Aneena Ananth
L4: the whole capture's Company main reading is equal to the retained…
503048e 
… loader

All 76,230 pinned submissions, 76,230 Company rows and 8,065,661 filing rows,
every main table equal (4 shards, 17:51-18:43 ET). The ticket records the
landed-parquet finding and the remaining L4 parts; L6 closes with #904.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FFKdv4nqKXKM9d5Cf2hEFk
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@paulananth