Profiler: nested documents, entities inside parts, false hierarchies, findings-first narration

[paulananth]

What this fixes

Profiling the insider ownership filings (Forms 3, 4 and 5; ticket 07) classed every part "unknown" and showed six hierarchies that did not exist. All of the causes are in the profiler, and each fix is generic. Operator: "Fix the profiler first (Recommended)"; standing instruction: "fix the skill once you find any issue".

Reading nested documents

• Two equal-length lists of records, such as a document's transactions and its holdings, were zipped together as one table. Each transaction got paired with an unrelated holding, and single-item lists were left behind as columns: 1,360 of 8,391 transactions were missing. Now only lists of plain values are read as a table stored by column.
• A nested record with no key of its own is identified by its parent's key and its place in the list. Classification now counts that as a key; the findings still mark it as designed.

Entities carried inside a part (new: embedded.py)

• An identifier column that repeats, with a name-like column in the same object that follows it on 95% of rows, becomes its own part (derived_from). The part it came from links to it.
• A column that follows only by coincidence (one value on nearly every row, or filled on fewer than half the identifier's rows) is not carried along.
• Rows that disagree with the kept value are reported as quality item carried_value_disagrees, never dropped silently.
• A tie keeps the smallest value, so reruns give the same result.
• The evidence is written to the findings (derived_evidence).

False hierarchies

• A yes/no flag spelled several ways (0, false, 1, true) is now a flag.
• A parent level needs at least 10 rows outside its most common value. Two rows that happen to hold their own values prove nothing.
• Columns that determine each other on 99% of rows (an id and its name, with a stray spelling) name one thing, not two levels.

Narration. Operator: "fix the skill to be more story telling ... i need to know what it is finding no more ... discussion and slop".

• The narration sections of data-profiling and data-onboarding now ask for each update to be a finding, with numbers and meaning, never a description of the next step.
• The profiler's progress lines say what each stage found: parts and row counts, entities carried, keys, links, classes and hierarchies.

Also

• The dataset name is kept; the report title used to read "None".
• A working copy the profiler creates for itself is now removed. Tests had left 188 behind, about 588 MB.
• findings.md gains derived_from and §3.2, and states how a nested record's designed key is treated.
• data-quality lists the new quality item.

Results

On the ownership cohort (3,367 filings):

• The transaction counts match a direct count: 8,391 non-derivative and 1,528 derivative.
• The issuer is a master (63 issuers, by CIK).
• The filing and its tables are transactions.
• The owner is reference at 0.857, just ahead of master at 0.833. It carries only a CIK and a name, which is indistinguishable from a code with its label. This is a question for the operator on ticket 07.
• 3 hierarchies remain, all plausible: an owner's street within its state, with city and postal code under it; and each indirect-ownership description with one security title.

Verification

• Tests: 206 profiling and data-quality tests pass, 15 of them new. Each new test fails without its fix.
• Genericity lint: passes.
• Three-axis review (Standards, Spec, GoF) of the first two commits is done, and the fixes are in. The GoF verdict was to leave the structure, and its suggested Part.is_input tidy-up is taken.
• Trial B (non-SEC): 52 of 52, with the same score lines as before.
• Trial A (SEC and GLEIF): still running. The first rerun ran out of disk and was stopped safely by the watchdog; the operator chose to prune the uv cache and retry. Its score goes on ticket 01d before merge.

Overlap guard

An earlier commit (the narration) ran while Grok held an uncommitted edit to the Examples section of data-onboarding/SKILL.md. The operator said "Commit both skills (Recommended)". After the rebase the guard is clear.

🤖 Generated with Claude Code

https://claude.ai/code/session_01FFKdv4nqKXKM9d5Cf2hEFk