Profiler: zips of tables, CSV not in UTF-8, files that are not one table

[paulananth]

Profiler fixes for the gaps found on ticket 07's ADV bulk files: the data-profiling skill, its findings spec and ticket 07. No platform code.

The gaps. A zip of CSV tables was refused. One Windows-1252 member made DuckDB stop the whole run with "CSV Error on Line 3".

What changed.

• Zips of tables. A zip of CSV or Parquet files gives each member as its own table, located at zip#member with its own sha256. Part names stay distinct when members or zips share a file name. Mac __MACOSX entries are left out.
• Encoding. A CSV that is not UTF-8 is copied as UTF-8, from Windows-1252 if that reads it, else Latin-1. Its encoding record says it was inferred, gives the first byte that is not UTF-8, and gives declare, the read.encoding the reading contract should state: windows-1252, [utf-8, windows-1252], or none. UTF-16 files and files with NUL bytes are not read.
• Shape. DuckDB reads leniently: a ragged file can load as one column, a preamble is skipped, and an open quote is kept as data. So each CSV is read again with Python's csv module. A file is not one table if it has a skipped preamble, a row with another field count, a quote that does not close, or one column whose first two lines hold another delimiter. Such a file is not read, and the reason gives the line as evidence.
• Not stopping. A table that is not read is listed in dataset.not_read with its reason, and the rest of the input is profiled.

Decision (operator, 2026-10-11): option 1. The profiler stays self-contained, with DuckDB pulled in only for the run (plan decision 3). It works out an encoding and names the contract setting; the Rust engine does the real reading.

Cost. The checks read each CSV in full several times, sampled files included: about three times a plain read (88 MB, 2 million rows: 9.4 s against 3.0 s). SKILL.md says so.

Open on the ticket. Gap 3: is a custodian held only inside custody rows found as its own entity? Not verified. The ADV files are no longer on this machine, and #899's entity finder may not catch broker-dealer numbers of varying length or the 15% of rows that carry no identifier.

Checks.

• 209 profiling tests pass, including the genericity lint.
• GoF consult: leave it (register routes by format; this is one more branch).
• Three-axis review, then a re-review of the fixes. Every finding is fixed, and the re-review found no false positive on 20 regular CSV shapes (multi-line quoted fields, TSV, semicolons, CRLF, CR-only line endings, a byte-order mark, no header, a sampled file).
• Trial B was not rerun: its data set is no longer here, and its SQLite form does not use the CSV path.

🤖 Generated with Claude Code