Repository navigation
Conversation
A null cell was written as the bare word null, the same bytes as the stored string "null". load csv then read every null back as the string "null", and refused a whole file when a nullable number column held a null. Write a null as an empty unquoted field, the spelling load csv, polars, DuckDB and Arrow's CSV writer use; an empty string stays "".
|
|
ragnorc
left a comment
There was a problem hiding this comment.
Review of #1440 at 5c74be7
Recommendation: merge after two small text fixes and a merge of main. The code change is correct. I found no blocking defect.
Why, what, how, impact
- Why:
query --format csvwrote a null cell asnull. A stored string"null"was written with the same bytes, so no reader could tell them apart.load csvthen read every null as the string"null", and refused a file with a null in a number column. - What: CSV output now writes a null as an empty field. An empty string stays
"". Other cells and other formats do not change. - How: a new
csv_cell(crates/omnigraph-cli/src/read_format.rs:382) maps a missing key or a JSON null to an empty field. Everything else goes throughcsv_escape(stringify_value(..))as before. - Impact: for files with two or more columns, CSV output now round-trips through
load csv. A script that matched the literalnullmust now read an empty field as null. Thechangedchangelog note says this.
Contract and root cause
The contract is "CSV output is text that load csv and common CSV readers read back with the same values". The violated invariant: one spelling (null) had two meanings. The empty field is the right spelling because it is the one the reader already uses. utf8_without_empty (crates/omnigraph/src/loader/csv.rs:1435) calls it "the CSV spelling of null, since the format has no other". polars, DuckDB and the Arrow CSV writer use it too. The fix is in the owning layer, the one CSV writer. I searched the workspace and render_csv is the only CSV writer. Served reads render through the same function on the client, so the fix covers direct and remote targets. The same kind of collision does not happen in TOON: toon.rs quotes a string null (line 579). table and kv are for people, not machines, so their null spelling is acceptable.
Should fix before merge (text only)
S1. The one-column claim in the PR description is wrong, and the limit is not documented. The PR says "A one-column file cannot feed load csv (it needs a key)". That is true only for keyed types. A node type without @key gets generated ids (loader/csv.rs:1604), so a one-column file loads. The null row is written as an empty line. The load csv record parser skips empty lines (loader/csv.rs:539), so the row is dropped without an error. I reproduced this with node Note { text: String? } and three rows ("a", null, "null"):
| Build | note.csv bytes |
load csv result |
|---|---|---|
base ccf273bd |
text\na\nnull\nnull\n |
3 rows; the null row reads back as the string "null" |
head 5c74be76 |
text\na\n\nnull\n |
affected_nodes: 2, success; the null row is gone |
So for this narrow shape, the PR turns a wrong value into a missing row. Neither is reported. I do not ask to change the writer: the empty line is the correct spelling, and polars and DuckDB read it back as a null row. The cause is in the reader: for a one-column file, an empty line is a record with one null field, not "no record". Please (a) correct the Risks bullet, and (b) add one sentence to the CLI reference (and the load csv section of docs/user/mutations/index.md) that says load csv skips empty lines, so a one-column file loses its null rows. Making load csv keep such records is a separate loader change, because the partition scanner depends on csv_core skipping empty lines. I suggest a findings issue for it.
Optional
- O1. Say how
load csvreads"". The new reference text sets "a null as an empty field" against "an empty string as""". A reader can infer that""loads back as an empty string. It does not:load csvreads both as null (utf8_without_empty). The PR description says this, but the docs do not. One clause would fix it: "load csvreads both as null; it has no spelling for an empty string". - O2. One owner for the round trip. The CSV null spelling is now defined in two crates: the writer (
csv_cell) and the reader (utf8_without_empty). The new unit test pins the writer's bytes. Nothing pins that the two agree. A shared constant is not worth the coupling. A small CLI integration test (query--format csv, thenload csv, then compare rows) next to the existingload csvtests incli_data.rswould make the agreement hold by test, and would also catch S1 if the loader changes later. After five similar changes (more types, more formats), that one test stays the single owner of the contract.
Liability
Removes: an ambiguous output spelling and a load failure on number columns. Adds: one 6-line helper and one test. No new API, state or invariant. The documented output format changes, and the changelog note covers that. Net liability goes down.
Validation
- GQT cannot express this contract: GQT asserts query rows, and the defect is in the CLI's text rendering of those rows. The PR's in-source test in
read_format.rsis the cheapest owner. - Regression before and after, clean detached worktree,
CARGO_TARGET_DIRshared warm target,cargo dev-test -p omnigraph-cli --lib -- read_format::tests::csv_keeps:ccf273bd(test only, on base25b51992): fails for the predicted reason:assertion left != right failed: null and the string "null", left"null", right"null".5c74be76(head): passes. All 9read_format/presentationCSV tests and the 26loader::csvtests pass.
- End to end with the CLI built at each commit (direct graphs,
query --format csv, thenload csvinto a fresh graph):node X { slug: String @key s: String? n: I32? }, rowss: "",snull,s: "null": base refuses (Error while parsing value 'null' as type 'Int32'); head loads all three rows,nulnull,word"null",emptynull (documented limit). This matches the PR.- The one-column result in S1 above.
- CI at
5c74be76: every job passes exceptDependency Guard (cargo deny)and soCI Gate. That failure is three RustSecunmaintainedadvisories (RUSTSEC-2026-0542, -0543, -0555).mainignores them indeny.tomlsince #1439 (5a1ec458), and main's Dependency Guard passes there. Mergemainto clear it. The PR is not the cause. - Not run: the full workspace suite and
system_remotelocally. CI covers them at this head.
No Decision needed threads. Temporary worktree only; nothing pushed.
load csv reads `""` as null and skips empty lines, so a one-column CSV result loses its null rows when loaded into a type without @key. Say so in the CLI reference, the load csv rules and the skill. Add cli_data::csv_query_output_round_trips_through_load_csv: rows written by query --format csv load back through load csv with the same values, so the writer's and the reader's null spelling stay in step.
|
Reply to the review of Two new commits: S1. Fixed. The finding is valid. A node type without O1. Taken. The same sentences say that O2. Taken. New test Dependency Guard. The merge of main brings #1439's Gates run locally on
CI on |
Summary
query --format csvwrote a null cell as the bare wordnull, the same bytes as the stored string"null". A reader could not tell them apart, andload csv(which reads an empty field as null) read every null back as the string"null". When a nullable number column held a null,load csvrefused the whole file.This change writes a null as an empty unquoted field. An empty string stays
""(#1375), and every other cell is unchanged.Findings queue Issue:
og-stress-csv-output-null-equals-string-null(beads graph, found by the stress workflow). It is not on GitHub, so there is noCloses #.Reproduction
Main
25b51992360d7858cc6bc7afaef9a198f869e022, debug CLI, a standalone graph withnode X { slug: String @key s: String? n: I32? }and three rows:s: "",snull,s: "null".Before:
load csvof that file into a fresh graph:Parser error: Error while parsing value 'null' as type 'Int32' for column 2 at line 3, exit 1. With the text column alone it loads, butnulbecomes the string"null"andemptybecomes null.After:
load csvloads all three rows:nulreads back as null,wordas the string"null", and the nulls innas null.emptyreads back as null, becauseload csvdocuments an empty field as null and has no spelling for an empty string. That limit is unchanged.The regression test
read_format::tests::csv_keeps_a_null_cell_distinct_from_the_string_nullfails on main (commit ccf273b:nullandnullcompared equal) and passes with the fix.Root cause
render_csv(crates/omnigraph-cli/src/read_format.rs) filled a missing key, which is how a JSON row spells null, withValue::Null. It then rendered it throughstringify_value, which spells nullnullfor the table and kv formats. CSV has no null literal, so the word collided with a string.Decision and alternatives
How common CSV writers and readers handle the three rows (measured with polars, DuckDB, pyarrow, pandas and Python
csv; details below):"null"kept apart byload csvround tripnull"null"; a number column refuses"null"load csv"null"round-trippolars
write_csv, DuckDBCOPY TOand Arrow's CSV writer all write a null as an empty field and an empty string as"". So the empty field is the one spelling the ecosystem and our ownload csvalready agree on. The null spelling in CSV was not documented: the reference only said that CSV writes an empty string as"".Alternatives rejected:
"null": standard readers drop quotes, so nothing changes for them.load csvwould stay broken, and number columns would still refuse.Risks
nullin CSV output must now treat an empty field as null. The changelog note ischangedfor that reason.csvgives[], and pandas and pyarrow skip it.load csvskips empty lines, so when such a file is loaded into a node type without@key(ids are generated), the null rows are dropped without an error. Before this change they loaded as the string"null". The CLI reference and theload csvdocs now state this limit; keeping those rows is a separateload csvchange.table,kv,json,jsonl,toonandarrowoutput are unchanged.Tests and gates run locally
read_format::tests::csv_keeps_a_null_cell_distinct_from_the_string_null(exact bytes).omnigraph-cliCSV, read_format, presentation and skill tests;omnigraph-skilltests.cargo fmt --all --check,cargo dev-clippy,scripts/check-docs.py,scripts/check-release-note.py, typos 1.50.1.load csvround trip above, with the CLI built before and after.Not run locally:
system_remote::remote_read_preserves_projection_order_in_json_and_csvcannot start a server on this 2-core box, and it fails the same way on main here. GQT, DST and the engine suites do not depend on CLI output formatting; CI runs them.Docs:
docs/user/cli/reference.md,docs/user/mutations/index.md,skills/omnigraph/references/commands.md,changelog.d/csv-null-empty-field.changed.md.Round trip test:
cli_data::csv_query_output_round_trips_through_load_csvwrites rows withquery --format csv, loads the file withload csvand compares the rows.Reader comparison
polars 2.1.0, DuckDB 2.0.0, pyarrow 26.0.1, pandas 3.1.1, Python 3.12 csv.
Workflow run: run-wf-reproduce-modernrelay-omnigraph-issue-20261117T1202Z-e94174
Response run: run-wf-respond-modernrelay-omnigraph-pr-20261117T1402Z-ba93a4