Skip to content

fix(cli): write a null as an empty field in CSV output - #1440

Open
ragnorc wants to merge 4 commits into
ModernRelay:mainfrom
ragnorc:fix/og-csv-null-empty-field
Open

ragnorc wants to merge 4 commits into
ModernRelay:mainfrom
ragnorc:fix/og-csv-null-empty-field

Conversation

@ragnorc

@ragnorc ragnorc commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Summary

query --format csv wrote a null cell as the bare word null, the same bytes as the stored string "null". A reader could not tell them apart, and load csv (which reads an empty field as null) read every null back as the string "null". When a nullable number column held a null, load csv refused the whole file.

This change writes a null as an empty unquoted field. An empty string stays "" (#1375), and every other cell is unchanged.

Findings queue Issue: og-stress-csv-output-null-equals-string-null (beads graph, found by the stress workflow). It is not on GitHub, so there is no Closes #.

Reproduction

Main 25b51992360d7858cc6bc7afaef9a198f869e022, debug CLI, a standalone graph with node X { slug: String @key s: String? n: I32? } and three rows: s: "", s null, s: "null".

Before:

slug,s,n
empty,"",1
nul,null,2
word,null,null

load csv of that file into a fresh graph: Parser error: Error while parsing value 'null' as type 'Int32' for column 2 at line 3, exit 1. With the text column alone it loads, but nul becomes the string "null" and empty becomes null.

After:

slug,s,n
empty,"",1
nul,,2
word,null,

load csv loads all three rows: nul reads back as null, word as the string "null", and the nulls in n as null. empty reads back as null, because load csv documents an empty field as null and has no spelling for an empty string. That limit is unchanged.

The regression test read_format::tests::csv_keeps_a_null_cell_distinct_from_the_string_null fails on main (commit ccf273b: null and null compared equal) and passes with the fix.

Root cause

render_csv (crates/omnigraph-cli/src/read_format.rs) filled a missing key, which is how a JSON row spells null, with Value::Null. It then rendered it through stringify_value, which spells null null for the table and kv formats. CSV has no null literal, so the word collided with a string.

Decision and alternatives

How common CSV writers and readers handle the three rows (measured with polars, DuckDB, pyarrow, pandas and Python csv; details below):

Encoding null vs "null" kept apart by load csv round trip
main: null as null no reader null becomes "null"; a number column refuses
quote the string: "null" no reader (all strip quotes) same as main
null as empty field (this PR) polars, DuckDB, load csv null and "null" round-trip

polars write_csv, DuckDB COPY TO and Arrow's CSV writer all write a null as an empty field and an empty string as "". So the empty field is the one spelling the ecosystem and our own load csv already agree on. The null spelling in CSV was not documented: the reference only said that CSV writes an empty string as "".

Alternatives rejected:

  • Quote the string "null": standard readers drop quotes, so nothing changes for them.
  • Docs only: the round trip into load csv would stay broken, and number columns would still refuse.

Risks

  • A script that matched the literal null in CSV output must now treat an empty field as null. The changelog note is changed for that reason.
  • A one-column result with a null row prints an empty line, as polars and DuckDB also do. polars and DuckDB read it back as a null row, but Python csv gives [], and pandas and pyarrow skip it. load csv skips empty lines, so when such a file is loaded into a node type without @key (ids are generated), the null rows are dropped without an error. Before this change they loaded as the string "null". The CLI reference and the load csv docs now state this limit; keeping those rows is a separate load csv change.
  • table, kv, json, jsonl, toon and arrow output are unchanged.

Tests and gates run locally

  • New: read_format::tests::csv_keeps_a_null_cell_distinct_from_the_string_null (exact bytes).
  • omnigraph-cli CSV, read_format, presentation and skill tests; omnigraph-skill tests.
  • cargo fmt --all --check, cargo dev-clippy, scripts/check-docs.py, scripts/check-release-note.py, typos 1.50.1.
  • End to end: the query to load csv round trip above, with the CLI built before and after.

Not run locally: system_remote::remote_read_preserves_projection_order_in_json_and_csv cannot start a server on this 2-core box, and it fails the same way on main here. GQT, DST and the engine suites do not depend on CLI output formatting; CI runs them.

Docs: docs/user/cli/reference.md, docs/user/mutations/index.md, skills/omnigraph/references/commands.md, changelog.d/csv-null-empty-field.changed.md.

Round trip test: cli_data::csv_query_output_round_trips_through_load_csv writes rows with query --format csv, loads the file with load csv and compares the rows.

Reader comparison
== main (null as null)
   python csv  empty=""      null="null"  "null"="null"
   pandas      empty=NULL    null=NULL    "null"=NULL
   polars      empty=""      null="null"  "null"="null"
   duckdb      empty=NULL    null="null"  "null"="null"
   pyarrow     empty=""      null="null"  "null"="null"
== A (null as empty field)
   python csv  empty=""      null=""      "null"="null"
   pandas      empty=NULL    null=NULL    "null"=NULL
   polars      empty=""      null=NULL    "null"="null"
   duckdb      empty=NULL    null=NULL    "null"="null"
   pyarrow     empty=""      null=""      "null"="null"
== B (quote the string null)
   python csv  empty=""      null="null"  "null"="null"
   pandas      empty=NULL    null=NULL    "null"=NULL
   polars      empty=""      null="null"  "null"="null"
   duckdb      empty=NULL    null="null"  "null"="null"
   pyarrow     empty=""      null="null"  "null"="null"

polars writes: 's\na\n\n""\nb\n'
duckdb writes: 's\na\n\n""\nb\n'
pyarrow writes: '"s"\n"a"\n\n""\n"b"\n'
pandas writes: 's\na\n""\n""\nb\n'

polars 2.1.0, DuckDB 2.0.0, pyarrow 26.0.1, pandas 3.1.1, Python 3.12 csv.

Workflow run: run-wf-reproduce-modernrelay-omnigraph-issue-20261117T1202Z-e94174
Response run: run-wf-respond-modernrelay-omnigraph-pr-20261117T1402Z-ba93a4


Devin Review

A null cell was written as the bare word null, the same bytes as the
stored string "null". load csv then read every null back as the string
"null", and refused a whole file when a nullable number column held a
null. Write a null as an empty unquoted field, the spelling load csv,
polars, DuckDB and Arrow's CSV writer use; an empty string stays "".
@ragnorc
ragnorc requested a review from aaltshuler as a code owner November 17, 2026 12:32

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no bugs or issues to report.

Devin Review

@ragnorc

ragnorc commented Nov 17, 2026

Copy link
Copy Markdown
Contributor Author

Dependency Guard (cargo deny) fails on three unmaintained advisories that RustSec published after main's last green run: RUSTSEC-2026-0542 (humantime), RUSTSEC-2026-0543 (chrono) and RUSTSEC-2026-0555 (chrono-tz). This PR changes no dependency (no Cargo.lock, Cargo.toml or deny.toml change). cargo deny --locked check advisories (cargo-deny 0.21.0) on a clean checkout of main 25b5199 fails with the same three. Failed run: https://github.com/ModernRelay/omnigraph/actions/runs/44084067985/job/134041768220

@ragnorc ragnorc left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review of #1440 at 5c74be7

Recommendation: merge after two small text fixes and a merge of main. The code change is correct. I found no blocking defect.

Why, what, how, impact

  • Why: query --format csv wrote a null cell as null. A stored string "null" was written with the same bytes, so no reader could tell them apart. load csv then read every null as the string "null", and refused a file with a null in a number column.
  • What: CSV output now writes a null as an empty field. An empty string stays "". Other cells and other formats do not change.
  • How: a new csv_cell (crates/omnigraph-cli/src/read_format.rs:382) maps a missing key or a JSON null to an empty field. Everything else goes through csv_escape(stringify_value(..)) as before.
  • Impact: for files with two or more columns, CSV output now round-trips through load csv. A script that matched the literal null must now read an empty field as null. The changed changelog note says this.

Contract and root cause

The contract is "CSV output is text that load csv and common CSV readers read back with the same values". The violated invariant: one spelling (null) had two meanings. The empty field is the right spelling because it is the one the reader already uses. utf8_without_empty (crates/omnigraph/src/loader/csv.rs:1435) calls it "the CSV spelling of null, since the format has no other". polars, DuckDB and the Arrow CSV writer use it too. The fix is in the owning layer, the one CSV writer. I searched the workspace and render_csv is the only CSV writer. Served reads render through the same function on the client, so the fix covers direct and remote targets. The same kind of collision does not happen in TOON: toon.rs quotes a string null (line 579). table and kv are for people, not machines, so their null spelling is acceptable.

Should fix before merge (text only)

S1. The one-column claim in the PR description is wrong, and the limit is not documented. The PR says "A one-column file cannot feed load csv (it needs a key)". That is true only for keyed types. A node type without @key gets generated ids (loader/csv.rs:1604), so a one-column file loads. The null row is written as an empty line. The load csv record parser skips empty lines (loader/csv.rs:539), so the row is dropped without an error. I reproduced this with node Note { text: String? } and three rows ("a", null, "null"):

Build note.csv bytes load csv result
base ccf273bd text\na\nnull\nnull\n 3 rows; the null row reads back as the string "null"
head 5c74be76 text\na\n\nnull\n affected_nodes: 2, success; the null row is gone

So for this narrow shape, the PR turns a wrong value into a missing row. Neither is reported. I do not ask to change the writer: the empty line is the correct spelling, and polars and DuckDB read it back as a null row. The cause is in the reader: for a one-column file, an empty line is a record with one null field, not "no record". Please (a) correct the Risks bullet, and (b) add one sentence to the CLI reference (and the load csv section of docs/user/mutations/index.md) that says load csv skips empty lines, so a one-column file loses its null rows. Making load csv keep such records is a separate loader change, because the partition scanner depends on csv_core skipping empty lines. I suggest a findings issue for it.

Optional

  • O1. Say how load csv reads "". The new reference text sets "a null as an empty field" against "an empty string as """. A reader can infer that "" loads back as an empty string. It does not: load csv reads both as null (utf8_without_empty). The PR description says this, but the docs do not. One clause would fix it: "load csv reads both as null; it has no spelling for an empty string".
  • O2. One owner for the round trip. The CSV null spelling is now defined in two crates: the writer (csv_cell) and the reader (utf8_without_empty). The new unit test pins the writer's bytes. Nothing pins that the two agree. A shared constant is not worth the coupling. A small CLI integration test (query --format csv, then load csv, then compare rows) next to the existing load csv tests in cli_data.rs would make the agreement hold by test, and would also catch S1 if the loader changes later. After five similar changes (more types, more formats), that one test stays the single owner of the contract.

Liability

Removes: an ambiguous output spelling and a load failure on number columns. Adds: one 6-line helper and one test. No new API, state or invariant. The documented output format changes, and the changelog note covers that. Net liability goes down.

Validation

  • GQT cannot express this contract: GQT asserts query rows, and the defect is in the CLI's text rendering of those rows. The PR's in-source test in read_format.rs is the cheapest owner.
  • Regression before and after, clean detached worktree, CARGO_TARGET_DIR shared warm target, cargo dev-test -p omnigraph-cli --lib -- read_format::tests::csv_keeps:
    • ccf273bd (test only, on base 25b51992): fails for the predicted reason: assertion left != right failed: null and the string "null", left "null", right "null".
    • 5c74be76 (head): passes. All 9 read_format/presentation CSV tests and the 26 loader::csv tests pass.
  • End to end with the CLI built at each commit (direct graphs, query --format csv, then load csv into a fresh graph):
    • node X { slug: String @key s: String? n: I32? }, rows s: "", s null, s: "null": base refuses (Error while parsing value 'null' as type 'Int32'); head loads all three rows, nul null, word "null", empty null (documented limit). This matches the PR.
    • The one-column result in S1 above.
  • CI at 5c74be76: every job passes except Dependency Guard (cargo deny) and so CI Gate. That failure is three RustSec unmaintained advisories (RUSTSEC-2026-0542, -0543, -0555). main ignores them in deny.toml since #1439 (5a1ec458), and main's Dependency Guard passes there. Merge main to clear it. The PR is not the cause.
  • Not run: the full workspace suite and system_remote locally. CI covers them at this head.

No Decision needed threads. Temporary worktree only; nothing pushed.

load csv reads `""` as null and skips empty lines, so a one-column CSV
result loses its null rows when loaded into a type without @key. Say so
in the CLI reference, the load csv rules and the skill.

Add cli_data::csv_query_output_round_trips_through_load_csv: rows
written by query --format csv load back through load csv with the same
values, so the writer's and the reader's null spelling stay in step.
@ragnorc

ragnorc commented Nov 17, 2026

Copy link
Copy Markdown
Contributor Author

Reply to the review of 5c74be76 (review 5939506987).

Two new commits: 93ad7cb3 merges main 5a1ec458 into this branch (no conflict), and 89bfb57e fixes the docs and adds a round-trip test. The CSV writer is unchanged.

S1. Fixed. The finding is valid. A node type without @key gets generated ids (loader/csv.rs None => Ok(generate_id())), and the load csv parser skips empty lines, so a one-column file loses its null rows without an error. The Risks bullet in the description now says this. The CLI reference (docs/user/cli/reference.md), the load csv rules in docs/user/mutations/index.md and the skill's references/commands.md now say that load csv skips empty lines, so a one-column result does not load its null rows back. I did not change the loader. I did not open a findings issue; that is left to a maintainer.

O1. Taken. The same sentences say that load csv reads "" as null too, because it has no spelling for an empty string.

O2. Taken. New test cli_data::csv_query_output_round_trips_through_load_csv: it writes three rows ("", null and "null", with a null in an I32 column) with query --format csv, loads the file with load csv into a fresh graph, and compares the rows. The test does not include the one-column case: dropping those rows is a loader defect, and a test should not pin it as correct.

Dependency Guard. The merge of main brings #1439's deny.toml entries for RUSTSEC-2026-0542, -0543 and -0555. Main's CI passes at 5a1ec458.

Gates run locally on 89bfb57e:

  • cargo dev-nextest -E 'package(omnigraph-cli) & (test(csv) | test(read_format)) & not test(system_remote)': 11 passed, including the new test and read_format::tests::csv_keeps_a_null_cell_distinct_from_the_string_null.
  • S1 with the CLI built from this branch: node Note { text: String? }, rows "a", null and "null". query --format csv writes text\na\n\nnull\n; load csv returns affected_nodes: 2 and reads back only the strings "a" and "null".
  • cargo dev-clippy, cargo fmt --all --check, scripts/check-docs.py (539 files), typos 1.50.1: clean.
  • Not run locally: cargo deny (not installed on this box) and the full workspace suite. CI runs both on this head.

CI on 89bfb57e: 27 passed, 4 skipped, none failed; the PR is mergeable (CLEAN). Dependency Guard and CI Gate pass.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant