Skip to content

refactor(engine): use Lance's system-column check instead of two copies - #1309

Merged
ragnorc merged 2 commits into
ModernRelay:mainfrom
ragnorc:simplify/system-columns-20261109
Nov 9, 2026
Merged

ragnorc merged 2 commits into
ModernRelay:mainfrom
ragnorc:simplify/system-columns-20261109

Conversation

@ragnorc

@ragnorc ragnorc commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

The engine kept two hand-written copies of the five virtual column names that the Lance scanner adds. This change makes both copies call lance_core::is_system_column, the Lance function that names the same five columns. Behavior does not change.

What changed

The five columns are _rowid, _rowaddr, _rowoffset, _row_created_at_version and _row_last_updated_at_version. Before this change, the workspace had three copies of the list, beside the list in Lance:

Copy Use After this change
omnigraph-compiler schema::is_reserved_storage_system_column Refuse a property with one of these names Kept. The compiler does not depend on Lance.
engine changes::model::is_reserved_storage_system_column and its test Drop these columns from change images, the schema fingerprint, export and push comparison Deleted. The five call sites call lance_core::is_system_column.
engine loader::is_reserved_physical_input_field (the five names and _tombstone) Refuse load input fields with these names Reduced to field == "_tombstone" || lance_core::is_system_column(field).

The engine copy existed only because the compiler copy is private to its crate. Its own comment said so, and its test said that a new Lance column must go into both lists "in one audited change". The engine removes the columns that Lance adds, so the Lance function is the correct owner for the engine.

The deleted test also had negative cases: _row, _rowdy, _row_custom, _row_created_at, id and rowid are not system columns. These cases move into the existing guard lance_surface_guards.rs::compiler_rejects_five_surveyed_lance_virtual_system_columns. That guard already tests that Lance reserves the five names and that the compiler refuses them. No check is removed.

The pull request also adds docs/simplify/20261109.md, the ranked list of candidates that this pass examined, with the reason each one was not taken. docs/dev/index.md links to it.

Kind of candidate

A special case that has a general owner: one list kept in three places, one of them a hand-written copy of a Lance function.

Why it is safe

  • At the pinned lance-core 13.0.0, is_system_column (src/lib.rs:93-98) is an exact matches! on the same five constants, with no prefix rule. The deleted functions were exact matches on the same five strings.
  • No public API, file format, HTTP route or documented behavior changes. The deleted function was pub(crate).
  • No test, guard or error path is weakened. The only deleted test tested only the deleted copy. Its negative cases now run against the Lance function in the surface guard.
  • No storage access changes. The DST storage-cost golden passes unchanged.

Liability

Removed: two copies of a list that had to change together by hand, and one test that tested only a copy.

Added: five call sites name a lance_core function directly. The engine already depends on lance-core and uses its column constants in other files. If a Lance upgrade adds a sixth column, the engine drops it automatically, which is what it needs. The compiler list still needs a manual change, as before.

Line counts

The engine definitions and the deleted test went from 52 lines to 3. The guard gained 13 lines. The code change adds 23 lines and removes 59. The report adds 174 lines of documentation, and the developer index gains 2.

Commands and results

All commands ran on this branch, on Linux, from the repository root unless a directory is named.

cargo fmt --all --check                                         pass
cargo dev-clippy                                                pass
cargo test -p omnigraph-gqt --locked --lib --test runner_dispatch   pass
cargo test -p omnigraph-gqt --locked                            pass (464 cases, 14 and 29 harness tests)
cargo clippy -p omnigraph-gqt --all-targets --locked -- -D warnings -W clippy::dbg_macro   pass
(cd crates/omnigraph-dst && cargo test --locked --test scenarios dst_bench_cost_count_golden)   pass
bash scripts/check-agents-md.sh                                 pass
python3 scripts/check-docs.py                                   pass
RUST_MIN_STACK=16777216 cargo dev-nextest --no-fail-fast        5148 passed, 10 failed, 41 skipped

The 10 failures of the workspace run come from the test machine, not from this change. The same 10 tests fail on unmodified main (f0df4673) on the same machine. They are tests of symlink refusal, unreadable files and host identity in omnigraph-bench, omnigraph-cli (cli_cluster::managed_lifecycle_capture_and_retry_records_refuse_symlinks_without_reading_targets) and omnigraph-engine (db::upgrade::tests::interrupted::unreadable_legacy_object_asks_for_a_rerun). The machine runs as root in a sandbox. typos is not installed on the machine, so CI runs that check.

Workflow run: run-wf-simplify-modernrelay-omnigraph-20261109T0046Z-a2a618

The engine kept two hand-written copies of Lance's five virtual column
names: `changes::model::is_reserved_storage_system_column` (a mirror of
the compiler's private function, with a test that pinned only itself)
and the loader's `is_reserved_physical_input_field`. Both now call
`lance_core::is_system_column`, which matches the same five names
exactly at Lance 13.0.0.

The deleted test's negative cases move into the existing Lance surface
guard, which already pins the five names and the compiler's refusal.

Adds docs/simplify/20261109.md, the ranked candidate report of this
simplification pass, linked from the developer index.

@ragnorc ragnorc left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review of 14ca568c

Why. The engine kept two hand-written copies of the five names of the Lance virtual columns. The compiler keeps a third copy. Copies that must change together can drift.

What. The engine copy in changes::model and its unit test are deleted. The five engine call sites and the loader now call lance_core::is_system_column. The negative cases of the deleted test move into the existing guard lance_surface_guards.rs::compiler_rejects_five_surveyed_lance_virtual_system_columns. The PR also adds a dated report, docs/simplify/20261109.md, and links it from docs/dev/index.md.

How it behaves. At the pinned lance-core 13.0.0, is_system_column (src/lib.rs:93-98) is an exact matches! on ROW_ID | ROW_ADDR | ROW_OFFSET | ROW_LAST_UPDATED_AT_VERSION | ROW_CREATED_AT_VERSION. These constants are the same five strings (src/lib.rs:25-33). The deleted engine function matched the same five strings exactly. So every call site returns the same result as before. I found no behavior change.

Impact. Correct. No public API, file format, HTTP route or user document changes. omnigraph-engine already depends on lance-core directly (crates/omnigraph/Cargo.toml:41). Lance itself uses this function for the same two jobs: to refuse a user column on write (lance-13.0.0/src/dataset/write/insert.rs:379, dataset/fragment.rs:2810), like the loader, and to remove system columns from a schema (dataset/mem_wal/scanner/generation_read.rs:299-310), like the engine call sites. So the engine now follows the upstream owner.

Blocking

None in the code.

Requested before merge (documentation placement, not correctness)

R1. docs/simplify/20261109.md does not fit the repository documentation rules. docs/dev/documentation.md gives each kind of content one owner. Open work, sequencing and ownership belong to the issue tracker. docs/dev/ does not own roadmaps or stale plans, and "Remove a plan after it lands". The report is mostly a ranked list of work not yet done (candidates 2-9, "a good next candidate"). It also holds facts that go stale quickly: a base SHA, line counts, and the names of open pull requests. docs/simplify/ is not in the content-ownership table, so nobody owns it. After five more passes of this kind, the developer index links five dated reports that nobody updates.

Suggested fix: put the "Taken" rationale in the PR body (it is already there, and the squash commit keeps it). Open one issue per candidate that should stay visible (2 and 3 look ready). Delete the file and the docs/dev/index.md link. If the team wants these reports as a series, first add the location and its rules to the table in docs/dev/documentation.md.

Optional

O1. "_tombstone" is the same kind of copy that this PR removes. The loader still has field == "_tombstone" || lance_core::is_system_column(field) (crates/omnigraph/src/loader/mod.rs:2849). That string is Lance's public lance::dataset::mem_wal::TOMBSTONE (lance-13.0.0/src/dataset/mem_wal.rs:75). Lance documents it as a physical column of MemWAL memtables only, not a system column. Omnigraph has no MemWAL consumer (docs/dev/lance.md). Use the constant, or add one line that says why the loader refuses a MemWAL-only name.

Related, and older than this PR (found by reading the code, not by a run): the legacy-vintage and persisted-contract name rules admit any _ name except the five (omnigraph-compiler/src/schema/parser.rs:182-206; parser_tests.rs keeps _row_id and _rowid2 readable). But validate_strict_input_fields checks is_reserved_physical_input_field before it checks the declared properties. So a legacy graph that declares a _tombstone property cannot load a value for it. This can be a separate issue.

O2. After a future Lance bump, the engine and the compiler can disagree, and no guard reports it. Before this PR, the engine copy and the compiler copy changed only together. Now the engine follows Lance automatically and the compiler does not. Assume that Lance adds a sixth name, _row_x. A current-vintage graph is safe, because the compiler refuses every _ property name. But a legacy-vintage graph or persisted contract can still declare _row_x. The engine would then remove that user property, without an error, from user_schema_fingerprint, change-feed images and logical_row_image (export, export.rs:393). That is a silent omission, and invariant 8 says such failures must be loud. The guard does not catch this case. It checks that the five known names are system columns and that six chosen near-names are not. It cannot detect a new name. The old engine test could not detect it either. So the risk is not new, but the report's sentence "the engine now drops it automatically, which is what the engine needs" hides it. I suggest that the report (or the compiler comment) says this instead: until someone updates the compiler list, the engine and the compiler disagree on legacy contracts, and step 2 of the Lance bump checklist owns that update. The cost is low and the risk is narrow, so I agree with the design choice.

O3. A misplaced doc comment, older than this PR. In omnigraph-compiler/src/schema/mod.rs:4-13, the paragraph "Names owned by Lance's virtual row-address and row-version columns ... every Lance bump still requires a source audit" is attached to is_reserved_system_column_name (the _ prefix rule), not to is_reserved_storage_system_column. The two /// blocks have no blank line between them, so rustdoc joins them. This PR makes that comment the authority for "the compiler keeps its copy", so moving the paragraph to the right function is a cheap fix.

Tradeoffs and liability

  • Removed: one hand-written list and its test. That test tested only the copy, and its text required a manual change in two places.
  • Added: five direct calls to a lance_core function. The name of this function is part of the substrate API that lance_surface_guards already pins. If a Lance bump renames it, compilation fails, which is the correct signal.
  • Not changed: the compiler copy. The compiler has no storage dependency, so this is correct. The guard ties the compiler to Lance in one direction only (see O2).
  • Five more changes of this kind: if each replaces a hand-written copy of a Lance name with the Lance symbol, liability goes down. If each adds a dated report under docs/ (R1), liability goes up.
  • Other open PRs: no open PR adds a call to the deleted changes::model::is_reserved_storage_system_column, is_reserved_physical_input_field or is_system_column. #785, #1175, #1219, #1235, #1275, #1282, #1291 and #1308 change some of the same files, but their changes do not depend on this one. 14ca568c merges cleanly into main at 2e7921c4 (git merge-tree). main adds no new callers after the merge base f0df4673.

Validation

  • Local, at exact head 14ca568c, in a clean detached worktree: cargo dev-nextest -E 'package(omnigraph-engine) & (test(/lance_surface_guards|compiler_rejects_five/) | test(/^changes::/) | test(/^loader::/) | test(/export::/) | test(/push::/) | test(/strict_graph_batch/))' --no-fail-fast. Result: 156 passed, 0 failed. These were engine lib changes:: (21), loader:: (52) and engine::push (9); integration changes:: (60) and export:: (13); and lance_surface_guards::compiler_rejects_five_surveyed_lance_virtual_system_columns, which now holds the moved negative cases.

  • cargo fmt --all --check passes.

  • GQT regression: not applicable. This PR fixes no bug and changes no behavior that rows, counts or errors show, so the before/after regression rule does not apply. The existing loader case strict_graph_batch_rejects_ambiguous_or_noncanonical_json_before_effects already covers the refusal of a reserved field (_rowid) at the top level and in data.

  • Exact-head CI on 14ca568c: 28 passed, 4 skipped, 0 failed.

  • Code inspection: lance-core-13.0.0/src/lib.rs (constants and is_system_column), the Lance callers listed above, lance-13.0.0/src/dataset/mem_wal.rs (TOMBSTONE), the compiler name rules (schema/mod.rs, schema/parser.rs, catalog/schema_ir.rs), and every changed call site. Upstream pages read in full: Row-ID lineage (format/table/row_id_lineage), Table schema (format/table/schema), Read/write guide. Also read: AGENTS.md, docs/dev/invariants.md, docs/dev/lance.md, docs/dev/testing.md (owners), docs/dev/documentation.md.

Limits

  • I did not run the full workspace, GQT, DST, clippy or the served and S3/Azure suites. For those I depend on the exact-head CI result above.
  • I did not run a before/after comparison. The change does not change behavior, and the predicate equality is proved by reading the pinned Lance source.
  • O1 (the second part) and O2 are based on code reading only. I did not build a legacy-vintage graph to show them.

@ragnorc
ragnorc merged commit 7b827da into ModernRelay:main Nov 9, 2026
34 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant