Repository navigation
fix(load): charge a wide vector its Arrow size in keyed load estimates - #1308
Conversation
The parse-time lower bound charged a 4-byte offset per array element, so a Float32 vector counted twice its bytes, and the strict projection charged a byte per validity bit. A load of Vector(3072) rows was refused at about half of write_max_bytes. Closes ModernRelay#484
…oad-estimate # Conflicts: # docs/releases/v0.15.0.md
ragnorc
left a comment
There was a problem hiding this comment.
Review of 10ffb47c: wide-vector keyed load estimates (#484)
Why. A keyed load of wide vectors was refused at about half of write_max_bytes. Two admission estimates that run before the Arrow batch exists counted a vector as larger than Arrow stores it.
What. The parse estimate (estimate_json_arrow_bytes) no longer charges a 4-byte offset for each list element. It charges 4 bytes only for each String item. The strict projection (projected_strict_column_bytes) charges one validity bit, not one byte, for each FixedSizeList child slot.
Impact. With the default 32 MiB, a direct keyed load now admits about 2,600 Vector(3072) rows, up from about 1,365. The batch check (MutationStaging::append_batch, get_array_memory_size) is unchanged and stays the authority.
Verdict. No blocking defects. The arithmetic is correct for vectors and numeric lists. The fix is at the layer that owns the defect, and it adds no new state, API or invariant. I ask for one GQT case (below). The other findings are optional.
Blocking
None.
Requested: add the GQT regression
The defect shows in an error and a row count. GQT can express it, because load generate has a vector column kind and a --- mutate step can set write_max_bytes. Under the testing rule, that makes GQT the cheapest tier. Please add this case as crates/omnigraph-gqt/cases/issue_484_wide_vector_load_fits_write_max_bytes.gqt:
# issue: 484
# red_on: main @ 2e7921c4: the 850-row load is refused with `keyed parsed entity bytes for node:Chunk` (the parse estimate charged an offset per vector element).
# notes: A keyed load of vectors is charged their Arrow size. With write_max_bytes at 1 MiB, 850 Vector(256) rows (about 0.85 MiB of Arrow) load; 1,100 rows are refused with the typed limit and publish nothing.
# notes: No omnigraph-server environment: served runs refuse settings steps and the HTTP load routes take no setting.
--- runner
timeout_ms: 10000
environments:
- target: omnigraph-engine
storage: local-filesystem
--- schema
node Chunk {
slug: String @key
embedding: Vector(256)?
}
--- seed
{"type":"Chunk","data":{"slug":"seed"}}
--- mutate
set write_max_bytes = 1048576;
--- expect ok
--- load generate: v1 seed: 1 mode: merge
tables:
- kind: node
name: Chunk
rows: 850
commits: 1
columns:
slug: {kind: key, prefix: c, width: 4}
embedding: {kind: vector, dimensions: 256}
--- expect ok
--- query
query fitting_rows() { match { $c: Chunk } return { count($c) as n } }
--- expect unordered
{"n": 851}
--- expect shape
n: I64
--- load generate: v1 seed: 2 mode: merge
tables:
- kind: node
name: Chunk
rows: 1100
commits: 1
columns:
slug: {kind: key, prefix: d, width: 4}
embedding: {kind: vector, dimensions: 256}
--- expect error: keyed parsed entity bytes
--- query
query rows_after_refusal() { match { $c: Chunk } return { count($c) as n } }
--- expect unordered
{"n": 851}
--- expect shape
n: I64
The Rust test consistency::wide_vector_loads_fit_write_max_bytes_by_their_arrow_size_issue_484 still has a reason to stay. It reaches the strict door (load_graph_batch) and the null-vector rows, and GQT cannot reach those with a session setting. Please say that in its doc comment and cross-reference the GQT case.
Optional findings
1. The class is closed for vectors, but not for the whole estimate. The violated invariant is "the parse estimate is a lower bound on the Arrow payload" (its doc comment says so). The estimate reads only the JSON shape, not the property type, so other shapes still break that bound:
- A
Datewritten as an ISO string (the form the docs prefer) costs 10 bytes, but Arrow stores 4. ADateTimestring costs 20 bytes, but Arrow stores 8. A[Date]item costs 14 bytes for 4 stored. The PR body says[Date]lists are "still a lower bound". That is true only for epoch-day integers. - A
Boolcosts 1 byte, but Arrow stores 1 bit. This applies at top level and in lists.
These are smaller than the vector case and need wide or list-heavy rows to matter, so they do not block this PR. But they come from the same cause. account_keyed_json_row already receives properties: &HashMap<String, PropType> and uses it for Blob. If the estimate took each property's Arrow type, as the strict projection already does, the lower bound would hold by construction for every type. Without that, each new shape needs its own arithmetic patch. After five more changes like this one, we would have five special cases in a function that does not know types.
2. One quantity, several owners. The Arrow size of a cell is computed in four places: the parse estimate (type-blind lower bound), the strict projection (typed upper bound, projected_strict_*), the strict vector allocation guard in build_column_from_json (rows × dim × 4), and the authority (get_array_memory_size). The first three keep separate width tables, which can drift. One small module could own them: one typed width table and one function per bound direction. It would make item 1 a one-line change and remove the drift risk. Its cost is one property lookup for each field.
3. Memory tradeoff of the parse spool (please state it under Risks). Phase 1 keeps every row as serde_json::Value until the batches are built. The doc comment says the estimate is "not a JSON DOM bound". But for a direct load, this estimate and the 8,192-row cap are the only whole-load limits before the spool fills. I measured size_of::<serde_json::Value>() = 72 bytes with this workspace's features (preserve_order is on). So one Vector(3072) row in the spool is about 221 KB, while its Arrow cells are 12 KB. If the maximum admitted rows go from about 1,365 to about 2,630, the peak spool for one such load goes from about 302 MB to about 582 MB. This is the intended effect of the fix, not a defect. The long-term fix that removes this liability is to append rows into Arrow builders as they are parsed, not into a JSON spool. That change would also remove the need for the parse estimate.
4. Doc scope. references/data.md now says "about 2,600 Vector(3072) rows fit one load". That is true for a direct load. The served /load route caps the request body at 32 MiB (INGEST_REQUEST_BODY_LIMIT_BYTES), and the PR body says a managed load has its own 32 MiB body cap. Real embeddings serialize at about 10 to 12 bytes per float, so one such body holds about 1,000 rows of Vector(3072). The repro reached 2,600 rows in 32.1 MB only because its values were short (about 4 bytes per float). Consider saying that the body cap applies first for served and managed loads.
5. Wording of the projection bound. "This stays at or above the built batch" is true per row at scale. It is not exact for tiny batches: Arrow rounds each buffer up to 64 bytes, and one Vector(1) row projects 21 bytes. The old formula had the same gap, and the batch check is the authority, so nothing breaks. Consider "per row, at or above the built batch's buffers".
Tradeoffs and liability
- Removed: a false refusal at about half (parse) and four fifths (strict) of the documented limit for every vector load. No new state, API, setting or invariant was added. The diff is small (+25/−13 in the loader).
- Added: a higher peak parse-spool memory for vector loads (item 3), and the same type-blind estimate design (items 1 and 2).
- Alignment: none of the other open PRs changes these estimates. #1291 edits load planning in the same file, and #1101 edits the same docs page in another section. Lance is not involved: both estimates run before any Lance call, and the authority measures Arrow buffers.
Validation (this review)
Isolated worktree at 10ffb47c, debug build, RUST_MIN_STACK=16777216. For "before", I reverted only crates/omnigraph/src/loader/mod.rs to the merge base 2e7921c4 and kept the new tests.
| Check | 2e7921c4 loader |
10ffb47c |
|---|---|---|
integration consistency::wide_vector_loads_fit_write_max_bytes_by_their_arrow_size_issue_484 --exact |
fails: present=true, strict=false, mode=Append … ResourceLimitExceeded { resource: "keyed parsed entity bytes for node:Chunk", limit: 1048576, actual: 1049070 } |
passes |
omnigraph-gqt <case above> |
fails at step 2 (load): keyed parsed entity bytes for node:Chunk: actual 1049070, limit 1048576 |
ok in 2.83 s |
integration consistency:: |
not run | 25 passed |
engine lib loader:: |
not run | 52 passed |
Both regressions fail before the fix for the predicted reason (the parse estimate) and pass after it. Exact-commit CI for 10ffb47c is green: all 28 non-skipped checks pass, including Test Workspace, GQ Logic Tests, DST pinned suite, both RustFS jobs and Azurite. The memory figures in item 3 are a measured size_of multiplied out, not a measured RSS. Items 1, 4 and 5 come from code inspection. I did not run served or managed loads. I made no changes to the PR branch.
|
I merged
|
|
I fixed the failing
|
Summary
A keyed
load(appendormerge) of wide vectors was refused at about half ofwrite_max_bytes. Two admission estimates that run before the Arrow batch is built counted a vector as bigger than Arrow stores it. Both now charge a vector its Arrow size. With the default 32 MiB, about 2,630Vector(3072)rows now fit in one load, up from about 1,365.Closes #484
Reproduction
On main
f0df46736c3598b6f70d30a6cb01abfa6320d788, schemanode Chunk { slug: String @key text: String embedding: Vector(3072) }, localfile://graph,omnigraph load --mode merge:Both inputs fit in 32 MiB of Arrow. The second one is the case in the issue comment from 2026-09-19. Verdict: reproduced, with a different cause. The report blamed a chunk planner. On main there is none: a direct load is one operation, bounded by the documented 8,192 entities and
write_max_bytesper type. What overshoots is the input estimates.The new test
consistency::wide_vector_loads_fit_write_max_bytes_by_their_arrow_size_issue_484fails on main:Root cause
crates/omnigraph/src/loader/mod.rsbounds a keyed load three times. The authority isMutationStaging::append_batch, which measures the built batch withget_array_memory_size. Two estimates run earlier:account_keyed_json_row→estimate_json_arrow_bytes). Its documentation says it is a lower bound on the row's Arrow payload. For a JSON array it charged(len + 1) × 4offset bytes plus the items. Arrow has a per-element offset only for String items. Numeric list items have none, and aFixedSizeListvector has no offsets at all. AVector(3072)row was charged 24,580 bytes for 12,288 bytes of Arrow. That is twice its size, so it was not a lower bound.preflight_strict_rows_arrow_bytes→projected_strict_column_bytes). For aFixedSizeListit charged16 + dim × (width + 1): a whole byte per child validity slot, where Arrow uses one bit. That is 15,376 bytes for aVector(3072)cell, a quarter too much. A null vector really does take its full width: the builder appendsdimnull children, so its values buffer is full-sized.Change
estimate_json_arrow_bytes: an array charges its items, plus 4 bytes for each String item (its offset in the child Utf8 array). The list's own row offset is left out, as a top-level String's offset already is. For vectors and[String],[I32]and[F32]lists it now equals the item bytes Arrow stores (a list's 4-byte row offset excepted). For 8-byte numbers it is lower. Bool values still count a byte each, as before.projected_strict_column_bytes: aFixedSizeListcharges16 + dim × width + ceil(dim / 8), which is the values plus validity bits. This stays at or above the built batch, so the projection still guards the builder.docs/user/mutations/index.mdandskills/omnigraph/references/data.mdsay that a vector counts its full width (dimension × 4 bytes) whether present or null. The issue comment asked for this formula.changelog.d/wide-vector-load-estimate.fixed.md, and the regenerateddocs/releases/v0.15.0.mdsnapshot.Decisions
Listitems keep theirwidth + 1projection. Their over-charge is per item and small for the list sizes the schema sees, and changing it would move unrelated limits. This change only touches what the issue covers.actualis unchanged. The issue comment notes thatactualis the running total at the crossing row, not the input's total. That is how every streaming bound in the loader reports, so changing it is a separate decision. It is noted on the issue.Before and after
--mode merge, local graph)Vector(3072)rowskeyed parsed entity bytes)Vector(3072)rowsstrict_input_arrow_bytes), nothing publishedVector(3072)?rows with 4.6 KB textstrict_input_arrow_bytes)keyed entities, 8,192 bound)Tests
crates/omnigraph/tests/integration/consistency.rs:wide_vector_loads_fit_write_max_bytes_by_their_arrow_size_issue_484.write_max_bytesis set to 1 MiB, and the schema isVector(256)?. 850 rows (about 0.85 MiB of Arrow) load with vectors present and with vectors null, throughload_jsonlandload_graph_batch, inappendandmergemode. 1,100 rows still refuse with the typed limit, and the manifest version and row count stay the same.Local results (this box, debug builds,
RUST_MIN_STACK=16777216as CI sets):cargo test -p omnigraph-engine --features failpoints --locked: lib 637 passed, 1 failed; integration 662, writes 67, failpoints 144, detached_commit_matrix 29, forbidden_apis 39, lance_surface_guards 73, schema_apply 50, search 36 and the rest all passed. The one failure,db::upgrade::tests::interrupted::unreadable_legacy_object_asks_for_a_rerun, makes a file unreadable with mode000. This box runs tests as root, and root can still read it. The test does not touch the loader.cargo test -p omnigraph-gqt -p omnigraph-gqt-core --locked(GQT parity against the reference engine): all passed, including the 464 corpus cases.cargo test -p omnigraph-dst --lockedfromcrates/omnigraph-dst: 56 passed, 30 ignored.cargo clippy -p omnigraph-engine --all-targets --locked -- -D warnings -W clippy::dbg_macroandcargo fmt --all --check: clean.2e7921c4. Only the release snapshot conflicted: I took main's version and regenerated it. Reran on the merged tree: integration 663 passed, writes 67 passed.ead05c40(eee0aa04). Again only the release snapshot conflicted: I took main's version and regenerated it withscripts/release_notes.py snapshot --target HEAD --date 2026-11-09 --replace(3f2081f0). No code changed in that merge.f6d7d6fcmergedmainbut keptmain's snapshot metadata, socheck-docs.pyfailed with "release notes changed after snapshot generation". The branch then merged main448c3ad7(4074cf08, no conflict) and regenerated the snapshot with the same command (b340afb7). Reranissue_484andwrite_max_bytesintegration tests on the merged tree: passed.scripts/check-docs.py,scripts/check-agents-md.shandscripts/release_notes.py verify --version v0.15.0 --target HEAD: OK.Risks and what was not verified
[I64],[F64]and[Date]-shaped lists the parse estimate is now lower than before, but still a lower bound.omnigraph-server) and managed loads. They use the same loader functions, but no served test was added. A managed load also has its own 32 MiB request-body cap, and that is unchanged.Workflow run: run-wf-reproduce-modernrelay-omnigraph-issue-20261109T0012Z-487b76
Response run: run-wf-respond-modernrelay-omnigraph-pr-20261109T0240Z-803ae5
Response run: run-wf-respond-modernrelay-omnigraph-pr-20261109T0340Z-2f0968