Skip to content

fix(load): charge a wide vector its Arrow size in keyed load estimates - #1308

Merged
ragnorc merged 10 commits into
ModernRelay:mainfrom
ragnorc:fix/484-wide-vector-load-estimate
Nov 9, 2026
Merged

ragnorc merged 10 commits into
ModernRelay:mainfrom
ragnorc:fix/484-wide-vector-load-estimate

Conversation

@ragnorc

@ragnorc ragnorc commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

Summary

A keyed load (append or merge) of wide vectors was refused at about half of write_max_bytes. Two admission estimates that run before the Arrow batch is built counted a vector as bigger than Arrow stores it. Both now charge a vector its Arrow size. With the default 32 MiB, about 2,630 Vector(3072) rows now fit in one load, up from about 1,365.

Closes #484

Reproduction

On main f0df46736c3598b6f70d30a6cb01abfa6320d788, schema node Chunk { slug: String @key text: String embedding: Vector(3072) }, local file:// graph, omnigraph load --mode merge:

2,600 rows, vectors present (32.1 MB JSONL, ~32.0 MB of Arrow):
resource limit exceeded for keyed parsed entity bytes for node:Chunk: actual 33572430, limit 33554432
disposition: {"effects":"none","action":"fix"}

1,687 rows, embedding: Vector(3072)? omitted, 4.6 KB of text each (~29 MB of Arrow):
resource limit exceeded for strict_input_arrow_bytes: actual 33565642, limit 33554432
disposition: {"effects":"none","action":"fix"}

Both inputs fit in 32 MiB of Arrow. The second one is the case in the issue comment from 2026-09-19. Verdict: reproduced, with a different cause. The report blamed a chunk planner. On main there is none: a direct load is one operation, bounded by the documented 8,192 entities and write_max_bytes per type. What overshoots is the input estimates.

The new test consistency::wide_vector_loads_fit_write_max_bytes_by_their_arrow_size_issue_484 fails on main:

present=true, strict=false, mode=Append: rows whose Arrow batch fits write_max_bytes must load:
ResourceLimitExceeded { resource: "keyed parsed entity bytes for node:Chunk", limit: 1048576, actual: 1049070 }

Root cause

crates/omnigraph/src/loader/mod.rs bounds a keyed load three times. The authority is MutationStaging::append_batch, which measures the built batch with get_array_memory_size. Two estimates run earlier:

  • Parse estimate (account_keyed_json_row → estimate_json_arrow_bytes). Its documentation says it is a lower bound on the row's Arrow payload. For a JSON array it charged (len + 1) × 4 offset bytes plus the items. Arrow has a per-element offset only for String items. Numeric list items have none, and a FixedSizeList vector has no offsets at all. A Vector(3072) row was charged 24,580 bytes for 12,288 bytes of Arrow. That is twice its size, so it was not a lower bound.
  • Strict projection (preflight_strict_rows_arrow_bytes → projected_strict_column_bytes). For a FixedSizeList it charged 16 + dim × (width + 1): a whole byte per child validity slot, where Arrow uses one bit. That is 15,376 bytes for a Vector(3072) cell, a quarter too much. A null vector really does take its full width: the builder appends dim null children, so its values buffer is full-sized.

Change

  • estimate_json_arrow_bytes: an array charges its items, plus 4 bytes for each String item (its offset in the child Utf8 array). The list's own row offset is left out, as a top-level String's offset already is. For vectors and [String], [I32] and [F32] lists it now equals the item bytes Arrow stores (a list's 4-byte row offset excepted). For 8-byte numbers it is lower. Bool values still count a byte each, as before.
  • projected_strict_column_bytes: a FixedSizeList charges 16 + dim × width + ceil(dim / 8), which is the values plus validity bits. This stays at or above the built batch, so the projection still guards the builder.
  • Docs: docs/user/mutations/index.md and skills/omnigraph/references/data.md say that a vector counts its full width (dimension × 4 bytes) whether present or null. The issue comment asked for this formula.
  • changelog.d/wide-vector-load-estimate.fixed.md, and the regenerated docs/releases/v0.15.0.md snapshot.

Decisions

  • Fix the estimates; do not raise or remove them. Both estimates exist on purpose: the parse estimate bounds the parse spool before the batch exists, and the strict projection bounds the builder allocation. The error was in their arithmetic, not their role. The batch-level check is unchanged and stays authoritative.
  • Vectors only in the strict projection. List items keep their width + 1 projection. Their over-charge is per item and small for the list sizes the schema sees, and changing it would move unrelated limits. This change only touches what the issue covers.
  • No automatic splitting. One load is one atomic commit, and that is documented. Splitting a file into several commits would change the semantics users get, so it is not part of this fix.
  • actual is unchanged. The issue comment notes that actual is the running total at the crossing row, not the input's total. That is how every streaming bound in the loader reports, so changing it is a separate decision. It is noted on the issue.

Before and after

Input (--mode merge, local graph) main this change
2,600 Vector(3072) rows refused (keyed parsed entity bytes) loaded
2,650 Vector(3072) rows refused refused (strict_input_arrow_bytes), nothing published
1,687 null Vector(3072)? rows with 4.6 KB text refused (strict_input_arrow_bytes) loaded
10,000 small rows refused (keyed entities, 8,192 bound) same

Tests

  • crates/omnigraph/tests/integration/consistency.rs: wide_vector_loads_fit_write_max_bytes_by_their_arrow_size_issue_484. write_max_bytes is set to 1 MiB, and the schema is Vector(256)?. 850 rows (about 0.85 MiB of Arrow) load with vectors present and with vectors null, through load_jsonl and load_graph_batch, in append and merge mode. 1,100 rows still refuse with the typed limit, and the manifest version and row count stay the same.

Local results (this box, debug builds, RUST_MIN_STACK=16777216 as CI sets):

  • The new test fails on main (above) and passes with the change.
  • cargo test -p omnigraph-engine --features failpoints --locked: lib 637 passed, 1 failed; integration 662, writes 67, failpoints 144, detached_commit_matrix 29, forbidden_apis 39, lance_surface_guards 73, schema_apply 50, search 36 and the rest all passed. The one failure, db::upgrade::tests::interrupted::unreadable_legacy_object_asks_for_a_rerun, makes a file unreadable with mode 000. This box runs tests as root, and root can still read it. The test does not touch the loader.
  • cargo test -p omnigraph-gqt -p omnigraph-gqt-core --locked (GQT parity against the reference engine): all passed, including the 464 corpus cases.
  • cargo test -p omnigraph-dst --locked from crates/omnigraph-dst: 56 passed, 30 ignored.
  • cargo clippy -p omnigraph-engine --all-targets --locked -- -D warnings -W clippy::dbg_macro and cargo fmt --all --check: clean.
  • After perf(engine): delete a branch merge's rows by the _rowid its walk read #1301 merged, the branch merged main 2e7921c4. Only the release snapshot conflicted: I took main's version and regenerated it. Reran on the merged tree: integration 663 passed, writes 67 passed.
  • After release(notes): link #1301 in the v0.15.0 snapshot #1310 changed the snapshot on main, the branch merged main ead05c40 (eee0aa04). Again only the release snapshot conflicted: I took main's version and regenerated it with scripts/release_notes.py snapshot --target HEAD --date 2026-11-09 --replace (3f2081f0). No code changed in that merge.
  • f6d7d6fc merged main but kept main's snapshot metadata, so check-docs.py failed with "release notes changed after snapshot generation". The branch then merged main 448c3ad7 (4074cf08, no conflict) and regenerated the snapshot with the same command (b340afb7). Reran issue_484 and write_max_bytes integration tests on the merged tree: passed.
  • scripts/check-docs.py, scripts/check-agents-md.sh and scripts/release_notes.py verify --version v0.15.0 --target HEAD: OK.

Risks and what was not verified

  • A load that used to be refused early may now get as far as the batch-level check. That check is the authority, and it is unchanged. For [I64], [F64] and [Date]-shaped lists the parse estimate is now lower than before, but still a lower bound.
  • Not verified: served (omnigraph-server) and managed loads. They use the same loader functions, but no served test was added. A managed load also has its own 32 MiB request-body cap, and that is unchanged.
  • Lance: not involved. Both estimates are Omnigraph code that runs before any Lance call. No pending Lance change affects this.
  • No storage reads or writes change, so the DST storage-cost golden is untouched.

Workflow run: run-wf-reproduce-modernrelay-omnigraph-issue-20261109T0012Z-487b76
Response run: run-wf-respond-modernrelay-omnigraph-pr-20261109T0240Z-803ae5
Response run: run-wf-respond-modernrelay-omnigraph-pr-20261109T0340Z-2f0968

The parse-time lower bound charged a 4-byte offset per array element, so a
Float32 vector counted twice its bytes, and the strict projection charged a
byte per validity bit. A load of Vector(3072) rows was refused at about half
of write_max_bytes.

Closes ModernRelay#484

@ragnorc ragnorc left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review of 10ffb47c: wide-vector keyed load estimates (#484)

Why. A keyed load of wide vectors was refused at about half of write_max_bytes. Two admission estimates that run before the Arrow batch exists counted a vector as larger than Arrow stores it.

What. The parse estimate (estimate_json_arrow_bytes) no longer charges a 4-byte offset for each list element. It charges 4 bytes only for each String item. The strict projection (projected_strict_column_bytes) charges one validity bit, not one byte, for each FixedSizeList child slot.

Impact. With the default 32 MiB, a direct keyed load now admits about 2,600 Vector(3072) rows, up from about 1,365. The batch check (MutationStaging::append_batch, get_array_memory_size) is unchanged and stays the authority.

Verdict. No blocking defects. The arithmetic is correct for vectors and numeric lists. The fix is at the layer that owns the defect, and it adds no new state, API or invariant. I ask for one GQT case (below). The other findings are optional.

Blocking

None.

Requested: add the GQT regression

The defect shows in an error and a row count. GQT can express it, because load generate has a vector column kind and a --- mutate step can set write_max_bytes. Under the testing rule, that makes GQT the cheapest tier. Please add this case as crates/omnigraph-gqt/cases/issue_484_wide_vector_load_fits_write_max_bytes.gqt:

# issue: 484
# red_on: main @ 2e7921c4: the 850-row load is refused with `keyed parsed entity bytes for node:Chunk` (the parse estimate charged an offset per vector element).
# notes: A keyed load of vectors is charged their Arrow size. With write_max_bytes at 1 MiB, 850 Vector(256) rows (about 0.85 MiB of Arrow) load; 1,100 rows are refused with the typed limit and publish nothing.
# notes: No omnigraph-server environment: served runs refuse settings steps and the HTTP load routes take no setting.

--- runner
timeout_ms: 10000
environments:
  - target: omnigraph-engine
    storage: local-filesystem

--- schema
node Chunk {
    slug: String @key
    embedding: Vector(256)?
}

--- seed
{"type":"Chunk","data":{"slug":"seed"}}

--- mutate
set write_max_bytes = 1048576;

--- expect ok

--- load generate: v1 seed: 1 mode: merge
tables:
  - kind: node
    name: Chunk
    rows: 850
    commits: 1
    columns:
      slug: {kind: key, prefix: c, width: 4}
      embedding: {kind: vector, dimensions: 256}
--- expect ok

--- query
query fitting_rows() { match { $c: Chunk } return { count($c) as n } }
--- expect unordered
{"n": 851}
--- expect shape
n: I64

--- load generate: v1 seed: 2 mode: merge
tables:
  - kind: node
    name: Chunk
    rows: 1100
    commits: 1
    columns:
      slug: {kind: key, prefix: d, width: 4}
      embedding: {kind: vector, dimensions: 256}
--- expect error: keyed parsed entity bytes

--- query
query rows_after_refusal() { match { $c: Chunk } return { count($c) as n } }
--- expect unordered
{"n": 851}
--- expect shape
n: I64

The Rust test consistency::wide_vector_loads_fit_write_max_bytes_by_their_arrow_size_issue_484 still has a reason to stay. It reaches the strict door (load_graph_batch) and the null-vector rows, and GQT cannot reach those with a session setting. Please say that in its doc comment and cross-reference the GQT case.

Optional findings

1. The class is closed for vectors, but not for the whole estimate. The violated invariant is "the parse estimate is a lower bound on the Arrow payload" (its doc comment says so). The estimate reads only the JSON shape, not the property type, so other shapes still break that bound:

  • A Date written as an ISO string (the form the docs prefer) costs 10 bytes, but Arrow stores 4. A DateTime string costs 20 bytes, but Arrow stores 8. A [Date] item costs 14 bytes for 4 stored. The PR body says [Date] lists are "still a lower bound". That is true only for epoch-day integers.
  • A Bool costs 1 byte, but Arrow stores 1 bit. This applies at top level and in lists.

These are smaller than the vector case and need wide or list-heavy rows to matter, so they do not block this PR. But they come from the same cause. account_keyed_json_row already receives properties: &HashMap<String, PropType> and uses it for Blob. If the estimate took each property's Arrow type, as the strict projection already does, the lower bound would hold by construction for every type. Without that, each new shape needs its own arithmetic patch. After five more changes like this one, we would have five special cases in a function that does not know types.

2. One quantity, several owners. The Arrow size of a cell is computed in four places: the parse estimate (type-blind lower bound), the strict projection (typed upper bound, projected_strict_*), the strict vector allocation guard in build_column_from_json (rows × dim × 4), and the authority (get_array_memory_size). The first three keep separate width tables, which can drift. One small module could own them: one typed width table and one function per bound direction. It would make item 1 a one-line change and remove the drift risk. Its cost is one property lookup for each field.

3. Memory tradeoff of the parse spool (please state it under Risks). Phase 1 keeps every row as serde_json::Value until the batches are built. The doc comment says the estimate is "not a JSON DOM bound". But for a direct load, this estimate and the 8,192-row cap are the only whole-load limits before the spool fills. I measured size_of::<serde_json::Value>() = 72 bytes with this workspace's features (preserve_order is on). So one Vector(3072) row in the spool is about 221 KB, while its Arrow cells are 12 KB. If the maximum admitted rows go from about 1,365 to about 2,630, the peak spool for one such load goes from about 302 MB to about 582 MB. This is the intended effect of the fix, not a defect. The long-term fix that removes this liability is to append rows into Arrow builders as they are parsed, not into a JSON spool. That change would also remove the need for the parse estimate.

4. Doc scope. references/data.md now says "about 2,600 Vector(3072) rows fit one load". That is true for a direct load. The served /load route caps the request body at 32 MiB (INGEST_REQUEST_BODY_LIMIT_BYTES), and the PR body says a managed load has its own 32 MiB body cap. Real embeddings serialize at about 10 to 12 bytes per float, so one such body holds about 1,000 rows of Vector(3072). The repro reached 2,600 rows in 32.1 MB only because its values were short (about 4 bytes per float). Consider saying that the body cap applies first for served and managed loads.

5. Wording of the projection bound. "This stays at or above the built batch" is true per row at scale. It is not exact for tiny batches: Arrow rounds each buffer up to 64 bytes, and one Vector(1) row projects 21 bytes. The old formula had the same gap, and the batch check is the authority, so nothing breaks. Consider "per row, at or above the built batch's buffers".

Tradeoffs and liability

  • Removed: a false refusal at about half (parse) and four fifths (strict) of the documented limit for every vector load. No new state, API, setting or invariant was added. The diff is small (+25/−13 in the loader).
  • Added: a higher peak parse-spool memory for vector loads (item 3), and the same type-blind estimate design (items 1 and 2).
  • Alignment: none of the other open PRs changes these estimates. #1291 edits load planning in the same file, and #1101 edits the same docs page in another section. Lance is not involved: both estimates run before any Lance call, and the authority measures Arrow buffers.

Validation (this review)

Isolated worktree at 10ffb47c, debug build, RUST_MIN_STACK=16777216. For "before", I reverted only crates/omnigraph/src/loader/mod.rs to the merge base 2e7921c4 and kept the new tests.

Check 2e7921c4 loader 10ffb47c
integration consistency::wide_vector_loads_fit_write_max_bytes_by_their_arrow_size_issue_484 --exact fails: present=true, strict=false, mode=Append … ResourceLimitExceeded { resource: "keyed parsed entity bytes for node:Chunk", limit: 1048576, actual: 1049070 } passes
omnigraph-gqt <case above> fails at step 2 (load): keyed parsed entity bytes for node:Chunk: actual 1049070, limit 1048576 ok in 2.83 s
integration consistency:: not run 25 passed
engine lib loader:: not run 52 passed

Both regressions fail before the fix for the predicted reason (the parse estimate) and pass after it. Exact-commit CI for 10ffb47c is green: all 28 non-skipped checks pass, including Test Workspace, GQ Logic Tests, DST pinned suite, both RustFS jobs and Azurite. The memory figures in item 3 are a measured size_of multiplied out, not a measured RSS. Items 1, 4 and 5 come from code inspection. I did not run served or managed loads. I made no changes to the PR branch.

@ragnorc

ragnorc commented Nov 9, 2026

Copy link
Copy Markdown
Contributor Author

I merged main (ead05c40) into this branch to clear a merge conflict. The branch can be merged again.

@ragnorc

ragnorc commented Nov 9, 2026

Copy link
Copy Markdown
Contributor Author

I fixed the failing Check AGENTS.md Links check on f6d7d6fc. CI is running on the new head b340afb7.

  • Cause: f6d7d6fc merged main but kept main's metadata line in docs/releases/v0.15.0.md, the generated release snapshot. That line did not list this PR's note, so scripts/check-docs.py failed with "release notes changed after snapshot generation; regenerate it".
  • Fix: 4074cf08 merges main (448c3ad7), with no conflict. b340afb7 regenerates the snapshot with python3 scripts/release_notes.py snapshot --target HEAD --date 2026-11-09 --replace. Nobody edited it by hand. Compared with main, the snapshot adds only this PR's fixed note.
  • No code changed. I did not rebase or force-push.
  • Checks on b340afb7: scripts/check-docs.py, scripts/check-agents-md.sh, release_notes.py verify --version v0.15.0 and scripts/check-release-note.py pass. cargo fmt --all --check is clean. The issue_484 and write_max_bytes integration tests pass on the merged tree.

@ragnorc
ragnorc added this pull request to the merge queue Nov 9, 2026
Merged via the queue into ModernRelay:main with commit 3f249cc Nov 9, 2026
33 of 34 checks passed
ragnorc added a commit to ragnorc/omnigraph that referenced this pull request Nov 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

CLI: load chunk planner overshoots the 32 MiB staging cap on wide vectors — single-file load of 3072-dim embeddings refused

1 participant