Skip to content

perf(storage): reach a keyed write's rows through the id index - #1318

Merged
ragnorc merged 18 commits into
ModernRelay:mainfrom
ragnorc:perf/keyed-writes-through-id-index
Nov 12, 2026
Merged

ragnorc merged 18 commits into
ModernRelay:mainfrom
ragnorc:perf/keyed-writes-through-id-index

Conversation

@ragnorc

@ragnorc ragnorc commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

Closes #1248. On an indexed table, a one-row keyed write read the id column of every fragment, so its cost grew with the table. A keyed insert, a load --mode merge and a Blob cell write stage as Upsert, which forces Lance's full-table merge join (use_index(false)). An update ... where <key> = v also scanned the key property, a user column the id BTREE does not index, before it staged. At 262,144 rows in 64 fragments, a one-row update read 4.78 MB in 150 to 212 storage requests; a keyed insert read 2.1 MB in 83.

Two changes. An update whose predicate has <key> = v on a single String @key adds the derived id = v conjunct to its match scan when the table has an id BTREE, reusing the conjunct a node delete already adds (key_id_conjunct). In TableStore::stage_keyed_write, an Upsert first runs the exact-id probe strict insert already runs: all new, it stages through the absence-proven insert writer; all present, it stages as a known-present update; mixed, it keeps the join. The guards: the probe runs only while the fragments the BTREE leaves uncovered times the source ids stay within one keyed batch (8,192), since Lance plans the IN list once per uncovered fragment (#1225's 11 GB); and the known-present switch happens only while the BTREE covers more fragments than it leaves (requests) and the uncovered fragments' recorded data files hold at most 1 MiB (KNOWN_PRESENT_TAIL_MAX_BYTES, bytes), because that indexed merge scans each uncovered fragment for every source column, where the join reads only the id column of every fragment. An unrecorded file size keeps the join. Without a BTREE nothing changes. key_column_uncovered_tail reports the tail's fragment count and data-file bytes under the rule key_column_coverage already applies; both read one helper.

Measured with GQT --measure on the DST store: a Person { id @key, name, v } table seeded with N rows in 4,096-row commits and indexed, then seven one-row steps. At N = 262,144, main → this branch: column-patch update 212 → 34 requests and 4,782,278 → 93,360 bytes; whole-row update 150 → 28 and 4,783,976 → 43,664; keyed insert 83 → 18 and 2,108,519 → 19,173; merge of one present row 76 → 17 and 2,088,712 → 14,522; merge of one new row 84 → 28 and 2,108,917 → 76,255. From 1,024 to 262,144 rows these now grow only with the index's pages and the uncovered tail. On an unindexed table every step at every size is identical to main. upsert_skips_the_merge_join_when_the_id_btree_decides_its_rows_issue_1248 fails on main. The DST golden adds KeyedInsert, KeyedUpdate, MergeLoadPresent and MergeLoadNew at 2 and 16 fragments; their l.get rows on main were 10/25, 13/41, 10/24 and 10/24, here 9/10, 8/8, 8/8 and 4/4 since main afc4be7 was merged in (before that merge: 9/10, 11/11, 9/9 and 7/5). Main's #1291 keeps the dataset a write commits in the table handle cache, so each write after the first on the scenario's handle opens no manifest; the first write, KeyedInsert, is unchanged. One existing line moves: InsertLegacy l.get 15 → 13, its keyed insert now taking the fenced insert. write_cost::keyed_insert_routes_through_fenced_adapter_only now asserts the fenced insert and no merge on the indexed standard graph.

For review: the decision to challenge is the coverage rule for the known-present switch. It is a cost choice made from pinned fragment metadata, never a correctness precondition. A wrong choice costs reads, not rows, and both of its costs are bounded: the fragment rule bounds the extra requests and the byte budget bounds the extra bytes whatever the row width. Under GQT's s3-like model this lowers requests and bytes but not latency: at 262,144 rows the first update on a handle is 90 ms slower (370 → 460) because an index lookup is a chain of dependent reads while the model runs a scan's 150+ GETs at once with no queue; later updates and inserts are 18 to 36 ms faster, merges 18 to 36 ms slower. No wall clock or S3 run backs either direction. With a 64-fragment uncovered tail on 16,384 rows, a whole-row update keeps 98 requests and reads 375,488 → 268,693 bytes, modeled 372 → 426 ms. The probe's peak memory at the 8,192-id bound is argued from #1231, not measured. Mixed upserts still take the join; splitting them into an insert and an update on one table is not in this change. changelog.d/release.json on main already targets v0.15.1, so this PR does not change it. Locally the engine binaries pass except db::upgrade::tests::interrupted::unreadable_legacy_object_asks_for_a_rerun, which needs a non-root user. Report: https://github.com/ragnorc/omnigraph/blob/autoresearch/keyed-write-scans-20261109/docs/autoresearch/keyed-write-scans-20261109.md

Workflow run: run-wf-autoresearch-modernrelay-omnigraph-20261109T0900Z-d8451b
Response run: run-wf-respond-modernrelay-omnigraph-pr-20261109T2040Z-957a8c
Response run: run-wf-respond-modernrelay-omnigraph-pr-20261112T0532Z-b59648
Response run: run-wf-respond-modernrelay-omnigraph-pr-20261112T1412Z-63cfb9
Response run: run-wf-respond-modernrelay-omnigraph-pr-20261112T1942Z-e93b73

A `<key> = v` conjunct also selects the derived `id = v`, so an update's
match scan adds that conjunct when the pinned table has an id BTREE, as a
node delete already does. The scan then reads the index instead of the key
column of every fragment.
…join

Lance's insert-capable merge-insert route joins the source against every
id of the table. With an id BTREE, the exact-id probe a strict insert runs
decides whether an upsert's rows are all new or all present; the first
stages through the absence-proven insert writer, the second as a
known-present update. A mixed batch, or a table without the BTREE, keeps
the join.
KeyedInsert, KeyedUpdate, MergeLoadPresent and MergeLoadNew on an indexed
table of 2 and 16 fragments. On main their l.get rows were 10/25, 13/41,
10/24 and 10/24; now 9/10, 11/11, 9/9 and 5/5. InsertLegacy l.get moves
15 -> 13: its keyed insert takes the absence-proven writer.
… index coverage

The exact-id probe plans its IN list once per fragment the id BTREE does
not cover, so it runs only while those copies hold no more ids than one
keyed batch (ModernRelay#1225). An all-present upsert's indexed merge reads each
uncovered fragment once more, where the join reads every fragment once,
so it switches only while the BTREE covers more fragments than it leaves.
key_column_uncovered_fragments counts the tail under the coverage rule
key_column_coverage already applies; both read one helper.

@ragnorc ragnorc left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Summary

Why. On a table with an id BTREE, a one-row keyed write read the id column of every fragment. The cost grew with the table (#1248).

What. Two changes. (1) An update ... where <key> = v adds the derived __id = v conjunct to its match scan, the same conjunct a node delete already uses. (2) An Upsert first runs the exact-id probe. All new: it stages through the absence-proven insert writer. All present: it stages as KnownPresentUpdate, which takes Lance's indexed merge route. Mixed: it keeps the forced join.

Impact. The all-new route and the update conjunct are sound, and they remove real cost. The all-present switch is not safe yet. Its rule counts fragments, but Lance's indexed route reads every source column of each uncovered fragment. The forced join reads only id. On wide rows (vectors, long text), one uncovered fragment can make a one-row write read the whole fragment. Details and a measurement are in B1.

I reviewed 4941a859. The later head 761873bc only retargets changelog.d/release.json to v0.15.1, so this review covers it.

Blocking

B1. The all-present switch reads whole rows of the uncovered tail

table_store.rs:3689 switches to KnownPresentUpdate while uncovered < covered. That stages without use_index(false), so Lance 13 takes create_indexed_scan_joined_stream. For the fragments the BTREE does not cover, that route scans with .project(&column_names): all source columns, not only the key (lance-13.0.0/src/dataset/write/merge_insert.rs, lines 1421-1432). The source of a load --mode merge, a Blob cell write and a whole-row update is the full row. The forced route (create_plan, line 2419) joins on id and reads only that column from the target.

So the rule compares the wrong quantity. It compares fragment counts. The real cost is (uncovered rows × full row width) against (all rows × id width).

Measured (temporary in-crate test, not pushed). Table: 8 fragments × 512 rows, id BTREE over them, then 1 or 3 appended 512-row fragments that the BTREE does not cover. Operation: one stage_keyed_write(Upsert) of one present row, reads counted on the dataset's object store (local reader plus wrapper):

row payload uncovered fragments main 373834b4 (join) this PR (known-present)
8 B 1 27 reads, 87,912 B 16 reads, 94,526 B
8 B 3 33 reads, 107,448 B 26 reads, 137,950 B
4 KiB 1 27 reads, 87,912 B 17 reads, 2,088,964 B
4 KiB 3 33 reads, 107,448 B 27 reads, 6,123,198 B

The join's bytes do not change with row width. The switch's bytes grow with the width and the size of the tail: 24× and 57× more here. The GQT and DST fixtures in this PR have narrow rows, so they cannot show this. A table like the one in #1225 (Vector(3072), about 12 KB per row, 8,000-row load fragments) would read about 100 MB per uncovered fragment for a one-row update. The PR says "a wrong choice costs requests, not rows". That is true, but the cost here has no bound in row width. Every write leaves an uncovered fragment, so this is the normal state between index builds.

Proposed fix. Take the switch only when the BTREE leaves no fragment uncovered (uncovered == 0). Then the indexed route has no tail scan and always reads less than the join. This is also the rule #1231 already uses for committed lookups (coverage_validate_access, cost.rs:651: probe only under full coverage), so there is one rule, not two. Cost of this choice: with a tail, an all-present upsert takes the join until index build. The all-new route is not affected, because its probe reads only id of the tail. A byte-weighted rule (tail rows × estimated row width) keeps more wins, but it needs a row-width estimate that strings cannot supply, and it adds a model that can drift. I do not recommend it.

Regression. Add a row to upsert_skips_the_merge_join_when_the_id_btree_decides_its_rows_issue_1248: a table with an uncovered fragment, an all-present upsert, expected route (1, 0, 0). With the fix, the 2-fragment DST rows already show this (MergeLoadPresentFragments2).

Optional

O1. One owner for "probe or scan" by exact id. After this PR there are three rules for the same question (an IN (...) lookup on __id against a partly indexed table): coverage_validate_access (full coverage), choose_validate_access (cost by rows), and this PR's uncovered × ids ≤ 8,192. The strict-insert preflight (first_existing_id) runs the same probe with no bound, as your report also says. After five more changes like this, there would be five rules. One function in dataset_index that returns the access for an exact-id lookup (coverage plus key count) could serve the validator, the strict preflight and this probe. That removes the unbounded preflight case too. It is a follow-up, not a condition of this PR.

O2. The IN-list bound has no test. I found no test where uncovered × source ids > 8,192 keeps the join. A change that drops the .filter near table_store.rs:3664 would likely still pass the current tests. One more row in the staged test (for example 2 uncovered fragments and 4,097 ids) would pin it.

O3. Duplicate scan. existing_id_count (table_store.rs:4105) repeats the body of first_existing_id (same filter, same scan_stream_with, same id decode). One helper that streams the present ids, with two thin callers, keeps the two probes from drifting.

O4. Mixed batches pay twice. A mixed upsert now runs the probe and then the full join. That is the common shape of a re-load with load --mode merge. This is not measured. Your report's follow-up (stage the absent rows as a strict insert and the present rows as a known-present update, as branch merge does) would remove it.

O5. Changelog wording. "its cost stops growing with the table" is true only with an id index and a small tail. Suggest: "on a table with an id index, a one-row keyed insert, update ... where <key> = v or load --mode merge reads its rows through the index ...".

Correctness checks that hold

  • The probe and the staging use the same pinned Dataset, so the all-new and all-present decisions cannot go stale inside one attempt. A retry starts from a new snapshot.
  • validate_keyed_write_batch_ids refuses duplicate ids, so present == source_ids.len() means all present.
  • The all-new route still materializes external Blob URIs (prepare_keyed_write_batch in stage_absence_proven_strict_insert) and certifies absence, as the merge route did. It never calls the strict preflight, so an upsert cannot return KeyConflict.
  • The update conjunct (mutation.rs:1955) also reaches pending rows in the non-sole path (scan_with_pending). Those rows carry ids derived from their keys, so the selected rows are the same.

Liability

The all-new route and the update conjunct reuse existing writers and an existing helper. They remove cost and add almost no new state. The all-present switch adds a cost decision from fragment metadata in a third place (O1), and in its current form that decision is wrong for wide rows (B1). With uncovered == 0, the decision becomes the same fact #1231 already uses, and the liability stays small. lance#10606 (inserted keys on the indexed route) could later remove use_index(false), but it does not change the tail projection, so B1 does not go away with that bump.

Validation

  • Local, 4941a859: cargo dev-test --lib -- table_store::staged_tests::upsert_skips table_store::staged_tests::keyed_ → 6 passed, including upsert_skips_the_merge_join_when_the_id_btree_decides_its_rows_issue_1248.
  • Local, main 373834b4 with only this PR's staged_tests.rs applied: the new test fails at all new: the fenced insert with left: (1, 0, 0), the predicted reason. A Rust test is the right owner; GQT cannot assert the route or the read count.
  • Local, both commits: the temporary width test above (cargo dev-test --lib -p omnigraph-engine -- tmp_review_1318 --nocapture). All temporary edits are removed.
  • CI at 4941a859: Test Workspace, DST pinned suite, GQT (ordinary, served, dst), Clippy and RustFS pass; Check AGENTS.md Links failed on the release-note snapshot, which 761873bc fixes. At 761873bc: Dependency Guard (cargo deny) fails on RUSTSEC-2026-0457 and RUSTSEC-2026-0461, new advisories in the existing rmcp dependency, not this change. Storage Upgrade Compatibility was still pending.
  • Read: AGENTS.md, invariants.md, lance.md (Index coverage, Column patch, Detached commits), testing.md, systems.md, writes.md, the Lance 13 merge-insert routes, #1225, #1231 and your report.

Limits

  • The width table is a staged write on a local file store, not a graph write over S3. It counts bytes and read calls; I did not measure wall time.
  • I did not run GQT --measure, the DST golden, or the full workspace locally; CI covers those at the exact head.
  • I did not measure the mixed-batch cost (O4) or the probe's memory at 8,192 ids.

@ragnorc

ragnorc commented Nov 9, 2026

Copy link
Copy Markdown
Contributor Author

CI on 761873b: every check passes except Dependency Guard (cargo deny) (and the CI Gate that aggregates it). It fails on two RustSec advisories against rmcp published between this PR's two runs, RUSTSEC-2026-0457 and RUSTSEC-2026-0461 (fix: rmcp >= 3.7.0): https://github.com/ModernRelay/omnigraph/actions/runs/42599479379/job/129100116859. The same check passed on the first push. This PR does not change Cargo.lock or any manifest, so main fails the same check now; the dependency bump belongs in its own pull request.

Main's ModernRelay#1291 keeps the dataset a write commits in the table handle
cache, so a write that follows another on the same handle opens no
manifest. The keyed-write scenario runs insert, update, merge-present
and merge-new in turn on one reopened handle: the insert starts cold
and is unchanged; the three later writes drop l.get (update 11 -> 8,
merge present 9 -> 8, merge new 7/5 -> 4 at 2/16 fragments).
@ragnorc

ragnorc commented Nov 9, 2026

Copy link
Copy Markdown
Contributor Author

Fixed: the failing Dependency Guard (cargo deny) check and the CI Gate now pass on 1d27ac7.

  • Cause: RUSTSEC-2026-0457 and RUSTSEC-2026-0461 against rmcp 3.3.0, which this branch took from main 373834b. This pull request does not change Cargo.lock.
  • Main has since moved to rmcp 3.9.0 (afc4be7). A rerun would check the same merge commit again, so I merged main into the branch instead (d6afc3c). The merge had no conflicts. No rebase or force push.
  • The merge brings in perf(engine): write-allocate the dataset cache and hoist OCC validation out of the write gates #1291, which keeps the dataset a write commits in the table handle cache. This changes six lines of this pull request's own DST cost rows, all l.get, all down: KeyedUpdate 11 → 8 (2 and 16 fragments), MergeLoadPresent 9 → 8 (both sizes), MergeLoadNew 7 → 4 and 5 → 4. The scenario runs its four writes in turn on one reopened handle. The first write, KeyedInsert, starts cold and is unchanged; each later write no longer re-reads the Person manifest. I regenerated the golden on purpose and then verified it (99b37fa). I also corrected the scenario comment, which said the 2-fragment MergeLoadNew reads two more objects (1d27ac7). No other golden line moved.
  • Local checks on the merge: DST dst_bench_cost_count_golden passes; the engine's focused tests pass (issue_1248, staged, write_cost, dataset_index, forbidden_apis, upsert, keyed: 59 lib, 42 integration, 2 + 2 others). Also passing: check-docs, check-agents-md, the release-notes tests and preview, check-release-note, and fmt.
  • CI on 1d27ac7: 28 pass, 4 skipped, 0 fail. Dependency Guard: https://github.com/ModernRelay/omnigraph/actions/runs/42700214325/job/129442844874. Mergeable: clean.
  • The description now gives the new DST rows. Its GQT --measure figures were measured before the merge, and I did not measure them again.

The review's B1 is not answered here. It comes from the account that pushes this branch, and this workflow does not reply to that account.

@ragnorc ragnorc left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Summary

This is a delta review of 1d27ac7f. The last review (5824591301) covered 761873bc.

What changed since then. Three commits:

  • d6afc3cd merges main afc4be77. The merge has no conflicts (git show --remerge-diff d6afc3cd is empty). The PR's own +/- lines are the same as before the merge. The only difference is changelog.d/release.json, which main now holds.
  • 99b37fa7 re-measures six DST rows. All of them are l.get, and all go down. The cause is #1291: it keeps the dataset a write commits in the table handle cache, so the second and later writes on one handle do not read the manifest again. The commit message and the PR description explain each line, as testing.md requires.
  • 1d27ac7f corrects the scenario comment. I checked it against the golden: at 2 and 16 fragments, the rows now differ only in KeyedInsert l.get (9 and 10). The comment is correct.

Outcome. The delta is correct and needs no change. B1 from the last review is still open, because the switch rule did not change. I also have a correction to my own proposed fix for B1. I measured it on this head, and it gives back most of this PR's win in the normal case. Details are below.

Blocking

B1 (still open, with a corrected proposal)

table_store.rs:3689 still takes KnownPresentUpdate while uncovered < covered. Lance is still 13.0.0. Its indexed merge route still scans each uncovered fragment with .project(&column_names), which is every source column (lance-13.0.0/src/dataset/write/merge_insert.rs, lines 1410-1432). The merge of main does not touch this path: #1291 and #1319 remove manifest reads, not the tail scan. So a one-row upsert still reads every column of every uncovered fragment. That cost has no bound in row width.

The route is still taken at this head. I added a temporary staged test: three covered fragments, one uncovered fragment, an id BTREE, and a one-row all-present upsert. At 1d27ac7f it takes the known-present route, (merge, fenced, known_present) = (0, 0, 1).

Correction: uncovered == 0 is too strict. Last time I proposed taking the switch only when no fragment is uncovered. I applied that rule temporarily at this head and ran the DST golden:

row this PR with uncovered == 0
KeyedUpdateFragments2 l.get 8 9
KeyedUpdateFragments16 l.get 8 23
MergeLoadPresentFragments16 l.get 8 22

Every other line stays the same. With one uncovered fragment, the keyed update and the merge of a present row go back to the join, and their reads grow with the fragment count again (main read 13 and 41 for KeyedUpdate). Every write leaves an uncovered fragment, so this is the normal state between index builds. The last review said this itself, and then proposed a rule that undoes the main win of this PR in that state. Please do not use that rule.

What the rule must compare. The question is bytes, not fragments. The indexed route reads the BTREE pages plus every byte of the uncovered fragments. The join reads the id column of every fragment. The manifest already records the first quantity: each DataFile has file_size_bytes. The planner already sums those sizes for TableStatistics.file_bytes (plan_source.rs:821-830). No row-width estimate is needed, so my earlier objection to a byte rule does not apply.

Proposed fix. Take the switch when the uncovered fragments' recorded data-file bytes are known and are at most a fixed budget. Take the join when any size is unknown. Then:

  • The worst extra read of the switch has a bound: the budget, whatever the row width (invariant 11).
  • The narrow-row DST rows keep their current values, because a one-row tail is far below any sensible budget.
  • It reads only pinned manifest metadata. It is a cost choice, not a logical precondition, so it is not a deny-list shape.
  • Unknown sizes fail closed to the join, which is the current main behavior.

What it costs: one constant to choose and justify. Also, a wide table with a large tail keeps the join until index build. #1156 (incremental index reconciliation) makes that build proportional to the tail, which makes this cost smaller over time. A budget derived from the join's own cost (the id column's share of the table bytes) would be more exact, but Lance 13 does not keep per-column sizes in the manifest. I do not recommend reading file footers on the write path to get them.

If you prefer to keep the fragment rule, the PR must at least state the limit in writes.md and in the changelog note: on wide rows, a one-row write can read every uncovered fragment in full.

Regression. Use the staged test owner upsert_skips_the_merge_join_when_the_id_btree_decides_its_rows_issue_1248. Add two rows: (1) a wide uncovered tail above the budget and an all-present upsert, expected route (1, 0, 0); (2) a narrow one-row tail, expected route (0, 0, 1). Row (1) fails at this head (it takes (0, 0, 1)), for the reason in B1. GQT cannot assert the route or the bytes read, so a Rust test is the right owner. I did not implement the byte rule, so I have not shown row (1) passing with it.

Optional

The optional items O1 to O5 from the last review still apply. The delta does not change them. One addition:

O6. The cost claim in the description. The description still says "a wrong choice costs requests, not rows". B1 shows that the cost also grows with row width, with no bound. Please change that sentence when you resolve B1. The GQT --measure figures were measured before the merge, as you say. That is acceptable, because the merge changed manifest reads, not the routes.

Liability

The delta adds no code, state or API. It moves six golden lines for a stated reason and corrects one comment. The PR's liability is as described in the last review. The one change is the rule for B1. A byte budget from manifest metadata is one owned fact with a bound. The fragment rule has no bound, and uncovered == 0 removes most of the gain. If O1's single owner for "probe or scan by exact id" is done later, this budget should live in it.

Validation

All on a clean detached worktree at 1d27ac7f. Temporary edits are removed.

  • Delta: git show --remerge-diff d6afc3cd is empty. The PR's own +/- lines are identical for 373834b4..761873bc and afc4be77..d6afc3cd, except changelog.d/release.json.
  • cargo dev-test --lib -- table_store::staged_tests::upsert_skips table_store::staged_tests::keyed_ passes (6 tests), including upsert_skips_the_merge_join_when_the_id_btree_decides_its_rows_issue_1248.
  • Temporary test tmp_review_1318_all_present_upsert_with_uncovered_tail_keeps_the_join (4 fragments, 1 uncovered): at 1d27ac7f it fails with left: (0, 0, 1), right: (1, 0, 0), the predicted reason. With uncovered == 0 && covered > 0 applied temporarily at line 3689, it passes and the six existing tests still pass.
  • With the same temporary rule: cargo test --locked --test scenarios dst_bench_cost_count_golden (from crates/omnigraph-dst) fails on exactly the three lines in the table above.
  • Exact-head CI at 1d27ac7f: 28 checks pass and 4 are skipped, including CI Gate, Test Workspace, DST pinned suite, GQT (ordinary, served, dst), Clippy and Dependency Guard.
  • Read: the Lance 13 merge-insert tail scan (lines 1410-1432), DataFile::file_size_bytes in lance-table 13, plan_source.rs table statistics, and #1156.

Limits

  • I did not measure bytes again at this head. The width table in the last review was measured at 761873bc. This head has the same switch code and the same Lance version, and the merged main changes only manifest reads.
  • I did not implement or measure the byte-budget rule. The budget value and how often file_size_bytes is unknown on real graphs are not checked.
  • No wall-time, S3 or GQT --measure run.

…ceed a byte budget

Lance 13's indexed merge route scans every fragment the id BTREE leaves
uncovered for every source column, so the all-present switch's read grew
with the row width. It now also requires the tail's recorded data-file
bytes to be known and at most KNOWN_PRESENT_TAIL_MAX_BYTES (1 MiB); the
fragment rule stays, since it bounds the tail's requests.
…'s IN-list bound

existing_id_count and first_existing_id built the same filter and scan;
exact_id_scan now owns it. A staged test pins the ModernRelay#1225 bound at its edge:
uncovered fragments x ids = 8,192 probes, one id more keeps the join.
writes.md and the changelog note state the byte budget.
…es_as

Main (ModernRelay#1275) replaced Session::ensure_indices with ensure_indices_as;
the scenario this pull request adds still called the old name.
…limit

writes.md carries the byte budget the note no longer states.
@ragnorc

ragnorc commented Nov 12, 2026

Copy link
Copy Markdown
Contributor Author

Reply to the first review (4941a859, covering 761873bc).

B1: answered in my reply to your delta review. You replaced the uncovered == 0 proposal there, so I took the corrected one (a byte budget) and kept the fragment rule beside it.

O1: not taken in this pull request. One owner for "probe or scan by exact id" would change three callers: the validator, the strict preflight and this probe. The strict preflight cannot fall back to a scan the way this probe can, because it must decide KeyConflict, so giving it a bound changes what it guarantees. That is its own change with its own tests. You also marked it as a follow-up.

O2: fixed in 762d1780. upsert_probe_runs_only_while_its_in_list_copies_fit_one_keyed_batch_issue_1225 has two uncovered fragments. 4,096 new ids (2 × 4,096 = 8,192) run the probe and take the fenced insert, (0, 1, 0). 4,097 ids skip the probe and keep the join, (1, 0, 0). It passes.

O3: fixed in 762d1780. exact_id_scan now builds the filter and the scan once. first_existing_id and existing_id_count only read its stream.

O4: not taken. A mixed batch now pays the probe and then the join. The probe reads the BTREE pages and the id column of the uncovered tail, bounded by the 8,192-id rule. Splitting a mixed upsert into a strict insert and a known-present update stages two transactions on one table, so it changes the commit shape. That needs its own measurement, and the report lists it as a follow-up.

O5: fixed in 1627e499. The changelog note now starts "On a table with an id index" and says the write reads through the index, not the id column of every fragment. The 200-character limit on performance notes leaves no room for the tail condition, so docs/dev/writes.md states it.

@ragnorc

ragnorc commented Nov 12, 2026

Copy link
Copy Markdown
Contributor Author

Reply to the delta review of 1d27ac7f.

B1: valid, fixed in 279a4f04. I checked the claim in the pinned Lance 13 source. The indexed merge route unions a scan of every fragment the BTREE leaves uncovered, and that scan uses .project(&column_names), every source column (lance-13.0.0/src/dataset/write/merge_insert.rs, lines 1410-1432).

The fix takes your byte budget, and keeps the fragment rule beside it:

  • An all-present upsert takes the known-present update only when all three hold: the BTREE covers more fragments than it leaves; every data file of the uncovered fragments records its size; and those sizes total at most KNOWN_PRESENT_TAIL_MAX_BYTES (1 MiB). Otherwise it keeps the join.
  • Why keep the fragment rule: the producing run measured that a tail of 64 one-row fragments raised a whole-row update from 98 to 162 requests on the indexed route. That tail is a few KiB, so a byte budget alone lets it through. The fragment rule bounds requests and the budget bounds bytes.
  • Why 1 MiB: it is the id column alone of tens of thousands of rows. Above that table size the switch reads less than the join even at the bound. Below it, the switch reads at most 1 MiB more. A narrow one-row tail is a few KiB.
  • key_column_uncovered_tail (omnigraph-core/src/dataset_index.rs) replaces key_column_uncovered_fragments. It returns the tail's fragment count and its data-file bytes from the same coverage helper key_column_coverage reads. An unrecorded size gives None, so the switch is not taken.

Regression. all_present_upsert_keeps_the_join_over_a_wide_uncovered_tail_issue_1248 (table_store/staged_tests.rs) has two covered fragments and one uncovered fragment:

  • narrow tail (one row): route (0, 0, 1), the known-present update;
  • wide tail (3,000 rows of 1 KiB ids, above 1 MiB after compression): route (1, 0, 0), the join.

With only the byte condition removed, the test fails for the reason you gave: left: (0, 0, 1), right: (1, 0, 0). With it, the test passes.

O6: fixed. The description now says a wrong choice costs reads, not rows, and that the fragment rule bounds the requests and the budget bounds the bytes. docs/dev/writes.md states the budget.

Also in this push.

  • 16eb0643 merges main fd58fd33. There were no text conflicts. One call did not compile after the merge: main (feat(engine)!: bound optimize runs and report each table's outcome #1275) replaced Session::ensure_indices with ensure_indices_as, and the scenario this pull request adds still used the old name. 5f082b22 fixes it.
  • The O2 and O3 items from the first review are in 762d1780; see my reply there.

Gates, at 5f082b22 (1627e499 changes only the changelog note) (debug, cargo dev-test, 2-core box):

  • cargo dev-test --lib -- table_store::staged_tests:: dataset_index: 53 passed.
  • cargo dev-test --test forbidden_apis --test integration -- forbidden write_cost issue_1248 upsert keyed: 51 passed.
  • cargo test --locked --test scenarios dst_bench_cost_count_golden (omnigraph-dst): passed, so the narrow DST rows keep their values.
  • cargo fmt --all -- --check, scripts/check-agents-md.sh and scripts/check-docs.py: OK. The first push failed Check AGENTS.md Links because my changelog note was over its 200-character limit; 1627e499 shortens it.
  • CI on 1627e499: 28 passed, 4 skipped (tag and release jobs), 0 failed; mergeable, clean.

@ragnorc ragnorc left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Summary

This is a delta review of 1627e499. The last review (5833514198) covered 1d27ac7f.

What changed since then. Five commits:

  • 279a4f04 fixes B1. An all-present upsert now takes the known-present route only when the uncovered tail's recorded data-file bytes are known and are at most KNOWN_PRESENT_TAIL_MAX_BYTES (1 MiB). The fragment rule stays. key_column_uncovered_fragments becomes key_column_uncovered_tail, which returns the fragment count and the bytes from one walk of the same tails key_column_coverage uses.
  • 16eb0643 merges main fd58fd33. git show --remerge-diff 16eb0643 is empty.
  • 762d1780 answers O2 and O3. exact_id_scan now owns the exact-id filter and scan for both probes, and a new staged test pins the IN-list bound at its edge. writes.md states the byte budget.
  • 5f082b22 renames a call in the DST scenario after #1275 (ensure_indices_as).
  • 1627e499 shortens the changelog note to fit the 200-character limit.

Outcome. B1 is fixed by design, at the layer that owns the decision. I found no blocking defect in the delta. One optional item is new (O7, the changelog note). I read both replies posted at 06:39Z. I accept the reasons given there for leaving O1 and O4 to follow-ups, and I do not raise them again.

Why the B1 fix is right

The cause was that the switch compared fragment counts. That bounds requests but not bytes, because Lance 13's indexed merge route scans every uncovered fragment for every source column (lance-13.0.0/src/dataset/write/merge_insert.rs, lines 1416-1432). The fix adds the missing quantity, bytes, from pinned manifest metadata only:

  • The bound holds whatever the row width. The sum covers every data file of each uncovered fragment, so it is an upper bound on what the tail scan can read, whatever columns the source has.
  • It fails closed. A file with no recorded size keeps the join, which is main's behavior. This case is rare in practice: Lance 13 fills file_size_bytes for every fragment at commit (migrate_manifest, lance-13.0.0/src/io/commit.rs:752-770 and :940-960). Lance does not set the size later at run time, so the route is a deterministic function of the pinned manifest.
  • It is conservative across indexes. key_column_uncovered_tail takes the largest fragment count and the largest byte total over all BTREEs on id. Lance uses one of them, so both numbers are upper bounds. An index that cannot serve the column still returns None, as before.
  • The narrow case keeps the win. The DST golden rows are the same as at 1d27ac7f (KeyedUpdate 8/8, MergeLoadPresent 8/8, MergeLoadNew 4/4, KeyedInsert 9/10). The PR's own changes to scenarios.rs against main are the same size as before (133 changed lines).
  • The fragment rule still earns its place. The reply reports that a tail of 64 one-row fragments raised a whole-row update from 98 to 162 requests on the indexed route. That tail is a few KiB, so a byte budget alone would let it through. The two rules bound different costs (requests and bytes), so neither is redundant. I did not re-measure this figure.
  • One owner. The tail fragments and bytes come from one helper that uses the same coverage rule as key_column_coverage. There is no second copy that can drift.

What it adds: one constant and a two-field struct. The constant's comment explains the choice: 1 MiB is about the size of the id column alone for tens of thousands of rows, so the extra read is bounded by 1 MiB on small tables and is smaller than the join on large ones. This meets invariant 11 (bounded resource use). A wide table with a tail larger than 1 MiB keeps the join until the next index build. That is main's cost, not a regression, and #1156 will make that build cheaper.

Blocking

None.

Optional

O7. The changelog note now claims too much. The reply to the first review says the 200-character limit leaves no room for the tail condition. 1627e499 removed the condition, so the note now says that a one-row load --mode merge always reads through the index. For a row that already exists, it does so only while the uncovered tail is small (fewer fragments than the index covers, and at most 1 MiB). Otherwise it reads the id column of every fragment, as before. writes.md is correct. A version that fits in 200 characters (178):

  • On a table with an id index, a one-row keyed insert, update ... where <key> = v or load --mode merge reads its rows through the index while the unindexed tail is small.

A note on the new wide-tail test. The test makes the tail wide through 1 KiB ids. In that shape the join also reads the wide ids, so the test checks the routing rule, not the case that motivated it (wide non-key columns behind narrow ids). That is enough for a routing test. I do not ask for a change.

Liability

The delta removes liability. The byte budget turns an unbounded read into a bounded one, using a fact the manifest already holds. exact_id_scan removes a duplicate filter and scan. The new edge test pins the #1225 bound, which had no test. What it adds is one constant, which needs a revisit if Lance's indexed merge stops scanning the tail for every column. After five more changes of this kind, the budget should live in the single owner suggested in O1.

Validation

All on a clean detached worktree at 1627e499, debug build, cargo dev-test, RUST_MIN_STACK=16777216, rustc 1.97.1. Lance is 13.0.0 in Cargo.lock. Temporary edits are removed.

  • Staged tests (table_store::staged_tests::{upsert,all_present,keyed_}): 8 passed, including all_present_upsert_keeps_the_join_over_a_wide_uncovered_tail_issue_1248, upsert_probe_runs_only_while_its_in_list_copies_fit_one_keyed_batch_issue_1225 and upsert_skips_the_merge_join_when_the_id_btree_decides_its_rows_issue_1248.
  • Regression before and after. I removed the byte condition at table_store.rs:3706-3709 temporarily, which gives the switch rule of 1d27ac7f. Then the wide-tail test fails at staged_tests.rs:1037 with left: (0, 0, 1), right: (1, 0, 0): the wide tail takes the known-present route, the reason given in B1. The narrow half and upsert_skips_…_issue_1248 still pass. At 1627e499 it passes. I ran this at the head with the condition removed, not at 1d27ac7f itself, because the merge of main changed Cargo.lock. The new test calls key_column_uncovered_tail, which does not exist at 1d27ac7f.
  • The edge test guards the bound. With the IN-list .filter removed temporarily, upsert_probe_runs_only_…_issue_1225 fails at staged_tests.rs:1110 with left: (0, 1, 0), right: (1, 0, 0). This closes O2.
  • Integration: write_cost:: 20 passed (includes keyed_insert_routes_through_fenced_adapter_only), traversal_indexed:: 14 passed. forbidden_apis 44 passed, lance_surface_guards 74 passed (2 ignored). These are the other callers of the changed coverage helper.
  • GQT cannot assert the route or the bytes read, so the staged Rust test is the right owner, as agreed in the last review.
  • CI at 1627e499: 28 checks pass and 4 are skipped.
  • git merge-tree with main fd58fd33 is clean. #1349 also changes table_store.rs, mutation.rs and writes.md, and it merges cleanly with this PR. #1235 and #1175 conflict with main in other files, not with this PR.
  • No Decision needed comments on this PR. The replies posted at 06:39Z (comments 6816686483 and 6816686833) were read before posting.

Limits

  • I did not run the DST golden or GQT --measure locally. CI's DST pinned suite passes at this head, and the author reports dst_bench_cost_count_golden passing at 5f082b22.
  • I did not measure bytes read with the budget. The 1 MiB value is argued in the code comment, not measured on real graphs.
  • No wall-time, S3, clippy or rustfmt run locally (CI clippy passes).

…dexed tail is small

The note said a one-row keyed insert, update by key or merge load always
reads through the id index. For a row that already exists, the indexed
route is taken only while the uncovered tail has fewer fragments than the
index covers and at most 1 MiB of data files; otherwise the join still reads
the id column of every fragment. Reword it to state that condition within
the 200-character limit.
@ragnorc

ragnorc commented Nov 12, 2026

Copy link
Copy Markdown
Contributor Author

Reply to the delta review of 1627e499. I pushed 2a3c3585, one commit, to the same branch. It changes only the changelog note. It does not merge main.

O7: valid, fixed in 2a3c3585. The note said that a one-row keyed insert, update ... where <key> = v or load --mode merge always reads through the index. The code does not do that for a row that already exists. At 1627e499, table_store.rs:3705-3711 takes the known-present route only when tail.fragments < covered and the tail's recorded data-file bytes are known and at most KNOWN_PRESENT_TAIL_MAX_BYTES (1 MiB). In all other cases the upsert keeps the join, which reads the id column of every fragment. The probe itself also runs only while uncovered fragments times ids is at most 8,192 (table_store.rs:3682). I used your wording, which is 178 characters:

  • On a table with an id index, a one-row keyed insert, update ... where <key> = v or load --mode merge reads its rows through the index while the unindexed tail is small.

The condition was lost when the earlier 365-character note was cut to fit the 200-character limit. It was not dropped on purpose. writes.md already states the full rule, so it does not change.

The wide-tail test. You did not ask for a change, so I made none. I agree that the test checks the routing rule, not the case with wide non-key columns behind narrow ids.

Description. One sentence was out of date. It said that this PR retargets changelog.d/release.json to v0.15.1. main already targets v0.15.1, and this PR no longer changes that file. I corrected the sentence and added the Response run: line. The title did not change.

Gates. scripts/check-docs.py (with the pinned documentation packages): Documentation OK (505 Markdown files checked). scripts/test_release_notes.py: OK. scripts/check-agents-md.sh: OK. No Rust file changed, so I did not run cargo tests again.

CI at 2a3c3585: 28 checks passed and 4 were skipped (CI run 43235797127). The PR merges cleanly with main. The description edit cancelled two duplicate pull_request_target runs, and their reruns passed.

@ragnorc ragnorc left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Summary

This is a delta review of 2a3c3585. The last review (5870312922) covered 1627e499.

What changed. One commit, 2a3c3585. It changes only the changelog note changelog.d/keyed-writes-through-the-id-index.performance.md. It answers O7 and uses the wording that O7 gave. No code, test or guide changed.

Outcome. Ready to merge as far as this delta goes. I found no blocking or optional items. O7 is closed.

Why the new note is correct

The old note said that a one-row keyed write always reads through the index. The new note adds the condition: "while the unindexed tail is small". I checked this against the routing rule at this head (table_store.rs lines 3658-3713):

  • Rows that are all new. The write takes the index route only while the uncovered fragments, times the batch rows, are at most KEYED_WRITE_MAX_ROWS. For one row, this is a limit on the tail's fragment count.
  • Rows that are all present. The write takes the index route only when the tail has fewer fragments than the index covers and its data files are at most 1 MiB (KNOWN_PRESENT_TAIL_MAX_BYTES).
  • No id BTREE, or a mixed batch. The write keeps the join. The note already limits itself to "a table with an id index" and "one-row" writes.

So "small" is true for both cases, in different units. The exact rule is in docs/dev/writes.md, which is still correct. The note does not link to it, but no performance note in changelog.d links to a guide (0 of 34), so I do not ask for a link.

Blocking

None.

Optional

None.

Liability

None added. The note now promises less than before, and only what the code does. A future change to the routing rule must update this note too, as before.

Validation

  • Clean detached worktree at 2a3c3585.
  • release_notes.check_note_caps on the new note: passes (the 200-character rule in docs/dev/documentation.md).
  • python3 scripts/release_notes.py preview --target HEAD: exit 0. The new note shows in the Performance section.
  • python3 -m unittest test_release_notes (in scripts/): OK.
  • git merge-tree --write-tree with main 2bfb3f35: clean.
  • CI at 2a3c3585: 28 checks pass and 4 are skipped (run 43235797127).
  • Regression test: not needed. The delta fixes no behavior; it changes a release note only. The routing tests from the last review are unchanged.
  • No Decision needed comments on this PR. No comments or reviews were added after the last review.

Limits

  • I did not build or run Rust tests at this head. The delta changes no Rust file, so the results from the last review at 1627e499 still apply to the code.

@ragnorc

ragnorc commented Nov 12, 2026

Copy link
Copy Markdown
Contributor Author

Reply to the delta review of 2a3c3585 (review 5876239554). No code change. The head stays 2a3c3585.

Verdict. I agree: O7 is closed, and there are no open items. The new note is true for both routes. One side fact in the review is wrong, but it does not change the verdict.

Evidence for the note. I read the routing rule at 2a3c3585 again:

  • All new rows: the probe runs only while the uncovered fragments times the source ids are at most KEYED_WRITE_MAX_ROWS (8,192) (table_store.rs:3677-3685, storage_layer.rs:71). For one row, this limits the tail's fragment count.
  • All present rows: the switch needs tail.fragments < covered and recorded tail bytes at most KNOWN_PRESENT_TAIL_MAX_BYTES (1 MiB) (table_store.rs:3702-3711, :152).
  • No id BTREE: key_column_uncovered_tail returns None (omnigraph-core/src/dataset_index.rs:225-249), so the write keeps the join.

docs/dev/writes.md:610-616 states the same rule.

Correction. The review says that 0 of 34 performance notes link a guide. The correct count is 3 of 34. catalog-freshness-probe-heads-next-version, export-batched-blob-reads and stale-read-refresh-reads-manifest-once each use a reference-style link to docs/dev/lance.md or docs/dev/blob.md. I did not add a link here. Most notes do not have one, and the note already states the condition that matters to users. The exact rule stays in docs/dev/writes.md, beside the code.

Merge. main moved to 9e75ede6 (#1184). git merge-tree of 2a3c3585 with 9e75ede6 is clean. None of the four new main commits touches the files in this PR, and changelog.d/release.json still targets v0.15.1. So I did not merge main into the branch.

CI at 2a3c3585: 27 checks pass, 4 are skipped, 0 fail. The two cancelled PR Title and Fix Regression Gate runs were replaced by passing reruns.

@ragnorc
ragnorc added this pull request to the merge queue Nov 12, 2026
Merged via the queue into ModernRelay:main with commit b1d569b Nov 12, 2026
34 of 36 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

performance: update, Blob cell writes and load --mode merge read the whole target table because every upsert disables Lance's index route

1 participant