Repository navigation
perf(storage): reach a keyed write's rows through the id index - #1318
Conversation
A `<key> = v` conjunct also selects the derived `id = v`, so an update's match scan adds that conjunct when the pinned table has an id BTREE, as a node delete already does. The scan then reads the index instead of the key column of every fragment.
…join Lance's insert-capable merge-insert route joins the source against every id of the table. With an id BTREE, the exact-id probe a strict insert runs decides whether an upsert's rows are all new or all present; the first stages through the absence-proven insert writer, the second as a known-present update. A mixed batch, or a table without the BTREE, keeps the join.
KeyedInsert, KeyedUpdate, MergeLoadPresent and MergeLoadNew on an indexed table of 2 and 16 fragments. On main their l.get rows were 10/25, 13/41, 10/24 and 10/24; now 9/10, 11/11, 9/9 and 5/5. InsertLegacy l.get moves 15 -> 13: its keyed insert takes the absence-proven writer.
… index coverage The exact-id probe plans its IN list once per fragment the id BTREE does not cover, so it runs only while those copies hold no more ids than one keyed batch (ModernRelay#1225). An all-present upsert's indexed merge reads each uncovered fragment once more, where the join reads every fragment once, so it switches only while the BTREE covers more fragments than it leaves. key_column_uncovered_fragments counts the tail under the coverage rule key_column_coverage already applies; both read one helper.
… the fenced insert
ragnorc
left a comment
There was a problem hiding this comment.
Summary
Why. On a table with an id BTREE, a one-row keyed write read the id column of every fragment. The cost grew with the table (#1248).
What. Two changes. (1) An update ... where <key> = v adds the derived __id = v conjunct to its match scan, the same conjunct a node delete already uses. (2) An Upsert first runs the exact-id probe. All new: it stages through the absence-proven insert writer. All present: it stages as KnownPresentUpdate, which takes Lance's indexed merge route. Mixed: it keeps the forced join.
Impact. The all-new route and the update conjunct are sound, and they remove real cost. The all-present switch is not safe yet. Its rule counts fragments, but Lance's indexed route reads every source column of each uncovered fragment. The forced join reads only id. On wide rows (vectors, long text), one uncovered fragment can make a one-row write read the whole fragment. Details and a measurement are in B1.
I reviewed 4941a859. The later head 761873bc only retargets changelog.d/release.json to v0.15.1, so this review covers it.
Blocking
B1. The all-present switch reads whole rows of the uncovered tail
table_store.rs:3689 switches to KnownPresentUpdate while uncovered < covered. That stages without use_index(false), so Lance 13 takes create_indexed_scan_joined_stream. For the fragments the BTREE does not cover, that route scans with .project(&column_names): all source columns, not only the key (lance-13.0.0/src/dataset/write/merge_insert.rs, lines 1421-1432). The source of a load --mode merge, a Blob cell write and a whole-row update is the full row. The forced route (create_plan, line 2419) joins on id and reads only that column from the target.
So the rule compares the wrong quantity. It compares fragment counts. The real cost is (uncovered rows × full row width) against (all rows × id width).
Measured (temporary in-crate test, not pushed). Table: 8 fragments × 512 rows, id BTREE over them, then 1 or 3 appended 512-row fragments that the BTREE does not cover. Operation: one stage_keyed_write(Upsert) of one present row, reads counted on the dataset's object store (local reader plus wrapper):
| row payload | uncovered fragments | main 373834b4 (join) |
this PR (known-present) |
|---|---|---|---|
| 8 B | 1 | 27 reads, 87,912 B | 16 reads, 94,526 B |
| 8 B | 3 | 33 reads, 107,448 B | 26 reads, 137,950 B |
| 4 KiB | 1 | 27 reads, 87,912 B | 17 reads, 2,088,964 B |
| 4 KiB | 3 | 33 reads, 107,448 B | 27 reads, 6,123,198 B |
The join's bytes do not change with row width. The switch's bytes grow with the width and the size of the tail: 24× and 57× more here. The GQT and DST fixtures in this PR have narrow rows, so they cannot show this. A table like the one in #1225 (Vector(3072), about 12 KB per row, 8,000-row load fragments) would read about 100 MB per uncovered fragment for a one-row update. The PR says "a wrong choice costs requests, not rows". That is true, but the cost here has no bound in row width. Every write leaves an uncovered fragment, so this is the normal state between index builds.
Proposed fix. Take the switch only when the BTREE leaves no fragment uncovered (uncovered == 0). Then the indexed route has no tail scan and always reads less than the join. This is also the rule #1231 already uses for committed lookups (coverage_validate_access, cost.rs:651: probe only under full coverage), so there is one rule, not two. Cost of this choice: with a tail, an all-present upsert takes the join until index build. The all-new route is not affected, because its probe reads only id of the tail. A byte-weighted rule (tail rows × estimated row width) keeps more wins, but it needs a row-width estimate that strings cannot supply, and it adds a model that can drift. I do not recommend it.
Regression. Add a row to upsert_skips_the_merge_join_when_the_id_btree_decides_its_rows_issue_1248: a table with an uncovered fragment, an all-present upsert, expected route (1, 0, 0). With the fix, the 2-fragment DST rows already show this (MergeLoadPresentFragments2).
Optional
O1. One owner for "probe or scan" by exact id. After this PR there are three rules for the same question (an IN (...) lookup on __id against a partly indexed table): coverage_validate_access (full coverage), choose_validate_access (cost by rows), and this PR's uncovered × ids ≤ 8,192. The strict-insert preflight (first_existing_id) runs the same probe with no bound, as your report also says. After five more changes like this, there would be five rules. One function in dataset_index that returns the access for an exact-id lookup (coverage plus key count) could serve the validator, the strict preflight and this probe. That removes the unbounded preflight case too. It is a follow-up, not a condition of this PR.
O2. The IN-list bound has no test. I found no test where uncovered × source ids > 8,192 keeps the join. A change that drops the .filter near table_store.rs:3664 would likely still pass the current tests. One more row in the staged test (for example 2 uncovered fragments and 4,097 ids) would pin it.
O3. Duplicate scan. existing_id_count (table_store.rs:4105) repeats the body of first_existing_id (same filter, same scan_stream_with, same id decode). One helper that streams the present ids, with two thin callers, keeps the two probes from drifting.
O4. Mixed batches pay twice. A mixed upsert now runs the probe and then the full join. That is the common shape of a re-load with load --mode merge. This is not measured. Your report's follow-up (stage the absent rows as a strict insert and the present rows as a known-present update, as branch merge does) would remove it.
O5. Changelog wording. "its cost stops growing with the table" is true only with an id index and a small tail. Suggest: "on a table with an id index, a one-row keyed insert, update ... where <key> = v or load --mode merge reads its rows through the index ...".
Correctness checks that hold
- The probe and the staging use the same pinned
Dataset, so the all-new and all-present decisions cannot go stale inside one attempt. A retry starts from a new snapshot. validate_keyed_write_batch_idsrefuses duplicate ids, sopresent == source_ids.len()means all present.- The all-new route still materializes external Blob URIs (
prepare_keyed_write_batchinstage_absence_proven_strict_insert) and certifies absence, as the merge route did. It never calls the strict preflight, so an upsert cannot returnKeyConflict. - The update conjunct (
mutation.rs:1955) also reaches pending rows in the non-sole path (scan_with_pending). Those rows carry ids derived from their keys, so the selected rows are the same.
Liability
The all-new route and the update conjunct reuse existing writers and an existing helper. They remove cost and add almost no new state. The all-present switch adds a cost decision from fragment metadata in a third place (O1), and in its current form that decision is wrong for wide rows (B1). With uncovered == 0, the decision becomes the same fact #1231 already uses, and the liability stays small. lance#10606 (inserted keys on the indexed route) could later remove use_index(false), but it does not change the tail projection, so B1 does not go away with that bump.
Validation
- Local,
4941a859:cargo dev-test --lib -- table_store::staged_tests::upsert_skips table_store::staged_tests::keyed_→ 6 passed, includingupsert_skips_the_merge_join_when_the_id_btree_decides_its_rows_issue_1248. - Local,
main373834b4with only this PR'sstaged_tests.rsapplied: the new test fails atall new: the fenced insertwithleft: (1, 0, 0), the predicted reason. A Rust test is the right owner; GQT cannot assert the route or the read count. - Local, both commits: the temporary width test above (
cargo dev-test --lib -p omnigraph-engine -- tmp_review_1318 --nocapture). All temporary edits are removed. - CI at
4941a859: Test Workspace, DST pinned suite, GQT (ordinary, served, dst), Clippy and RustFS pass;Check AGENTS.md Linksfailed on the release-note snapshot, which761873bcfixes. At761873bc:Dependency Guard (cargo deny)fails on RUSTSEC-2026-0457 and RUSTSEC-2026-0461, new advisories in the existingrmcpdependency, not this change.Storage Upgrade Compatibilitywas still pending. - Read: AGENTS.md, invariants.md, lance.md (Index coverage, Column patch, Detached commits), testing.md, systems.md, writes.md, the Lance 13 merge-insert routes, #1225, #1231 and your report.
Limits
- The width table is a staged write on a local file store, not a graph write over S3. It counts bytes and read calls; I did not measure wall time.
- I did not run GQT
--measure, the DST golden, or the full workspace locally; CI covers those at the exact head. - I did not measure the mixed-batch cost (O4) or the probe's memory at 8,192 ids.
|
CI on 761873b: every check passes except |
Main's ModernRelay#1291 keeps the dataset a write commits in the table handle cache, so a write that follows another on the same handle opens no manifest. The keyed-write scenario runs insert, update, merge-present and merge-new in turn on one reopened handle: the insert starts cold and is unchanged; the three later writes drop l.get (update 11 -> 8, merge present 9 -> 8, merge new 7/5 -> 4 at 2/16 fragments).
|
Fixed: the failing Dependency Guard (cargo deny) check and the CI Gate now pass on 1d27ac7.
The review's B1 is not answered here. It comes from the account that pushes this branch, and this workflow does not reply to that account. |
ragnorc
left a comment
There was a problem hiding this comment.
Summary
This is a delta review of 1d27ac7f. The last review (5824591301) covered 761873bc.
What changed since then. Three commits:
d6afc3cdmerges mainafc4be77. The merge has no conflicts (git show --remerge-diff d6afc3cdis empty). The PR's own+/-lines are the same as before the merge. The only difference ischangelog.d/release.json, which main now holds.99b37fa7re-measures six DST rows. All of them arel.get, and all go down. The cause is #1291: it keeps the dataset a write commits in the table handle cache, so the second and later writes on one handle do not read the manifest again. The commit message and the PR description explain each line, astesting.mdrequires.1d27ac7fcorrects the scenario comment. I checked it against the golden: at 2 and 16 fragments, the rows now differ only inKeyedInsertl.get(9 and 10). The comment is correct.
Outcome. The delta is correct and needs no change. B1 from the last review is still open, because the switch rule did not change. I also have a correction to my own proposed fix for B1. I measured it on this head, and it gives back most of this PR's win in the normal case. Details are below.
Blocking
B1 (still open, with a corrected proposal)
table_store.rs:3689 still takes KnownPresentUpdate while uncovered < covered. Lance is still 13.0.0. Its indexed merge route still scans each uncovered fragment with .project(&column_names), which is every source column (lance-13.0.0/src/dataset/write/merge_insert.rs, lines 1410-1432). The merge of main does not touch this path: #1291 and #1319 remove manifest reads, not the tail scan. So a one-row upsert still reads every column of every uncovered fragment. That cost has no bound in row width.
The route is still taken at this head. I added a temporary staged test: three covered fragments, one uncovered fragment, an id BTREE, and a one-row all-present upsert. At 1d27ac7f it takes the known-present route, (merge, fenced, known_present) = (0, 0, 1).
Correction: uncovered == 0 is too strict. Last time I proposed taking the switch only when no fragment is uncovered. I applied that rule temporarily at this head and ran the DST golden:
| row | this PR | with uncovered == 0 |
|---|---|---|
KeyedUpdateFragments2 l.get |
8 | 9 |
KeyedUpdateFragments16 l.get |
8 | 23 |
MergeLoadPresentFragments16 l.get |
8 | 22 |
Every other line stays the same. With one uncovered fragment, the keyed update and the merge of a present row go back to the join, and their reads grow with the fragment count again (main read 13 and 41 for KeyedUpdate). Every write leaves an uncovered fragment, so this is the normal state between index builds. The last review said this itself, and then proposed a rule that undoes the main win of this PR in that state. Please do not use that rule.
What the rule must compare. The question is bytes, not fragments. The indexed route reads the BTREE pages plus every byte of the uncovered fragments. The join reads the id column of every fragment. The manifest already records the first quantity: each DataFile has file_size_bytes. The planner already sums those sizes for TableStatistics.file_bytes (plan_source.rs:821-830). No row-width estimate is needed, so my earlier objection to a byte rule does not apply.
Proposed fix. Take the switch when the uncovered fragments' recorded data-file bytes are known and are at most a fixed budget. Take the join when any size is unknown. Then:
- The worst extra read of the switch has a bound: the budget, whatever the row width (invariant 11).
- The narrow-row DST rows keep their current values, because a one-row tail is far below any sensible budget.
- It reads only pinned manifest metadata. It is a cost choice, not a logical precondition, so it is not a deny-list shape.
- Unknown sizes fail closed to the join, which is the current main behavior.
What it costs: one constant to choose and justify. Also, a wide table with a large tail keeps the join until index build. #1156 (incremental index reconciliation) makes that build proportional to the tail, which makes this cost smaller over time. A budget derived from the join's own cost (the id column's share of the table bytes) would be more exact, but Lance 13 does not keep per-column sizes in the manifest. I do not recommend reading file footers on the write path to get them.
If you prefer to keep the fragment rule, the PR must at least state the limit in writes.md and in the changelog note: on wide rows, a one-row write can read every uncovered fragment in full.
Regression. Use the staged test owner upsert_skips_the_merge_join_when_the_id_btree_decides_its_rows_issue_1248. Add two rows: (1) a wide uncovered tail above the budget and an all-present upsert, expected route (1, 0, 0); (2) a narrow one-row tail, expected route (0, 0, 1). Row (1) fails at this head (it takes (0, 0, 1)), for the reason in B1. GQT cannot assert the route or the bytes read, so a Rust test is the right owner. I did not implement the byte rule, so I have not shown row (1) passing with it.
Optional
The optional items O1 to O5 from the last review still apply. The delta does not change them. One addition:
O6. The cost claim in the description. The description still says "a wrong choice costs requests, not rows". B1 shows that the cost also grows with row width, with no bound. Please change that sentence when you resolve B1. The GQT --measure figures were measured before the merge, as you say. That is acceptable, because the merge changed manifest reads, not the routes.
Liability
The delta adds no code, state or API. It moves six golden lines for a stated reason and corrects one comment. The PR's liability is as described in the last review. The one change is the rule for B1. A byte budget from manifest metadata is one owned fact with a bound. The fragment rule has no bound, and uncovered == 0 removes most of the gain. If O1's single owner for "probe or scan by exact id" is done later, this budget should live in it.
Validation
All on a clean detached worktree at 1d27ac7f. Temporary edits are removed.
- Delta:
git show --remerge-diff d6afc3cdis empty. The PR's own+/-lines are identical for373834b4..761873bcandafc4be77..d6afc3cd, exceptchangelog.d/release.json. cargo dev-test --lib -- table_store::staged_tests::upsert_skips table_store::staged_tests::keyed_passes (6 tests), includingupsert_skips_the_merge_join_when_the_id_btree_decides_its_rows_issue_1248.- Temporary test
tmp_review_1318_all_present_upsert_with_uncovered_tail_keeps_the_join(4 fragments, 1 uncovered): at1d27ac7fit fails withleft: (0, 0, 1),right: (1, 0, 0), the predicted reason. Withuncovered == 0 && covered > 0applied temporarily at line 3689, it passes and the six existing tests still pass. - With the same temporary rule:
cargo test --locked --test scenarios dst_bench_cost_count_golden(fromcrates/omnigraph-dst) fails on exactly the three lines in the table above. - Exact-head CI at
1d27ac7f: 28 checks pass and 4 are skipped, including CI Gate, Test Workspace, DST pinned suite, GQT (ordinary, served, dst), Clippy and Dependency Guard. - Read: the Lance 13 merge-insert tail scan (lines 1410-1432),
DataFile::file_size_bytesinlance-table13,plan_source.rstable statistics, and #1156.
Limits
- I did not measure bytes again at this head. The width table in the last review was measured at
761873bc. This head has the same switch code and the same Lance version, and the merged main changes only manifest reads. - I did not implement or measure the byte-budget rule. The budget value and how often
file_size_bytesis unknown on real graphs are not checked. - No wall-time, S3 or GQT
--measurerun.
…ceed a byte budget Lance 13's indexed merge route scans every fragment the id BTREE leaves uncovered for every source column, so the all-present switch's read grew with the row width. It now also requires the tail's recorded data-file bytes to be known and at most KNOWN_PRESENT_TAIL_MAX_BYTES (1 MiB); the fragment rule stays, since it bounds the tail's requests.
…'s IN-list bound existing_id_count and first_existing_id built the same filter and scan; exact_id_scan now owns it. A staged test pins the ModernRelay#1225 bound at its edge: uncovered fragments x ids = 8,192 probes, one id more keeps the join. writes.md and the changelog note state the byte budget.
…es_as Main (ModernRelay#1275) replaced Session::ensure_indices with ensure_indices_as; the scenario this pull request adds still called the old name.
…limit writes.md carries the byte budget the note no longer states.
|
Reply to the first review ( B1: answered in my reply to your delta review. You replaced the O1: not taken in this pull request. One owner for "probe or scan by exact id" would change three callers: the validator, the strict preflight and this probe. The strict preflight cannot fall back to a scan the way this probe can, because it must decide O2: fixed in O3: fixed in O4: not taken. A mixed batch now pays the probe and then the join. The probe reads the BTREE pages and the O5: fixed in |
|
Reply to the delta review of B1: valid, fixed in The fix takes your byte budget, and keeps the fragment rule beside it:
Regression.
With only the byte condition removed, the test fails for the reason you gave: O6: fixed. The description now says a wrong choice costs reads, not rows, and that the fragment rule bounds the requests and the budget bounds the bytes. Also in this push.
Gates, at
|
ragnorc
left a comment
There was a problem hiding this comment.
Summary
This is a delta review of 1627e499. The last review (5833514198) covered 1d27ac7f.
What changed since then. Five commits:
279a4f04fixes B1. An all-present upsert now takes the known-present route only when the uncovered tail's recorded data-file bytes are known and are at mostKNOWN_PRESENT_TAIL_MAX_BYTES(1 MiB). The fragment rule stays.key_column_uncovered_fragmentsbecomeskey_column_uncovered_tail, which returns the fragment count and the bytes from one walk of the same tailskey_column_coverageuses.16eb0643merges mainfd58fd33.git show --remerge-diff 16eb0643is empty.762d1780answers O2 and O3.exact_id_scannow owns the exact-id filter and scan for both probes, and a new staged test pins theIN-list bound at its edge.writes.mdstates the byte budget.5f082b22renames a call in the DST scenario after #1275 (ensure_indices_as).1627e499shortens the changelog note to fit the 200-character limit.
Outcome. B1 is fixed by design, at the layer that owns the decision. I found no blocking defect in the delta. One optional item is new (O7, the changelog note). I read both replies posted at 06:39Z. I accept the reasons given there for leaving O1 and O4 to follow-ups, and I do not raise them again.
Why the B1 fix is right
The cause was that the switch compared fragment counts. That bounds requests but not bytes, because Lance 13's indexed merge route scans every uncovered fragment for every source column (lance-13.0.0/src/dataset/write/merge_insert.rs, lines 1416-1432). The fix adds the missing quantity, bytes, from pinned manifest metadata only:
- The bound holds whatever the row width. The sum covers every data file of each uncovered fragment, so it is an upper bound on what the tail scan can read, whatever columns the source has.
- It fails closed. A file with no recorded size keeps the join, which is main's behavior. This case is rare in practice: Lance 13 fills
file_size_bytesfor every fragment at commit (migrate_manifest,lance-13.0.0/src/io/commit.rs:752-770and:940-960). Lance does not set the size later at run time, so the route is a deterministic function of the pinned manifest. - It is conservative across indexes.
key_column_uncovered_tailtakes the largest fragment count and the largest byte total over all BTREEs onid. Lance uses one of them, so both numbers are upper bounds. An index that cannot serve the column still returnsNone, as before. - The narrow case keeps the win. The DST golden rows are the same as at
1d27ac7f(KeyedUpdate8/8,MergeLoadPresent8/8,MergeLoadNew4/4,KeyedInsert9/10). The PR's own changes toscenarios.rsagainst main are the same size as before (133 changed lines). - The fragment rule still earns its place. The reply reports that a tail of 64 one-row fragments raised a whole-row update from 98 to 162 requests on the indexed route. That tail is a few KiB, so a byte budget alone would let it through. The two rules bound different costs (requests and bytes), so neither is redundant. I did not re-measure this figure.
- One owner. The tail fragments and bytes come from one helper that uses the same coverage rule as
key_column_coverage. There is no second copy that can drift.
What it adds: one constant and a two-field struct. The constant's comment explains the choice: 1 MiB is about the size of the id column alone for tens of thousands of rows, so the extra read is bounded by 1 MiB on small tables and is smaller than the join on large ones. This meets invariant 11 (bounded resource use). A wide table with a tail larger than 1 MiB keeps the join until the next index build. That is main's cost, not a regression, and #1156 will make that build cheaper.
Blocking
None.
Optional
O7. The changelog note now claims too much. The reply to the first review says the 200-character limit leaves no room for the tail condition. 1627e499 removed the condition, so the note now says that a one-row load --mode merge always reads through the index. For a row that already exists, it does so only while the uncovered tail is small (fewer fragments than the index covers, and at most 1 MiB). Otherwise it reads the id column of every fragment, as before. writes.md is correct. A version that fits in 200 characters (178):
- On a table with an
idindex, a one-row keyedinsert,update ... where <key> = vorload --mode mergereads its rows through the index while the unindexed tail is small.
A note on the new wide-tail test. The test makes the tail wide through 1 KiB ids. In that shape the join also reads the wide ids, so the test checks the routing rule, not the case that motivated it (wide non-key columns behind narrow ids). That is enough for a routing test. I do not ask for a change.
Liability
The delta removes liability. The byte budget turns an unbounded read into a bounded one, using a fact the manifest already holds. exact_id_scan removes a duplicate filter and scan. The new edge test pins the #1225 bound, which had no test. What it adds is one constant, which needs a revisit if Lance's indexed merge stops scanning the tail for every column. After five more changes of this kind, the budget should live in the single owner suggested in O1.
Validation
All on a clean detached worktree at 1627e499, debug build, cargo dev-test, RUST_MIN_STACK=16777216, rustc 1.97.1. Lance is 13.0.0 in Cargo.lock. Temporary edits are removed.
- Staged tests (
table_store::staged_tests::{upsert,all_present,keyed_}): 8 passed, includingall_present_upsert_keeps_the_join_over_a_wide_uncovered_tail_issue_1248,upsert_probe_runs_only_while_its_in_list_copies_fit_one_keyed_batch_issue_1225andupsert_skips_the_merge_join_when_the_id_btree_decides_its_rows_issue_1248. - Regression before and after. I removed the byte condition at
table_store.rs:3706-3709temporarily, which gives the switch rule of1d27ac7f. Then the wide-tail test fails atstaged_tests.rs:1037withleft: (0, 0, 1),right: (1, 0, 0): the wide tail takes the known-present route, the reason given in B1. The narrow half andupsert_skips_…_issue_1248still pass. At1627e499it passes. I ran this at the head with the condition removed, not at1d27ac7fitself, because the merge of main changedCargo.lock. The new test callskey_column_uncovered_tail, which does not exist at1d27ac7f. - The edge test guards the bound. With the
IN-list.filterremoved temporarily,upsert_probe_runs_only_…_issue_1225fails atstaged_tests.rs:1110withleft: (0, 1, 0),right: (1, 0, 0). This closes O2. - Integration:
write_cost::20 passed (includeskeyed_insert_routes_through_fenced_adapter_only),traversal_indexed::14 passed.forbidden_apis44 passed,lance_surface_guards74 passed (2 ignored). These are the other callers of the changed coverage helper. - GQT cannot assert the route or the bytes read, so the staged Rust test is the right owner, as agreed in the last review.
- CI at
1627e499: 28 checks pass and 4 are skipped. git merge-treewith mainfd58fd33is clean. #1349 also changestable_store.rs,mutation.rsandwrites.md, and it merges cleanly with this PR. #1235 and #1175 conflict with main in other files, not with this PR.- No
Decision neededcomments on this PR. The replies posted at 06:39Z (comments 6816686483 and 6816686833) were read before posting.
Limits
- I did not run the DST golden or GQT
--measurelocally. CI's DST pinned suite passes at this head, and the author reportsdst_bench_cost_count_goldenpassing at5f082b22. - I did not measure bytes read with the budget. The 1 MiB value is argued in the code comment, not measured on real graphs.
- No wall-time, S3, clippy or rustfmt run locally (CI clippy passes).
…dexed tail is small The note said a one-row keyed insert, update by key or merge load always reads through the id index. For a row that already exists, the indexed route is taken only while the uncovered tail has fewer fragments than the index covers and at most 1 MiB of data files; otherwise the join still reads the id column of every fragment. Reword it to state that condition within the 200-character limit.
|
Reply to the delta review of O7: valid, fixed in
The condition was lost when the earlier 365-character note was cut to fit the 200-character limit. It was not dropped on purpose. The wide-tail test. You did not ask for a change, so I made none. I agree that the test checks the routing rule, not the case with wide non-key columns behind narrow ids. Description. One sentence was out of date. It said that this PR retargets Gates. CI at |
ragnorc
left a comment
There was a problem hiding this comment.
Summary
This is a delta review of 2a3c3585. The last review (5870312922) covered 1627e499.
What changed. One commit, 2a3c3585. It changes only the changelog note changelog.d/keyed-writes-through-the-id-index.performance.md. It answers O7 and uses the wording that O7 gave. No code, test or guide changed.
Outcome. Ready to merge as far as this delta goes. I found no blocking or optional items. O7 is closed.
Why the new note is correct
The old note said that a one-row keyed write always reads through the index. The new note adds the condition: "while the unindexed tail is small". I checked this against the routing rule at this head (table_store.rs lines 3658-3713):
- Rows that are all new. The write takes the index route only while the uncovered fragments, times the batch rows, are at most
KEYED_WRITE_MAX_ROWS. For one row, this is a limit on the tail's fragment count. - Rows that are all present. The write takes the index route only when the tail has fewer fragments than the index covers and its data files are at most 1 MiB (
KNOWN_PRESENT_TAIL_MAX_BYTES). - No
idBTREE, or a mixed batch. The write keeps the join. The note already limits itself to "a table with anidindex" and "one-row" writes.
So "small" is true for both cases, in different units. The exact rule is in docs/dev/writes.md, which is still correct. The note does not link to it, but no performance note in changelog.d links to a guide (0 of 34), so I do not ask for a link.
Blocking
None.
Optional
None.
Liability
None added. The note now promises less than before, and only what the code does. A future change to the routing rule must update this note too, as before.
Validation
- Clean detached worktree at
2a3c3585. release_notes.check_note_capson the new note: passes (the 200-character rule indocs/dev/documentation.md).python3 scripts/release_notes.py preview --target HEAD: exit 0. The new note shows in the Performance section.python3 -m unittest test_release_notes(inscripts/): OK.git merge-tree --write-treewith main2bfb3f35: clean.- CI at
2a3c3585: 28 checks pass and 4 are skipped (run 43235797127). - Regression test: not needed. The delta fixes no behavior; it changes a release note only. The routing tests from the last review are unchanged.
- No
Decision neededcomments on this PR. No comments or reviews were added after the last review.
Limits
- I did not build or run Rust tests at this head. The delta changes no Rust file, so the results from the last review at
1627e499still apply to the code.
|
Reply to the delta review of Verdict. I agree: O7 is closed, and there are no open items. The new note is true for both routes. One side fact in the review is wrong, but it does not change the verdict. Evidence for the note. I read the routing rule at
Correction. The review says that 0 of 34 Merge. CI at |
Closes #1248. On an indexed table, a one-row keyed write read the
idcolumn of every fragment, so its cost grew with the table. A keyedinsert, aload --mode mergeand a Blob cell write stage asUpsert, which forces Lance's full-table merge join (use_index(false)). Anupdate ... where <key> = valso scanned the key property, a user column the id BTREE does not index, before it staged. At 262,144 rows in 64 fragments, a one-row update read 4.78 MB in 150 to 212 storage requests; a keyed insert read 2.1 MB in 83.Two changes. An update whose predicate has
<key> = von a single String@keyadds the derivedid = vconjunct to its match scan when the table has an id BTREE, reusing the conjunct a node delete already adds (key_id_conjunct). InTableStore::stage_keyed_write, anUpsertfirst runs the exact-id probe strict insert already runs: all new, it stages through the absence-proven insert writer; all present, it stages as a known-present update; mixed, it keeps the join. The guards: the probe runs only while the fragments the BTREE leaves uncovered times the source ids stay within one keyed batch (8,192), since Lance plans theINlist once per uncovered fragment (#1225's 11 GB); and the known-present switch happens only while the BTREE covers more fragments than it leaves (requests) and the uncovered fragments' recorded data files hold at most 1 MiB (KNOWN_PRESENT_TAIL_MAX_BYTES, bytes), because that indexed merge scans each uncovered fragment for every source column, where the join reads only theidcolumn of every fragment. An unrecorded file size keeps the join. Without a BTREE nothing changes.key_column_uncovered_tailreports the tail's fragment count and data-file bytes under the rulekey_column_coveragealready applies; both read one helper.Measured with GQT
--measureon the DST store: aPerson { id @key, name, v }table seeded with N rows in 4,096-row commits and indexed, then seven one-row steps. At N = 262,144, main → this branch: column-patch update 212 → 34 requests and 4,782,278 → 93,360 bytes; whole-row update 150 → 28 and 4,783,976 → 43,664; keyed insert 83 → 18 and 2,108,519 → 19,173; merge of one present row 76 → 17 and 2,088,712 → 14,522; merge of one new row 84 → 28 and 2,108,917 → 76,255. From 1,024 to 262,144 rows these now grow only with the index's pages and the uncovered tail. On an unindexed table every step at every size is identical to main.upsert_skips_the_merge_join_when_the_id_btree_decides_its_rows_issue_1248fails on main. The DST golden addsKeyedInsert,KeyedUpdate,MergeLoadPresentandMergeLoadNewat 2 and 16 fragments; theirl.getrows on main were 10/25, 13/41, 10/24 and 10/24, here 9/10, 8/8, 8/8 and 4/4 since main afc4be7 was merged in (before that merge: 9/10, 11/11, 9/9 and 7/5). Main's #1291 keeps the dataset a write commits in the table handle cache, so each write after the first on the scenario's handle opens no manifest; the first write,KeyedInsert, is unchanged. One existing line moves:InsertLegacyl.get15 → 13, its keyed insert now taking the fenced insert.write_cost::keyed_insert_routes_through_fenced_adapter_onlynow asserts the fenced insert and no merge on the indexed standard graph.For review: the decision to challenge is the coverage rule for the known-present switch. It is a cost choice made from pinned fragment metadata, never a correctness precondition. A wrong choice costs reads, not rows, and both of its costs are bounded: the fragment rule bounds the extra requests and the byte budget bounds the extra bytes whatever the row width. Under GQT's
s3-likemodel this lowers requests and bytes but not latency: at 262,144 rows the first update on a handle is 90 ms slower (370 → 460) because an index lookup is a chain of dependent reads while the model runs a scan's 150+ GETs at once with no queue; later updates and inserts are 18 to 36 ms faster, merges 18 to 36 ms slower. No wall clock or S3 run backs either direction. With a 64-fragment uncovered tail on 16,384 rows, a whole-row update keeps 98 requests and reads 375,488 → 268,693 bytes, modeled 372 → 426 ms. The probe's peak memory at the 8,192-id bound is argued from #1231, not measured. Mixed upserts still take the join; splitting them into an insert and an update on one table is not in this change.changelog.d/release.jsonon main already targets v0.15.1, so this PR does not change it. Locally the engine binaries pass exceptdb::upgrade::tests::interrupted::unreadable_legacy_object_asks_for_a_rerun, which needs a non-root user. Report: https://github.com/ragnorc/omnigraph/blob/autoresearch/keyed-write-scans-20261109/docs/autoresearch/keyed-write-scans-20261109.mdWorkflow run: run-wf-autoresearch-modernrelay-omnigraph-20261109T0900Z-d8451b
Response run: run-wf-respond-modernrelay-omnigraph-pr-20261109T2040Z-957a8c
Response run: run-wf-respond-modernrelay-omnigraph-pr-20261112T0532Z-b59648
Response run: run-wf-respond-modernrelay-omnigraph-pr-20261112T1412Z-63cfb9
Response run: run-wf-respond-modernrelay-omnigraph-pr-20261112T1942Z-e93b73