Repository navigation
perf(engine): pass an unfiltered scan's limit to Lance - #1342
Conversation
A Limit directly over the projection of a table scan with no pushed filter, no full-text query and no runtime filter now reaches Lance as Scanner::limit, so Lance plans only the fragments that hold the first live rows instead of opening up to fragment_readahead fragments before the engine drops the stream. Closes ModernRelay#1335
ragnorc
left a comment
There was a problem hiding this comment.
Recommendation: ready to merge after one test change (non-blocking). I found no blocking defect in the code. This note is a comment because the reviewing account is the PR author.
Reviewed commit 6be33daa918c7abc417811ba374c47dc5846d46d. The base is 2c4b2f12c87b41a58c1b32ab44678fcf34e66e2a.
Why, what, how, impact
- Why. A
limitover an unfiltered table scan stopped the engine early, but Lance had already started reads on up tofragment_readaheadfragments (16 local, 128 on an object store). - What. When the
Limitsits directly over the projection of a read with no pushed filter, no full-text query and no runtime filter,stream_batchesnow also gives its row count to Lance asScanner::limit. - How. Lance 13 then takes
Scanner::plan_scan_range→plan_metadata_scan_range(lance-13.0.0/src/dataset/scanner.rs:3159-3200). It keeps only the fragments that hold the first live rows, from manifest row and deletion counts. - Impact. Fewer fragment opens and range reads for unfiltered
limitqueries. The rows do not change.
Correctness
I checked the cases where a Lance limit could drop rows that the engine still needs. None applies:
- A filter the scan does not push.
build_lance_filter_exprskips a filter it cannot express, sofilter_expr.is_none()alone would not prove "no filter". But the optimizer pushes a conjunct only whenPlanSource::filter_pushablesays yes (optimizer.rs:1991,plan_source.rs:450), and that calls the sameir_expr_to_df_expr. A filter that is not pushed stays as aFilternode. Thenlimit_over_scandoes not match, because it requiresLimit → Projection → Scan. Soreads_every_row()is true only when no filter exists. - Offset or pagination.
PhysicalNode::Limithas no offset.Page(withresume) is a different node, so it does not get the Lance limit. - Rows that the projection removes.
Projectionhas no filter and no distinct. - Runtime sieve. The sieve runs after Lance, and
!markedexcludes it. - Full-text query.
reads_every_row()excludes it. This is correct: a limit there would become the search's k, and it could cut rows before a post-filter. - Ordering and old formats. Lance does not use the range plan when there is an ordering, a filter, v1 files or incomplete fragment counts. In these cases it still applies
LimitExecafter the read, so the rows are correct.
Validation
Local runs used a detached worktree at the exact head. The build ran cargo dev-test --test integration scan_limit -- --nocapture.
| Engine source | unfiltered_limit_reads_only_the_first_fragment |
pushed_limit_skips_deleted_rows_and_spans_fragments |
|---|---|---|
base 2c4b2f12 (pipelined.rs and scan.rs reverted, PR tests kept) |
fails as predicted: limit 1 read 15 of 25 data files |
passes |
head 6be33daa |
passes (1 of 25) | passes |
This gives the before/after proof for the cost claim. The base run read 15 files, not the 16–17 in the PR text, because read-ahead timing varies. The failure reason is the same.
Deletion-vector path (temporary probe, now removed). I added a probe test and then removed it:
- Deleting
limit_pushdown_0created 0 deletion files. Lance drops a fragment when its only row is deleted, so the PR's deletion test never reaches the deletion-vector count inplan_metadata_scan_range. - Deleting
Bobfrom the 4-row loaded fragment created 1 deletion file. After that:limit 3returned[Alice, Charlie, Diana]. It read 1 data file at head and 17 at base.limit 4returned[Alice, Charlie, Diana, limit_pushdown_1]. It read 2 data files at head and 18 at base.
The code handles a real deletion vector correctly. The PR's test does not check it.
Exact-commit CI. All checks pass, including Test Workspace, Lint (clippy), GQT (ordinary/served/dst), DST pinned suite and RustFS S3. Storage Upgrade Compatibility was still pending when I posted.
Not verified. I did not measure object-store reads. I did not run clippy, rustfmt or the GQT corpus locally (CI covers them).
GQT. The rows do not change by design. GQT can check rows and plan facts, but --measure and --baseline only report numbers and never fail a case. So GQT cannot express this cost contract. The Rust cost test is the right owner.
Findings
No blocking findings.
O1 (recommended): test a real deletion vector. In pushed_limit_skips_deleted_rows_and_spans_fragments_issue_1335, delete a row from a multi-row fragment. For example, delete Bob from the loaded fragment and assert that limit 4 returns 4 live rows without Bob. The current deletes remove whole single-row fragments. They never test the manifest deletion count, which is the one new input this change gives to row selection.
O2: use the existing cost owners. docs/dev/testing.md gives two rules that apply here:
- "object-store counters belong in
tests/helpers/cost.rs". The new module readsIOTrackerrequests directly.helpers/cost.rsalready hasraw_io_probesanddrain_probed_iooverProbedStores. A smalldistinct_data_files_readhelper there would keep one place for this plumbing. - The DST storage-cost golden: "Add or update a row when a change … changes what an existing one reads." A labeled scenario such as
ScanLimitUnfiltered, on a reopened handle with_-prefixed setup, would record the exact calls. It runs onshared-memory://, so it also covers the object-store path that the PR lists as not measured.
O3 (design, optional): the condition copies a Lance rule. For correctness, only two things must be true: no step after Lance removes rows (the sieve), and the limit must not change search semantics (full-text). A pushed prefilter is safe, because Lance applies limit after its own filter (Lance read guide; Scanner::create_plan). reads_every_row() also excludes pushed filters only because Lance 13 does not prune when a filter exists. That makes OmniGraph depend on Lance's internal pruning rule.
Lance 13's FilteredReadExec already has scan_range_after_filter for exact index results (filtered_read.rs:842-866), but Scanner does not set it yet. If the condition were "no runtime sieve and no full-text query", a later Lance could prune filtered limits with no OmniGraph change, and the gate would name only OmniGraph's own invariant.
This is a tradeoff, not a defect. The current gate is easier to read and keeps the filtered path exactly as it was. I did not test the wider condition. It would need the GQT same as v1 corpus.
O4 (nit). filter.is_none() && !marked: filter comes from the runtime-filter slot (scan/mod.rs:246), so it is Some only when the scan is marked. !marked alone is enough.
O5 (PR text). The "Local gates" list still says cargo dev-test: WORKSPACE_PLACEHOLDER. CI's Test Workspace passed on this head, so please fill in or remove that line.
Liability
- Removes: wasted read-ahead on every unfiltered
limit. - Adds: one predicate (
NodeRead::reads_every_row) and one call through the existingScanTuning::limit. It adds no new state, API or plan field. - Ownership: the decision lives where the final Lance read is known, which is one owner. Five similar pushdowns would all add clauses to this same gate, so it should stay a single named predicate.
- Visibility: explain output does not show that the limit reached Lance. A GQT
expect planline therefore cannot pin it. This is acceptable while the DST golden (O2) carries the cost contract.
Context
- No other open PR changes
pipelined.rs,NodeRead::configure/planorlower.rs::limit_over_scan. - #1316 and #1334 (GQ 3) touch
lower.rsandengine/scan.rs, but only to addIRExpr::Neg/Callmatch arms. They do not conflict with this change. - Under GQ 3, a new expression kind that
ir_expr_to_df_exprcannot push stays aFilternode. Correctness point 1 therefore still holds. - #1257 (filtered limit) is the natural next step, and O3 is the change that would let it follow Lance.
Summary
A
limitdirectly over the projection of an unfiltered table scan now reaches Lance asScanner::limit. With no filter and no ordering, Lance plans only the fragments that hold the first live rows from the manifest's row and deletion counts, somatch { $p: Person } return { $p.name } limit 1reads one fragment instead of up tofragment_readaheadfragments (16 on a local store, 128 on an object store) before the engine drops the stream.Closes #1335
Reproduction
On main
2c4b2f12c87b41a58c1b32ab44678fcf34e66e2a, the new testscan_limit_cost::unfiltered_limit_reads_only_the_first_fragment_issue_1335builds aPersontable of 25 fragments (the loaded rows, then 24 single-row inserts), opens a fresh handle, and counts the distinct data files a cold read touches at the table store's own tracker (which also sees Lance's direct local reader):limit 1, main: 16 or 17 of 25 files, depending on read-ahead timing;limit 1, this branch: 1 of 25.Root cause
Lowering::limit_over_scan(engine/lower.rs) finds theLimitover the projection over a scan, but only passes its rows as the scan's gather target (ScanExec::with_gather_rows), which decides when a batch is sent, not what Lance reads.stream_batches(engine/operators/scan/pipelined.rs) configured the scanner throughNodeRead::planwith no limit, so Lance'sFilteredReadExecstartedfragment_readaheadfragment reads (io_parallelism * 2,lance-13.0.0/src/io/exec/filtered_read.rs:604-606) before the engine stopped by dropping the stream.The change
NodeRead::reads_every_row(engine/scan.rs): no pushed filter and no full-text query. A gate's eligible-id list becomes a pushed filter, so it is excluded too.stream_batchespasses theLimit's rows toScanTuning::limitonly when the read reads every row, carries no runtime-filter marker and no filled sieve. Ranked scans never take the pipelined path.Scanner::plan_scan_range,lance-13.0.0/src/dataset/scanner.rs:3159-3166), and the runtime sieve runs after Lance, so those cases keep the early stop alone.Rows do not change: Lance reads fragments in order, so the rows are the same first rows the early stop returned, and a
limitwithoutordermay return any rows anyway.Decisions and alternatives
Limit, but onlyNodeReadknows whether its final Lance read has a filter (pushed IR filters, the gate's eligible ids, hoisted full-text queries). Deciding there keeps one condition in one place.with_lance_limitonScanExec. It would duplicategather_rowswith the same value; the comment onstream_batchesnow says the gather target is also Lance's limit on an unfiltered read.Tests
scan_limit_cost::unfiltered_limit_reads_only_the_first_fragment_issue_1335(new moduletests/integration/scan_limit_cost.rs): fails on main (16–17 of 25 files), passes here (1 of 25).scan_limit_cost::pushed_limit_skips_deleted_rows_and_spans_fragments_issue_1335: with two inserted rows deleted,limit 6spans fragments and returns 6 live rows, none deleted; a filteredlimit 1(not pushed) still finds the row in the last fragment.docs/dev/execution.md(theScanrow) states when the limit reaches Lance. Changelog:changelog.d/unfiltered-limit-reaches-lance.performance.md.Local gates (base
2c4b2f12)cargo fmt --all --check: clean.cargo dev-clippy(workspace, all targets,-D warnings -W clippy::dbg_macro): clean.cargo test -p omnigraph-gqt --locked: 467 corpus cases passed (theirexpect same as v1steps compare rows with the reference engine), harness tests passed.cargo test -p omnigraph-dst --locked(fromcrates/omnigraph-dst): passed, includingdst_bench_cost_count_golden, so no storage-cost golden line changed (no labeled scenario runs an unfilteredlimit).cargo dev-test --no-fail-fastwithRUST_MIN_STACK=16777216as CI sets it: 5,166 passed, 10 failed, all from the local sandbox (gVisor, running as root), none on the scan path: sevenomnigraph-benchtests (noCpus_allowed_listin/proc/self/status, symlink refusals),cli_cluster::managed_lifecycle_capture_and_retry_records_refuse_symlinks_without_reading_targets(symlink refusal), anddb::upgrade::tests::interrupted::unreadable_legacy_object_asks_for_a_rerun(root reads achmod 000object). CI on this pull request is the authoritative run for those.scripts/check-docs.py,scripts/check-agents-md.sh: OK.Risks and what was not verified
writer_version, or with missing fragment row counts, falls back inside Lance to its unpruned path, so the change cannot make such reads worse.--measurebaselines recorded elsewhere for unfilteredlimitqueries will show fewer reads.Workflow run: run-wf-reproduce-modernrelay-omnigraph-issue-20261110T1212Z-de86c0