Skip to content

perf(engine): pass an unfiltered scan's limit to Lance - #1342

Merged
ragnorc merged 1 commit into
ModernRelay:mainfrom
ragnorc:fix/1335-scan-limit-pushdown
Nov 10, 2026
Merged

ragnorc merged 1 commit into
ModernRelay:mainfrom
ragnorc:fix/1335-scan-limit-pushdown

Conversation

@ragnorc

@ragnorc ragnorc commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

Summary

A limit directly over the projection of an unfiltered table scan now reaches Lance as Scanner::limit. With no filter and no ordering, Lance plans only the fragments that hold the first live rows from the manifest's row and deletion counts, so match { $p: Person } return { $p.name } limit 1 reads one fragment instead of up to fragment_readahead fragments (16 on a local store, 128 on an object store) before the engine drops the stream.

Closes #1335

Reproduction

On main 2c4b2f12c87b41a58c1b32ab44678fcf34e66e2a, the new test scan_limit_cost::unfiltered_limit_reads_only_the_first_fragment_issue_1335 builds a Person table of 25 fragments (the loaded rows, then 24 single-row inserts), opens a fresh handle, and counts the distinct data files a cold read touches at the table store's own tracker (which also sees Lance's direct local reader):

  • full scan: 25 of 25 files (the meter sees every fragment);
  • limit 1, main: 16 or 17 of 25 files, depending on read-ahead timing;
  • limit 1, this branch: 1 of 25.

Root cause

  • Lowering::limit_over_scan (engine/lower.rs) finds the Limit over the projection over a scan, but only passes its rows as the scan's gather target (ScanExec::with_gather_rows), which decides when a batch is sent, not what Lance reads.
  • stream_batches (engine/operators/scan/pipelined.rs) configured the scanner through NodeRead::plan with no limit, so Lance's FilteredReadExec started fragment_readahead fragment reads (io_parallelism * 2, lance-13.0.0/src/io/exec/filtered_read.rs:604-606) before the engine stopped by dropping the stream.

The change

  • NodeRead::reads_every_row (engine/scan.rs): no pushed filter and no full-text query. A gate's eligible-id list becomes a pushed filter, so it is excluded too.
  • stream_batches passes the Limit's rows to ScanTuning::limit only when the read reads every row, carries no runtime-filter marker and no filled sieve. Ranked scans never take the pipelined path.
  • Lance itself pushes no limit through a filter or an ordering (Scanner::plan_scan_range, lance-13.0.0/src/dataset/scanner.rs:3159-3166), and the runtime sieve runs after Lance, so those cases keep the early stop alone.

Rows do not change: Lance reads fragments in order, so the rows are the same first rows the early stop returned, and a limit without order may return any rows anyway.

Decisions and alternatives

  • Decide at execution, not in lowering. The lowering knows the Limit, but only NodeRead knows whether its final Lance read has a filter (pushed IR filters, the gate's eligible ids, hoisted full-text queries). Deciding there keeps one condition in one place.
  • Not chosen: keep the early stop only (the issue's alternative). It leaves the read-ahead Lance started.
  • Not chosen: a separate with_lance_limit on ScanExec. It would duplicate gather_rows with the same value; the comment on stream_batches now says the gather target is also Lance's limit on an unfiltered read.

Tests

  • scan_limit_cost::unfiltered_limit_reads_only_the_first_fragment_issue_1335 (new module tests/integration/scan_limit_cost.rs): fails on main (16–17 of 25 files), passes here (1 of 25).
  • scan_limit_cost::pushed_limit_skips_deleted_rows_and_spans_fragments_issue_1335: with two inserted rows deleted, limit 6 spans fragments and returns 6 live rows, none deleted; a filtered limit 1 (not pushed) still finds the row in the last fragment.
  • Docs: docs/dev/execution.md (the Scan row) states when the limit reaches Lance. Changelog: changelog.d/unfiltered-limit-reaches-lance.performance.md.

Local gates (base 2c4b2f12)

  • cargo fmt --all --check: clean.
  • cargo dev-clippy (workspace, all targets, -D warnings -W clippy::dbg_macro): clean.
  • cargo test -p omnigraph-gqt --locked: 467 corpus cases passed (their expect same as v1 steps compare rows with the reference engine), harness tests passed.
  • cargo test -p omnigraph-dst --locked (from crates/omnigraph-dst): passed, including dst_bench_cost_count_golden, so no storage-cost golden line changed (no labeled scenario runs an unfiltered limit).
  • cargo dev-test --no-fail-fast with RUST_MIN_STACK=16777216 as CI sets it: 5,166 passed, 10 failed, all from the local sandbox (gVisor, running as root), none on the scan path: seven omnigraph-bench tests (no Cpus_allowed_list in /proc/self/status, symlink refusals), cli_cluster::managed_lifecycle_capture_and_retry_records_refuse_symlinks_without_reading_targets (symlink refusal), and db::upgrade::tests::interrupted::unreadable_legacy_object_asks_for_a_rerun (root reads a chmod 000 object). CI on this pull request is the authoritative run for those.
  • scripts/check-docs.py, scripts/check-agents-md.sh: OK.

Risks and what was not verified

  • Object-store read counts were not measured; the gain there follows from the same Lance path (128 fragments of read-ahead at the cloud default).
  • A table written before Lance recorded writer_version, or with missing fragment row counts, falls back inside Lance to its unpruned path, so the change cannot make such reads worse.
  • GQT --measure baselines recorded elsewhere for unfiltered limit queries will show fewer reads.

Workflow run: run-wf-reproduce-modernrelay-omnigraph-issue-20261110T1212Z-de86c0

A Limit directly over the projection of a table scan with no pushed
filter, no full-text query and no runtime filter now reaches Lance as
Scanner::limit, so Lance plans only the fragments that hold the first
live rows instead of opening up to fragment_readahead fragments before
the engine drops the stream.

Closes ModernRelay#1335

@ragnorc ragnorc left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Recommendation: ready to merge after one test change (non-blocking). I found no blocking defect in the code. This note is a comment because the reviewing account is the PR author.

Reviewed commit 6be33daa918c7abc417811ba374c47dc5846d46d. The base is 2c4b2f12c87b41a58c1b32ab44678fcf34e66e2a.

Why, what, how, impact

  • Why. A limit over an unfiltered table scan stopped the engine early, but Lance had already started reads on up to fragment_readahead fragments (16 local, 128 on an object store).
  • What. When the Limit sits directly over the projection of a read with no pushed filter, no full-text query and no runtime filter, stream_batches now also gives its row count to Lance as Scanner::limit.
  • How. Lance 13 then takes Scanner::plan_scan_range → plan_metadata_scan_range (lance-13.0.0/src/dataset/scanner.rs:3159-3200). It keeps only the fragments that hold the first live rows, from manifest row and deletion counts.
  • Impact. Fewer fragment opens and range reads for unfiltered limit queries. The rows do not change.

Correctness

I checked the cases where a Lance limit could drop rows that the engine still needs. None applies:

  1. A filter the scan does not push. build_lance_filter_expr skips a filter it cannot express, so filter_expr.is_none() alone would not prove "no filter". But the optimizer pushes a conjunct only when PlanSource::filter_pushable says yes (optimizer.rs:1991, plan_source.rs:450), and that calls the same ir_expr_to_df_expr. A filter that is not pushed stays as a Filter node. Then limit_over_scan does not match, because it requires Limit → Projection → Scan. So reads_every_row() is true only when no filter exists.
  2. Offset or pagination. PhysicalNode::Limit has no offset. Page (with resume) is a different node, so it does not get the Lance limit.
  3. Rows that the projection removes. Projection has no filter and no distinct.
  4. Runtime sieve. The sieve runs after Lance, and !marked excludes it.
  5. Full-text query. reads_every_row() excludes it. This is correct: a limit there would become the search's k, and it could cut rows before a post-filter.
  6. Ordering and old formats. Lance does not use the range plan when there is an ordering, a filter, v1 files or incomplete fragment counts. In these cases it still applies LimitExec after the read, so the rows are correct.

Validation

Local runs used a detached worktree at the exact head. The build ran cargo dev-test --test integration scan_limit -- --nocapture.

Engine source unfiltered_limit_reads_only_the_first_fragment pushed_limit_skips_deleted_rows_and_spans_fragments
base 2c4b2f12 (pipelined.rs and scan.rs reverted, PR tests kept) fails as predicted: limit 1 read 15 of 25 data files passes
head 6be33daa passes (1 of 25) passes

This gives the before/after proof for the cost claim. The base run read 15 files, not the 16–17 in the PR text, because read-ahead timing varies. The failure reason is the same.

Deletion-vector path (temporary probe, now removed). I added a probe test and then removed it:

  • Deleting limit_pushdown_0 created 0 deletion files. Lance drops a fragment when its only row is deleted, so the PR's deletion test never reaches the deletion-vector count in plan_metadata_scan_range.
  • Deleting Bob from the 4-row loaded fragment created 1 deletion file. After that:
    • limit 3 returned [Alice, Charlie, Diana]. It read 1 data file at head and 17 at base.
    • limit 4 returned [Alice, Charlie, Diana, limit_pushdown_1]. It read 2 data files at head and 18 at base.

The code handles a real deletion vector correctly. The PR's test does not check it.

Exact-commit CI. All checks pass, including Test Workspace, Lint (clippy), GQT (ordinary/served/dst), DST pinned suite and RustFS S3. Storage Upgrade Compatibility was still pending when I posted.

Not verified. I did not measure object-store reads. I did not run clippy, rustfmt or the GQT corpus locally (CI covers them).

GQT. The rows do not change by design. GQT can check rows and plan facts, but --measure and --baseline only report numbers and never fail a case. So GQT cannot express this cost contract. The Rust cost test is the right owner.

Findings

No blocking findings.

O1 (recommended): test a real deletion vector. In pushed_limit_skips_deleted_rows_and_spans_fragments_issue_1335, delete a row from a multi-row fragment. For example, delete Bob from the loaded fragment and assert that limit 4 returns 4 live rows without Bob. The current deletes remove whole single-row fragments. They never test the manifest deletion count, which is the one new input this change gives to row selection.

O2: use the existing cost owners. docs/dev/testing.md gives two rules that apply here:

  • "object-store counters belong in tests/helpers/cost.rs". The new module reads IOTracker requests directly. helpers/cost.rs already has raw_io_probes and drain_probed_io over ProbedStores. A small distinct_data_files_read helper there would keep one place for this plumbing.
  • The DST storage-cost golden: "Add or update a row when a change … changes what an existing one reads." A labeled scenario such as ScanLimitUnfiltered, on a reopened handle with _-prefixed setup, would record the exact calls. It runs on shared-memory://, so it also covers the object-store path that the PR lists as not measured.

O3 (design, optional): the condition copies a Lance rule. For correctness, only two things must be true: no step after Lance removes rows (the sieve), and the limit must not change search semantics (full-text). A pushed prefilter is safe, because Lance applies limit after its own filter (Lance read guide; Scanner::create_plan). reads_every_row() also excludes pushed filters only because Lance 13 does not prune when a filter exists. That makes OmniGraph depend on Lance's internal pruning rule.

Lance 13's FilteredReadExec already has scan_range_after_filter for exact index results (filtered_read.rs:842-866), but Scanner does not set it yet. If the condition were "no runtime sieve and no full-text query", a later Lance could prune filtered limits with no OmniGraph change, and the gate would name only OmniGraph's own invariant.

This is a tradeoff, not a defect. The current gate is easier to read and keeps the filtered path exactly as it was. I did not test the wider condition. It would need the GQT same as v1 corpus.

O4 (nit). filter.is_none() && !marked: filter comes from the runtime-filter slot (scan/mod.rs:246), so it is Some only when the scan is marked. !marked alone is enough.

O5 (PR text). The "Local gates" list still says cargo dev-test: WORKSPACE_PLACEHOLDER. CI's Test Workspace passed on this head, so please fill in or remove that line.

Liability

  • Removes: wasted read-ahead on every unfiltered limit.
  • Adds: one predicate (NodeRead::reads_every_row) and one call through the existing ScanTuning::limit. It adds no new state, API or plan field.
  • Ownership: the decision lives where the final Lance read is known, which is one owner. Five similar pushdowns would all add clauses to this same gate, so it should stay a single named predicate.
  • Visibility: explain output does not show that the limit reached Lance. A GQT expect plan line therefore cannot pin it. This is acceptable while the DST golden (O2) carries the cost contract.

Context

  • No other open PR changes pipelined.rs, NodeRead::configure/plan or lower.rs::limit_over_scan.
  • #1316 and #1334 (GQ 3) touch lower.rs and engine/scan.rs, but only to add IRExpr::Neg/Call match arms. They do not conflict with this change.
  • Under GQ 3, a new expression kind that ir_expr_to_df_expr cannot push stays a Filter node. Correctness point 1 therefore still holds.
  • #1257 (filtered limit) is the natural next step, and O3 is the change that would let it follow Lance.

@ragnorc
ragnorc added this pull request to the merge queue Nov 10, 2026
Merged via the queue into ModernRelay:main with commit 62b03de Nov 10, 2026
34 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

performance: pass a query's limit to Lance when the scan under it has no filter, so it reads only the fragments it needs

1 participant