You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
performance: pass a query's limit to Lance when the scan under it has no filter, so it reads only the fragments it needs #1335
A limit directly over the projection over a table scan never reaches Lance.
limit_over_scan finds that shape and passes the limit only as the number of rows the scan gathers before it sends a batch (lower.rs:355-366, 641-643, pipelined.rs:180).
NodeRead::configure never calls Scanner::limit (scan.rs:255-281). The engine stops by dropping the stream.
By then Lance has started up to fragment_readahead fragments (2 × I/O parallelism: 16 on a local store, 128 on an object store) and scheduled their reads, up to the scan's 64 MiB I/O buffer (lance-13.0.0/src/io/exec/filtered_read.rs:603-606).
Proposed change
Pass Scanner::limit(Some(k), None) when the scan under the Limit has no filter, no runtime filter and no ranking.
Upstream primitive at the pin: with no filter and no ordering, Scanner::plan_scan_range (lance-13.0.0/src/dataset/scanner.rs:3159-3200) takes plan_metadata_scan_range (:3110-3155). It keeps only the fragments that hold the first k live rows, using manifest row and deletion counts, so limit 10 reads one fragment's first pages.
Gain: fewer fragment opens and range reads for limit queries without a filter. The rows are the same: fragments are read in order, and a limit without order may return any rows.
Cost and risk: small. The runtime filter is applied by the engine after Lance, so a Lance limit there would drop rows; the condition must exclude it. Lance itself pushes nothing when a filter exists (scanner.rs:3160-3162). DST storage-cost lines and GQT --measure baselines for such queries would change.
Scan lowering and NodeRead (engine/lower.rs, engine/scan.rs), the DST storage-cost golden. Related: #1257 (the filtered case), #887 (early stop), #1165 (ordered keyset limit, a different case).
Alternatives considered
Keep early stop only. It leaves the read-ahead that Lance started before the stream was dropped.
A query with a limit and no filter reads more of the table than it returns. Reproduced on main 2c4b2f12. The new test unfiltered_limit_reads_only_the_first_fragment_issue_1335 builds a table of 25 fragments (file groups that Lance stores separately), and a cold limit 1 read opens 16 or 17 of them.
Why
Omnigraph never tells Lance about the limit. It reads, then stops the stream after the first rows (pipelined.rs). By then Lance has already started reads on up to 16 fragments on a local disk, or 128 on an object store.
Pull request
#1342 gives the limit to Lance when the read has no filter. Lance then reads only the fragments that hold the first rows, so the test reads 1 of 25 files.
Commands, evidence and Lance notes
Test module: crates/omnigraph/tests/integration/scan_limit_cost.rs. It counts distinct data files at the table store's own I/O tracker, which also sees Lance's direct local reads. Full scan: 25 of 25 files. limit 1 on main: 16 or 17 of 25. With perf(engine): pass an unfiltered scan's limit to Lance #1342: 1 of 25.
Lance 13: with no filter and no ordering, Scanner::plan_scan_range selects fragments from the manifest's row and deletion counts (scanner.rs:3159-3200). With a filter it pushes nothing (scanner.rs:3161), so filtered reads keep the current early stop. No pending Lance change is needed for this.
A second test checks that deleted rows never fill the limit, that a limit can span fragments, and that a filtered limit still finds a row in the last fragment.
Local gates: fmt, clippy, GQT corpus (467 cases, with the v1 comparison), DST suite with the storage-cost golden unchanged.
Activity
ragnorc commentedon Nov 10, 2026
What happens
A query with a
limitand no filter reads more of the table than it returns. Reproduced on main2c4b2f12. The new testunfiltered_limit_reads_only_the_first_fragment_issue_1335builds a table of 25 fragments (file groups that Lance stores separately), and a coldlimit 1read opens 16 or 17 of them.Why
Omnigraph never tells Lance about the limit. It reads, then stops the stream after the first rows (pipelined.rs). By then Lance has already started reads on up to 16 fragments on a local disk, or 128 on an object store.
Pull request
#1342 gives the limit to Lance when the read has no filter. Lance then reads only the fragments that hold the first rows, so the test reads 1 of 25 files.
Commands, evidence and Lance notes
crates/omnigraph/tests/integration/scan_limit_cost.rs. It counts distinct data files at the table store's own I/O tracker, which also sees Lance's direct local reads. Full scan: 25 of 25 files.limit 1on main: 16 or 17 of 25. With perf(engine): pass an unfiltered scan's limit to Lance #1342: 1 of 25.cargo dev-test --test integration scan_limit_cost::Scanner::plan_scan_rangeselects fragments from the manifest's row and deletion counts (scanner.rs:3159-3200). With a filter it pushes nothing (scanner.rs:3161), so filtered reads keep the current early stop. No pending Lance change is needed for this.perf(engine): pass an unfiltered scan's limit to Lance (ModernRelay#1342