Skip to content

performance: pass a query's limit to Lance when the scan under it has no filter, so it reads only the fragments it needs #1335

Closed

Description

@ragnorc

Problem

A limit directly over the projection over a table scan never reaches Lance.

  • limit_over_scan finds that shape and passes the limit only as the number of rows the scan gathers before it sends a batch (lower.rs:355-366, 641-643, pipelined.rs:180).
  • NodeRead::configure never calls Scanner::limit (scan.rs:255-281). The engine stops by dropping the stream.
  • By then Lance has started up to fragment_readahead fragments (2 × I/O parallelism: 16 on a local store, 128 on an object store) and scheduled their reads, up to the scan's 64 MiB I/O buffer (lance-13.0.0/src/io/exec/filtered_read.rs:603-606).

Proposed change

Pass Scanner::limit(Some(k), None) when the scan under the Limit has no filter, no runtime filter and no ranking.

Upstream primitive at the pin: with no filter and no ordering, Scanner::plan_scan_range (lance-13.0.0/src/dataset/scanner.rs:3159-3200) takes plan_metadata_scan_range (:3110-3155). It keeps only the fragments that hold the first k live rows, using manifest row and deletion counts, so limit 10 reads one fragment's first pages.

  • Gain: fewer fragment opens and range reads for limit queries without a filter. The rows are the same: fragments are read in order, and a limit without order may return any rows.
  • Cost and risk: small. The runtime filter is applied by the engine after Lance, so a Lance limit there would drop rows; the condition must exclude it. Lance itself pushes nothing when a filter exists (scanner.rs:3160-3162). DST storage-cost lines and GQT --measure baselines for such queries would change.
  • Evidence: code inspection only. The measurements in query: three read paths still build SQL filter strings where a typed Expr exists next to them #1257 show that limit(1) with a filter is not pushed down, which matches scanner.rs:3160-3162.

Affected areas

Scan lowering and NodeRead (engine/lower.rs, engine/scan.rs), the DST storage-cost golden. Related: #1257 (the filtered case), #887 (early stop), #1165 (ordered keyset limit, a different case).

Alternatives considered

Keep early stop only. It leaves the read-ahead that Lance started before the stream was dropped.

Finding ID: L-scan-limit-pushdown · Suggested size: S
Audited commit: 2c4b2f12 · Pinned: lance 13.0.0, datafusion 54.0.0, arrow 58.4.1
Report: underused-primitives-20261110.md (focus: scans and query execution)

Activity

  1. added
    needs-triageAwaiting maintainer triage
    featureFeature proposal
    performanceCorrect result, wrong cost: work scales with the table, not with the delta or the query
    on Nov 10, 2026
  2. ragnorc commented on Nov 10, 2026

    @ragnorc
    ContributorAuthor

    What happens

    A query with a limit and no filter reads more of the table than it returns. Reproduced on main 2c4b2f12. The new test unfiltered_limit_reads_only_the_first_fragment_issue_1335 builds a table of 25 fragments (file groups that Lance stores separately), and a cold limit 1 read opens 16 or 17 of them.

    Why

    Omnigraph never tells Lance about the limit. It reads, then stops the stream after the first rows (pipelined.rs). By then Lance has already started reads on up to 16 fragments on a local disk, or 128 on an object store.

    Pull request

    #1342 gives the limit to Lance when the read has no filter. Lance then reads only the fragments that hold the first rows, so the test reads 1 of 25 files.

    Commands, evidence and Lance notes
    • Test module: crates/omnigraph/tests/integration/scan_limit_cost.rs. It counts distinct data files at the table store's own I/O tracker, which also sees Lance's direct local reads. Full scan: 25 of 25 files. limit 1 on main: 16 or 17 of 25. With perf(engine): pass an unfiltered scan's limit to Lance #1342: 1 of 25.
    • Run: cargo dev-test --test integration scan_limit_cost::
    • Lance 13: with no filter and no ordering, Scanner::plan_scan_range selects fragments from the manifest's row and deletion counts (scanner.rs:3159-3200). With a filter it pushes nothing (scanner.rs:3161), so filtered reads keep the current early stop. No pending Lance change is needed for this.
    • A second test checks that deleted rows never fill the limit, that a limit can span fragments, and that a filtered limit still finds a row in the last fragment.
    • Local gates: fmt, clippy, GQT corpus (467 cases, with the v1 comparison), DST suite with the storage-cost golden unchanged.
  3. added a commit that references this issue on Nov 10, 2026
    62b03de
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    featureFeature proposalneeds-triageAwaiting maintainer triageperformanceCorrect result, wrong cost: work scales with the table, not with the delta or the query

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions