Skip to content

Stop the budget walk on settled blocks, and carry its cut out to the response - #26

Merged
mo4islona merged 1 commit into
masterfrom
fix/gap-33-budget-walk-cut
Sep 9, 2026
Merged

mo4islona merged 1 commit into
masterfrom
fix/gap-33-budget-walk-cut

Conversation

@mo4islona

@mo4islona mo4islona commented Sep 5, 2026

Copy link
Copy Markdown
Collaborator

Closes gap 33 in spec/GAPS.md — an S1: wrong results that look right.

The bug

A table whose sort key leads with the block number is scanned in waves that stop
once the budget is crossed. Three things went wrong on that path.

The walk stopped on the wrong weight. It weighed everything it had read, but
part of that belongs to the block the next row group still owns, and that block
cannot be emitted. The blocks that could be emitted were then often lighter
than the budget, so apply_weight_limit never fired.

The stop point went nowhere. scan_budget returned rows only; the boundary
it had computed was dropped. Every other table had been read for the whole
range, so block selection ran past that boundary: the blocks above it came back
carrying the other tables' rows and none of this one's, and lastBlock named the
top of them. A client resuming from lastBlock + 1 never asks for the rows in
between — a 200, with no way to tell.

The wave width reached the answer. A wave is as wide as the rayon pool, so
the same query on the same chunk answered differently at different core counts,
against INV-O13.

There was a fourth, one layer down. The walk trusted the catalog's sort_key.
One archiver wrote statediffs sorted by address while the catalog declares it
block-led, and on that layout the walk drew a boundary out of row groups that
overlap.

The fix

The walk now stops on the weight of settled blocks — those no unread row group
can still add to. That estimate is a lower bound on what apply_weight_limit
charges, so crossing the budget on settled weight guarantees the exact trim lands
at or below the cut. The emitted set is therefore the one a full scan would have
produced, whatever the wave width — which is what closes the determinism half.

scan_budget returns BudgetScan { batches, complete_through }, and
complete_through caps both the header scan and block selection — the same shape
the two-phase pre-scan path already had.

Layout is now read from the row-group statistics rather than the declared sort
key: waves only when the groups' block ranges don't overlap beyond the boundary
block two neighbours share, otherwise a plain full scan, which is what the
reference does on every layout.

Evidence

The query the gap was measured on, run through qdiff against the reference on
the real EVM chunk (stateDiffs with a kind filter, 24550605–24550820):

before after
blocks 73 72
lastBlock 24550820 24550676
vs. reference DIFFER MATCH

Before, it diverged at 1 and 17 threads and agreed at 2–16. After, it matches at
1, 2, 3, 8 and 17.

A sweep of 12 EVM queries × 4 pool sizes agrees with the reference except for two
cases that also diverge on master — gap 44, the weight model, where this
engine's answer is an exact prefix of the reference's.

Five new tests, each checked by mutating the engine to confirm it fails without
the fix:

  • paging_a_partitioned_chunk_loses_no_block — pages a block-partitioned chunk
    end to end at 1, 2 and 17 threads; the concatenation must be the whole chunk
    (INV-B4, INV-B7)
  • the_pool_size_does_not_move_a_page_boundary — page boundaries don't move with
    the pool (INV-O13)
  • an_overlapping_layout_is_read_whole_rather_than_cut — the address-sorted
    shape falls back to a full scan
  • a_wrapped_int32_statistic_draws_no_cut — see the third commit
  • a_cut_never_outruns_the_rows_behind_itcomplete_through's contract stated
    directly, over every budget and wave width: a block at or below the cut is a
    block the walk read in full

The synthetic chunk needed reshaping to reach the bug at all: the block two row
groups share has to be much heavier than its neighbours, or the exact trim fires
before the cut and nothing is observable. That is why the first version of the
test passed against the broken engine.

Full suite green, including the fixture-gated tests. make fmt, make lint and
make spec-check all pass; INV-B4 and INV-B7 move from P to C, property
coverage 0.71 → 0.73.

Perf-neutral on the one benchmark that takes this path (evm/usdc_traces+diffs,
six interleaved runs a side): 0.995 small, 0.996 big.

Third commit: inverted statistics

Reviewed and confirmed: the new boundary detector could still draw a cut it had
no business drawing.

Widening a stored block number reinterprets the bits, so a signed Int32 past
2³¹ reads back as the block it is. The row group's statistics do not survive
that. A writer comparing the wrapped values signed records the block above the
wrap as the minimum and the highest block below it as the maximum, and widened
back the pair is min > max — a range leaving out most of the blocks the group
holds. Ordering on it puts the group after row groups it precedes, and its
understated maximum clears the overlap check, so the walk would settle blocks
that group has not been read for.

Reproduced on a two-row-group file with a signed Int32 block column: raw
statistics min = -2147483648, max = 2147483647, widened to
min = 2147483648 > max = 2147483647.

The walk now refuses to order a file on such a pair and reads the table whole.

Fourth commit: three more from review

A narrowed integer weighed nothing. The weight model resolved the block
number and the *_size companion through its own downcast chain — four widths
where IntColumn carries eight, which is the duplication src/integers.rs
exists to end. A UInt16 size read as zero, so every row weighed its fixed part
alone and the response the budget was meant to cap went out whole; a UInt16
block number read as absent, so the row belonged to no block and was never
charged at all. Both now resolve through the one list, which also reaches
fork.rs, arrow_out.rs and the exact trim.
a_narrow_size_column_still_reaches_the_budget puts 480 logs of 65 535 bytes
against a 20 MB cap.

The inverted-statistic guard sat behind the reader it was guarding.
select_row_groups prunes on the same widened bounds and runs first, so a group
whose reported range starts above its own end is dropped by any query with a
toBlock below that start — with the rows it holds inside the range, on every
scan path, not just the walk. The pruner now keeps a group it cannot bound.
an_inverted_statistic_does_not_prune_a_row_group_away.

The walk claimed a cut it could not stand behind. retain_blocks_below gives
up on a block number that is null or unreadable, and the weight that crossed the
budget came off that same column — so the stop rested on numbers nothing could
resolve. It reads on instead. Returning no cut and stopping would have been
worse: that is the original bug again.
an_unreadable_block_number_yields_no_cut.

Gap 31's entry gains the fix this argues for: refuse such a chunk with
MalformedChunkData rather than teach four readers about the wrap — with what
refusing gives up, since the rows themselves widen correctly. Also recorded there:
from_block of zero is treated as no lower bound, which is true for an unsigned
column and false for this one.

For review

The sorted_blocks cap and the header_to_block cap are not distinguishable by
mutation under the new stop rule — I checked. They are kept deliberately: they
enforce BudgetScan::complete_through's stated contract, and the stop rule is
what makes them free. The two agree only while the walk's estimate stays under
the exact model, and a block whose header the chunk is missing is already outside
that. Removing them is defensible; it would leave the property resting on an
invariant nothing checks.

@mo4islona
mo4islona force-pushed the fix/gap-33-budget-walk-cut branch 6 times, most recently from 2795a45 to 3003f87 Compare September 9, 2026 09:23
A block-sorted table used to be scanned in waves that stopped once the cumulative
weight crossed the budget, and that cut never reached block selection. Every
other table had been read for the whole range, so the blocks above the cut came
back complete-looking with none of the walked table's rows, and `lastBlock` named
the top of them — a client resuming from there never asks for the rows in between.
The wave was as wide as the rayon pool, so the same query on the same chunk
answered differently on workers with different core counts.

The budget is now applied over ranges rather than over one table's reading. A
range is read for every requested table and every relation at once, the blocks in
it are weighed by the exact model, and the range is extended until the budget is
exhausted or the request ends. Nothing is ever emitted that some table was not
read for, because a range is only ever complete, and there is no estimate to be a
lower bound on anything: the model that decides the page is the model that
charges it. Row-group statistics only suggest how much to read next — the largest
suggestion any table makes, so a table stored in one group does not force the
others to be read again for every range — and a single unfiltered table keeps the
narrow size pre-scan, which is what stops a full scan from decoding wide columns
for blocks no response can carry.

Underneath sat a second fault of one shape: several readers of one thing, each
deciding for itself what an unreadable value means. Five readers of the
block-number value carried their own downcast chains, four widths where
`IntColumn` has eight, and none refused a null: the placeholder read as block 0,
differently in each of them. Three readers of the row-group statistic each acted
on a pair that reads back inverted — a block past 2³¹ in a signed `Int32`,
recorded by a writer comparing the stored values — and the relation pruner does
not merely risk dropping such a group: every key at or above an inverted minimum
is above the inverted maximum by construction, so the overlap test cannot pass
and the group is dropped whatever the query asked.

So each thing is read in one place, and read before anything acts on it.
`block_bounds` states a row group's range or states nothing, and every pruner
reads through it — widening the statistic at the column's width rather than the
statistic's, since parquet carries a narrowed integer's bounds in an `Int32`
whatever the column is. `BlockNumbers` resolves the block-number column, and
`ensure_block_numbers_readable` runs it at the scan's entry, over every row group
rather than the selected ones, reading the column itself where the file states no
null count — a check that believes an absent count is a zero has not checked
anything. A chunk erroring on one query shape and under-answering on another is
worse than either, because a client cannot tell them apart.

A statistic also has to belong to the column asked for. Parquet counts a file's
columns in leaves and Arrow counts them in top-level fields, and the two agree
only while every field is a primitive; the bounds were fetched by the Arrow
index, so one `List<Struct<…>>` shifted every field after it. On the Solana
chunks in this tree that put `fee_payer` — the table's leading sort key — on the
bounds of `loaded_addresses.readonly`, and on an EVM chunk carrying `access_list`
it put `sighash` and `status` inside `signature`. A filter compared against a
stranger's bounds concludes no row can match and skips the row group holding its
own rows. Bounds now come from the leaf a column's name names, and a nested field
maps to none: a reader that cannot interpret a statistic declines to prune rather
than guesses.

The weight model dedups by comparing key values rather than by trusting a hash,
so two rows the hash cannot tell apart are still two rows.

The tests had to be able to fail. The budget was a constant and the range reads
had no off switch, so nothing could ask whether the response depends on them;
`ExecOptions` carries both, and the law is stated directly: over generated
queries, at budgets that straddle what the corpus weighs, the answer equals a
whole read's. That law is satisfied by a read that never stops early, so the
response carries how far it read and the law asserts on it — and, because one
early stop anywhere is not the mechanism running everywhere, each requested table
is asked about on its own. The refusals are pinned from every path a query can
reach a chunk by, in both writings of it, stating a null count and stating none;
the leaf indexing from a chunk carrying a two-leaf column ahead of the one being
filtered on; the row comparison from a pair whose hashes are equal by
construction.

Closes gap 33, and the collision half of gap 44. Gap 31 gains the fix this argues
for — refuse the chunk rather than teach four readers about the wrap — with what
refusing gives up.

Co-Authored-By: Claude Code <noreply@anthropic.com>
Co-Authored-By: Codex <codex@openai.com>
@mo4islona
mo4islona force-pushed the fix/gap-33-budget-walk-cut branch from bb2ad39 to febf3b4 Compare September 9, 2026 10:43
@mo4islona
mo4islona merged commit 4d638e0 into master Sep 9, 2026
2 checks passed
@mo4islona
mo4islona deleted the fix/gap-33-budget-walk-cut branch September 9, 2026 10:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant