Stop the budget walk on settled blocks, and carry its cut out to the response - #26
Merged
Merged
Conversation
mo4islona
force-pushed
the
fix/gap-33-budget-walk-cut
branch
6 times, most recently
from
September 9, 2026 09:23
2795a45 to
3003f87
Compare
A block-sorted table used to be scanned in waves that stopped once the cumulative weight crossed the budget, and that cut never reached block selection. Every other table had been read for the whole range, so the blocks above the cut came back complete-looking with none of the walked table's rows, and `lastBlock` named the top of them — a client resuming from there never asks for the rows in between. The wave was as wide as the rayon pool, so the same query on the same chunk answered differently on workers with different core counts. The budget is now applied over ranges rather than over one table's reading. A range is read for every requested table and every relation at once, the blocks in it are weighed by the exact model, and the range is extended until the budget is exhausted or the request ends. Nothing is ever emitted that some table was not read for, because a range is only ever complete, and there is no estimate to be a lower bound on anything: the model that decides the page is the model that charges it. Row-group statistics only suggest how much to read next — the largest suggestion any table makes, so a table stored in one group does not force the others to be read again for every range — and a single unfiltered table keeps the narrow size pre-scan, which is what stops a full scan from decoding wide columns for blocks no response can carry. Underneath sat a second fault of one shape: several readers of one thing, each deciding for itself what an unreadable value means. Five readers of the block-number value carried their own downcast chains, four widths where `IntColumn` has eight, and none refused a null: the placeholder read as block 0, differently in each of them. Three readers of the row-group statistic each acted on a pair that reads back inverted — a block past 2³¹ in a signed `Int32`, recorded by a writer comparing the stored values — and the relation pruner does not merely risk dropping such a group: every key at or above an inverted minimum is above the inverted maximum by construction, so the overlap test cannot pass and the group is dropped whatever the query asked. So each thing is read in one place, and read before anything acts on it. `block_bounds` states a row group's range or states nothing, and every pruner reads through it — widening the statistic at the column's width rather than the statistic's, since parquet carries a narrowed integer's bounds in an `Int32` whatever the column is. `BlockNumbers` resolves the block-number column, and `ensure_block_numbers_readable` runs it at the scan's entry, over every row group rather than the selected ones, reading the column itself where the file states no null count — a check that believes an absent count is a zero has not checked anything. A chunk erroring on one query shape and under-answering on another is worse than either, because a client cannot tell them apart. A statistic also has to belong to the column asked for. Parquet counts a file's columns in leaves and Arrow counts them in top-level fields, and the two agree only while every field is a primitive; the bounds were fetched by the Arrow index, so one `List<Struct<…>>` shifted every field after it. On the Solana chunks in this tree that put `fee_payer` — the table's leading sort key — on the bounds of `loaded_addresses.readonly`, and on an EVM chunk carrying `access_list` it put `sighash` and `status` inside `signature`. A filter compared against a stranger's bounds concludes no row can match and skips the row group holding its own rows. Bounds now come from the leaf a column's name names, and a nested field maps to none: a reader that cannot interpret a statistic declines to prune rather than guesses. The weight model dedups by comparing key values rather than by trusting a hash, so two rows the hash cannot tell apart are still two rows. The tests had to be able to fail. The budget was a constant and the range reads had no off switch, so nothing could ask whether the response depends on them; `ExecOptions` carries both, and the law is stated directly: over generated queries, at budgets that straddle what the corpus weighs, the answer equals a whole read's. That law is satisfied by a read that never stops early, so the response carries how far it read and the law asserts on it — and, because one early stop anywhere is not the mechanism running everywhere, each requested table is asked about on its own. The refusals are pinned from every path a query can reach a chunk by, in both writings of it, stating a null count and stating none; the leaf indexing from a chunk carrying a two-leaf column ahead of the one being filtered on; the row comparison from a pair whose hashes are equal by construction. Closes gap 33, and the collision half of gap 44. Gap 31 gains the fix this argues for — refuse the chunk rather than teach four readers about the wrap — with what refusing gives up. Co-Authored-By: Claude Code <noreply@anthropic.com> Co-Authored-By: Codex <codex@openai.com>
mo4islona
force-pushed
the
fix/gap-33-budget-walk-cut
branch
from
September 9, 2026 10:43
bb2ad39 to
febf3b4
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes gap 33 in
spec/GAPS.md— an S1: wrong results that look right.The bug
A table whose sort key leads with the block number is scanned in waves that stop
once the budget is crossed. Three things went wrong on that path.
The walk stopped on the wrong weight. It weighed everything it had read, but
part of that belongs to the block the next row group still owns, and that block
cannot be emitted. The blocks that could be emitted were then often lighter
than the budget, so
apply_weight_limitnever fired.The stop point went nowhere.
scan_budgetreturned rows only; the boundaryit had computed was dropped. Every other table had been read for the whole
range, so block selection ran past that boundary: the blocks above it came back
carrying the other tables' rows and none of this one's, and
lastBlocknamed thetop of them. A client resuming from
lastBlock + 1never asks for the rows inbetween — a 200, with no way to tell.
The wave width reached the answer. A wave is as wide as the rayon pool, so
the same query on the same chunk answered differently at different core counts,
against INV-O13.
There was a fourth, one layer down. The walk trusted the catalog's
sort_key.One archiver wrote
statediffssorted by address while the catalog declares itblock-led, and on that layout the walk drew a boundary out of row groups that
overlap.
The fix
The walk now stops on the weight of settled blocks — those no unread row group
can still add to. That estimate is a lower bound on what
apply_weight_limitcharges, so crossing the budget on settled weight guarantees the exact trim lands
at or below the cut. The emitted set is therefore the one a full scan would have
produced, whatever the wave width — which is what closes the determinism half.
scan_budgetreturnsBudgetScan { batches, complete_through }, andcomplete_throughcaps both the header scan and block selection — the same shapethe two-phase pre-scan path already had.
Layout is now read from the row-group statistics rather than the declared sort
key: waves only when the groups' block ranges don't overlap beyond the boundary
block two neighbours share, otherwise a plain full scan, which is what the
reference does on every layout.
Evidence
The query the gap was measured on, run through
qdiffagainst the reference onthe real EVM chunk (
stateDiffswith akindfilter, 24550605–24550820):lastBlockDIFFERMATCHBefore, it diverged at 1 and 17 threads and agreed at 2–16. After, it matches at
1, 2, 3, 8 and 17.
A sweep of 12 EVM queries × 4 pool sizes agrees with the reference except for two
cases that also diverge on
master— gap 44, the weight model, where thisengine's answer is an exact prefix of the reference's.
Five new tests, each checked by mutating the engine to confirm it fails without
the fix:
paging_a_partitioned_chunk_loses_no_block— pages a block-partitioned chunkend to end at 1, 2 and 17 threads; the concatenation must be the whole chunk
(INV-B4, INV-B7)
the_pool_size_does_not_move_a_page_boundary— page boundaries don't move withthe pool (INV-O13)
an_overlapping_layout_is_read_whole_rather_than_cut— the address-sortedshape falls back to a full scan
a_wrapped_int32_statistic_draws_no_cut— see the third commita_cut_never_outruns_the_rows_behind_it—complete_through's contract stateddirectly, over every budget and wave width: a block at or below the cut is a
block the walk read in full
The synthetic chunk needed reshaping to reach the bug at all: the block two row
groups share has to be much heavier than its neighbours, or the exact trim fires
before the cut and nothing is observable. That is why the first version of the
test passed against the broken engine.
Full suite green, including the fixture-gated tests.
make fmt,make lintandmake spec-checkall pass; INV-B4 and INV-B7 move from P to C, propertycoverage 0.71 → 0.73.
Perf-neutral on the one benchmark that takes this path (
evm/usdc_traces+diffs,six interleaved runs a side): 0.995 small, 0.996 big.
Third commit: inverted statistics
Reviewed and confirmed: the new boundary detector could still draw a cut it had
no business drawing.
Widening a stored block number reinterprets the bits, so a signed
Int32past2³¹ reads back as the block it is. The row group's statistics do not survive
that. A writer comparing the wrapped values signed records the block above the
wrap as the minimum and the highest block below it as the maximum, and widened
back the pair is
min > max— a range leaving out most of the blocks the groupholds. Ordering on it puts the group after row groups it precedes, and its
understated maximum clears the overlap check, so the walk would settle blocks
that group has not been read for.
Reproduced on a two-row-group file with a signed
Int32block column: rawstatistics
min = -2147483648, max = 2147483647, widened tomin = 2147483648 > max = 2147483647.The walk now refuses to order a file on such a pair and reads the table whole.
Fourth commit: three more from review
A narrowed integer weighed nothing. The weight model resolved the block
number and the
*_sizecompanion through its own downcast chain — four widthswhere
IntColumncarries eight, which is the duplicationsrc/integers.rsexists to end. A
UInt16size read as zero, so every row weighed its fixed partalone and the response the budget was meant to cap went out whole; a
UInt16block number read as absent, so the row belonged to no block and was never
charged at all. Both now resolve through the one list, which also reaches
fork.rs,arrow_out.rsand the exact trim.a_narrow_size_column_still_reaches_the_budgetputs 480 logs of 65 535 bytesagainst a 20 MB cap.
The inverted-statistic guard sat behind the reader it was guarding.
select_row_groupsprunes on the same widened bounds and runs first, so a groupwhose reported range starts above its own end is dropped by any query with a
toBlockbelow that start — with the rows it holds inside the range, on everyscan path, not just the walk. The pruner now keeps a group it cannot bound.
an_inverted_statistic_does_not_prune_a_row_group_away.The walk claimed a cut it could not stand behind.
retain_blocks_belowgivesup on a block number that is null or unreadable, and the weight that crossed the
budget came off that same column — so the stop rested on numbers nothing could
resolve. It reads on instead. Returning no cut and stopping would have been
worse: that is the original bug again.
an_unreadable_block_number_yields_no_cut.Gap 31's entry gains the fix this argues for: refuse such a chunk with
MalformedChunkDatarather than teach four readers about the wrap — with whatrefusing gives up, since the rows themselves widen correctly. Also recorded there:
from_blockof zero is treated as no lower bound, which is true for an unsignedcolumn and false for this one.
For review
The
sorted_blockscap and theheader_to_blockcap are not distinguishable bymutation under the new stop rule — I checked. They are kept deliberately: they
enforce
BudgetScan::complete_through's stated contract, and the stop rule iswhat makes them free. The two agree only while the walk's estimate stays under
the exact model, and a block whose header the chunk is missing is already outside
that. Removing them is defensible; it would leave the property resting on an
invariant nothing checks.