Repository navigation
perf(engine): fold a correlated block's inner rows as they stream - #1341
Conversation
The per-row path of a correlated block drained its whole inner output into a Vec, charging every batch to the query pool, before folding it. Fold each inner batch into the SubqueryAggregate as the stream yields it, as the key-set path already does, so the block's peak is one batch. Closes ModernRelay#1338
ragnorc
left a comment
There was a problem hiding this comment.
Review of #1341 at e9f733cb
Why. A correlated block on the per-row path, such as max($b.x) { $a knows $b $a.name = $a.name } > "", kept its whole inner output in memory before it folded it. A hub with a large fan-out was refused with query_memory_bytes, although the fold needs only one batch at a time (#1338).
What. The operator now folds each inner batch into SubqueryAggregate when the stream yields it, and then drops the batch. The key-set path in the same file already works this way.
How. AntiJoinMaskExec replaces drain(...) plus absorb_inner_batches(&[RecordBatch]) with a while let loop over the inner stream and absorb_inner_batch(&RecordBatch) (anti_join.rs:372-394, subquery_aggregate.rs:455). The per-row accumulators are now reserved before the inner plan runs (anti_join.rs:363-366).
Impact. The block's peak memory goes from "all inner rows" to "one inner batch plus one accumulator per outer row". Results do not change. No API, format or setting changes.
Recommendation: the code change is correct and I have no blocking finding in it. I ask for one addition before merge: a .gqt regression case (R1 below). The repository rule asks for it, and the reason the PR gives for leaving it out is not correct.
Correctness (checked by reading the code and by running tests)
- Fold order. The batches arrive in the same order as before, and
absorbfolds rows in that order.min/maxtherefore still keep the first signed zero. The extremum is an ownedScalarValue(subquery_aggregate.rs:126), so dropping a batch does not leave a reference into its buffers. - Slot guard.
OuterReferenceExectakes the tagged batch on its first poll (anti_join.rs:130). The guard is created afterslot.filland is dropped only after the stream ends. On an error, the inner stream is dropped inside theasyncblock before the guard is dropped. This is the same order as before. - Accounting. The tagged outer batch is still held under the reservation. An inner batch in flight is not charged to this operator. The key-set path does the same, and
docs/dev/execution.mdnow says so for both strategies. Theinput_rowsmetric is kept. - Earlier refusal. The accumulators are now reserved before the inner plan runs. A pool that is too small for them now refuses before any inner I/O. This is better than before.
- The whole class. I looked for other "drain everything, then fold one batch at a time" paths in the engine. The remaining
draincallers aredrain_one(the outer side, which must be one batch to tag it),sort.rs(a full sort with nofetchneeds all its input), andvalidate.rs(a write must judge all rows before it continues). None of them folds per batch. This PR closes the only case of this defect.
Required before merge
R1. Add a .gqt regression case. GQT can select the pool. The test comment (engine_v2_memory.rs:2192) and the PR body say "GQT cannot select the pool". That is not correct: set query_memory_bytes = N; is a request setting that applies to read queries, and the corpus already uses it (cases/query_memory_bytes_sizes_the_pool.gqt, cases/v2/rrf_narrow_limit_memory.gqt). The seed generator can build the hub (repeat, literal and key columns), and --- expect plan can pin anti join $a: correlated. AGENTS.md asks for the cheapest tier that catches the defect, and here the defect shows as an error. I wrote this case and ran it in a temporary copy of the tree:
# issue: 1338
# red_on: 2c4b2f12: the correlated block drains its 16 MiB inner output under the pool and is refused with query_memory_bytes.
# notes: A correlated block on the per-row path folds each inner batch as it streams: 16,384
# notes: neighbours with a 1 KiB payload (16 MiB of inner rows) run under a 12 MiB query pool.
# notes: The conjunct that reads the outer row keeps the block off the key-set path, and max
# notes: keeps it off the bulk row count.
--- runner
timeout_ms: 10000
environments:
- target: omnigraph-engine
storage: local-filesystem
--- schema
node Person {
name: String @key
payload: String
}
edge Knows: Person -> Person
--- seed generate: v1 seed: 0
tables:
- kind: node
name: Person
rows: 16385
commits: 9
batch_rows: 2048
columns:
name: {kind: key, prefix: p, width: 5}
payload: {kind: repeat, text: x, count: 1024}
- kind: edge
name: Knows
rows: 16384
commits: 8
batch_rows: 2048
start: 1
from: {kind: literal, value: p00000}
to: {kind: key, prefix: p, width: 5}
columns: {}
--- query
set engine = v2;
set query_memory_bytes = 12582912;
query hubs() {
match {
$a: Person { name: "p00000" }
max($b.payload) { $a knows $b $a.name = $a.name } > ""
}
return { $a.name }
}
--- expect unordered
{"a.name": "p00000"}
--- expect shape
a.name: String
--- expect plan
anti join $a: correlated
As crates/omnigraph-gqt/cases/issue_1338_correlated_block_folds_inner_batches.gqt:
- With the engine files of main
2c4b2f12: fails at step 1 withresource limit exceeded for query_memory_bytes: actual 12682022, limit 12582912(2.99 s). This is the predicted reason. - At
e9f733cb: passes (3.31 s, inside the 10 s case budget, so it can stay in the ordinary corpus).
Keep the Rust _issue_1338 test as well. Its assert_released probe is a mechanism assertion that GQT cannot make. Change its comment to give that reason. The older comment at engine_v2_memory.rs:463 makes the same claim about pool selection; its second reason (the collector's reservation owner) is still valid.
Optional suggestions
O1. One owner for "fold the inner stream, never hold it". After this change, the two strategies contain the same loop: take a batch, record input_rows, skip an empty batch, find a tag per row, evaluate the argument, absorb (anti_join.rs:376-392 and anti_join.rs:438-458). Only the tag source is different: the correlated path reads the tag column, and the key-set path maps the row's key to a group. One function, for example fold_inner(stream, &mut aggregate, tags: impl Fn(&RecordBatch) -> Result<UInt32Array>, arg, params, reservation) in subquery_aggregate.rs, would make "the inner stream is folded, not held" true by construction for both strategies and for any future one. absorb_inner_batch and the key-set loop would then share their empty-batch and length checks. If five more strategies of this kind were added, each would otherwise copy the loop and could reintroduce a drain. The cost is one closure parameter. This is not needed for correctness.
O2. Accumulator heap is not counted (this PR did not cause it). reservation.entries::<ValueAccumulator>(num_rows) (anti_join.rs:365) charges only the size of the enum. A min/max over a String keeps an owned value per outer row, on the heap and not charged. In this test that is one 1 KiB string. With many outer rows and long text, it can be large. Now that the inner batches are not held, this is the largest part of the fold state that the pool does not count. A follow-up could charge the extremum's size when it is replaced. This applies to both strategies.
O3. Remaining bound. The outer side is still drained into one batch (drain_one, anti_join.rs:319) and tagged, so peak memory is the outer rows, the accumulators, one inner batch and the inner operators' own working sets. The PR's table shows this: at smaller pools, expand hop and hydrate_nodes refuse first. That is correct and outside this PR's scope. I mention it only so that nobody reads "streams" as "bounded by one batch".
Liability
- Removed: a
Vec<RecordBatch>that grew with fan-out, and its per-batch charges. The fold now matches the key-set path, so the two strategies share one memory model. - Added: no new state, API, invariant or setting. The source lines are about even (+45/-37 in the two engine files). The only new obligation is the guard lifetime, and the PR's comment states it.
- Alternative (lower to DataFusion
AggregateExec). That would give streaming and spilling from the substrate (invariant 1). Waiting for #1260's float-semantics decision is reasonable. This change does not make that move harder. - Other open PRs. #1316 does not touch
anti_join.rsorsubquery_aggregate.rs. #1275 and #1282 also add tests at the end ofengine_v2_memory.rs, so expect a small textual conflict there. Lance is not involved: the change is after the scan, on DataFusion record batch streams.
Validation
All runs are local, in a clean detached worktree at e9f733cb, unless stated otherwise.
| What | Commit | Command | Result |
|---|---|---|---|
| New Rust test, before | engine files of 2c4b2f12, test from e9f733cb |
cargo test --workspace --exclude omnigraph-gqt --exclude omnigraph-gqt-served --exclude omnigraph-dst --locked --features omnigraph-engine/failpoints,omnigraph-cluster/failpoints -p omnigraph-engine --test integration -- engine_v2_memory::correlated_block_folds |
fails after the plan assertion (correlated) passes: ResourceLimitExceeded { resource: "query_memory_bytes", limit: 12582912, actual: 12587283 } |
| New Rust test, after | e9f733cb |
same | passes |
engine_v2_memory:: module |
e9f733cb |
same, filter engine_v2_memory::, RUST_MIN_STACK=16777216 |
32 passed |
| Engine integration, related filters | e9f733cb |
test binary with filters anti_join, block, correlat, negation |
1, 8, 3 and 5 passed; 0 failed |
| GQT probe (R1), before / after | 2c4b2f12 engine files / e9f733cb |
cargo test -p omnigraph-gqt --locked --test gq_logic_tests -- issue_1338 |
fails for the predicted reason / passes |
| Complete GQT corpus with the probe | e9f733cb |
gq_logic_tests with no filter |
468 passed, 0 failed |
| CI | e9f733cb |
all checks | 28 pass, 4 skipped, 0 fail |
Limits. I did not run Clippy, rustfmt or the engine unit tests locally; CI ran them on this exact commit and they pass. I did not measure peak memory directly. As in the PR, admission under a pool smaller than the inner output is the evidence. I made no performance claims. I removed all temporary edits, and I pushed nothing to this branch.
…p for both strategies A GQT case runs a correlated block whose 16 MiB of inner rows exceed its 12 MiB query pool: holding the inner output was refused with query_memory_bytes, and the streaming fold answers the row. The Rust test stays for the reservation release that GQT cannot assert, and its comment and an older one no longer claim GQT cannot select the pool. fold_inner is now the one loop that folds an inner stream batch by batch for the correlated and key-set strategies, which differ only in how a row finds its group, so no strategy can hold its inner output again.
|
Addressed at R1. Added your case unchanged as
O1. Done. O2 and O3 are outside this PR's scope, as you note. The accumulator heap of a text Checked locally on 1.97.1: fmt, |
ragnorc
left a comment
There was a problem hiding this comment.
Review of #1341 at b49e232a (delta since e9f733cb)
Why. A correlated block on the per-row path kept its whole inner output in memory before it folded it, so a hub with a large fan-out was refused with query_memory_bytes (#1338). My first review (at e9f733cb) found the fold change correct and asked for a .gqt regression (R1). It also suggested one owner for the inner loop (O1).
What changed since e9f733cb. Two commits. 96c7ccf8 merges main with no conflict: its remerge diff is empty. b49e232a does three things:
- It adds the R1 case as
crates/omnigraph-gqt/cases/issue_1338_correlated_block_folds_inner_batches.gqt. The file is byte-identical to the probe in my first review. - It corrects the two test comments in
engine_v2_memory.rs. - It adds
fold_inner(anti_join.rs:450-474), the one loop over an inner stream for both strategies, and removesabsorb_inner_batch.
Impact. Results and memory model are unchanged from e9f733cb. The new function makes "fold the inner stream, never hold it" true by construction for both strategies.
Recommendation: one blocking item, B1. CI is red at this head because of this commit, and the fix is a rename. The design and the R1 follow-up are correct.
Required before merge
B1. fold_inner's closure parameter tags trips the graph-write chokepoint guard. forbidden_apis::graph_visible_write_chokepoints_are_registered fails in Test Workspace and in Storage Upgrade Compatibility, so CI Gate fails too:
engine/operators/anti_join.rs `.tags(`: observed 1, no registered protocol
The cause is the call tags(&batch) at anti_join.rs:464. The guard's structural scanner (CallInventory::visit_expr_call, forbidden_apis.rs:1765) records every function call by its final path identifier. It cannot tell a local closure from Lance's native-ref Dataset::tags(). The .tags( primitive key is the bare identifier tags, from primitive_inventory_key. So the closure call counts as a native-ref access in a file that has no registered protocol.
Rename the parameter, for example group_of, and its mention in the doc comment above. Do not register anti_join.rs in the chokepoint registry. That would record a durable-effect protocol for code that has none, and it would hide a real .tags( call if one were added to this file later. The local let tags = … and aggregate.absorb(&tags, …) do not trip the guard (a binding, and a method call named absorb).
I checked this fix in my worktree. At b49e232a the guard fails locally with the same message. With only the parameter renamed to group_of (three lines in anti_join.rs), the whole forbidden_apis binary passes: 44 passed, 0 failed. I restored the file afterwards.
Also note that nextest stopped at the first failure in Test Workspace (3140/5316 tests run), so CI has no evidence for the remaining 2,176 workspace tests at this head. The rerun after the rename will provide it.
Delta correctness (code inspection and local tests)
- Same fold, both paths.
fold_innerkeeps the order of the old loops: take a batch, recordinput_rows, skip an empty batch, find each row's group, evaluate the argument in the inner scope,absorb. The correlated path used to skip an empty batch insideabsorb_inner_batchbefore it read the tag column. It now skips it infold_innerbefore it calls the closure. Same behavior, and an empty batch with no tag column still does not fail. - Errors. The two
manifest_internalerrors for a missing or non-UInt32tag column keep their text. They now go throughexternalonce, intag_array, as the other inner errors already did. tag_arrayclones..cloned()on the downcastUInt32Arraycopies the array'sArcs, not its values. This is cheap, and it lets one closure type (FnMut(&RecordBatch) -> DfResult<UInt32Array>) serve both strategies. The key-set closure already built an owned array.- Slot guard. The
FilledSlotguard is still created afterslot.filland dropped afterfold_innerreturns, so the inner stream still ends before the tagged batch is released. - Docs.
docs/dev/execution.md(theAntiJoinrow) already describes both strategies as folding each batch without holding it. Nothing indocs/or the skill names the removedabsorb_inner_batch.
Optional
O1 (new). The guard's false positive is a general hazard, not only this rename. Any local function or closure named after a durable primitive (tags, commit, restore, and the rest of DURABLE_PRIMITIVES) in a scanned file will trip the guard. A rename is the right fix here. If this happens again, the guard could skip calls whose path is a single segment bound by a closure or function parameter in scope. That costs a scope walk in a test file and is not needed now.
Earlier O2 (the uncounted heap of a text min/max accumulator) and O3 (the drained outer side) stay out of scope, as the author said.
Liability
- Removed: two copies of the inner loop and
absorb_inner_batch. The two engine files go +69/−56: the shared loop costs a few lines more than two copies, and the doc comment explains it. A third strategy now gets streaming by callingfold_inner, not by copying the loop. - Added: one closure parameter and
tag_array. No new state, API, setting or invariant. - Other open PRs.
git merge-treeagainstmainfd58fd33is clean. #1345 (which changeskey_groups.rs) merges cleanly with this head. #1316 and #1334 conflict with it only onuselines insubquery_aggregate.rs(IRExprmoves to the test module here, and they renamequery::asttoquery::value). Both already conflict withmainincrates/omnigraph-gqt/README.md. - Lance is not involved: the change is after the scan, on DataFusion record batch streams.
Validation
All local runs are in a clean detached worktree at b49e232a, debug, the cargo dev-test feature set, RUST_MIN_STACK=16777216, rustc 1.97.1.
| What | Commit | Command | Result |
|---|---|---|---|
| Chokepoint guard, as pushed | b49e232a |
cargo dev-test -p omnigraph-engine --test forbidden_apis -- graph_visible_write_chokepoints_are_registered |
fails: engine/operators/anti_join.rs .tags(: observed 1, no registered protocol (same as CI) |
forbidden_apis, with B1's rename |
b49e232a + rename of the tags parameter |
cargo dev-test -p omnigraph-engine --test forbidden_apis |
44 passed, 0 failed |
| GQT regression, before | b49e232a with anti_join.rs and subquery_aggregate.rs of main ad41b299 (the merge parent) |
cargo test -p omnigraph-gqt --locked --test gq_logic_tests -- issue_1338 |
fails at step 1: resource limit exceeded for query_memory_bytes: actual 12682022, limit 12582912. This is the predicted reason, now against current main |
| GQT regression, after | b49e232a |
same | passes (2.35 s, inside the 10 s budget) |
| Engine integration, related filters | b49e232a |
engine integration binary with filters engine_v2_memory::, block, correlat, negation, anti_join |
32, 8, 3, 5 and 1 passed; 0 failed |
| CI, complete GQT corpus | b49e232a |
GQ Logic Tests |
472 passed, issue_1338 in 3.43 s |
| CI, other checks | b49e232a |
all checks | 25 pass, 4 skipped, 3 fail (Test Workspace, Storage Upgrade Compatibility, CI Gate, all from B1). Clippy, rustfmt and DST pass. The in-source subquery_aggregate (5), key_groups (2) and report (19) unit tests passed before nextest stopped |
Limits. I did not run the full workspace graph locally. The 2,176 workspace tests that CI did not reach at this head are not verified by me either; the rerun after B1 will cover them. I did not run Clippy or rustfmt locally; CI ran them on this exact commit and they pass. The Rust _issue_1338 test's before/after proof is from my first review (it is unchanged except its comment); at this head it passes inside the engine_v2_memory:: run above. I made no performance claims. I removed all temporary edits, and I pushed nothing to this branch.
The chokepoint guard records every call by its final path identifier, so the closure call tags(&batch) counted as Lance's native-ref .tags( in a file with no registered protocol, failing forbidden_apis::graph_visible_write_chokepoints_are_registered.
|
Reply to the review of B1: valid, fixed in
O1: declined for now. I agree with you that a rename is the right fix here. This is the first time a local name has tripped the guard. A scope-aware skip would add a scope walk to the guard's scanner, and that walk could also hide a real call. If the same false positive appears again, it will be worth doing. Earlier O2 and O3 stay out of scope, as before. Checks on
|
ragnorc
left a comment
There was a problem hiding this comment.
Review of #1341 at 4acd86bf (delta since b49e232a)
Why. My review at b49e232a found one blocking item, B1: the closure parameter tags in fold_inner tripped the graph-write chokepoint guard, so CI failed. The streaming fold itself and the R1 GQT case were correct.
What changed since b49e232a. Two commits.
37b514a3renames the closure parametertagstogroup_of(anti_join.rs:447,:456,:464). This is the rename B1 asked for, three lines, and nothing else.4acd86bfmergesmain2bfb3f35. It brings onlyREADME.mdchanges frommain. Its remerge diff is empty, so the merge resolved nothing by hand.
Impact. No change to behavior, results or the memory model since b49e232a. anti_join.rs is not in the chokepoint registry, which is correct: the file has no durable effect, and a real .tags( call added there later will still fail the guard.
Recommendation: ready to merge. B1 is resolved, and CI at this head is green. No blocking findings. No new optional findings.
B1: resolved
- The guard counts calls by their final path identifier (
CallInventory::visit_expr_call,forbidden_apis.rs:1765; key fromprimitive_inventory_key,forbidden_apis.rs:1875). After the rename, the only call through the closure isgroup_of(&batch), which is not a durable primitive. The local bindinglet tags = …is not a call, so it does not count. - The doc comment above
fold_innernamesgroup_of, so the comment and the code agree. - The fix is at the right layer. It removes the false match. It does not add a registry entry that would claim a durable-effect protocol for this file.
O1 (declined in the reply): accepted. O1 was optional. The reason holds: a scope-aware skip adds a walk to the guard's scanner, and a wrong skip could hide a real call. This is the first false positive of its kind, so a rename is the cheaper fix until it recurs.
Liability
The delta adds nothing: no state, API, setting or registry entry. It removes the only CI failure. The PR as a whole keeps the assessment from my earlier review: one shared inner loop (fold_inner) in place of two copies, so a third strategy gets streaming by calling it, not by copying it.
Other open PRs
git merge-treeof this head with #1345: clean.- #1316 and #1334: conflict only in
crates/omnigraph-gqt/README.mdand theuselines ofsubquery_aggregate.rs, as atb49e232a. - #1362 (new since my last review): conflicts only at the end of
crates/omnigraph/tests/integration/engine_v2_memory.rs, where both PRs append tests. Keep both tests when the second one merges.
Validation
Local runs are in a clean detached worktree at 4acd86bf, debug build, RUST_MIN_STACK=16777216.
| What | Commit | Command | Result |
|---|---|---|---|
| Chokepoint guard, before the rename | 4acd86bf with anti_join.rs of b49e232a |
cargo dev-test -p omnigraph-engine --test forbidden_apis -- graph_visible_write_chokepoints_are_registered |
fails: engine/operators/anti_join.rs .tags(: observed 1, no registered protocol (the B1 reason) |
| Chokepoint guard, after | 4acd86bf |
same | passes |
forbidden_apis, whole binary |
4acd86bf |
cargo dev-test -p omnigraph-engine --test forbidden_apis |
44 passed, 0 failed |
| GQT regression for #1338 | 4acd86bf |
cargo test -p omnigraph-gqt --locked --test gq_logic_tests -- issue_1338 |
passes (3.41 s, inside the 10 s budget) |
| CI, all checks | 4acd86bf |
GitHub Actions | 28 passed, 4 skipped, 0 failed. Test Workspace, Storage Upgrade Compatibility and CI Gate now pass, so the 2,176 workspace tests that CI did not reach at b49e232a ran at this head |
Regression proof. The delta fixes a source guard, not query behavior, so GQT cannot express it. The guard test is the cheapest owner, and it is already in the tree. I put back the b49e232a version of anti_join.rs (only the three renamed lines differ), and the guard failed for the predicted reason. With the head's file it passed. I restored the file; git status is clean.
Limits. I did not run the full workspace suite locally; CI covers it at this head. The before/after proof for the streaming fold (the issue_1338 GQT case and the Rust memory test) is in my earlier reviews; this delta does not touch that code path except the rename. I made no performance claims. I removed all temporary edits, and I pushed nothing to this branch.
|
Reply to the delta review of B1: fixed, as the review confirms. Nothing more to do. O1: closed. The review accepts the reason it was declined. No change. Merge state. No code changed, so I ran no gates. CI on |
|
Merged Conflict. There was one conflict, at the end of Checks at
CI. At
I did not change the test. The flake belongs to #1361. The review findings do not change. B1 and O1 stay as in my earlier replies. |
ragnorc
left a comment
There was a problem hiding this comment.
Review of #1341 at 5313732f (delta since 4acd86bf)
Why. My review at 4acd86bf (review 5873132563) found no blocking and no optional items. Since then the head moved to 942aa0c4, and then to 5313732f while I reviewed. This review checks only that delta.
What changed since 4acd86bf. Two merge commits and no other change. 942aa0c4 merges main at bbabbd7d (#1365, #1184, #1364, #1362). 5313732f merges main at 568180aff (#1346). The PR's own diff against its merge base is the same, line for line. Only the position of the new test in engine_v2_memory.rs and one context line in docs/dev/execution.md moved.
The merge resolved one conflict by hand, at the end of crates/omnigraph/tests/integration/engine_v2_memory.rs. Both this PR and #1362 append a test there. The resolution keeps both tests: correlated_block_folds_inner_batches_as_they_stream_issue_1338, then #1362's a_query_that_fits_a_budget_fits_every_larger_budget_issue_1361. That is the outcome my last review asked for. git show --remerge-diff shows no other hand edit in 942aa0c4, and none at all in 5313732f. #1362 and #1346 also edit other rows of docs/dev/execution.md; this PR's AntiJoin row applies beside them unchanged.
#1346 changes FilterExec, which can run inside a correlated block's inner plan (the issue_1338 query has one inner filter). It changes how one batch is filtered, not how batches stream. So the fold still sees each inner batch once and in order. The issue_1338 Rust test passes at 5313732f.
Impact. No change to behavior, results or the memory model since 4acd86bf.
Recommendation: ready to merge. No blocking findings. No new optional findings. CI at this head is green. At 942aa0c4 one test failed in CI. It is intermittent on main and not caused by this PR (see below).
CI: issue_1361 failed at 942aa0c4, not caused by this PR
I agree with the reply on this flake, and I checked it on my own:
- The failing query (
$p $e:knows $q,return { max($e.payload) }) has no correlated block, so it never runsAntiJoinMaskExec.SubqueryAggregateis used only fromanti_join.rs. Those two files are all the source this PR changes. - At
942aa0c4the refusal was at 5 MiB, ingraph take output(cannot reserve 358966 bytes: 4928521 bytes are reserved, query limit is 5242880 bytes), the same place as the failure in #1367's run 43301358856. - Locally it passed 6 of 6 times at
942aa0c4and 5 of 5 at5313732f. It passed in CI at5313732f.
#1361 is closed, but its regression test still fails sometimes on main. That needs its own follow-up, not a change in this PR.
Other open PRs
5313732f merges cleanly with current main 7747bd3b (#1348). I also ran git merge-tree of 942aa0c4 with main 568180aff and with the head of each of the 37 other open PRs. Then I compared each result with the same PR merged into main.
- The only conflicts this PR adds are in #1316 and #1334 (drafts): the
uselines ofsubquery_aggregate.rs, the same as at4acd86bf. Theircrates/omnigraph-gqt/README.mdconflict is withmaintoo. - #1175 (draft) conflicts with this head in
engine_v2_memory.rs, but it conflicts withmainin that file as well, and in 34 others. So this PR adds no new work for it.
Validation
Local runs are in a clean detached worktree, debug build, RUST_MIN_STACK=16777216.
| What | Commit | Command | Result |
|---|---|---|---|
| Both tests at the hand-resolved spot, plus #1362's unit test | 942aa0c4 |
cargo dev-nextest -E 'package(omnigraph-engine) & (test(issue_1338) | test(issue_1361))' |
3 passed, 6 runs of 6 |
All of engine_v2_memory |
942aa0c4 |
cargo dev-nextest -E 'package(omnigraph-engine) & binary(integration) & test(/^engine_v2_memory::/)' |
33 passed, 0 failed |
| Same tests, after #1346 | 5313732f |
same two commands | 3 passed, 5 runs of 5; engine_v2_memory 34 passed, 0 failed |
| PR diff unchanged | 4acd86bf, 942aa0c4, 5313732f |
diff of git diff $(git merge-base <c> origin/main) <c> |
only index lines, the moved hunk and one context line differ |
| CI, all checks | 5313732f |
GitHub Actions (CI 43309969887, GQ Logic Tests 43309969575, DST 43309969293) | 27 passed, 4 skipped, 0 failed, including Test Workspace with issue_1361, and GQ Logic Tests with the issue_1338 GQT case |
| CI, all checks | 942aa0c4 |
GitHub Actions | 26 passed, 4 skipped, Test Workspace and CI Gate failed on issue_1361 only. GQ Logic Tests (with the issue_1338 GQT case), DST, clippy and the S3 and Azure jobs passed |
Regression proof. This delta is a merge with no change to the PR's code. So there is no new fix to prove. The before/after proof for the streaming fold (the issue_1338 GQT case and the Rust memory test, red on 2c4b2f12, green on the PR) is in my earlier reviews and still applies.
Limits. I did not run the full workspace suite or the GQT corpus locally; CI ran them. I did not find the cause of the issue_1361 flake; I only showed that this PR's code is not on its path. I made no performance claims. I made no edits in the worktree, and I pushed nothing to this branch.
|
Reply to the delta review of The review has no blocking items and no optional items, so there is nothing to fix. I checked the parts that could have changed since it was written:
The For a maintainer: nothing to decide. The review recommends merge. |
Merges main 9fb5b48..4f7c207 (17 commits; old merge base 9fb5b48). Carried into edition-3 code: - ModernRelay#1341 (fold a correlated block's inner rows as they stream): anti_join.rs takes main's streaming fold_inner on both strategies and keeps the branch's comparison against the outer row (reads_outer_row, keep_mask_against, keep_rows_against) in each; issue_1338 GQT case and the engine_v2_memory test respelled in edition 3. - ModernRelay#1389 (sort merge reservation): engine fix merged as is; its three GQT cases respelled (property braces to `where`). - ModernRelay#1406 (widen an edge @card): RelaxCardinality beside the branch's ChangeFullTextAnalysis and ChangeVectorDistance; schema_apply test queries respelled; skill and schema reference keep both lists. - ModernRelay#1398, ModernRelay#1410 (gqt --check, closed diagnostic set): main's parser taken whole, the branch's changes re-applied (same as reference, the parse refusal as a step under `expect error:`, a section's called reads); diagnostic rules, process.rs and grammar/v1.txt say `same as reference`; four tests/invalid fixtures respelled. - ModernRelay#1416 (bench over a shallow clone): the _served workloads and scenarios deleted as main did; served tests keep the edition-3 write spelling; the catalog identity fixture regenerated. Other main commits touch no query-language code and merge unchanged. The DST cost golden matches the merged tree; its moved lines are ModernRelay#1390's.
Summary
A correlated block on the per-row path (
max($b.x) { $a knows $b … } > …, or any block whose inner reads the outer row) drained its whole inner output into aVec, charging every batch to the query pool, and only then folded it. Its peak memory was the whole fan-out, so a large fan-out was refused withquery_memory_bytesalthough the fold needs one batch at a time.This change folds each inner batch into the
SubqueryAggregateas the stream yields it and drops it, as the key-set path in the same file already does.Closes #1338
Reproduction
New test
engine_v2_memory::correlated_block_folds_inner_batches_as_they_stream_issue_1338: one hub with 16,384 neighbours, each with a 1 KiB payload (16 MiB of inner output), and the queryunder a 12 MiB query pool. The test asserts that the plan uses the
correlatedstrategy, so the block cannot move to the key-set or bulk path without the test noticing.2c4b2f12:ResourceLimitExceeded { resource: "query_memory_bytes", limit: 12582912, actual: 12587123 }. With a 24 MiB pool, main is still refused (actual: 25376187).assert_released).Verdict: reproduced as the issue describes.
Root cause
AntiJoinMaskExec::executeincrates/omnigraph/src/engine/operators/anti_join.rscalleddrain(inner.execute(..), &reservation). That held every inner batch under the reservation, and only then calledabsorb_inner_batches(subquery_aggregate.rs), which folded the batches one by one. Holding the batches was not needed: the fold is per batch and order-preserving.Decisions
min/maxstill keep the first signed zero and the results are the same. Each in-flight batch is not held under the reservation, the same askey_set_mask. The aggregate's per-row accumulators are now reserved before the inner stream starts instead of after it, so a pool too small for them refuses before the inner plan runs.FilledSlotguard lives until the inner stream ends.OuterReferenceExectakes the tagged outer batch when its stream is first polled, so the guard must cover the whole stream, as before.absorb_inner_batches(&[RecordBatch])becameabsorb_inner_batch(&RecordBatch). This was its only caller.AggregateExec, which also streams and spills. The issue notes that this waits on the float-semantics decision in query: correlated block aggregates fold row by row; decide their float semantics, then use DataFusion's grouped accumulators #1260.Lance: not involved. The change is in the engine's operator, after Lance has produced the batches; no pending Lance change affects it.
Before and after
2c4b2f12query_memory_bytes)actual25,376,187)actual8,661,403)With smaller pools, the refusals in this branch come from the inner expand and hydration (
expand hop,hydrate_nodes), which hold their own working sets; they are not changed here.Tests and checks
cases/issue_1338_correlated_block_folds_inner_batches.gqt(changed in review): 16 MiB of inner rows under a 12 MiBquery_memory_bytes, refused on2c4b2f12and answered here._issue_1338Rust test intests/integration/engine_v2_memory.rs, the owner of pool acceptance tests, for what GQT cannot assert: the run releases every reservation it took.fold_inneris the one inner-stream loop of both strategies (changed in review);absorb_inner_batchis removed. Its group closure is namedgroup_of, because the chokepoint guard reads a call namedtags(as Lance's native-refDataset::tags()(changed in review).cargo fmt --all --checkandcargo dev-clippy: pass.tests/integration), run with CI'sRUST_MIN_STACK=16777216: 665 passed, 0 failed, 9 ignored.db::upgrade::tests::interrupted::unreadable_legacy_object_asks_for_a_rerun, which makes a file unreadable with mode000; this box runs as root, so the file stays readable. It is in code this change does not touch.docs/dev/execution.md(theAntiJoinrow) and achangelog.dperformance note are updated.Risks and what was not verified
Workflow run: run-wf-reproduce-modernrelay-omnigraph-issue-20261110T1142Z-e43d0f
Response run: run-wf-respond-modernrelay-omnigraph-pr-20261112T1012Z-4bf986
Response run: run-wf-respond-modernrelay-omnigraph-pr-20261112T1712Z-71dbf1
Response run: run-wf-respond-modernrelay-omnigraph-pr-20261112T2007Z-4a902a
Response run: run-wf-respond-modernrelay-omnigraph-pr-20261113T0112Z-4da4e0