Repository navigation
Conversation
ragnorc
left a comment
There was a problem hiding this comment.
Review of #1431 at 64d74072
Recommendation: fix two things before merge. The change is correct for the path it targets. But the same refusal stays on a second, common merge path, and the charge the new bound compares against depends on the setting that raises it.
What the change does and why
A three-way branch merge keeps the scalar columns of every changed row in memory while it checks constraints. That hold had a fixed 32 MiB ceiling, so write_max_bytes could not raise it (#1427). Now the ceiling is max(32 MiB, write_max_bytes), taken from the WriteBudget that the merge already captures once. The diff is small: one constant is removed, and one constructor now takes the budget (merge.rs:3396). The docs say what the bound covers, that it is operation-wide, and how to raise it.
I agree with the decisions in the description. A lower write_max_bytes must not lower a hold of rows that the merge only reads (write_max_bytes_merge_pointer_adoption.gqt shows why). Also, a separate setting would add a knob with the same meaning. The server ceiling claim is true: refuse_raised_budgets (omnigraph-server/src/lib.rs:1179) caps a request at max(server, default).
Blocking
B1. The pure-insert path keeps a fixed 32 MiB cap, so the issue's refusal is still there.
scan_proven_pure_inserts_for_validation refuses any single validation batch above KEYED_WRITE_MAX_BYTES (merge.rs:3696). It does this before it charges the batch to MergeValidationBudget. This path runs when the target table equals the merge base and the source only inserted rows. That is the usual case when main moved only other types. The scanner's batch_size_bytes is only a target, so it cannot split a row. A row whose scalar columns pass 32 MiB therefore gets a batch above the cap, whatever write_max_bytes is.
Reproduction (a temporary case, not pushed). The schema is Doc plus node Other { slug: String @key }. main inserts one Other. b0 inserts one Doc with 9 × 4 MiB strings under set write_max_bytes = 268435456. Then set write_max_bytes = 268435456; branch merge b0 into main; runs:
FAIL step 4 (branch merge): resource limit exceeded for branch-merge pure-insert
validation batch bytes for node:Doc: actual 37750414, limit 33554432
A control case uses the same data, except that main inserts a Doc instead of an Other. That sends the table to the rewrite path. It fails before this PR (retained validation delta bytes: actual 37750414, limit 33554432) and merges at 64d74072. So the same data and the same setting merge or fail depending only on which type main touched.
The broken invariant is "the merge-validation hold has one bound, and the user can reach it". The per-batch check is a second bound on the same hold. It is redundant: a batch above the aggregate limit already fails reserve. And it is pinned to the constant. Delete it and let MergeValidationBudget own the only byte bound. Keep the row-count check. Add a pure-insert row to issue_1427_*.gqt (main writes another type), so the gate covers both paths.
B2. The charge depends on the layout that the same setting controls, so the refusal's actual cannot guide the fix.
reserve_projected_batch charges get_array_memory_size() (merge.rs:3417). That counts every backing buffer of a sliced batch, including bytes of rows that belong to other batches. The description already reports this. The PR makes it matter more, because write_max_bytes now changes both the bound and the staged layout. I measured this with the PR's own case, changing only the merge's setting:
| merge setting | result |
|---|---|
| default (32 MiB) | refused, actual 37758380 |
write_max_bytes = 41943040 (40 MiB, above that actual) |
refused, actual 75516760 |
write_max_bytes = 268435456 |
merged |
A user who follows the refusal (and the new merge.md text) sets a value above the reported actual, and is refused again at twice that value. The case itself needs 256 MiB for about 36 MiB of rows. The repo's resource refusals report actual as the smallest limit that would admit the work (see the load csv text in settings.md). This refusal does not do that any more once the setting is raised.
The fix already exists in this file. The write path compacts a slice when its physical size passes its logical size (merge.rs:4779–4785, using table_store::batch_slice_memory_size). Do the same in reserve_projected_batch: compact the batch with take_record_batch when get_array_memory_size() passes the logical size plus a small slack, then charge and return the compacted batch. Both validation paths go through that one function, so one change fixes both. The charge then equals what the validator really keeps, it does not depend on staging, and it also frees the unused page bytes that the retained slices pin today. Add one assertion: the PR's case, merged at a setting just above the default refusal's actual, succeeds.
If you prefer to ship B2 separately, then merge.md and the changelog should not suggest that a setting near the refused actual is enough. Open the follow-up issue before merge.
Optional
- O1. "How many bytes does this batch hold" is answered in three ways in
merge.rs:get_array_memory_size,batch_slice_memory_size, andwrite_batch_bytes. Each caller picks one. If one helper (for exampleretained_batch_bytes, which compacts when needed) owns "bytes a retained batch costs", B2 cannot come back through a new caller. - O2. Other fixed
KEYED_WRITE_MAX_BYTESuses remain in the merge (LINEAGE_CANDIDATE_MAX_BYTES, the delete-chunk filter bound atmerge.rs:363). I did not find one that blocks a user-raised merge the way B1 does. Still, a one-line comment at each site saying why it stays fixed would stop the next reader from repeating #1427's question.
Liability
Net, the PR removes liability. One constant and one Default impl go away, and the bound now follows the documented knob. It adds one asymmetric rule (max), which is documented and has precedents. Five more changes like this one would each find another fixed constant on a merge path. B1 is the first of those. With B1 fixed (one owner for the validation byte bound) and B2 fixed (one measure of retained bytes), that class closes by design.
Validation
All runs were local, in a clean worktree at 64d740726f8d (merge base f5b6c761), debug, Linux x86_64. Command: cargo test -p omnigraph-gqt --locked --test gq_logic_tests -- <filter>.
- At
64d74072:issue_1427_*ok.issue_856_*(2) ok.write_max_bytes_merge*(4, including pointer adoption and the Blob refusal) ok. - Before the fix (
merge.rstaken fromf5b6c761, everything else at head):issue_1427_*FAILS at step 8 withretained validation delta bytes: actual 37758380, limit 33554432, which is the predicted reason. - The temporary cases above (pure-insert, rewrite control, the 40 MiB variant) ran both ways as reported. All temporary edits are reverted, and nothing was pushed.
- CI at
64d74072, when I checked: 20 checks passed, including GQT (ordinary), Fix Regression Gate and clippy. GQT (dst), GQT (served), Storage Upgrade Compatibility, the format fence and the aws server tests were still pending.
Limitations: I did not trace in the pinned Lance decoder why one-row batches share a buffer. The doubling comes from measured charges, not from reading the code. I did not run a served merge, or S3/Azure storage.
… kept rows The pure-insert validation scan refused any batch above a fixed 32 MiB before the shared allowance saw it, so a row past 32 MiB stayed refused whatever write_max_bytes was. MergeValidationBudget is now the only byte bound on what validation retains. The allowance charged get_array_memory_size, which counts every buffer a sliced batch shares with others. A retained batch is now copied to its own rows when it holds more, and charged that size, so the refusal's actual no longer depends on the staged layout that write_max_bytes also changes. A delete chunk planned under a raised write_max_bytes is checked against the bound it was planned with, not the fixed default. Review: ModernRelay#1431 (review)
|
Reply to the review of B1. Valid, fixed. I reproduced it: with the engine from B2. Valid, fixed as you proposed. O1. Taken, in part. O2. Taken. One site was not fine. Delete chunks are planned with Gates at
Not verified: a served merge, and S3 or Azure storage. |
ragnorc
left a comment
There was a problem hiding this comment.
Review of #1431 at 0a74214c
This follows the review of 64d74072 (5931107253) and your reply. Merge base f5b6c761. I read the whole delta 64d74072..0a74214c (4 files, +113 −35) and reran the gate.
Recommendation: fix one thing before merge. The delta fixes B1, B2 and O2 as described, and I checked each one. But the B2 fix lets one more shape get past validation. That shape then fails in publication with an internal error, where before it got a typed refusal. The fix is one line, using the helper this delta adds. Also, the docs leave out fast forwards.
What the delta does
- B1 (fixed). The pure-insert validation scan no longer has its own 32 MiB per-batch byte check.
MergeValidationBudgetis now the only byte bound on what validation keeps. The case has a new pure-insert step (main writes onlyOther), and it passes. - B2 (fixed).
reserve_projected_batchtakes the batch and returns the batch to keep. If the batch holds more than its own rows,table_store::compact_retained_batchcopies it first. The charge is the size of the kept batch. The rewrite-path step now merges at 40 MiB, just above the default refusal'sactual. - O1 (partly taken).
batch_holds_only_its_rowsnow owns the rule for "does this batch hold only its rows". Validation and the single-slice path innext_exact_staged_chunkboth use it. I agree thatwrite_batch_bytesanswers a different question. - O2 (taken). Each
IdChunkrecords the filter bound it was planned under, andcheck_boundsuses that bound. Comments explain why the lineage bounds stay fixed (a miss only falls back to the walk).
Blocking
B1. Publication still counts a null's span twice. The B2 fix now lets a merge reach that code, and it fails there as an internal error.
Shape: in one statement, b0 inserts an 18 MiB Doc (nine 2 MiB strings) and a second Doc with those columns null. Both rows land in one fragment. main writes its own Doc, so the merge goes through the rewrite path. This is the shape in #1433's issue_1427_merge_rows_sharing_a_fragment_with_nulls.gqt. I ran that case as a temporary file. Results:
| engine | default setting | write_max_bytes = 35651584 (34 MiB) |
41943040 (40 MiB) |
|---|---|---|---|
f5b6c761 (base) |
refused: retained validation delta bytes: actual 37755302, limit 33554432 |
(setting cannot raise it) | (setting cannot raise it) |
64d74072 |
same refusal | refused: actual 37755302, limit 35651584 |
refused: actual 75510604, limit 41943040 |
0a74214c |
fails: branch merge reconstructed a keyed chunk outside its planned bound: 2 rows / 37754978 bytes |
same failure | merged |
At the head, validation now admits about 18.9 MB of real values, so the B2 fix works here as intended. The failure comes later, in next_exact_staged_chunk (merge.rs:4809). The two rows come back as two slices, and that path joins them with concat_batches without compacting them first. concat copies each slot's whole offset range, so the null row copies the wide row's bytes a second time: 37,754,978 bytes for about 18.9 MB of values. The error is ManifestError { kind: Internal }. It names no resource and gives no action, so a user or agent cannot tell that write_max_bytes would fix it. I checked the effect: main stays at 1 row after the failure, and a retry at 40 MiB merges (3 rows). So nothing is published, but the error class is now wrong.
The broken invariant is the one O1 names: what a batch costs has one measure, and every site that keeps or rebuilds batches uses it. The single-slice path above line 4809 already compacts. The multi-slice path does not.
Fix (I tested it): before concat_batches, pass each slice through compact_retained_batch:
let slices = slices
.into_iter()
.map(|s| crate::table_store::compact_retained_batch(s).map(|(b, _)| b))
.collect::<Result<Vec<_>>>()?;With that one change, the whole temporary case passes at the default setting: the three-way merge, then a validated fast forward of a second pair. These also still pass: issue_1427_* (1), issue_856_* (2), write_max_bytes* (16) and merge* (69) GQT cases. Please add this shape to the regression gate, as a step in issue_1427_*.gqt or as #1433's case file. It fails at 0a74214c for the reason above. #1433 fixes the same thing ("Publication no longer copies a null's span"), with take before concat.
Before merge
D1. The docs say only a non-fast-forward merge holds the delta. A validated fast forward holds it too. merge.rs:5975 runs build_merge_changeset for every fast forward except a narrow proven pure-insert whose checks are only identity-backed @keys. So @range, enum, non-key @unique and edges all keep validation on. A temporary case shows this: two 18 MiB rows on b0, the schema has @range(n, 0..100), main is unchanged. At 0a74214c the fast forward is refused at the default with retained validation delta bytes, and it fast-forwards at 40 MiB. The red_on of issue_856_* was also a fast forward. Please fix the scope in docs/user/branching/merge.md:213 ("a merge that is not a fast forward"), the write_max_bytes row in docs/user/queries/settings.md:87 ("a three-way branch merge"), skills/omnigraph/references/data.md:50, and the changelog line.
D2. Two open PRs close #1427. The review of #1433 (5932755660) already covers this, so I give only the cost here. If this PR lands first, the user guide, settings reference, skill and changelog promise "raise write_max_bytes" as the way through. #1433 keeps a fixed 32 MiB (of keys) and documents a staged merge instead, so it would have to take that promise back. Choose the order before you merge either one.
Optional
- O1. A delete chunk's filter-byte bound guards a string that no delete builds. Since #1295, a delete chunk becomes
_rowid IN (…)(merge.rs:341; its only use is at:4582).filter(), which builds theid IN (…)string, runs only for candidate chunks (:1867,:2818). So for delete chunks,filter_bytesand the newmax_filter_bytesonly check each other. Also, at a lowwrite_max_bytes(the pointer-adoption case uses 128), a long deleted id is refused asbranch-merge delete filter bytes, for a filter that is never built. If delete chunks were split only by row count and retained bytes, the field and this whole mismatch class would go away. That is outside this PR's issue, so a follow-up is fine. - O2. A refused batch is copied before it is refused.
compact_retained_batchcopies beforereserveruns, so for a moment the batch that crosses the limit is held twice. The logical size is already computed insidebatch_holds_only_its_rows. If you reserve that size first and copy after, a refusal costs no copy. This is small, because the scan already holds the batch.
Liability
The delta removes liability. A parallel per-batch bound is gone, a 64 KiB slack constant now has one name, and one helper decides when a batch is copied. With B1 fixed, every validation and publication site in the merge measures retained bytes the same way. Five more changes like this would each add a site that keeps or rebuilds batches. That class stays closed only if each new site calls compact_retained_batch. A sentence on the helper that says so would make that the rule for the next author. If #1433 lands instead, Sweep takes over the validation side, and this helper's remaining job is publication.
Validation
All runs were local, in a clean detached worktree, on Linux x86_64, debug, local filesystem. GQT: cargo test -p omnigraph-gqt --locked --test gq_logic_tests -- <filter>. Before-states replace only exec/merge.rs and table_store.rs, the only engine files this PR changes.
- At
0a74214c:issue_1427_*ok,issue_856_*(2) ok,write_max_bytes_merge*(4) ok.cargo test -p omnigraph-engine --lib -- branch_merge_delete: 3 pass, including the new…_passes_its_own_bound. - Before the PR (
f5b6c761engine, cases from the head):issue_1427_*FAILS at step 8 withretained validation delta bytes: actual 37758380, limit 33554432, the predicted reason. - Before the delta (
64d74072engine):issue_1427_*FAILS at step 8 withactual 75516760, limit 41943040, as your reply says. The new unit test, added to64d74072'smerge.rs, fails withdelete chunk exceeds its planned byte bound: 37748754 bytes. - Temporary cases (not pushed; removed): the null-span shape at default, 34 MiB and 40 MiB on all three engines (table in B1); an effect check (main unchanged, retry merges); and the
@rangefast forward (D1). The candidate fix for B1 was applied, tested as above, then reverted. - CI at
0a74214c: 27 checks pass and 4 are skipped, none failing.
Limitations: I did not read the pinned Lance decoder to see why it returns the two rows as slices over one buffer. The doubling is measured, and it matches #1433's diagnosis. I did not run the full engine suite with the candidate fix, a served merge, or S3/Azure storage.
… chunk A keyed publication chunk built from several scan slices concatenated them as they were. A null row whose offsets span a wide row's bytes then copied those bytes a second time, and the chunk failed its own planned bound as an internal error. Each slice now goes through compact_retained_batch first, as validation and the single-slice path already do. Docs: a validated fast forward holds the validation delta too, not only a three-way merge.
|
Reply to the review of B1. Valid, fixed as you proposed. I reproduced it with your shape: at D1. Valid, fixed. I checked D2. Needs a decision. It is D1 in the O1 (delete-chunk filter bound). Declined here, agreed as a follow-up. You are right that O2 (copy before refusal). Declined. The charge is the size of the batch that is kept, measured after the copy. That is the rule B2 set. Reserving the logical size first would add a second measure at the one site that decides the charge, and the two can differ (offsets, validity buffers, the 64 KiB slack). The extra cost is one batch for the moment before a refusal, and the scan already holds that batch. Gates at
Not verified: the full engine suite locally (CI ran it), a served merge, and S3 or Azure storage. |
Decision neededD1. Which pull request should close #1427? Two open pull requests fix it in different ways: this one raises the validation limit with
Recommendation: option 2, because it removes the cause (the wide delta held in memory) instead of moving the limit, and it leaves one rule in the code; keep this pull request open as the fallback until the other one is ready. Reply with the option number for each decision, or say what you want instead. |
ragnorc
left a comment
There was a problem hiding this comment.
Review of #1431 at 28c06264 (delta since 0a74214c)
This follows the review of 0a74214c (5933882191) and your reply. Merge base f5b6c761 (unchanged). I read the whole delta, one commit (7 files, +95 −7), and ran the new regression before and after the fix.
Recommendation: the delta has no blocking defect. It fixes B1 and D1 of the previous review as described, and I checked both. Two optional points follow. Which PR closes #1427 waits on Decision needed D1; I do not raise it again here.
What the delta does
- B1 (fixed). In
next_exact_staged_chunk(merge.rs:4808-4817), the multi-slice path now passes each slice throughtable_store::compact_retained_batchbeforeconcat_batches. A null row whose offsets span a wide row's bytes is copied with its own rows only, so the publication chunk no longer holds those bytes twice. The new caseissue_1427_publication_compacts_rows_sharing_a_fragment_with_nulls.gqtis the regression gate. It fails before the fix for the predicted reason and passes after it (see Validation). - D1 (fixed).
docs/user/branching/merge.md, thewrite_max_bytesrow indocs/user/queries/settings.md,skills/omnigraph/references/data.mdand the changelog note now say that a fast forward also holds the delta, unless it is the narrow pure-insert case. I compared the stated exemption withproven_fast_forward_needs_no_validationandnode_inserts_need_no_validation(merge.rs:3722-3761):@range,@check, non-key@unique, enum and an outgoing@cardminimum above 0 keep validation on, and every edge table does too. The text matches those checks.
Blocking
None.
Optional
O1. The new comment on compact_retained_batch says every site goes through it. Three sites do not, and the rule now has four spellings. table_store.rs:8946-8947 says: "Every site that keeps or rebuilds scanned batches (validation, and the merge's publication chunks) goes through this." These sites keep or rebuild scanned batches with their own copy rule:
- the single-slice branch of the same function,
merge.rs:4792-4807: it keeps the slice if it is whole, within budget andbatch_holds_only_its_rows, and otherwise copies it withtake_record_batch; StagedTableWriter::push_row,merge.rs:780-783: it always copies one row withtake_record_batch;next_proven_insert_batch,table_store.rs:8783-8816: it copies a partial range withcopy_batch_rangeand keeps a whole scanner batch.
I read all three. Each copies before it concatenates or keeps, so none of them doubles a null's span today. This is not a defect. But the comment states a rule that the code does not follow, and the next author will trust it. The smallest coherent change is at the site this delta already edits: compact every slice, single or multiple, with compact_retained_batch. Then use the one slice, or concat the slices. The extra conditions of the single-slice branch are not needed for correctness. A partial slice of a larger parent fails batch_holds_only_its_rows and is copied. A slice over the budget is refused by the bound check at merge.rs:4821-4829 whichever copy made it. That removes one spelling and makes the comment true for publication. Alternatively, narrow the comment to the two sites that use the helper. push_row and the proven-insert stream can stay as they are; they copy for a different reason (one row, or one bounded range).
O2. The docs state the fast-forward exemption as certain. The code also needs a proof of the branch's history. merge.md:213-216 says a fast forward holds the delta "unless it only inserts nodes of types with no @range, …". proven_fast_forward_needs_no_validation holds only when every candidate table is AdoptPureInserts or AdoptSourceState { PureInserts }. That needs try_proven_pure_insert_history (merge.rs:985) to certify each transaction since the base as a fenced insert. Any miss falls back to full validation, for example a row inserted and then updated on the branch, or transaction files that are not there. So a branch whose net effect is "only inserts" into unconstrained types can still be validated and refused. Suggested wording: "A fast forward skips this only when every commit on the branch since the fork inserted nodes of types with no …". Also, "required-edge constraint" is not a schema term. "an outgoing edge type with a @card minimum above 0" names what node_inserts_need_no_validation checks. The same applies to the skill line ("unless it only inserts nodes with no constraint beyond @key").
Liability
The delta removes liability. It adds 8 lines and no new state, API or invariant, and the publication side now measures a kept slice as validation does. With B1 fixed, I know of no merge site that charges a batch differently from the bytes it keeps. Five more changes of this kind would each add a site that keeps or rebuilds scanned batches. The class stays closed only if each new site copies before it keeps. O1 makes that one rule at publication instead of two, and makes the comment that tells the next author about it true.
Validation
All runs were local, in a clean detached worktree at 28c06264, on Linux x86_64, debug, local filesystem. GQT: cargo test -p omnigraph-gqt --locked --test gq_logic_tests -- <filter>.
- At
28c06264:issue_14272 of 2 pass, including the new case (1.76 s).issue_8562 of 2,write_max_bytes16 of 16 andmerge69 of 69 pass.cargo fmt --all --checkpasses. - Before the fix:
crates/omnigraph/src/exec/merge.rsandcrates/omnigraph/src/table_store.rsfrom0a74214c(the only engine files the delta changes), with the case from the head. The new case FAILS at step 4:branch merge failed: branch merge reconstructed a keyed chunk outside its planned bound: 2 rows / 37754978 bytes. That is the predicted reason (the null row's span copied a second time). The otherissue_1427case passes. I restored both files and ran the head again: 2 of 2 pass. - By reading: the docs exemption against
merge.rs:3722-3761, and the three copy sites in O1. - CI at
28c06264: 27 checks pass and 4 are skipped, none failing.
Limitations: I did not run the engine's own merge suites locally; your reply reports 186 of 186 and CI Test Workspace passes. I did not check which publication path the case's fast-forward step takes, so I do not claim it exercises the fixed branch; step 4 does. I did not read the pinned Lance decoder to see why it returns the two rows as slices over one buffer (as in the previous review). I did not run a served merge or S3/Azure storage. No Lance surface changes in the delta.
Summary
A three-way
branch mergeholds the scalar columns of every changed row in memory while it checks constraints. That allowance was a fixed 32 MiB constant (MERGE_VALIDATION_MAX_BYTES), so neither--set write_max_bytesnorOMNIGRAPH_WRITE_MAX_BYTEScould raise it. A branch whose delta had grown past 32 MiB could never be merged, even though the refusal saidaction: fix. The allowance is now the larger of 32 MiB and thewrite_max_bytesvalue the merge captures once for the operation. Raising the setting lifts it. The default and lowered values behave exactly as before.Closes #1427
Reproduction
crates/omnigraph-gqt/cases/issue_1427_three_way_merge_validation_follows_write_max_bytes.gqtis derived from the #856 case. Main gets its own write afterb0forks, so the last merge is three-way. That case:mainunchanged (2 rows), thenset write_max_bytes = 41943040;(40 MiB, just above the default refusal'sactual) and checks thatmainhas 4 rows, thenb1, writes only anOtherrow onmainand one 36 MiBDocrow onb1, so the merge takes the pure-insert path; it refuses at the default and merges at 40 MiB.On clean main
8ab65713a0894fd5b4514e1b26f95c267daa50f3:With this change:
ok issue_1427_three_way_merge_validation_follows_write_max_bytes 2.02s.Root cause
build_merge_changeset(crates/omnigraph/src/exec/merge.rs) built itsMergeValidationBudgetwithDefault, which setlimit: MERGE_VALIDATION_MAX_BYTES(= KEYED_WRITE_MAX_BYTES).branch_merge_as_with_optionsalready capturesWriteBudget::from_settings(self.settings())and passes it intobranch_merge_impl. The candidate staging and the validator's keyed checks use it, but the retained-delta budget never received it. The two values matched only by default;session.rstests that the defaults are equal.Change
MergeValidationBudget::new(write_budget)replacesDefault, withlimit = write_budget.bytes().max(KEYED_WRITE_MAX_BYTES).build_merge_changesettakes the merge'sWriteBudget, and the separate constant is gone. The operation-wide, cross-table scope and the charge-before-retain order are unchanged.0a74214c). The pure-insert scan no longer refuses a batch above a fixed 32 MiB before the shared allowance sees it; it keeps only its row-count check.MergeValidationBudget::reserve_projected_batchcopies a batch that holds more than its own rows (table_store::compact_retained_batch, the same 64 KiB rule the write path used) and charges that size, so the charge is what the validator keeps and does not depend on the staged layout. A delete chunk is checked against the byte bound it was planned with, not the fixed default.docs/user/branching/merge.md: says what the allowance bounds, that it covers the whole merge (chunking the branch's loads does not split it), and how to raise it.docs/user/queries/settings.md: thewrite_max_bytesrow says a value above 32 MiB raises this bound and a lower value leaves it at 32 MiB.skills/omnigraph/references/data.md: the per-load bounds note now names the merge allowance and the setting that raises it.changelog.d/merge-validation-follows-write-max-bytes.fixed.md.Decisions
write_max_bytes(chosen). A raised setting lifts the ceiling, which is what bug: a true three-way branch merge is capped at 32 MiB of retained validation delta, and the cap is not reachable by write_max_bytes #1427 needs. The default and lowered values keep today's behaviour, so nothing a user relies on changes. A served request stays capped by the server's ceiling for the setting. The settings reference already has rules that act only above the default (an overwrite load's typed batch) or in one direction only (history_release_bytes).write_max_bytesexactly (rejected; it was the first commit here). A lowered setting would then also bound validation. Two existing owners show why that is wrong:write_max_bytes_merge_pointer_adoption.gqtdocuments that a merge which adopts a table by pointer and writes no row is not bounded bywrite_max_bytes(it failed withactual 462, limit 128), andmerge_fast_forward::write_max_bytes_merge_keeps_independent_payload_and_chunk_allowancessets 8,190/8,191 bytes to test the payload and chunk accounts. Validation is a hold of rows the merge reads, not one of its write accounts, so a lower write allowance should not refuse it.write_max_bytesis already the documented way to raise merge limits.ChangeSet, so this is a validator redesign. That belongs in its own issue if a raised allowance is not enough.Tests
issue_1427_*). It also proves the default still refuses with no effect.issue_856_repeat_fast_forward_past_validation_budgetandwrite_max_bytes_merge(engine and DST seeds 0 and 42).merge_fast_forward::branch_merge_validation_delta_is_aggregate_bounded_pre_armstill asserts the 32 MiB default;write_max_bytes_merge_pointer_adoption.gqtandwrite_max_bytes_merge_keeps_independent_payload_and_chunk_allowancesstill prove that a lowered setting does not lower it.Local gates at
0a74214c(Linux x86_64, debug)cargo fmt --all --check,cargo dev-clippy,scripts/check-docs.pyandscripts/check-release-note.py: pass.cargo dev-nextest -E 'package(omnigraph-engine)': 1890 of 1892 pass.db::upgrade::tests::interrupted::unreadable_legacy_object_asks_for_a_rerunfails because root ignoreschmod 000;composite_flow_multi_branch_sequential_mergesoverflowed its stack without CI'sRUST_MIN_STACK=16777216and passes with it.cargo test -p omnigraph-gqt --locked --test gq_logic_tests: 478 of 478.64d74072, new tests): the case fails at step 8 withactual 75516760, limit 41943040; with step 8 at 256 MiB it fails at step 13 withpure-insert validation batch bytes for node:Doc: actual 37754504, limit 33554432; the new unit testbranch_merge_delete_chunk_planned_under_raised_write_max_bytes_passes_its_own_boundfails.Local gates (Linux x86_64, debug, at 7a326ec)
cargo fmt --all --check: pass.cargo dev-nextest -E 'package(omnigraph-engine) | package(omnigraph-seams)'withRUST_MIN_STACK=16777216as CI sets it: 1912 of 1913 pass. The one failure,db::upgrade::tests::interrupted::unreadable_legacy_object_asks_for_a_rerun, useschmod 000, which root ignores on this box; it is unrelated to this change.cargo test -p omnigraph-gqt --locked: 478 of 478 cases pass, on the engine and DST targets.cargo test --lockedincrates/omnigraph-dst): pass;lane_b::also passes withRUST_MIN_STACKunset.cargo dev-clippyand the GQT clippy: pass.scripts/check-docs.pyandscripts/check-release-note.pyfor this title: pass.Shared scan buffers (fixed in
0a74214c)Before
0a74214c, a raised allowance staged the two large rows together, and each one-row validation batch was charged the whole buffer it shared with the other row: about 75.5 MB for about 37.8 MB kept. At 40 MiB the case was refused withactual 75516760, so a user who set a value above the default refusal'sactualwas refused again. The batch is now copied to its own rows before it is charged.Publication chunks keep only their rows (fixed in
28c06264)At
0a74214c, a merge whose staged rows came back as several slices of one decode buffer rebuilt its publication chunk withconcat_batchesover the raw slices. A null row whose offsets span a wide row's bytes then copied those bytes a second time, and the chunk failed its own planned bound as an internal error (branch merge reconstructed a keyed chunk outside its planned bound: 2 rows / 37754978 bytes). Each slice now goes throughtable_store::compact_retained_batchfirst, as validation and the single-slice path do. Regression gate:issue_1427_publication_compacts_rows_sharing_a_fragment_with_nulls.gqt(a three-way merge, then a validated fast forward, both at the default setting); it fails at0a74214cat step 4 with that error.The docs now also say that a validated fast forward (one with constraints to check) holds the validation delta, not only a three-way merge.
Risks and what was not verified
write_max_bytesabove 32 MiB now also raises the memory a three-way merge may hold for validation. On a server, a request cannot go above the process ceiling.settingsoverride (the engine path is shared; the GQT case runs the embedded engine), and S3 or Azure storage.Workflow run: run-wf-reproduce-modernrelay-omnigraph-issue-20261116T2347Z-2e4a19
Response run: run-wf-respond-modernrelay-omnigraph-pr-20261117T0202Z-83c4bf
Response run: run-wf-respond-modernrelay-omnigraph-pr-20261117T0627Z-6c033e