Repository navigation
Conversation
Several writes on one branch publish in one __manifest compare-and-swap as one graph commit. A per-branch publisher admits queued entries by footprint, composes same-table strict inserts by key into one detached commit, and publishes through the unchanged single-commit publisher. Motivated by the same-branch ceiling measured for the shared schema gate (#783) and by #784.
Footprints include execution reads (write skew through zero-match scans); edge non-key @unique is exclusive; the materialized and effective heads are distinct; the rollback is cap one plus horizon zero; the retry contract is stated as it is; the append-only class covers insertion-only upserts, which is how the benchmark's keyed inserts are staged; carrying exact ids is a prerequisite.
# Conflicts: # docs/rfcs/README.md
…#794 #783 is merged, so its dependency leaves blocked_on. The admission matrix's outcome cells become .gqt concurrent blocks, with commit-count assertions kept in Rust. The concurrent block cannot name a spawned task's requests, so where the publisher's requests are attributed is now an unresolved question with a recommendation. #813's captured-authority reuse is noted.
…ommit) The Lance half of the RFC's composition guard. Keyed upserts of new keys staged against P0 and P1 compose into one detached commit from P1: raw staged fragments carry id 0 and an empty row-id sequence, the key filters' union is a bitwise OR under one Bloom configuration, and the result is the union of rows with fresh, unique row and fragment ids and one stamp. The RFC records the correction the first run made (the staged fragments do carry a row-id sequence, empty for an insert) and the OR rule for the filter.
# Conflicts: # docs/rfcs/README.md
|
Evidence for the RFC's instrument gate, from a measurement-only spike. It is a local branch and will not be proposed. Spike.
Method.
Medians of 3. The repetitions agree within a few percent, and every run verified. Against the gate.
At 2 writers the gain is small and the p99 rises. The two writers mostly alternate batches of one; baseline's p99 there counts only the writer that was not starved. Caveats.
Raw records: three rounds of |
# Conflicts: # docs/rfcs/README.md
The spike that reached the instrument gate ran a leader-follower write group with leader handoff. Under the accepted server runtime RFC an HTTP write is an owned operation a disconnect never drops, which narrows the cancellation objection, and a spawned publisher would be a child producer no single operation owns. A dropped embedded leader refuses its batch before any effect and reports Uncertain after one. Publication requests run on the leader's own task, which closes the attribution question.
|
The memory caveat from the spike results is explained, and it is not caused by batching. Peak RSS follows the commit count the same way with and without the batcher; the spike just made more commits in the window. Same binaries as before (
Lance's session caches stay small throughout: 21 MB metadata at 4,648 commits. The growth comes from two engine terms that every write path pays, now filed:
For this RFC: the batcher adds no memory term of its own (its admission chain is bounded by its horizon). The measured gain still holds. The memory bound before step 4 now depends on #859 and #860, not on the write group. |
ragnorc
left a comment
There was a problem hiding this comment.
Recommendation: request changes to the RFC's append-only admission rule before implementation. I found one blocking design gap, described inline. This PR adds an RFC and a Lance guard. It does not enable group commit in the engine.
Reviewed head: f275780092ea6599c2813fc1ce742d23afa81c22.
For a human reader: today, each request pays the fixed cost of publishing a graph commit. Concurrent imports into one branch can spend much of their time waiting for storage round trips. This proposal lets waiting requests share that cost. A publisher combines compatible table changes, then publishes one graph commit for the whole batch. Each successful request receives that shared commit ID. A lone request starts immediately, without a timer to collect more requests.
The important promise is that batching must preserve a result that some serial order can produce. Combining files correctly is only one part of that promise. The admission rule must also prove that earlier writes cannot change what each request read. The inline finding shows why a transaction that only adds rows does not prove that its request only read its own keys.
Concrete tradeoffs:
- Concurrent ingestion into a few keyed types on one branch can amortize storage round trips and catalog publication. Low contention offers less benefit. The single-writer path therefore needs the proposed cost gate.
- Requests share a commit ID and lose separate intermediate history states. The one-actor restriction preserves attribution, but limits batching across actors.
- A batch shares publication failure and uncertain-outcome handling. Its leader's cancellation also affects other requests. The proposed owned HTTP operations and cancellation rules need the listed failure tests.
- Exact inserted-ID sets, queued payloads, and the recent-effect log consume memory. Caps bound that cost by refusing some batching opportunities. Their values still need workload evidence.
- Table-level exclusion keeps general predicate reads simple and safe, at the cost of avoidable conflicts. The append relaxation should apply only when the engine proves the narrower read contract.
Liability: the design reuses one graph publication door, existing receipts, collector records, and Lance transactions. It avoids a second durable commit protocol. That removes potential format and recovery liability. It also adds a real concurrency validator, queue ownership, fairness rules, and bounded historical state. Those are lasting obligations. Five similar extensions should use one recorded read contract and one admission proof. They should not add five operation-specific exceptions. The PR's net addition of 1,105 lines is mainly design text and a guard, so line count alone does not measure its runtime liability.
The source checks support the substrate approach. I checked the distributed-write and transaction documentation against pinned Lance 11.0.0, revision ab6b5bbe46009ed78746b444df8db59a8bc5d842. The detached commit path builds from the supplied dataset. It does not supply the graph's predicate-read proof. The RFC correctly requires raw fragments before ID assignment and a current admitted base. Its separate gates for change metadata, certificate composition, collector safety, and engine behavior remain necessary.
Validation:
- Local, exact head: the new
group_commit_composes_inserts_staged_against_older_pins_onto_a_later_pinguard passed. - Local, exact head: the existing mixed insert/update test passed before and after the temporary probe. The probe confirmed append-only Lance effects and the serial-order counterexample. I restored all temporary source edits. The checkout is clean.
- Local: formatting, AGENTS links, documentation checks, and RFC spelling checks passed.
- CI: the workspace job passed and includes the new guard. It tested merge commit
4300122, which contains this PR head. This differs from the exact-head local checks.
Limits: no production batcher exists here, so the probe tests existing request semantics and the proposed admission rule's premises. It does not execute a group-commit implementation. I did not repeat the cloud, throughput, fairness, or memory instruments. The author's measurements support further work on concurrent ingestion. They do not prove general workload performance or the admission contract.
I found no second blocking issue. Retain semantic read information through staging. Add the mixed insert/zero-match-update case to the admission test matrix.
| - **Append-only**, when all of these hold: | ||
| - the write on that table is a strict insert, or an upsert; | ||
| - its staged transaction on that table adds fragments and removes or | ||
| updates none, so an upsert matched no existing key; |
There was a problem hiding this comment.
[P2] Preserve predicate reads when classifying append-only entries
Classify the whole request from its reads as well as its staged writes. These conditions can admit a mixed insert/update request as append-only when its update matches no rows.
Consider a keyed Person table with neither Left nor Right. T1 inserts Left with age 0, then updates Right to age 1. T2 inserts Right with age 0, then updates Left to age 1. Both execute against the same base. Each produces only new fragments, with distinct inserted IDs. The rule at lines 370–383 therefore allows their combined result: both ages remain 0. T1 then T2 produces ages (1, 0). T2 then T1 produces (0, 1). Neither serial order produces (0, 0).
I reproduced those premises in the existing mixed insert/update test. Both independent Lance transactions were Operation::Update with new fragments and no removed or updated fragments. Both serial orders gave the different results above.
The code explains this result. execute_update returns before staging a batch when the predicate matches nothing. ensure_path records the stricter update kind. However, stage_all_with_concurrency discards op_kinds. Recording table names alone cannot distinguish this predicate read from an insert's key check.
Please require any arbitrary predicate read to make the entry exclusive, including a zero-match update or an update that changes only pending inserts. Preserve the existing operation information, or record an equivalent read classification, through staging. Add this case to the required admission tests. If “updates” at line 280 already intends this rule, state that it covers executed operations with no write effect. The current definition classifies written tables and leaves that requirement unstated.
# Conflicts: # crates/omnigraph/tests/lance_surface_guards.rs
|
Brought up to date with Review of
The PR description's Admission bullet states the executed-operation rule. The Evidence section notes the guard re-run on Lance 13.0.0. |
ragnorc
left a comment
There was a problem hiding this comment.
Recommendation: ready after one small edit to the test matrix, plus the title fix that CI requires. I found no blocking design defect in this commit.
Reviewed head: 0069e925020522bd5125e4394268ebcbc1afb2f2. The earlier review covered f275780092ea6599c2813fc1ce742d23afa81c22. Since then there is one content commit, 0069e925 (one file, docs/rfcs/2026-09-28-group-commit.md, 70 lines added and 7 removed), and one merge of main, 5356ab49. The merge changes only lance_surface_guards.rs. It keeps both sides' appended tests.
Why, what, how
Why. Group commit lets waiting writes share one graph publication. This is safe only when the combined result equals some serial order of the writes. The earlier review showed a gap. A request that inserts one key and runs a zero-match update stages only appends. So the old rule called it append-only, but its update predicate had read the other request's key as absent.
What. The class of a table now comes from the operations that execution ran, not only from the staged transaction. An executed update or delete makes the entry exclusive. This holds when the predicate matched nothing, and when the update changed only the entry's own pending inserts. Rollout step 1 now carries op_kinds through staging. The admission text states the counterexample, and the test matrix adds two cases.
Impact. No engine code changes. The rule is conservative. It refuses batching only for requests that contain an update or delete. Bulk ingestion (inserts, keyed upserts, Append and Merge loads) keeps the full benefit.
Checks against the code at this head
ensure_pathrecords the kind before any row is read.execute_updatecallsopen_table_for_mutation(..., MutationOpKind::Update, ...)beforescan_with_pending. It returns beforeappend_batchwhen no row matches (crates/omnigraph/src/exec/mutation.rs,execute_update). The RFC's claim is correct.- The general case is covered, not only this example.
MutationOpIRhas three operations:Insert,UpdateandDelete(crates/omnigraph-compiler/src/ir/mod.rs).Updatealways recordsMutationOpKind::Update, and both delete paths recordDelete. A keyed insert recordsMergeand an unkeyed insert recordsInsert. So every arbitrary predicate read inmutaterecords a strict kind. Loads have no predicate. An Overwrite load recordsSchemaRewriteand stays exclusive. - The upgrade rule in
ensure_pathchanges a kind only from non-strict to strict (strict_pre_stage_version_check). So "strictest" is a two-level order, andUpdatethenDeletestaysUpdate. That is enough, because the class needs only that split. The append-only condition is the same as!strict_pre_stage_version_check()on the recorded kind. You can name that existing predicate in rollout step 1, so that one function owns the split. stage_all_with_concurrencydestructuresop_kinds: _. A search ofcrates/finds no other reader. See the optional finding below.
Findings
Should fix (test matrix, inline at lines 682-687). The new case's expected result conflicts with the RFC's own Errors section. Both entries contain an update. The Errors section says that an update returns the conflict to the caller and does not re-prepare. So under contention one entry publishes and the other gets ReadSetChanged. The final rows are then that one entry's result alone: Left at age 0 with no Right, or the reverse. They are not (1, 0) or (0, 1). Those results occur only if the refused request is submitted again. As written, a correct batcher fails this test. A batcher that re-prepares updates without telling the caller passes it. That breaks the current retry contract. Please state the expected result as: exactly one entry publishes, the other gets ReadSetChanged with no effect, and the state is never both keys at age 0. You can also add that a resubmission then gives (1, 0) or (0, 1). The write-skew case above it already uses this form ("the second fails with the conflict").
Optional (inline at line 253). Today op_kinds has no reader. Its doc comment in exec/staging.rs says it "drives the op-kind-aware drift check in StagedMutation::commit_all", but commit_all does not read it. Rollout step 1 makes group commit its first real consumer. Please have that step rewrite the doc comment to name the new contract (the executed-operation class). Otherwise a later cleanup can delete the field as dead state and silently weaken admission. This is the same kind of fix as the step's existing note about StagedWrite's dangling doc comment.
Required by CI, not a design issue. The PR Title check fails on this head: [rfc-type] docs(rfc) is not the RFC type. The fix it prints is to replace the type and scope with rfc, for example rfc: group commit, with the Lance composition guard.
Tradeoffs and liability
- The commit adds no new state, API or invariant. It reuses
MutationStaging.op_kinds, which execution already fills. That is one source of truth for "what did this request read through a predicate". It is better than a second read-classification record that could drift from execution. - It adds one obligation:
op_kindsmust survive staging, and every future mutation operation must record a kind before its first row read. If five more operations of this kind are added (for example a conditional insert or an insert from a match), each one stays safe as long as it records a strict kind when it reads through a predicate. Thedebug_assertinopen_table_for_mutationalready enforces a related contract. A matching assertion (or a single owner function that maps an operation to its kind) would make this hold by construction. That decision can wait for the implementation. - Cost: requests that mix an insert with an update no longer batch. This is the correct trade. An update reads through a predicate, and the engine cannot prove that predicate is independent of concurrent appends without a predicate-lock design. That design would add much more liability.
Context from other open PRs
#1291 moves the OCC probe of revalidate_write_txn out of the gates and adds revalidate_write_txn_with. It does not touch op_kinds, so it does not conflict with this commit. If it lands first, publisher step 2 ("Revalidate once") should name the entry point that then exists. That is context, not a requirement on this PR.
Validation
- Code inspection at
0069e925:exec/staging.rs(ensure_path,stage_all_with_concurrency),exec/mutation.rs(open_table_for_mutation,execute_insert,execute_update, the delete paths),db/mod.rs(MutationOpKind), andomnigraph-compiler/src/ir/mod.rs(MutationOpIR). - CI at the exact head
0069e925: 27 checks pass, includingTest Workspace(which runslance_surface_guards, so it covers the guard on Lance 13.0.0),GQ Logic TestsandCI Gate.PR Titlefails, as described above. - Local:
bash scripts/check-agents-md.shpasses. I did not runcheck-docs.pyortyposlocally, because this host lacks the pinned documentation environment andtypos. CI's documentation checks pass on this head. - GQT regression rule: this commit changes only RFC text, and no batcher exists yet, so there is no defect to show before and after a fix. The earlier review's probe already showed the counterexample's premises on current request semantics. I did not repeat it, because this commit does not change engine behavior.
Limitations: I did not run the cloud, throughput, fairness or memory instruments. The design remains unimplemented, so these findings concern the proposed contract and its tests, not running code.
| - A mixed insert and zero-match update on one keyed table: T1 inserts | ||
| `Left` and updates `Right`, T2 inserts `Right` and updates `Left`, both | ||
| from one base where neither key exists. Both staged transactions only | ||
| add fragments, and both entries are exclusive by their recorded | ||
| `Update`. They never share a batch, and the rows are one of the two | ||
| serial results, `(1, 0)` or `(0, 1)`, never `(0, 0)`. |
There was a problem hiding this comment.
This expected result conflicts with the Errors section above: an update returns ReadSetChanged to the caller and does not re-prepare. Both entries here contain an update. So under contention one publishes and the other is refused with no effect. The final rows are then Left at 0 with no Right, or the reverse, and not (1, 0) or (0, 1). Those occur only after the refused request is submitted again.
Suggested wording: "They never share a batch. Exactly one publishes, the other fails with the conflict and has no effect, and the state never holds both keys at age 0. A resubmission of the refused request then gives (1, 0) or (0, 1)." Without this change, a correct batcher fails the test, and one that silently retries updates passes it.
| outranks an insert. Execution records it before any row is read, so an | ||
| update whose predicate matched nothing still records `Update` | ||
| (`execute_update` returns before staging a batch, `exec/mutation.rs`). | ||
| Today `stage_all_with_concurrency` drops `op_kinds`; the entry carries it; |
There was a problem hiding this comment.
Optional: today nothing reads op_kinds. Its doc comment in exec/staging.rs still says it drives the drift check in StagedMutation::commit_all, but commit_all does not read it. Because group commit becomes its first real consumer, please have rollout step 1 rewrite that doc comment to name the executed-operation class. Otherwise a cleanup can remove the field as dead state and silently weaken admission. The append-only test is the same as !strict_pre_stage_version_check() on the recorded kind, so you can name that predicate as the single owner of the split.
Draft RFC for step 3 of RFC 0067's throughput path: several writes on one branch publish in one
__manifestcompare-and-swap. Also includes the Lance half of the RFC's composition guard, a test inlance_surface_guards.rs.Why
PR #783 measured the same-branch ceiling with
concurrent-writeson the local filesystem and on RustFS at +30 ms per round trip:What it decides
A batch is one graph commit. RFC 0067 sketched N commit rows per publication instead. A code survey of
mainplus engine: shared schema gate and the write critical section (RFC 2026-09-18) #783 found that every mapping from a commit to its__manifestversion assumes one commit per version: snapshot reads, diffs, the change feed's head check and one-transaction proof, merge bases and collector roots throughpinned_graph_commit, parent resolution, head rows incommit::overwrite, the lost-ack read-back, andExactGraphHead. One commit per batch changes none of them, and needs no change to the storage format, the wire or the API.Same-table inserts compose by key. Every benchmark writer inserts into one table, so batching only writes to different tables would batch nothing.
insertinto the@keytypeChunkis staged as an upsert. So composition covers strict inserts and upserts whose staged transaction only appends, meaning no key matched.Detached commits happen after admission and are stamped with the batch's head. RFC 0067's sketch had writers commit before submitting. The collector's staging rule would then sweep an admitted entry whose captured head had fallen behind. Committing after admission leaves that rule unchanged.
Admission. An entry is admitted when:
The footprint is its written tables, every table its execution opened (including a predicate scan that matched nothing), and the tables its validation read. For append-only entries the rule is relaxed to disjoint inserted ids. An entry is append-only only when every operation it executed was an insert or an upsert: an executed update or delete, even one that matched nothing and staged nothing, makes it exclusive (the
MutationOpKindthatensure_pathrecords, carried through staging). The batch is conflict-serializable in admission order.Scope.
Review
An independent Codex review of the first draft found two ways the admission rule was unsound, plus five further gaps. All were verified against the code and fixed in
970f3f68:@unique;The RFC's decision log lists each one.
Evidence so far
group_commit_composes_inserts_staged_against_older_pins_onto_a_later_pinpins the substrate fact composition rests on, on Lance 11.0.0. Two keyed upserts of new keys are staged against pin P0, and a third against P1, where P1 is another insert that landed detached from P0. All three are committed as one detached transaction from P1. The test asserts:Its first run corrected a claim from the review: the staged fragments do carry a row-id sequence, but it is empty for an insert, and Lance fills it at commit. The engine half is still open: markers, the change feed's fast path, and merge's insert-absence certificate over a composed commit.
Checks
check-docs.py,typosandcheck-agents-md.shpass. The RFC 0067 edit adds a pointer in its Unresolved questions, away from the lines #783 changes.The guard passes locally, and
cargo clippy -p omnigraph-engine --test lance_surface_guards -- -D warningsandcargo fmt --all --checkare clean.mainhas been merged in: the ruleset forbids force-pushes, so the branch is updated by merge.