Skip to content

docs(rfc): group commit, with the Lance composition guard - #785

Closed
ragnorc wants to merge 14 commits into
mainfrom
rfc/group-commit
Closed

ragnorc wants to merge 14 commits into
mainfrom
rfc/group-commit

Conversation

@ragnorc

@ragnorc ragnorc commented Sep 28, 2026 •

Copy link
Copy Markdown
Contributor

Draft RFC for step 3 of RFC 0067's throughput path: several writes on one branch publish in one __manifest compare-and-swap. Also includes the Lance half of the RFC's composition guard, a test in lance_surface_guards.rs.

Why

PR #783 measured the same-branch ceiling with concurrent-writes on the local filesystem and on RustFS at +30 ms per round trip:

What it decides

  • A batch is one graph commit. RFC 0067 sketched N commit rows per publication instead. A code survey of main plus engine: shared schema gate and the write critical section (RFC 2026-09-18) #783 found that every mapping from a commit to its __manifest version assumes one commit per version: snapshot reads, diffs, the change feed's head check and one-transaction proof, merge bases and collector roots through pinned_graph_commit, parent resolution, head rows in commit::overwrite, the lost-ack read-back, and ExactGraphHead. One commit per batch changes none of them, and needs no change to the storage format, the wire or the API.

  • Same-table inserts compose by key. Every benchmark writer inserts into one table, so batching only writes to different tables would batch nothing.

    • The benchmark's insert into the @key type Chunk is staged as an upsert. So composition covers strict inserts and upserts whose staged transaction only appends, meaning no key matched.
    • It uses Lance's documented distributed-write shape: the raw, unassigned fragments are committed in one transaction, and Lance assigns fragment ids, row ids and version metadata at commit.
  • Detached commits happen after admission and are stamped with the batch's head. RFC 0067's sketch had writers commit before submitting. The collector's staging rule would then sweep an admitted entry whose captured head had fallen behind. Committing after admission leaves that rule unchanged.

  • Admission. An entry is admitted when:

    • every commit since its capture is one this publisher made, within a bounded horizon;
    • no writer since then has touched its footprint.

    The footprint is its written tables, every table its execution opened (including a predicate scan that matched nothing), and the tables its validation read. For append-only entries the rule is relaxed to disjoint inserted ids. An entry is append-only only when every operation it executed was an insert or an upsert: an executed update or delete, even one that matched nothing and staged nothing, makes it exclusive (the MutationOpKind that ensure_path records, carried through staging). The batch is conflict-serializable in admission order.

  • Scope.

    • A batch holds the writes of one actor, and a write with an expected head publishes alone.
    • Merge, schema apply, index builds and Optimize are unchanged.

Review

An independent Codex review of the first draft found two ways the admission rule was unsound, plus five further gaps. All were verified against the code and fixed in 970f3f68:

  • write skew through zero-match scans;
  • edge non-key @unique;
  • the materialized versus effective head;
  • the rollback, which is now a cap of one plus a horizon of zero;
  • the retry contract;
  • keyed inserts staged as upserts;
  • inserted ids that aren't carried on the staged write.

The RFC's decision log lists each one.

Evidence so far

group_commit_composes_inserts_staged_against_older_pins_onto_a_later_pin pins the substrate fact composition rests on, on Lance 11.0.0. Two keyed upserts of new keys are staged against pin P0, and a third against P1, where P1 is another insert that landed detached from P0. All three are committed as one detached transaction from P1. The test asserts:

  • the raw staged fragments carry fragment id 0 and an empty row-id sequence;
  • the key filters share one Bloom configuration, so their union is a bitwise OR;
  • the result is exactly the union of rows, with P1's rows unchanged;
  • new row ids and fragment ids are unique and above P1's;
  • the composed rows share one version stamp.

Its first run corrected a claim from the review: the staged fragments do carry a row-id sequence, but it is empty for an insert, and Lance fills it at commit. The engine half is still open: markers, the change feed's fast path, and merge's insert-absence certificate over a composed commit.

Checks

check-docs.py, typos and check-agents-md.sh pass. The RFC 0067 edit adds a pointer in its Unresolved questions, away from the lines #783 changes.

The guard passes locally, and cargo clippy -p omnigraph-engine --test lance_surface_guards -- -D warnings and cargo fmt --all --check are clean. main has been merged in: the ruleset forbids force-pushes, so the branch is updated by merge.

Several writes on one branch publish in one __manifest compare-and-swap as
one graph commit. A per-branch publisher admits queued entries by footprint,
composes same-table strict inserts by key into one detached commit, and
publishes through the unchanged single-commit publisher. Motivated by the
same-branch ceiling measured for the shared schema gate (#783) and by #784.
Footprints include execution reads (write skew through zero-match scans);
edge non-key @unique is exclusive; the materialized and effective heads are
distinct; the rollback is cap one plus horizon zero; the retry contract is
stated as it is; the append-only class covers insertion-only upserts, which
is how the benchmark's keyed inserts are staged; carrying exact ids is a
prerequisite.
…#794

#783 is merged, so its dependency leaves blocked_on. The admission matrix's
outcome cells become .gqt concurrent blocks, with commit-count assertions kept
in Rust. The concurrent block cannot name a spawned task's requests, so where
the publisher's requests are attributed is now an unresolved question with a
recommendation. #813's captured-authority reuse is noted.
…ommit)

The Lance half of the RFC's composition guard. Keyed upserts of new keys
staged against P0 and P1 compose into one detached commit from P1: raw
staged fragments carry id 0 and an empty row-id sequence, the key filters'
union is a bitwise OR under one Bloom configuration, and the result is the
union of rows with fresh, unique row and fragment ids and one stamp.

The RFC records the correction the first run made (the staged fragments do
carry a row-id sequence, empty for an insert) and the OR rule for the filter.
@ragnorc ragnorc changed the title docs(rfc): group commit rfc: group commit, with the Lance composition guard Sep 30, 2026
@ragnorc ragnorc changed the title rfc: group commit, with the Lance composition guard docs(rfc): group commit, with the Lance composition guard Sep 30, 2026
@ragnorc
ragnorc marked this pull request as ready for review September 30, 2026 13:30
Recheck against main after #823, #824 and #833: each batch entry's
outcome carries the evidence the same failure carries today, an HTTP
disconnect never removes an entry, and server admission bounds the
HTTP side of the queue.
…ifest

#834 removed the schema-apply sentinel; revalidation no longer reads it
and an insert makes 13 requests. The next lever now cites #837.
@ragnorc

ragnorc commented Oct 1, 2026

Copy link
Copy Markdown
Contributor Author

Evidence for the RFC's instrument gate, from a measurement-only spike. It is a local branch and will not be proposed.

Spike.

  • An eligible mutation joins its branch's batch. Eligible means: one table, a keyed write that only appends, no caller head precondition, one actor.
  • The leader revalidates under the gates. It reuses the first entry's own capture when that capture is still current, and captures fresh otherwise.
  • Admission: an entry is admitted if it was captured at the current head, or at an earlier head reached only through this batcher's own publications, with exact id sets checked against everything published since.
  • The leader composes the admitted entries' raw staged fragments into one detached commit (as group_commit_composes_inserts_staged_against_older_pins_onto_a_later_pin pins) and publishes once. Every admitted entry's receipt is that commit; refused entries re-prepare through the existing loop.
  • A writer leads only the batch holding its own entry, then promotes the next waiter (write-group handoff).
  • Not covered: cancellation of a leader, multi-table writes, updates and deletes, the effect log's footprint rules, the engine half of the composition guard.

Method.

  • concurrent-writes, one branch, --rows 64 --dims 8, 5 s warm-up and a 30 s window, release builds with RUSTFLAGS= empty.
  • Baseline main 97f293e8 against the spike on the same base. Runs alternate between the two binaries, 3 repetitions each, and every run verified exact row counts and readback.
  • RustFS sits behind toxiproxy with 15 ms each way.
  • An earlier round ran the spike binary with the flag off at 1 and 8 writers. It matched baseline, so the gain is the batching, not the build.
Backend Writers Baseline commits/s Spike commits/s Gain Per-writer max/min p99 service __manifest writes per commit
RustFS +30 ms 1 1.83 1.83 1.0× 1.0 → 1.0 579 → 577 ms 3.05 → 3.05
2 1.87 2.30 1.2× 56 → 1.0 574 → 1,008 ms 3.05 → 3.09
4 1.47 4.63 3.1× 44 → 1.0 738 → 958 ms 3.27 → 1.53
8 0.87 9.23 10.6× 26 → 1.0 1,185 → 984 ms 3.23 → 0.77
16 0.50 13.63 27× 15 → 1.3 2,068 → 1,788 ms 6.00 → 0.52
local 1 31.8 32.6 1.0× 1.0 → 1.0 47 → 45 ms 4.00 → 4.00
2 29.1 43.2 1.5× 1.8 → 1.0 1,006 → 67 ms 4.01 → 4.01
8 21.5 65.0 3.0× 231 → 1.0 6,947 → 180 ms 4.05 → 2.42

Medians of 3. The repetitions agree within a few percent, and every run verified.

Against the gate.

  • Eight writers on one branch beat one writer: 9.23 against 1.83 commits/s on RustFS, and 65.0 against 32.6 locally. Today they do not (0.87 and 21.5).
  • One writer is within noise of today, and a batch of one makes the same requests per write.
  • Per-writer commit counts are within a factor of 1.3 of each other; today they differ by up to 231×, which is performance: same-branch writers starve under object-store latency #784.
  • Retries after a moved head fall from hundreds or thousands per window to zero.

At 2 writers the gain is small and the p99 rises. The two writers mostly alternate batches of one; baseline's p99 there counts only the writer that was not starved.

Caveats.

  • Memory. Process peak RSS rises with the commit rate: 86 → 233 MB at 16 writers on RustFS, and 313 MB → 1.4 GB at 8 writers locally, where the spike made 3× the commits in the window. That has to be explained, and bounded, before step 4.
  • Iterations. The first two spike versions had a lone-writer recapture and a leader that kept leading, costing 2 writers and fairness. Both are fixed above, and both point at decisions the RFC already makes: a batch of one is today's path, and no writer publishes on others' behalf at its own expense.
  • What remains of the gate. The correctness gates (the engine half of the composition guard, the admission matrix, failpoints and the DST concurrent universe) are untouched by this evidence.

Raw records: three rounds of concurrent-writes JSONL (baseline, spike, flag-off control), available on request.

The spike that reached the instrument gate ran a leader-follower write
group with leader handoff. Under the accepted server runtime RFC an HTTP
write is an owned operation a disconnect never drops, which narrows the
cancellation objection, and a spawned publisher would be a child producer
no single operation owns. A dropped embedded leader refuses its batch
before any effect and reports Uncertain after one. Publication requests
run on the leader's own task, which closes the attribution question.
@ragnorc

ragnorc commented Oct 3, 2026

Copy link
Copy Markdown
Contributor Author

The memory caveat from the spike results is explained, and it is not caused by batching.

Peak RSS follows the commit count the same way with and without the batcher; the spike just made more commits in the window. Same binaries as before (97f293e8 and the spike on top of it), local, peak RSS taken at the end of the write window:

arm commits peak RSS
baseline, 1 writer 1,625 / 2,316 / 3,380 215 / 386 / 918 MB
spike, 1 writer 1,967 427 MB
spike, 8 writers 2,201 / 3,449 / 4,648 782 / 1,235 / 2,018 MB

Lance's session caches stay small throughout: 21 MB metadata at 4,648 commits. The growth comes from two engine terms that every write path pays, now filed:

For this RFC: the batcher adds no memory term of its own (its admission chain is bounded by its horizon). The measured gain still holds. The memory bound before step 4 now depends on #859 and #860, not on the write group.

@ragnorc ragnorc left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Recommendation: request changes to the RFC's append-only admission rule before implementation. I found one blocking design gap, described inline. This PR adds an RFC and a Lance guard. It does not enable group commit in the engine.

Reviewed head: f275780092ea6599c2813fc1ce742d23afa81c22.

For a human reader: today, each request pays the fixed cost of publishing a graph commit. Concurrent imports into one branch can spend much of their time waiting for storage round trips. This proposal lets waiting requests share that cost. A publisher combines compatible table changes, then publishes one graph commit for the whole batch. Each successful request receives that shared commit ID. A lone request starts immediately, without a timer to collect more requests.

The important promise is that batching must preserve a result that some serial order can produce. Combining files correctly is only one part of that promise. The admission rule must also prove that earlier writes cannot change what each request read. The inline finding shows why a transaction that only adds rows does not prove that its request only read its own keys.

Concrete tradeoffs:

  • Concurrent ingestion into a few keyed types on one branch can amortize storage round trips and catalog publication. Low contention offers less benefit. The single-writer path therefore needs the proposed cost gate.
  • Requests share a commit ID and lose separate intermediate history states. The one-actor restriction preserves attribution, but limits batching across actors.
  • A batch shares publication failure and uncertain-outcome handling. Its leader's cancellation also affects other requests. The proposed owned HTTP operations and cancellation rules need the listed failure tests.
  • Exact inserted-ID sets, queued payloads, and the recent-effect log consume memory. Caps bound that cost by refusing some batching opportunities. Their values still need workload evidence.
  • Table-level exclusion keeps general predicate reads simple and safe, at the cost of avoidable conflicts. The append relaxation should apply only when the engine proves the narrower read contract.

Liability: the design reuses one graph publication door, existing receipts, collector records, and Lance transactions. It avoids a second durable commit protocol. That removes potential format and recovery liability. It also adds a real concurrency validator, queue ownership, fairness rules, and bounded historical state. Those are lasting obligations. Five similar extensions should use one recorded read contract and one admission proof. They should not add five operation-specific exceptions. The PR's net addition of 1,105 lines is mainly design text and a guard, so line count alone does not measure its runtime liability.

The source checks support the substrate approach. I checked the distributed-write and transaction documentation against pinned Lance 11.0.0, revision ab6b5bbe46009ed78746b444df8db59a8bc5d842. The detached commit path builds from the supplied dataset. It does not supply the graph's predicate-read proof. The RFC correctly requires raw fragments before ID assignment and a current admitted base. Its separate gates for change metadata, certificate composition, collector safety, and engine behavior remain necessary.

Validation:

  • Local, exact head: the new group_commit_composes_inserts_staged_against_older_pins_onto_a_later_pin guard passed.
  • Local, exact head: the existing mixed insert/update test passed before and after the temporary probe. The probe confirmed append-only Lance effects and the serial-order counterexample. I restored all temporary source edits. The checkout is clean.
  • Local: formatting, AGENTS links, documentation checks, and RFC spelling checks passed.
  • CI: the workspace job passed and includes the new guard. It tested merge commit 4300122, which contains this PR head. This differs from the exact-head local checks.

Limits: no production batcher exists here, so the probe tests existing request semantics and the proposed admission rule's premises. It does not execute a group-commit implementation. I did not repeat the cloud, throughput, fairness, or memory instruments. The author's measurements support further work on concurrent ingestion. They do not prove general workload performance or the admission contract.

I found no second blocking issue. Retain semantic read information through staging. Add the mixed insert/zero-match-update case to the admission test matrix.

- **Append-only**, when all of these hold:
- the write on that table is a strict insert, or an upsert;
- its staged transaction on that table adds fragments and removes or
updates none, so an upsert matched no existing key;

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Preserve predicate reads when classifying append-only entries

Classify the whole request from its reads as well as its staged writes. These conditions can admit a mixed insert/update request as append-only when its update matches no rows.

Consider a keyed Person table with neither Left nor Right. T1 inserts Left with age 0, then updates Right to age 1. T2 inserts Right with age 0, then updates Left to age 1. Both execute against the same base. Each produces only new fragments, with distinct inserted IDs. The rule at lines 370–383 therefore allows their combined result: both ages remain 0. T1 then T2 produces ages (1, 0). T2 then T1 produces (0, 1). Neither serial order produces (0, 0).

I reproduced those premises in the existing mixed insert/update test. Both independent Lance transactions were Operation::Update with new fragments and no removed or updated fragments. Both serial orders gave the different results above.

The code explains this result. execute_update returns before staging a batch when the predicate matches nothing. ensure_path records the stricter update kind. However, stage_all_with_concurrency discards op_kinds. Recording table names alone cannot distinguish this predicate read from an insert's key check.

Please require any arbitrary predicate read to make the entry exclusive, including a zero-match update or an update that changes only pending inserts. Preserve the existing operation information, or record an equivalent read classification, through staging. Add this case to the required admission tests. If “updates” at line 280 already intends this rule, state that it covers executed operations with no write effect. The current definition classifies written tables and leaves that requirement unstated.

@ragnorc

ragnorc commented Nov 8, 2026

Copy link
Copy Markdown
Contributor Author

Brought up to date with main (merge 5356ab49). The conflict was in lance_surface_guards.rs, where both sides appended tests at the end of the file. I kept main's row_identity_discovery_stamps_row_ids_and_rewrites_lance_13, followed by this PR's group-commit guard, unchanged. On Lance 13.0.0 the guard passes, as does the whole lance_surface_guards binary (73 passed, 2 ignored). cargo dev-clippy and cargo fmt --all --check are clean.

Review of f2757800:

  • [P2] Preserve predicate reads when classifying append-only entries. Fixed in 0069e925.
    • Each entry now carries the strictest MutationOpKind per opened table. ensure_path records it before any row is read, so a zero-match update still records Update. Today stage_all_with_concurrency drops op_kinds; rollout step 1 now carries it through staging.
    • A table is append-only only when every operation execution ran on it was an insert or an upsert. An executed update or delete makes the entry exclusive, whether or not that table was written. That covers one whose predicate matched nothing and one that changed only the entry's own pending inserts.
    • Admission states your Left/Right counterexample and why the executed-operation class excludes it.
    • The admission matrix gains that case, which must give (1, 0) or (0, 1) and never (0, 0). It also gains insert-then-own-update as exclusive.
    • The decision log records the change and the superseded first exclusive item.

The PR description's Admission bullet states the executed-operation rule. The Evidence section notes the guard re-run on Lance 13.0.0.

@ragnorc ragnorc left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Recommendation: ready after one small edit to the test matrix, plus the title fix that CI requires. I found no blocking design defect in this commit.

Reviewed head: 0069e925020522bd5125e4394268ebcbc1afb2f2. The earlier review covered f275780092ea6599c2813fc1ce742d23afa81c22. Since then there is one content commit, 0069e925 (one file, docs/rfcs/2026-09-28-group-commit.md, 70 lines added and 7 removed), and one merge of main, 5356ab49. The merge changes only lance_surface_guards.rs. It keeps both sides' appended tests.

Why, what, how

Why. Group commit lets waiting writes share one graph publication. This is safe only when the combined result equals some serial order of the writes. The earlier review showed a gap. A request that inserts one key and runs a zero-match update stages only appends. So the old rule called it append-only, but its update predicate had read the other request's key as absent.

What. The class of a table now comes from the operations that execution ran, not only from the staged transaction. An executed update or delete makes the entry exclusive. This holds when the predicate matched nothing, and when the update changed only the entry's own pending inserts. Rollout step 1 now carries op_kinds through staging. The admission text states the counterexample, and the test matrix adds two cases.

Impact. No engine code changes. The rule is conservative. It refuses batching only for requests that contain an update or delete. Bulk ingestion (inserts, keyed upserts, Append and Merge loads) keeps the full benefit.

Checks against the code at this head

  • ensure_path records the kind before any row is read. execute_update calls open_table_for_mutation(..., MutationOpKind::Update, ...) before scan_with_pending. It returns before append_batch when no row matches (crates/omnigraph/src/exec/mutation.rs, execute_update). The RFC's claim is correct.
  • The general case is covered, not only this example. MutationOpIR has three operations: Insert, Update and Delete (crates/omnigraph-compiler/src/ir/mod.rs). Update always records MutationOpKind::Update, and both delete paths record Delete. A keyed insert records Merge and an unkeyed insert records Insert. So every arbitrary predicate read in mutate records a strict kind. Loads have no predicate. An Overwrite load records SchemaRewrite and stays exclusive.
  • The upgrade rule in ensure_path changes a kind only from non-strict to strict (strict_pre_stage_version_check). So "strictest" is a two-level order, and Update then Delete stays Update. That is enough, because the class needs only that split. The append-only condition is the same as !strict_pre_stage_version_check() on the recorded kind. You can name that existing predicate in rollout step 1, so that one function owns the split.
  • stage_all_with_concurrency destructures op_kinds: _. A search of crates/ finds no other reader. See the optional finding below.

Findings

Should fix (test matrix, inline at lines 682-687). The new case's expected result conflicts with the RFC's own Errors section. Both entries contain an update. The Errors section says that an update returns the conflict to the caller and does not re-prepare. So under contention one entry publishes and the other gets ReadSetChanged. The final rows are then that one entry's result alone: Left at age 0 with no Right, or the reverse. They are not (1, 0) or (0, 1). Those results occur only if the refused request is submitted again. As written, a correct batcher fails this test. A batcher that re-prepares updates without telling the caller passes it. That breaks the current retry contract. Please state the expected result as: exactly one entry publishes, the other gets ReadSetChanged with no effect, and the state is never both keys at age 0. You can also add that a resubmission then gives (1, 0) or (0, 1). The write-skew case above it already uses this form ("the second fails with the conflict").

Optional (inline at line 253). Today op_kinds has no reader. Its doc comment in exec/staging.rs says it "drives the op-kind-aware drift check in StagedMutation::commit_all", but commit_all does not read it. Rollout step 1 makes group commit its first real consumer. Please have that step rewrite the doc comment to name the new contract (the executed-operation class). Otherwise a later cleanup can delete the field as dead state and silently weaken admission. This is the same kind of fix as the step's existing note about StagedWrite's dangling doc comment.

Required by CI, not a design issue. The PR Title check fails on this head: [rfc-type] docs(rfc) is not the RFC type. The fix it prints is to replace the type and scope with rfc, for example rfc: group commit, with the Lance composition guard.

Tradeoffs and liability

  • The commit adds no new state, API or invariant. It reuses MutationStaging.op_kinds, which execution already fills. That is one source of truth for "what did this request read through a predicate". It is better than a second read-classification record that could drift from execution.
  • It adds one obligation: op_kinds must survive staging, and every future mutation operation must record a kind before its first row read. If five more operations of this kind are added (for example a conditional insert or an insert from a match), each one stays safe as long as it records a strict kind when it reads through a predicate. The debug_assert in open_table_for_mutation already enforces a related contract. A matching assertion (or a single owner function that maps an operation to its kind) would make this hold by construction. That decision can wait for the implementation.
  • Cost: requests that mix an insert with an update no longer batch. This is the correct trade. An update reads through a predicate, and the engine cannot prove that predicate is independent of concurrent appends without a predicate-lock design. That design would add much more liability.

Context from other open PRs

#1291 moves the OCC probe of revalidate_write_txn out of the gates and adds revalidate_write_txn_with. It does not touch op_kinds, so it does not conflict with this commit. If it lands first, publisher step 2 ("Revalidate once") should name the entry point that then exists. That is context, not a requirement on this PR.

Validation

  • Code inspection at 0069e925: exec/staging.rs (ensure_path, stage_all_with_concurrency), exec/mutation.rs (open_table_for_mutation, execute_insert, execute_update, the delete paths), db/mod.rs (MutationOpKind), and omnigraph-compiler/src/ir/mod.rs (MutationOpIR).
  • CI at the exact head 0069e925: 27 checks pass, including Test Workspace (which runs lance_surface_guards, so it covers the guard on Lance 13.0.0), GQ Logic Tests and CI Gate. PR Title fails, as described above.
  • Local: bash scripts/check-agents-md.sh passes. I did not run check-docs.py or typos locally, because this host lacks the pinned documentation environment and typos. CI's documentation checks pass on this head.
  • GQT regression rule: this commit changes only RFC text, and no batcher exists yet, so there is no defect to show before and after a fix. The earlier review's probe already showed the counterexample's premises on current request semantics. I did not repeat it, because this commit does not change engine behavior.

Limitations: I did not run the cloud, throughput, fairness or memory instruments. The design remains unimplemented, so these findings concern the proposed contract and its tests, not running code.

Comment on lines +682 to +687
- A mixed insert and zero-match update on one keyed table: T1 inserts
`Left` and updates `Right`, T2 inserts `Right` and updates `Left`, both
from one base where neither key exists. Both staged transactions only
add fragments, and both entries are exclusive by their recorded
`Update`. They never share a batch, and the rows are one of the two
serial results, `(1, 0)` or `(0, 1)`, never `(0, 0)`.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This expected result conflicts with the Errors section above: an update returns ReadSetChanged to the caller and does not re-prepare. Both entries here contain an update. So under contention one publishes and the other is refused with no effect. The final rows are then Left at 0 with no Right, or the reverse, and not (1, 0) or (0, 1). Those occur only after the refused request is submitted again.

Suggested wording: "They never share a batch. Exactly one publishes, the other fails with the conflict and has no effect, and the state never holds both keys at age 0. A resubmission of the refused request then gives (1, 0) or (0, 1)." Without this change, a correct batcher fails the test, and one that silently retries updates passes it.

outranks an insert. Execution records it before any row is read, so an
update whose predicate matched nothing still records `Update`
(`execute_update` returns before staging a batch, `exec/mutation.rs`).
Today `stage_all_with_concurrency` drops `op_kinds`; the entry carries it;

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Optional: today nothing reads op_kinds. Its doc comment in exec/staging.rs still says it drives the drift check in StagedMutation::commit_all, but commit_all does not read it. Because group commit becomes its first real consumer, please have rollout step 1 rewrite that doc comment to name the executed-operation class. Otherwise a cleanup can remove the field as dead state and silently weaken admission. The append-only test is the same as !strict_pre_stage_version_check() on the recorded kind, so you can name that predicate as the single owner of the split.

@ragnorc

ragnorc commented Nov 14, 2026

Copy link
Copy Markdown
Contributor Author

Superseded by #1405 (RFC: group commit with known-key admission), which consolidates this work with the other research drafts. The per-branch writer-state fix lands separately in #1407.

@ragnorc ragnorc closed this Nov 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant