Skip to content

perf(engine): write-allocate the dataset cache and hoist OCC validation out of the write gates - #1291

Merged
azimafroozeh merged 4 commits into
ModernRelay:mainfrom
azimafroozeh:0423.1-write-read-set
Nov 9, 2026
Merged

azimafroozeh merged 4 commits into
ModernRelay:mainfrom
azimafroozeh:0423.1-write-read-set

Conversation

@azimafroozeh

Copy link
Copy Markdown
Collaborator

The version-keyed dataset cache is read-allocate only, so a writer drops the Lance dataset for the version it committed and the next open re-reads the manifest (a HEAD and a GET, twice past one block); the OCC validation probe of the branch head runs inside the latch-held critical section, after the gate-free staging puts.

Write-allocate: commit_all inserts the committed dataset under the key the publish registers (path, branch, version, detached version, e_tag), an immutable pin dropped by branch control or refresh. Validation hoisted: the probe is issued concurrently with the staging puts and reused under the gates while the installed coordinator's manifest incarnation still matches it; the __manifest compare-and-swap still refuses a cross-process publish.

write_invalidates_table_cache_for_changed_table 4 reads → 0, measure_counts_schema_contract_requests_issue_817 1 HEAD → 0; crates/omnigraph-dst/cost_table.txt: nine scenarios drop, Optimize +2 GETs (the held dataset's first index-section read).

For review: the probe is trusted on the installed coordinator's manifest incarnation (version, branch id, e_tag or timestamp); wrong if a store returns neither e_tag nor timestamp and two images share a version. A writer that loses the head between probe and gates re-probes under them: two requests more. load keeps its probe under the gates. Staging's .dataset() chokepoint count 1 → 2 (a clone, no write). docs/dev/writes.md and the release note carry the contract; no flag, endpoint or format change.

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 6, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-06T21:19:31.328741Z c4a8ede PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@ragnorc ragnorc left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review of head c4a8ede7.

Summary

Why. A write dropped the Lance dataset of the version it had just committed, so the next open of that pin read the manifest again. The write also probed the branch head of __manifest under its gates, inside the latch-held critical section.

What. Two changes:

  1. Write-allocate. commit_all puts each committed detached dataset into the table handle cache, under the key that the published registration will carry.
  2. Hoisted probe. A mutation sends its branch-head probe concurrently with its staging puts. Under the gates, it reuses that answer when this handle's installed coordinator still has the incarnation that the probe saw.

Impact. Fewer requests per write and per read-after-write: write_invalidates_table_cache_for_changed_table goes from 4 reads to 0, issue 817's repeated HEAD from 1 to 0, and nine DST scenarios drop. Optimize gains 2 GETs.

Verdict. The write-allocate part is sound. The key is version-exact and the pin is immutable, so a lost publish only leaves an entry that no open can reach. Two items block merge: the branch does not merge with main, and the hoisted probe changes the --if-commit contract when another handle or process publishes during staging.

Blocking

B1. Conflicts with main, and the clean part of the merge does not type-check

  • GitHub reports CONFLICTING. git merge-tree against main de02c281 shows text conflicts in crates/omnigraph-dst/cost_table.txt (_verify l.get: #1276 made it 1399, this PR makes it 1385) and docs/dev/writes.md.
  • crates/omnigraph-core/src/handle_cache.rs merges without a text conflict, but the result does not type-check. The new TableHandleCache::insert calls inner.insert(key, dataset) with a Dataset. Since #1276, TableHandleCacheInner::insert takes HeldHandle { dataset, manifest_bytes }, so the byte budget never weighs this entry. (I found this by reading the merged file. I did not compile the merge.)
  • Because of the conflict, CI on this head ran only PR Title, Regression and Review. No build, workspace test, GQT, DST or clippy job ran on this code.

After the rebase, regenerate the cost golden on top of #1276 and explain each changed line again.

Suggested shape after the rebase (it removes liability). #1276 already has one place where a publish changes the branch's working set: evict_superseded_handles in table_ops.rs, after the publish. Admit the new pin in the same place, with one cache method (for example replace(old_key, new_key, dataset)) that builds keys with TableHandleKey::new and measures manifest_bytes the way a miss does. The committed datasets already reach the publish step in CommittedMutation::detached, which is discarded today (detached: _ in exec/mutation.rs:1412 and loader/mod.rs:1504). This gives three things:

  • The key comes from the DatasetUpdate that is actually published, not from a second derivation in staging.rs, so the two cannot drift. A drift would show up only as a silent miss.
  • A lost publish leaves no entry that uses count and byte budget.
  • "A publish moves the branch from its base pin to its new pin" then has one owner in the cache.

The next operation gets the same benefit.

B2. A conditional update/delete returns ReadSetChanged (409) instead of PreconditionFailed (412) when another handle or process publishes during staging

Contract. skills/omnigraph/references/remote-ops.md says: "Any intervening branch commit makes the condition fail without effects. The CLI exits 4; HTTP returns 412." The existing test conditional_update_and_delete_races_return_precondition_failed_before_effects states that the under-gate check must map update and delete to PreconditionFailed and not leak ReadSetChanged.

Cause. revalidate_write_txn_with (db/omnigraph.rs:2132-2139) trusts the early probe when installed.manifest_incarnation() still matches early.held. That incarnation belongs to this handle. The write gates are shared by every handle on the root in the process (see cross_handle_branch_gate_serializes_post_effect_publish), but the installed coordinator is not shared. So a publish between the probe and the gates, by a second handle on the same root or by another process, does not change this handle's incarnation.

What happens next:

  1. Revalidation keeps the stale head.
  2. The caller-head check at db/omnigraph.rs:2168 passes.
  3. The detached commits run.
  4. The publisher's ExactGraphHead check refuses with ReadSetChanged.

Update and delete are not retryable, so this internal classification reaches the caller. The description says "a same-process publish installs a new incarnation". That is true only for the same handle.

Evidence. I added a temporary copy of the existing test with one change: the winner writes through a second Omnigraph::open on the same root. The writer runs a conditional set_age and is parked at mutation.post_stage_pre_effect_gate. That is after staging and the early probe, and before the gates. The winner inserts a Company, then the writer is released. Command:

cargo test -p omnigraph-engine --locked --features failpoints --test failpoints review_probe -- --nocapture

  • Base d794b725 (the PR's six source files reverted in the same worktree): PreconditionFailed { branch: "main", expected: <captured head>, actual: Some(<winner head>) }, which is the documented outcome.
  • Head c4a8ede7: Manifest(ManifestError { kind: Conflict, … details: Some(ReadSetChanged { member: "graph_head:main", … }) }).

In the same race, an unconditional write now also fails after its detached commits instead of before them. That is still effect-free for the graph, but it leaves orphaned detached versions for cleanup.

Fix at the owner. The caller's precondition belongs to publication, not to where the probe runs. When caller_expected_graph_head is set, map a graph-head ReadSetChanged from the publisher to PreconditionFailed, with the actual head, as revalidation already does. This holds for every race path. It also closes the shorter cross-process window between the under-gate probe and the CAS that exists on main today. I found that window by reading the code only; I could not reproduce it in-process, because same-process handles share the gates.

Skipping the early probe only for conditional writes would be a symptomatic patch: it leaves that window open.

Regression test. Extend conditional_update_and_delete_races_return_precondition_failed_before_effects with a second-handle winner, for both set_age and remove_person. It must fail at c4a8ede7 and pass after the fix. GQT cannot express this case, because sessions on separate handles are not supported yet (crates/omnigraph-gqt/README.md), so the failpoints test is the cheapest owner.

Optional

  • O1. One owner for the freshness decision. revalidate_write_txn_with adds a third path beside write_authority_for_known_branch(branch, fresh), which already computes held, probe and matches. Pass the early answer into that function as one three-state input (probe now / probed earlier / trust installed). Then one function decides when a head counts as current. With the current shape, five more callers give five copies of the rule.
  • O2. Make the incarnation root-scoped if in-process multi-handle contention matters. For example, keep a publication counter with the root-scoped branch gate. A same-process publish by any handle then invalidates the early answer before effects, instead of after detached commits.
  • O3. State the tradeoff in docs/dev/writes.md. Under contention from another handle or process, a lost race now costs one detached commit per table and fails after effects, not before them. With one server writer per root this case is probably rare. One sentence makes the tradeoff visible.
  • O4. loader/mod.rs now gives plan_load the cache-backed snapshot, not only plan_write. That is fine, but the description does not mention it.
  • O5. Overlap with open PRs. #1275 edits the same cost_table.txt lines and writes.md, and it changes optimize. Re-measure the Optimize +2 GETs line after whichever lands second.

Validation

All runs used an isolated worktree at c4a8ede7, a debug build and a shared target directory.

  • cargo test -p omnigraph-gqt --locked --test gq_logic_tests -- writer_parked_in_its_early_probe: 2 passed. -- two_table_mutation_fails_before_its_gates: 1 passed (DST seeds 0 and 42, replay matched).
  • cargo test -p omnigraph-gqt --locked --test runner_dispatch -- measure_counts_schema_contract_requests_issue_817: 1 passed.
  • cargo test -p omnigraph-engine --locked --features failpoints --test forbidden_apis: 39 passed.
  • … --test integration warm_read_cost::write_invalidates_table_cache_for_changed_table: passed.
  • … --test failpoints conditional_update_and_delete_races_return_precondition_failed_before_effects (same-handle winner): passed.
  • The B2 probe as described above: passes at base, fails at head. I removed it afterwards.
  • git merge-tree --write-tree with main de02c281: two conflicts, plus the type mismatch from B1 (by inspection).
  • CI on this head: only 3 checks ran, because of the conflict.

Limits. I did not build or test the merged tree. I did not run the full workspace gate, the DST suite (the cost golden values are the PR's own), clippy, or S3/Azure. I did not measure latency, so the critical-section gain is not verified here. I did not test cross-process races directly.

@ragnorc ragnorc left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review of head 2487c5e0. This follows my review of c4a8ede7.

Summary

Why and impact (unchanged). The PR keeps each Lance dataset a write commits in the table handle cache, and sends the branch-head probe beside the staging puts instead of under the write gates. Fewer object-store requests per write and per read-after-write: nine DST scenarios drop, and Optimize gains 2 GETs.

What changed since c4a8ede7. The commit was rebased onto 0851d796, which includes #1276. It is now c502dd59. Then main de02c281 was merged in as 2487c5e0. The merge needed no manual resolution (git show --remerge-diff is empty). The range-diff touches three places:

  • TableHandleCache::insert now builds its key with TableHandleKey::new and stores a HeldHandle with measured manifest_bytes. The byte budget now counts the entry.
  • cost_table.txt _verify l.get is re-measured on top of #1276: 1399 → 1389.
  • docs/dev/writes.md is rebased onto #1276's eviction paragraph.

revalidate_write_txn_with, early_revalidation and the mutation path are the same as in c4a8ede7.

Verdict. Old B1 is fixed. Old B2 is still there: I reproduced it on this head. A new docs check failure also blocks merge, because of when the PR adds its release note.

Blocking

B1. Old B2 is still there: a conditional update or delete returns ReadSetChanged (409), not PreconditionFailed (412), when another handle publishes during staging

The code did not change, so the cause is the one I described before. revalidate_write_txn_with (crates/omnigraph/src/db/omnigraph.rs) trusts the early probe when installed.manifest_incarnation() still matches early.held. That incarnation belongs to this handle. A publish by a second handle on the same root, or by another process, does not change it. Revalidation then keeps the stale head. The caller-head check passes, the detached commits run, and the publisher's CAS refuses with ReadSetChanged. Update and delete are not retryable, so the 409 reaches the caller. The documented contract is 412 (skills/omnigraph/references/remote-ops.md, and the test conditional_update_and_delete_races_return_precondition_failed_before_effects).

The description still says "the __manifest compare-and-swap still refuses a cross-process publish". That is true, but for a conditional write the refusal has the wrong type.

Evidence on this head. I rebuilt the earlier temporary probe as a copy of the existing test with one change: the winner writes through a second Omnigraph::open on the same root. The writer runs a conditional set_age, then a conditional remove_person, and is parked at mutation.post_stage_pre_effect_gate. That is after staging and the early probe, and before the gates. The winner inserts a Company, then the writer is released.

cargo test -p omnigraph-engine --locked --features failpoints --test failpoints review_probe -- --nocapture

  • Base: the PR's six source files set to main de02c281 in the same worktree. Both cases return PreconditionFailed { branch: "main", expected: <captured head>, actual: Some(<winner head>) }. 1 passed.
  • Head 2487c5e0: both cases return Manifest(ManifestError { kind: Conflict, … details: Some(ReadSetChanged { member: "graph_head:main", … }) }). 1 failed.

Fix at the owner (unchanged). The caller's precondition belongs to publication. When caller_expected_graph_head is set, map a graph-head ReadSetChanged from the publisher to PreconditionFailed, with the actual head. This covers every race path. It also closes the shorter cross-process window between the under-gate probe and the CAS that exists on main today. Do not skip the early probe only for conditional writes: that is a symptomatic patch, and it leaves that window open.

Regression test. Extend conditional_update_and_delete_races_return_precondition_failed_before_effects with a winner on a second Omnigraph::open, for both set_age and remove_person. The probe above is that test. It must fail at this head and pass after the fix. GQT cannot express separate handles, so the failpoints test is the cheapest owner.

B2. Check AGENTS.md Links fails because the new release note arrived after the v0.15.0 snapshot

CI on 2487c5e0: python3 scripts/check-docs.py fails with release notes: release notes changed after snapshot generation; regenerate it, and CI Gate fails with it. All other checks pass.

Cause: #1283 committed the v0.15.0 snapshot (docs/releases/v0.15.0.md), which records the digest of each note in the cycle (changelog.d/release.json still names v0.15.0). This PR adds changelog.d/write-keeps-committed-handle.performance.md. I compared the recorded and expected note sets at this head: the only difference is that file (recorded 64, expected 65). check-docs.py passes on main de02c281.

This is not a code defect. It is a choice for the release owner, and the PR cannot merge before one of these happens:

  • v0.15.0 is tagged and the next cycle opens (release.json bumped, docs/dev/documentation.md step 8). The note then belongs to the next release. No change to this PR is needed beyond a rerun.
  • The note ships in v0.15.0. Then regenerate the snapshot with python3 scripts/release_notes.py snapshot --target HEAD --date <date> --replace and commit it with this PR.

Every other open PR that adds a note will hit the same check until one of these happens.

Optional

  • O1. Admission happens before publication, and eviction after it. commit_all inserts the new pin before the publisher runs. #1276's evict_superseded_handles removes the base pin only after publication. The two compose correctly: the eviction key is the exact base key, so it never removes the new pin. But a lost publish now leaves an entry that uses count and byte budget until LRU removes it. It is harmless for correctness. The key is built from the same values as the adjacent DatasetUpdate, so the drift risk I raised before is now small. My earlier suggestion still applies if you want one owner: admit the pin next to evict_superseded_handles, from the published DatasetUpdate and CommittedMutation::detached. If you keep the current shape, say in docs/dev/writes.md that the writer admits the committed pin at commit time. The section describes only the eviction half today, and the description says writes.md carries the contract.
  • O2. Old O1 to O4 (one owner for the freshness decision, root-scoped incarnation, the tradeoff sentence in writes.md, the plan_load snapshot change in the description) are not addressed. They still apply. The fix for B1 removes most of the reason for the O3 sentence, but a lost race still costs one detached commit per table.
  • O3. Overlap with open PRs. git merge-tree with #1275 (0c9a0a18), #1282 (565978da) and #1224 (ce5c89ed) is clean. #1275 still changes Optimize costs and writes.md, so re-measure the cost golden after whichever lands second.

Validation

  • Isolated worktree at 2487c5e0. Debug build with a shared target directory.
  • Range-diff c4a8ede7 → c502dd59, and git show --remerge-diff 2487c5e0 (empty).
  • B1 probe (cargo test -p omnigraph-engine --locked --features failpoints --test failpoints review_probe -- --nocapture): passes at base de02c281 (PR source files reverted), fails at 2487c5e0 for both set_age and remove_person.
  • python3 scripts/check-docs.py with the pinned docs packages: fails at 2487c5e0 with the release-note error, passes at de02c281. A digest comparison of docs/releases/v0.15.0.md against HEAD shows only the new note as different.
  • CI on this head: 26 pass, 4 skipped, 2 fail (Check AGENTS.md Links, CI Gate). Test Workspace, Lint (clippy), the GQT jobs, DST pinned suite, S3/Azure and storage-upgrade jobs pass.
  • git merge-tree --write-tree with #1275, #1282 and #1224 heads: no conflicts.

Limits. I did not run the full workspace gate, DST, GQT, clippy or S3/Azure locally; CI covers them on this head. I did not measure latency. I did not test cross-process races directly; the second-handle probe covers the same cause. I removed the temporary probe afterwards.

@ragnorc ragnorc left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review of head 51f9c4a9. This follows my reviews of c4a8ede7 and 2487c5e0. It covers only what changed since 2487c5e0.

Summary

What changed. The branch was rebased onto main 2d8effab. The perf commit is now 9ae5b473, and git range-diff shows it is the same patch as c502dd59. The main delta since the old merge (de02c281..2d8effab, #1293) changes only release CI, install docs and release notes. It touches none of this PR's files. git diff 2487c5e0 51f9c4a9 -- crates/ Cargo.toml Cargo.lock is empty, so the engine code is the same as the code I reviewed. A new commit, 51f9c4a9, deletes changelog.d/write-keeps-committed-handle.performance.md.

Verdict. Old B2 (the docs check) now passes. Old B1 is still there, because the code did not change. Withdrawing the note causes a new PR Title failure and leaves a release-note gap that needs a decision before merge.

Blocking

B1. Old B1 is still there: a conditional update or delete returns ReadSetChanged (409), not PreconditionFailed (412), when another handle publishes during staging

revalidate_write_txn_with and the early probe are the same as at 2487c5e0, so my reproduction and the cause are unchanged. Please see B1 in the review of 2487c5e0. The fix is also unchanged: when caller_expected_graph_head is set, map a graph-head ReadSetChanged from the publisher to PreconditionFailed with the actual head. Add the regression test as a second-Omnigraph::open winner in conditional_update_and_delete_races_return_precondition_failed_before_effects, for both set_age and remove_person.

I did not run the probe again. The crates/ tree is the same as the tree on which it failed, so the result cannot be different.

Release note: decide before merge

51f9c4a9 fixes the old docs failure. But PR Title now fails at this head: "a perf pull request adds a changelog.d note, or carries the skip-changelog label" (run). PR Title is not a required check, so it does not stop the merge. That is the risk:

  • The skip-changelog label is not correct here. docs/dev/documentation.md keeps it for typed changes with "nothing users can see". This change removes requests on every write and read-after-write, and it adds 2 GETs to Optimize. Users can see both.
  • v0.15.0 is not tagged yet, and release.json still names v0.15.0. If the v0.15.0 tag lands on a commit that includes this PR, the change ships in v0.15.0 with no note. A note that a later PR adds would then appear in the wrong release.

Two clean paths:

  1. Merge after v0.15.0 is tagged and the next cycle opens. Revert 51f9c4a9 at that time, so the note goes into the next release. This is what the commit title says.
  2. Ship in v0.15.0. Restore the note and regenerate the snapshot with python3 scripts/release_notes.py snapshot --target HEAD --date <date> --replace in this PR.

Do not merge with the note withdrawn and the label added.

Optional

  • O1. Old optional items. O1 to O3 of the 2487c5e0 review are not addressed. They still apply.
  • O2. New overlap with #1272 (d3336e7b, patch an update's assigned columns). It changes exec/staging.rs, exec/mutation.rs, loader/mod.rs, forbidden_apis.rs, docs/dev/writes.md and cost_table.txt. git merge-tree with this head is clean. Its new PendingMode::ColumnPatch is staged through prepare_pending_table and committed by the same StagedMutation::commit_all, so this PR's handle admission and early probe cover it with no extra work. The DST cost golden needs a new measurement after the second of #1272, #1275 and #1299 lands. All three change cost_table.txt or write costs.

Validation

  • git range-diff de02c281..2487c5e0 2d8effab..51f9c4a9: the perf commit is unchanged (=). The only new commit is the note deletion.
  • git diff 2487c5e0 51f9c4a9 -- crates/ Cargo.toml Cargo.lock: empty. The full tree diff is #1293 plus the note deletion.
  • git merge-tree --write-tree with the current heads of every open PR that shares a file: clean with #1299, #1282, #1275, #1272 and #1235. #1217, #1184, #1175 and #818 conflict, but each of them already conflicts with main alone, so this PR does not cause it.
  • CI on 51f9c4a9: Check AGENTS.md Links and CI Gate now pass. Test Workspace, Lint (clippy), the GQT jobs, the DST pinned suite, S3/Azure and storage-upgrade jobs pass. PR Title fails as described above. Release tag jobs are skipped.

Limits. I did not build or run tests locally for this head, because its engine code is the same as the code I tested at 2487c5e0. I did not measure latency. I did not run #1272 and this PR together; the overlap note above comes from code reading and merge-tree.

@azimafroozeh
azimafroozeh force-pushed the 0423.1-write-read-set branch from 51f9c4a to 8f6f6c4 Compare November 9, 2026 10:47

@ragnorc ragnorc left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review of head 8f6f6c43. This follows my review of 51f9c4a9. It covers only what changed since then.

Summary

What changed. The branch was rebased onto main 373834b4, which is the v0.15.0 tag. git range-diff shows that the two earlier commits are the same patches (9ae5b473 = ccbe8f0e, 51f9c4a9 = d7f64038). One new commit, 8f6f6c43, does two things:

  1. It restores changelog.d/write-keeps-committed-handle.performance.md. The bytes are the same as the note that d7f64038 removed (blob 02ebb252).
  2. It opens the next cycle in changelog.d/release.json: version v0.15.1, base refs/tags/v0.15.0.

Why this matters. v0.15.0 is now tagged and published. The note can no longer go into v0.15.0, so it must go into the next cycle. This is path 1 from my last review. PR Title now passes.

Verdict. The release-note item from my last review is closed. B1 is still open. I reproduced it again on this head, on the new main base.

Blocking

B1. Still open: a conditional update or delete returns ReadSetChanged (409), not PreconditionFailed (412), when another handle publishes during staging

The PR's own code did not change. The cause and the fix are the same as in my reviews of 2487c5e0 and 51f9c4a9: when caller_expected_graph_head is set, map a graph-head ReadSetChanged from the publisher to PreconditionFailed, with the actual head. Add the second-handle winner to conditional_update_and_delete_races_return_precondition_failed_before_effects for set_age and remove_person.

main changed under the PR, including the update path (#1272). So I ran the probe again on this head instead of relying on the earlier result.

Evidence on this head. The probe is a copy of conditional_update_and_delete_races_return_precondition_failed_before_effects with one change: the winner writes through a second Omnigraph::open on the same root. The writer runs a conditional set_age, then a conditional remove_person, and is parked at mutation.post_stage_pre_effect_gate. The winner inserts a Company, then the writer is released.

cargo test --workspace --exclude omnigraph-gqt --exclude omnigraph-gqt-served --exclude omnigraph-dst --locked --features omnigraph-engine/failpoints,omnigraph-cluster/failpoints --test failpoints review_probe -- --nocapture

  • Base: the PR's six source files (handle_cache.rs, db/mod.rs, db/omnigraph.rs, exec/mutation.rs, exec/staging.rs, loader/mod.rs) set to main 373834b4 in the same worktree. Both cases return PreconditionFailed { branch: "main", expected: <captured head>, actual: Some(<winner head>) }. 1 passed.
  • Head 8f6f6c43: both cases return Manifest(ManifestError { kind: Conflict, … details: Some(ReadSetChanged { member: "graph_head:main", … }) }). 1 failed.

So the result does not depend on #1272 or the other main changes. GQT cannot express two handles on one root, so the failpoints test stays the cheapest owner for the regression.

Optional

  • O1. Open the v0.15.1 cycle once, with the package version. docs/dev/documentation.md step 8 opens a cycle with two edits: release.json and the workspace package version. Every crate is still 0.15.0. Edge builds stay refused until the package version is above the latest stable release (scripts/edge_release.py check-version). This PR, #1316, #1317 and #1318 each carry the same release.json hunk and none bumps the crates. The identical hunks merge cleanly (git merge-tree with #1318 is clean), so this does not block. But a small cycle-opening PR that does both edits gives the release one owner. It also lets each feature PR keep only its note. This is a decision for the release owner, not a defect of this PR.
  • O2. #1272 now on main. My last review predicted that #1272's PendingMode::ColumnPatch would get the write-allocate path at no extra cost. That is true on this head: ColumnPatch becomes PreparedPendingTable::Keyed (crates/omnigraph/src/exec/staging.rs:959-977) and commits through StagedMutation::commit_all, which inserts the committed dataset into the handle cache (staging.rs:1300-1317). The DST pinned suite passes with the PR's cost_table.txt, so the cost golden needs no new measurement for #1272.
  • O3. Old optional items. The old O1 to O4 (one owner for the freshness decision, root-scoped incarnation, the tradeoff sentence in writes.md, the plan_load snapshot note) are not addressed. They still apply.

Validation

  • git range-diff 2d8effab..51f9c4a9 373834b4..8f6f6c43: both old commits =; one new commit.
  • git show 8f6f6c43: release.json (2 lines) and the restored note (1 line). No code.
  • Isolated worktree at 8f6f6c43, pinned docs packages:
    • python3 scripts/release_notes.py preview --target HEAD: # Omnigraph v0.15.1, previous v0.15.0, base 373834b4, one selected note (write-keeps-committed-handle.performance.md) under Performance.
    • python3 scripts/check-docs.py: Documentation OK (476 Markdown files checked).
    • python3 scripts/test_release_notes.py: 96 tests, OK.
  • B1 probe: see B1 above.
  • git merge-tree --write-tree against the open PRs that share a file: clean with #1318, #1316, #1317, #1282 and #1275. #1235, #1217, #1184 and #1175 conflict, but each of them already conflicts with main alone.
  • CI on 8f6f6c43 (run): Test Workspace, Lint (clippy), the GQT jobs, the DST pinned suite, S3/Azure, storage-upgrade and PR Title pass. Dependency Guard (cargo deny) and CI Gate fail. The cause is three new rmcp 3.3.0 advisories (RUSTSEC-2026-0457, RUSTSEC-2026-0461, RUSTSEC-2026-0462). This PR does not change Cargo.lock, and #1318 fails the same way. The last main run (373834b4, 06:47 UTC) passed before these advisories arrived. A fix belongs in a separate dependency change.

Limits. I did not run the full workspace gate, DST, GQT, clippy or S3/Azure locally; CI covers them on this head. I did not measure latency. I removed the temporary probe afterwards.

@azimafroozeh
azimafroozeh added this pull request to the merge queue Nov 9, 2026

@ragnorc ragnorc left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review of head 565d116a. This follows my review of 8f6f6c43. It covers only what changed since then.

Summary

What changed. One new commit, 565d116a ("review"), on top of 8f6f6c43. No rebase: main is still 373834b4. It does two things:

  1. B1 fix. commit_updates_on_branch_with_expected (crates/omnigraph/src/db/omnigraph/table_ops.rs) now passes the publisher's error through caller_precondition_at_publish on both publish paths (bound branch and other branch). When the write carries the caller's --if-commit head and the publisher refuses with ReadSetChanged on graph_head:<branch>, the error becomes PreconditionFailed { branch, expected, actual }. actual is the head the publisher read. The failpoints test conditional_update_and_delete_races_return_precondition_failed_before_effects gains the two second-handle cases that my probe used. docs/dev/writes.md states the contract and the cost of the lost race.
  2. Dependency fix. rmcp goes from =3.3.0 to =3.9.0 for RUSTSEC-2026-0457, -0461 and -0462. ServerInfo becomes ServerConfig (the alias was deprecated in rmcp 3.4.0). A new note, changelog.d/rmcp-advisories.fixed.md, records it.

Why this matters. B1 was the only blocking item. A conditional update or delete now gets the same terminal 412 whether the race is caught under the gates or at the manifest CAS. The rmcp bump turns Dependency Guard (cargo deny) and CI Gate green.

Verdict. Ready to merge. B1 is fixed by design at the publish call, and I proved the regression: it fails without the fix for the predicted reason and passes on this head. No blocking items remain. The optional items below are cheap, and O1 (land the rmcp bump separately) helps every other open PR.

Blocking

None.

Optional

  • O1. Land the rmcp bump on its own, first. The advisories affect main and every open PR, not only this one. #1318 already fails cargo deny. #1316, #1317, #1311, #1282 and #1275 passed only because their last runs came before the advisories. Each of them fails on its next push until main has the bump. If the bump stays here, it reaches main only when this perf change merges, and the squash subject perf(engine): … hides a security fix. A separate fix(deps) PR made from these four hunks (Cargo.lock, omnigraph-mcp/Cargo.toml, omnigraph-mcp/src/lib.rs, the note) fixes main now and gives the fix one owner. Then rebase this PR on it.
  • O2. The test's doc comment now describes only half of the cases. It says the "authoritative under-gate check" maps both shapes. In the second-handle cases, the refusal comes from the publisher's CAS, after one detached commit per table. The final assertion compares the published pin and the linear latest_version_id. A detached commit moves neither, so "must fail before any Person effect" means no graph-visible effect in those cases. Please change the comment to name both refusal points, as writes.md now does.
  • O3. Only the bound-branch path is tested. The mapping is also added to the non-bound path (table_ops.rs, the commit_updates_on_branch_with_expected branch for a branch the handle is not bound to). No test reaches it with a second-handle winner. One more loop row, a conditional set_age on a named branch with the winner on a second handle, would cover it at almost no cost.
  • O4. Old optional items. O1 to O4 from my review of 51f9c4a9 (one owner for the freshness decision, root-scoped incarnation, the tradeoff sentence in writes.md, the plan_load snapshot note) still apply. The new writes.md paragraph covers part of the third one.

Tradeoffs and liability

  • The B1 fix is at the right layer. The caller's precondition is checked in two places: under the gates (revalidate_write_txn_with) and, after the early probe, implicitly by the publisher's ExactGraphHead CAS. The fix makes the second place give the first place's answer. It does not add a probe, a lock or a retry. It is one function in the module that owns the publish call. It adds one coupling: it matches the publisher's member string graph_head:<branch> (GraphHeadExpectation::read_set_member, omnigraph-catalog/src/publisher.rs). The under-gate check and the merge path already build the same string, so this is the third copy of that format. A typed member (for example a ReadSetMember::GraphHead { branch } in ManifestConflictDetails) would make the match hold by construction. That is optional and belongs with a change that touches the conflict type anyway.
  • The mapping is sound. At prepare time, the write refuses unless the effective head equals the caller's expected head (exec/mutation.rs). So when the publisher later reports that the graph head moved, the caller's expected head no longer holds either. A PreconditionFailed is then always correct, and it is terminal, which matches the update/delete rule that a moved head is never replayed.
  • The cost of the hoisted probe is now written down. A publish from another handle or process between the early probe and the gates is refused at the CAS, after the detached commits, not before them. The collector reclaims those commits. That is within invariant 5 (pre-publication effects stay unreachable), and writes.md now says it.
  • rmcp 3.9.0. I compared the two crate sources. StreamableHttpServerConfig::default() has the same allowed_hosts; Omnigraph still sets legacy_session_mode, json_response, max_request_body_bytes, hosts and origins itself; the new validate_empty_origin_allowlist defaults to false. ProtocolVersion::KNOWN_VERSIONS is the same set. LATEST moves from 2025-11-25 to 2026-07-28, but initialize negotiation still answers a supported client version with that version and otherwise falls back to the newest version that has initialize, 2025-11-25. So the note's "The MCP endpoint is unchanged" holds for initialize clients. The server MCP tests and the CLI relay tests pass in CI on this head.

Validation

  • git show --stat 565d116a: 7 files, +80/−23. git merge-base with origin/main is still 373834b4.
  • B1 regression, after the fix. Isolated worktree at 565d116a:
    cargo dev-test --test failpoints conditional_update_and_delete_races_return_precondition_failed_before_effects -- --nocapture
    (the .cargo/config.toml alias: --workspace --exclude omnigraph-gqt --exclude omnigraph-gqt-served --exclude omnigraph-dst --locked --features omnigraph-engine/failpoints,omnigraph-cluster/failpoints). Result: 1 passed. All four rows (set_age, remove_person, each on the same handle and on a second handle) return PreconditionFailed with the winner's head.
  • B1 regression, before the fix. The same worktree and command, with only crates/omnigraph/src/db/omnigraph/table_ops.rs set to 8f6f6c43 (the new test kept). Result: 1 failed, at failpoints.rs:2000: conditional set_age race (winner on a second handle: true) must be PreconditionFailed, got: write authority 'graph_head:main' changed during preparation (expected hb1….2…, current hb1….3…). This is the predicted reason: the publisher's ReadSetChanged reached the caller. The two same-handle rows passed before reaching it. I restored the file afterwards; the worktree is clean at 565d116a.
  • GQT cannot express this case: its --- concurrent writers share one engine handle. So the failpoints test stays the right owner, as in my last review.
  • rmcp: read rmcp-3.9.0/CHANGELOG.md (3.4.0 to 3.9.0) and compared StreamableHttpServerConfig::default(), ProtocolVersion and negotiate_protocol_version in the 3.3.0 and 3.9.0 crate sources. See the last tradeoff item.
  • git merge-tree --write-tree with #1318, #1317, #1316, #1311, #1302, #1282 and #1275: all clean. #1184 conflicts, but it already conflicts with main alone.
  • CI on 565d116a (run): every check passes, including Dependency Guard (cargo deny), CI Gate, Test Workspace, Lint (clippy), the GQT jobs, the DST pinned suite, RustFS S3 (default and failpoints), Azurite, storage upgrade and PR Title.

Limits. I ran only the focused failpoints test locally. CI covers the workspace gate, clippy, GQT, DST and the S3/Azure suites on this head. I did not run the MCP endpoint against a live client, and I did not measure latency. The new commit does not touch a Lance API, so no new Lance page applies beyond my earlier reviews.

Merged via the queue into ModernRelay:main with commit cf2dce5 Nov 9, 2026
32 checks passed
ragnorc added a commit to ragnorc/omnigraph that referenced this pull request Nov 9, 2026
Main's ModernRelay#1291 keeps the dataset a write commits in the table handle
cache, so a write that follows another on the same handle opens no
manifest. The keyed-write scenario runs insert, update, merge-present
and merge-new in turn on one reopened handle: the insert starts cold
and is unchanged; the three later writes drop l.get (update 11 -> 8,
merge present 9 -> 8, merge new 7/5 -> 4 at 2/16 fragments).
azimafroozeh pushed a commit to azimafroozeh/omnigraph that referenced this pull request Nov 12, 2026
…nRelay#1318)

* perf(mutation): an update narrows a keyed match through the id index

A `<key> = v` conjunct also selects the derived `id = v`, so an update's
match scan adds that conjunct when the pinned table has an id BTREE, as a
node delete already does. The scan then reads the index instead of the key
column of every fragment.

* perf(storage): an all-new or all-present upsert skips the full-table join

Lance's insert-capable merge-insert route joins the source against every
id of the table. With an id BTREE, the exact-id probe a strict insert runs
decides whether an upsert's rows are all new or all present; the first
stages through the absence-proven insert writer, the second as a
known-present update. A mixed batch, or a table without the BTREE, keeps
the join.

* test(dst): pin one-row keyed write costs at two table sizes

KeyedInsert, KeyedUpdate, MergeLoadPresent and MergeLoadNew on an indexed
table of 2 and 16 fragments. On main their l.get rows were 10/25, 13/41,
10/24 and 10/24; now 9/10, 11/11, 9/9 and 5/5. InsertLegacy l.get moves
15 -> 13: its keyed insert takes the absence-proven writer.

* perf(storage): bound the upsert probe and the known-present switch by index coverage

The exact-id probe plans its IN list once per fragment the id BTREE does
not cover, so it runs only while those copies hold no more ids than one
keyed batch (ModernRelay#1225). An all-present upsert's indexed merge reads each
uncovered fragment once more, where the join reads every fragment once,
so it switches only while the BTREE covers more fragments than it leaves.
key_column_uncovered_fragments counts the tail under the coverage rule
key_column_coverage already applies; both read one helper.

* test(dst): MergeLoadNewFragments2 reads 7 after the coverage-bounded switch

* docs(writes): the upsert probe and the keyed update conjunct

* test(write_cost): a keyed insert on the indexed standard graph stages the fenced insert

* docs(changelog): shorten the keyed-writes note

* release(notes): select new notes for v0.15.1 now v0.15.0 is tagged

* test(dst): re-measure keyed write costs after merging main

Main's ModernRelay#1291 keeps the dataset a write commits in the table handle
cache, so a write that follows another on the same handle opens no
manifest. The keyed-write scenario runs insert, update, merge-present
and merge-new in turn on one reopened handle: the insert starts cold
and is unchanged; the three later writes drop l.get (update 11 -> 8,
merge present 9 -> 8, merge new 7/5 -> 4 at 2/16 fragments).

* test(dst): say why the keyed-write rows now match at both sizes

* fix(writes): keep the id join when the uncovered tail's data files exceed a byte budget

Lance 13's indexed merge route scans every fragment the id BTREE leaves
uncovered for every source column, so the all-present switch's read grew
with the row width. It now also requires the tail's recorded data-file
bytes to be known and at most KNOWN_PRESENT_TAIL_MAX_BYTES (1 MiB); the
fragment rule stays, since it bounds the tail's requests.

* refactor(writes): one exact-id scan for both id probes; pin the probe's IN-list bound

existing_id_count and first_existing_id built the same filter and scan;
exact_id_scan now owns it. A staged test pins the ModernRelay#1225 bound at its edge:
uncovered fragments x ids = 8,192 probes, one id more keeps the join.
writes.md and the changelog note state the byte budget.

* test(dst): build the keyed-write scenario's indexes with ensure_indices_as

Main (ModernRelay#1275) replaced Session::ensure_indices with ensure_indices_as;
the scenario this pull request adds still called the old name.

* docs(changelog): keep the keyed-writes note within the 200-character limit

writes.md carries the byte budget the note no longer states.

* docs(changelog): say the keyed-write index route holds while the unindexed tail is small

The note said a one-row keyed insert, update by key or merge load always
reads through the id index. For a row that already exists, the indexed
route is taken only while the uncovered tail has fewer fragments than the
index covers and at most 1 MiB of data files; otherwise the join still reads
the id column of every fragment. Reword it to state that condition within
the 200-character limit.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants