Skip to content

fix(cluster): release the lock after a converged direct apply - #1360

Open
ragnorc wants to merge 8 commits into
ModernRelay:mainfrom
ragnorc:fix/1359-direct-apply-releases-lock
Open

ragnorc wants to merge 8 commits into
ModernRelay:mainfrom
ragnorc:fix/1359-direct-apply-releases-lock

Conversation

@ragnorc

@ragnorc ragnorc commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

Summary

A direct cluster apply that converges now releases its cluster lock, so the first local server starts without cluster force-unlock. It uses the same check a clean server shutdown uses (#899): all of the apply's writes run in one storage scope, the scope closes, every request it accepted settles, and none is uncertain. Every other case keeps the lock as before.

Closes #1359

Reproduction

On main 825213be, with the issue's steps (a fresh local root, one graph):

$ omnigraph cluster apply --target ./data2 --config ./source --json      # status complete, converged true
$ omnigraph cluster status --target ./data2 --json                       # lock_id: 01M76WWW1KPQTCAZM3TNSNNYGY
$ omnigraph-server --cluster ./data2 --bind 127.0.0.1:18787 --unauthenticated
[state_lock_held] __cluster/lock.json: cluster state lock already exists (lock id 01M76WWW1KPQTCAZM3TNSNNYGY) ...
Location: crates/omnigraph-server/src/settings.rs:79                     # exit 1

The released 0.15.0 CLI shows the same lock_id after a converged apply. Verdict: reproduced.

The failing test on main is deployment::tests::bootstrap_and_add_graph_use_one_v2_protocol_without_reset_or_unlock_issue_1359 (cluster crate). It fails at its first new assertion with left: Some("01M76WWPM0R1E109VGKW4Y7DKR"), right: None.

Root cause

apply_direct_bundle (crates/omnigraph-cluster/src/deployment/execution.rs) released the lock only when no effect had started. Once an apply created a graph or changed a schema, it returned with the lock held, because it had no proof that its writes had settled. The server has that proof: admit_serving_snapshot gives its store a StorageIoScope, and clean shutdown (crates/omnigraph-server/src/lib.rs) closes the scope, waits for idle, refuses on uncertainty and then calls release_after_settlement. The direct apply built its store without a scope, so it could not make the same check.

Change

  • apply_direct_bundle gives its store a fresh StorageIoScope after bootstrap_ledger, before admission. Graph opens and creation go through store.graph_storage, and ledger writes go through the store adapter, so every write of the deployment is in that scope.
  • After execution, a Complete result with converged: true calls release_after_settlement. That call already closes the scope, waits for idle and returns cluster_admission_io_uncertain when a request is uncertain; it also refuses while another admission handle is alive.
  • A store without a scope (Azure, where with_io_scope is a no-op) keeps the lock, as the server does on Azure.
  • An Outstanding result, an unconverged result, or an error keeps the lock, unchanged.
  • If release fails, apply returns an error that names the converged deployment and the held lock, so the operator knows the apply landed and which lock to reconcile.
  • The CLI's custody line now says the lock is released when the deployment converges and its writes settle, and kept otherwise.

Decisions

  • Release by default, not a new --release-lock flag. The issue offered both. The default is the only one that removes the step from every first-run guide, and it adds no API surface. The proof already exists and is what servers use to release on local storage and S3.
  • Only converged results release. A non-converged complete result is a partial outcome that an operator may want to inspect under the lock, so it keeps the current behaviour. The release could be widened later.
  • Not in scope: maintenance commands (Optimizing a served cluster requires stopping the server, and each direct run keeps its lock #1082), direct data writes and reconciliation (apply --deployment-id --writers-stopped) keep their locks. They could use the same mechanism later.

Before and after

After the change, the same steps:

$ omnigraph cluster apply --target ./data3 --config ./source --json      # complete, converged true
$ omnigraph cluster status --target ./data3 --json                       # lock_id: None
$ omnigraph-server --cluster ./data3 --bind 127.0.0.1:18788 --unauthenticated
GET /readyz -> {"ready":true,"status":"serving","served_graph_count":1,...}

After a clean SIGTERM, lock_id is still None.

Tests

  • Extended owner, renamed for the regression gate: bootstrap_and_add_graph_use_one_v2_protocol_without_reset_or_unlock_issue_1359. Bootstrap and a second apply that adds a graph now run with no manual unlock between them. The test asserts no lock after each converged apply, then admits a serving owner.
  • cli_cluster::cluster_apply_bootstraps_v2_and_releases_its_lock_issue_1359 (was ..._retains_exact_owner): the CLI apply prints its custody line, then leaves no lock.json, and cluster status reports a null lock_id.
  • Tests that force-unlocked right after a converged apply now assert release. In tests/failpoints.rs that is the new assert_released helper. unlock_offline stays strict after failed or reconciled applies, so those tests still prove failures keep the lock.
  • Tests whose later steps need a held lock (offline_deployment_captures_schema_queries_and_retains_original_lookup, app_tests::direct_deployment_delivery_failure_retains_one_result_and_original_identity, cli_cluster_e2e::cluster_e2e_offline_deployment_has_root_only_receipts_and_explicit_unlock) now assert release first. They then model a later writer's kept lock: an admission dropped without release, or write_cluster_lock.
  • The CLI test helper unlock_storage_fixture tolerates a released lock.
  • admission::tests::uncertain_scope_keeps_the_lock_through_a_settled_release_issue_1359: a write abandoned in flight makes release_after_settlement refuse cluster_admission_io_uncertain and keep the lock. The direct apply relies on that check inside the release, since it checks nothing before the call.
  • S3 journeys: cluster_e2e_s3_direct_root_deploys_unlocks_and_maintains_issue_1163 and the shared object_storage_offline_deployment in s3_cluster.rs now expect the lock to be released on S3. On the first push, RustFS showed the release on real S3: the old assertion found no lock_id. Azure runs the same journey and still expects the lock to be kept, because the apply has no tracked storage scope there.

Local results on a 2-core box:

  • omnigraph-cluster: all 179 tests pass, including the failpoints and identity-recovery suites.
  • omnigraph-server and omnigraph-skill: all pass.
  • cargo dev-clippy, cargo fmt --all --check and scripts/check-docs.py pass.
  • CI on cdda4e0c: all 28 checks pass, including Test Workspace, RustFS S3 Integration (default and failpoints), Azurite Azure Integration and Storage Upgrade Compatibility.
  • In omnigraph-cli, 70 tests fail locally, and they fail the same way with the fix reverted. 68 are the support harness's 30-second server-readiness timeout ("server did not become ready or report its bound address", empty stderr). The other two are managed_lifecycle_capture_and_retry_records_refuse_symlinks_without_reading_targets and system_local::cluster_server_boot_ignores_local_config_in_cwd ("could not find Cargo.toml"). CI is the evidence for those.

Docs and skill

The user guides (deployment.md writer topology, clusters/index.md direct bootstrap, operations/s3-cluster-roots.md), the developer guides (control-plane.md concurrency, the invariants support boundary, the architecture.md CLI delivery paragraph) and skills/omnigraph (SKILL.md, cluster.md, commands.md, server-policy.md) now say a converged direct apply releases its lock, and that Azure keeps it. The testing guide says the direct-apply delivery tests inspect the released lock. The CLI's custody line says the lock is never released on Azure. A changelog note is in changelog.d/direct-apply-releases-lock.changed.md.

Risks

  • Scripts that force-unlock after apply will fail. The current first-run pipe (force-unlock "$(cluster status ... | sed ...)") now finds no lock and fails with state_lock_missing. Omnigraph Studio's first-run screen (spec 029) shows that step; it should drop it once this ships.
  • S3 direct applies now run inside a storage scope. The scope disables hidden mutation retries, as it does for the server. A transient S3 error that a retry used to absorb can now fail the apply. That is fail-closed: the lock is kept. The scoped path is the one servers already use in production on S3.
  • Not verified here: a live S3 or RustFS run of the direct apply, an injected uncertain request at the apply level (the uncertainty refusal itself is covered by storage-level tests and release_settled), and the CLI system journeys this box cannot spawn.

Workflow run: run-wf-reproduce-modernrelay-omnigraph-issue-20261112T0312Z-10668a
Response run: run-wf-respond-modernrelay-omnigraph-pr-20261112T0922Z-606fa2
Response run: run-wf-respond-modernrelay-omnigraph-pr-20261112T1637Z-d25653
Response run: run-wf-respond-modernrelay-omnigraph-pr-20261113T0002Z-0f999c
Response run: run-wf-respond-modernrelay-omnigraph-pr-20261114T0027Z-d64a8f
Response run: run-wf-respond-modernrelay-omnigraph-pr-20261115T2312Z-366c63
Response run: run-wf-respond-modernrelay-omnigraph-pr-20261117T0307Z-f021ec

A direct cluster apply kept its admission lock after it succeeded, so the
first local server refused to start until cluster force-unlock released it.

The apply now runs its control and graph writes in one storage scope. When
the deployment completes converged, it releases the lock through the same
settlement check a clean server shutdown uses: the scope closes, every
accepted request settles, and none is uncertain. A failed, outstanding or
unconverged deployment, an uncertain scope, and Azure keep the lock as before.

Closes ModernRelay#1359
The RustFS journeys still force-unlocked after a converged direct apply,
which now leaves no lock on S3 either.
…journey

Azure keeps its stopped-process boundary, so a converged direct apply there
still retains the lock; only S3 releases it.

@ragnorc ragnorc left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review of cdda4e0

Why. After a direct cluster apply, the first server refused to start until the operator ran cluster force-unlock (#1359).

What. A direct apply now runs all its writes in one StorageIoScope. When the deployment ends Complete with converged: true, it releases its lock. It uses the same settlement check a clean server shutdown uses: close the scope, wait for idle, refuse if a request is uncertain. Azure, failed, outstanding and unconverged results keep the lock as before.

How it fits. The fix is at the right layer. apply_direct_bundle owns the deployment's admission, so it is the one place that knows both the logical outcome (converged) and the storage outcome (the scope). It adds no new state and no new API. It reuses release_after_settlement → ClusterStore::release_settled, which already closes and drains a tracked scope and refuses cluster_admission_io_uncertain. I checked the claim that every write of the deployment is in the scope: the lock and ledger writes use store.adapter, which with_io_scope rebinds, and every graph open and create in deployment/execution.rs goes through store.graph_storage(...), which uses self.io_scope(). No Omnigraph::open(uri) or unscoped storage_for_uri call remains in non-test cluster code. bootstrap_ledger runs before the scope, but it holds and releases its own lock after awaited control writes, as on main.

A good side effect: S3 direct applies now use the single-attempt client for conditional and mutable keys. On main, a hidden retry of a conditional ledger or lock write could not prove what happened to the first attempt. Now such a failure ends the apply with the lock kept. That is the scope's documented contract, and the server already runs this way.

Verdict. The design is correct and I found no correctness defect. One small blocking item: two skill statements still contradict the new behavior (B1). The rest is optional.

Blocking

B1. Two skill statements still say a direct apply keeps its lock. AGENTS.md asks that every skills/omnigraph statement about a changed behavior is reconciled in the same change, because omnigraph skill export ships it with the binary.

  • skills/omnigraph/SKILL.md:204: "Each direct writer on the root takes the admission lock and keeps it; once that writer has stopped and its I/O has settled, release its exact lock_id ... before the next writer or the server starts." This line is about the inline s3:// root, where a converged apply now releases.
  • skills/omnigraph/references/cluster.md:74-75: "Direct apply creates graphs, not rows ... It retains its exact admission lock after completion." This contradicts lines 46-48 of the same file, which the PR updated.

Also stale, but a developer guide: docs/dev/testing.md:216 says direct-apply delivery tests inspect the "retained lock". direct_deployment_delivery_failure_retains_one_result_and_original_identity now asserts release and then models a later owner's lock.

Optional

O1. Give "settle, then release" one owner. The rule now lives in two places with two different checks:

  • Server (crates/omnigraph-server/src/lib.rs:4322-4345): tests canonical_root().starts_with("az://") for Azure, closes and drains the scope, checks is_uncertain() itself, waits with wait_for_exclusive_owner(), then calls release_after_settlement.
  • Direct apply (execution.rs:1721-1738): tests io_scope().is_none() for Azure, and relies on the uncertainty check inside ClusterStore::release_settled. It does not wait for the exclusive owner; a live clone gives cluster_admission_in_use at once. That is fail-closed and no clone exists on this path today.

The PR names maintenance (#1082) and reconciliation as next users. A single ClusterAdmission method (for example one that returns Released or RetainedUntracked, or an error) would hold the Azure rule, the scope check and the exclusive-owner wait in one place, and both callers would use it. Also, the doc comment on release_after_settlement (admission.rs:390-399) still says "This method does not establish that proof". With a tracked scope it now does establish the storage half. Please say what it checks and what the caller must still prove (the logical outcome).

O2. Report what happened to the lock. The custody line is printed before execution (report_id), so it can only say "released when the deployment converges and its writes settle, else retained". Two cases read wrong:

  • On Azure a converged apply keeps the lock, but nothing printed after completion says so. The operator sees converged: true and the line above, and can conclude the lock was released.
  • If release fails, the CLI prints {"ok": false, "diagnostics": [...]} for a deployment that converged. The message does name the deployment and the lock, which is good, but a script that reads ok sees a failed apply.

Consider one line after completion ("Admission lock released" or "Admission lock retained: "), or a lock field in the result. Then the output is the one source of truth for the next step.

O3. Pin the uncertainty refusal with a test. The uncertainty branch in ClusterStore::release_settled (store.rs:1250-1262) had no test anywhere. The server checks uncertainty before it calls this, so the direct apply is the first caller that depends on it. I checked it with a temporary test in the admission.rs tests module, and then removed the test:

#[tokio::test]
async fn review_probe_uncertain_scope_keeps_the_lock() {
    let dir = state_fixture(2, false);
    let root = dir.path().to_str().unwrap();
    let store = crate::store::ClusterStore::for_storage_root(root).unwrap()
        .with_io_scope(omnigraph_storage::StorageIoScope::new()).unwrap();
    let owner = acquire_with_store(&store, ClusterAdmissionPurpose::Deployment)
        .await.unwrap().unwrap();
    let graph = dir.path().join("graphs/knowledge.omni");
    let adapter = owner.store().graph_storage(graph.to_str().unwrap()).unwrap();
    let payload = "x".repeat(64 << 20);
    let write = adapter.write_text(graph.join("_refs/probe.json").to_str().unwrap(), &payload);
    let _ = tokio::time::timeout(std::time::Duration::ZERO, write).await; // abandon a mutable-key write
    let error = owner.release_after_settlement().await.unwrap_err();
    assert_eq!(error.code, "cluster_admission_io_uncertain");
    assert!(dir.path().join("__cluster/lock.json").exists());
}

Result at cdda4e0: uncertain=true pending=0, release refused with cluster_admission_io_uncertain, and the lock file stayed. It passes. A test like this would stop a later edit from removing the check inside release_settled because "the caller already checked", as the server does.

Validation

All at exact head cdda4e0, in a clean detached worktree. Debug profile, cargo dev-nextest feature selection, RUST_MIN_STACK=16777216, rustc 1.97.1, 2-core box.

  • cargo dev-nextest -E 'package(omnigraph-cluster) | (package(omnigraph-cli) & (test(/issue_1359/) | test(direct_deployment_delivery_failure) | test(cluster_e2e_offline_deployment_has_root_only_receipts_and_explicit_unlock) | test(cluster_force_unlock_wrong_id_exits_nonzero)))': 183 passed, 0 failed. That is 179 cluster tests, including tests/failpoints.rs and tests/identity_recovery.rs, plus the 4 CLI tests.
  • Regression proof. I reverted only crates/omnigraph-cluster/src/deployment/execution.rs and crates/omnigraph-cli/src/commands/cluster/mod.rs to the merge base 825213b and kept the PR's tests:
    • deployment::tests::bootstrap_and_add_graph_use_one_v2_protocol_without_reset_or_unlock_issue_1359 failed at tests.rs:66: left: Some("01M7715MRJFBF4EE9Z7KBQ1PWQ"), right: None. This is the predicted reason: the converged apply kept its lock.
    • cli_cluster::cluster_apply_bootstraps_v2_and_releases_its_lock_issue_1359 failed at cli_cluster.rs:3106: __cluster/lock.json still exists.
    • With the files restored (git status clean), both pass.
  • GQT. This contract cannot be expressed in GQT. It is about cluster admission and lock custody between processes, not query rows, counts or errors. The owners are the cluster's in-source deployment tests and the CLI's cli_cluster, as the PR uses.
  • CI at cdda4e0: 28 passed, 4 skipped (tag/release jobs). Clippy, fmt, docs and the S3 journey ran there. I did not run them locally.
  • Code inspection. Every graph open and create in deployment/execution.rs uses store.graph_storage. Lock and ledger writes use the rebound store.adapter. Leases are server-only (lease.rs:1-4), so the new scope adds no lease fence to a direct apply.
  • Merge check. git merge-tree is clean against main ad41b29. Against the other open PRs, #1360 adds no conflicts: #1318, #1341, #1345, #1346, #1348, #1349 and #1357 merge clean, and #1175, #1184, #1217, #1234, #1235 and #1316 conflict only in files this PR does not touch. #1217 changes the S3 lease and takeover. It does not interact with this release path, because the released marker is still written by exact-version CAS.
  • Not run: the S3 and Azure suites (no bucket or container here), and a fault injected into the apply itself (see O3 for the store-level check). typos was not run.

Liability

Removes: a manual step from every first-run guide, and a scripted force-unlock that parsed cluster status output. Adds: one more release call site with its own Azure test, and a breaking change for scripts that still force-unlock after apply (the PR says this; state_lock_missing is a clear failure). No new state, API or invariant. After five more changes of this kind (maintenance per #1082, load, reconciliation), the risk is five copies of "if scoped, release, map the error". O1 removes that risk.

…in the uncertain release

Answer the review of cdda4e0 (ModernRelay#1360):
- B1: SKILL.md and references/cluster.md no longer say a converged direct
  apply keeps its lock; docs/dev/testing.md says the delivery tests inspect
  the released lock.
- O1: release_after_settlement's doc comment says what it proves with a
  tracked scope and what the caller must still prove.
- O2: the custody line and changelog note say Azure never releases.
- O3: admission test that an in-flight write abandoned in the scope makes
  the release refuse cluster_admission_io_uncertain and keep the lock.
@ragnorc

ragnorc commented Nov 12, 2026

Copy link
Copy Markdown
Contributor Author

Reply to the review of cdda4e0c. Pushed c3271af9, one commit, on the same branch with no merge of main: GitHub reports it mergeable, and the 5 commits on main since the merge base touch none of these files.

B1: valid, fixed. Both skill statements contradicted the change, and so did the testing guide.

  • skills/omnigraph/SKILL.md (direct targets): a converged cluster apply releases the lock once its writes have settled, never on Azure. Maintenance, and an apply that fails or does not converge, keep it. The force-unlock step now applies to a kept lock.
  • skills/omnigraph/references/cluster.md (direct bootstrap): the "retains its exact admission lock after completion" sentence now says the same as lines 46-48 of that file.
  • docs/dev/testing.md: the direct-apply delivery tests inspect "the released lock of the converged apply". That matches direct_deployment_delivery_failure_retains_one_result_and_original_identity, which asserts status.lock_id.is_none() and then models a later owner's lock.
  • After the edit, no statement in skills/omnigraph says a converged direct apply keeps its lock (checked with grep for keep/retain near "lock").

O1: partly taken.

  • Taken: the doc comment on release_after_settlement now says what it proves itself with a tracked scope (close, drain, refuse cluster_admission_io_uncertain), and what the caller must still prove: the logical outcome, and settlement when there is no tracked scope.
  • Not taken now: one shared "settle, then release" method. The two callers differ in one deliberate way. The server calls wait_for_exclusive_owner, whose doc says the caller must keep its own deadline, and the server's process watchdog bounds that wait. The CLI apply has no watchdog (no watchdog in omnigraph-cli/src or omnigraph-cluster/src). A shared method that waits would turn a leaked admission clone into a hung CLI, where today it fails at once with cluster_admission_in_use and keeps the lock. The settlement checks themselves already have one owner, ClusterStore::release_settled. The method is worth adding with the next caller (Optimizing a served cluster requires stopping the server, and each direct run keeps its lock #1082), with the wait deadline as a parameter.

O2: partly taken.

  • Taken: the custody line now reads "released when the deployment converges and its writes settle (never on Azure), else retained". On Azure, ClusterStore::with_io_scope returns the store unscoped, so release_converged_deployment always keeps the lock. The line no longer suggests a release there. The changelog note says so too.
  • Not taken: ok: false after a failed release stays. A converged apply whose release fails still holds the lock, and the server cannot start until someone reconciles it. A non-zero exit stops a script before it starts a server that would fail with state_lock_held. The message names the deployment and the lock. A lock field in the result would change the deployment result shared with the HTTP API, which is more than this fix needs.

O3: taken. admission::tests::uncertain_scope_keeps_the_lock_through_a_settled_release_issue_1359 is your probe, in the existing admission owner. It abandons a 64 MiB write to a _refs/ key in flight, then asserts that release_after_settlement refuses cluster_admission_io_uncertain and that __cluster/lock.json still names the owner's lock.
With the uncertainty check in ClusterStore::release_settled removed (if false && scope.is_uncertain()), the test fails at release_after_settlement().await.unwrap_err(): the release succeeds and drops the lock. With the check restored it passes.

Gates (debug, cargo dev-nextest, RUST_MIN_STACK=16777216, this box):

  • package(omnigraph-cluster): 180 passed (179 before, plus the new test).
  • package(omnigraph-skill) and the CLI tests for the custody line, the skill and the lock (cli_cluster::cluster_apply_bootstraps_v2_and_releases_its_lock_issue_1359, app::tests::direct_deployment_delivery_failure_retains_one_result_and_original_identity, cli_cluster_e2e::cluster_e2e_offline_deployment_has_root_only_receipts_and_explicit_unlock, cli_cluster::cluster_force_unlock_wrong_id_exits_nonzero, skill::tests::*, cli_schema_config::skill_export_*): 15 passed.
  • cargo fmt --all --check, cargo dev-clippy, scripts/check-docs.py (the changelog note is at the 200-character limit) and scripts/check-agents-md.sh pass. typos is not installed here; CI ran it.
  • CI at c3271af9: 28 passed, 4 skipped (tag and release jobs). Mergeable.

I did not run the S3 or Azure suites locally; CI ran RustFS and Azurite.

@ragnorc ragnorc left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review of c3271af

Scope. This review covers the delta since my review of cdda4e0c: one commit, c3271af9, which answers B1 and O1–O3. I do not repeat the earlier findings. The design verdict from that review stands: the fix is at the right layer and reuses the settled-release proof the server uses.

What changed. The commit edits two skill passages and the testing guide (B1), rewrites the doc comment on ClusterAdmission::release_after_settlement (O1), adds "(never on Azure)" to the custody line and the changelog note (O2), and adds admission::tests::uncertain_scope_keeps_the_lock_through_a_settled_release_issue_1359 (O3). It changes no production logic.

Verdict. Ready. B1 is resolved. No blocking findings. Two optional documentation items remain, both one-line edits.

Replies to the earlier items

  • B1: resolved. skills/omnigraph/SKILL.md:204 and skills/omnigraph/references/cluster.md:74-79 now agree with lines 46-48 of cluster.md and with docs/user/clusters/index.md:76-82. docs/dev/testing.md:216 now names the released lock.
  • O1: I accept the deferral. I checked the reason: watchdog appears only in crates/omnigraph-server/src/lib.rs, not in omnigraph-cli/src or omnigraph-cluster/src. wait_for_exclusive_owner (admission.rs:371-377) tells the caller to keep its own deadline. A shared method that waits would therefore need a deadline parameter, or it would turn a leaked clone into a hung CLI. Adding that method together with the next caller (#1082) is a reasonable time to do it. The new doc comment (admission.rs:390-401) is accurate: ClusterStore::release_settled (store.rs:1250-1263) closes the scope, waits for idle and refuses cluster_admission_io_uncertain, which is what the comment says.
  • O2: I accept both parts. ClusterStore::with_io_scope returns the store unscoped on Azure (store.rs:191-193), so release_converged_deployment returns at io_scope().is_none() (execution.rs:1725-1727) and keeps the lock. The custody line now says so. Keeping ok: false when release fails is the safer exit for a script.
  • O3: resolved. The test is in the existing admission owner, as asked. See Validation for its before/after proof.

Blocking

None.

Optional

O4. One developer-guide sentence still says the lock is retained after a delivery failure. docs/dev/architecture.md:231: "the CLI captures any diagnostic delivery failure, lets that one core call finish, then returns the failure with its original ID, root and retained lock." The PR does not touch this file. The test that owns this path, app_tests::direct_deployment_delivery_failure_retains_one_result_and_original_identity, now asserts status.lock_id.is_none() after the converged apply (app_tests.rs:1701-1703). Suggested wording: "…returns the failure with its original ID and root; a converged apply has still released its lock."

O5. Three skill passages leave out the Azure exception. SKILL.md:204, references/cluster.md:74-79, the custody line, the changelog note and the user guide all say a converged apply never releases on Azure. These three do not:

  • skills/omnigraph/SKILL.md:324-329 (Cluster deployment): "A direct apply that converges releases its admission lock once its writes have settled".
  • skills/omnigraph/references/commands.md:505-506: "Each direct writer except a converged cluster apply keeps its admission lock".
  • skills/omnigraph/references/server-policy.md:24-28: the list of writers that keep the lock names "a cluster apply that fails or does not converge" but not a converged apply on Azure.

An agent that reads one of these on Azure can skip the unlock step. The server then refuses with state_lock_held and names the lock, so this fails closed. It is still the kind of stale skill text AGENTS.md asks to reconcile in the same change, because omnigraph skill export ships it. Adding "(never on Azure)" to each passage makes all of them match.

Validation

All checks ran at exact head c3271af9, in a clean detached worktree. Debug profile, cargo dev-nextest feature selection, RUST_MIN_STACK=16777216, rustc 1.97.1, 2-core box.

  • cargo dev-nextest -E 'package(omnigraph-cluster) | package(omnigraph-skill) | (package(omnigraph-cli) & (test(/issue_1359/) | test(direct_deployment_delivery_failure) | test(cluster_e2e_offline_deployment_has_root_only_receipts_and_explicit_unlock) | test(cluster_force_unlock_wrong_id_exits_nonzero) | test(/skill_export/)))': 188 passed, 0 failed. That is 180 cluster tests (179 before, plus the new one), 1 skill test and 7 CLI tests.
  • Proof for the new test (O3). I changed only crates/omnigraph-cluster/src/store.rs:1257 to if false && scope.is_uncertain() { and ran cargo dev-nextest -E 'test(=admission::tests::uncertain_scope_keeps_the_lock_through_a_settled_release_issue_1359)'. It failed at admission.rs:880: called Result::unwrap_err() on an Ok value: (). That is the predicted reason: without the check, the release succeeds while a write is uncertain. After git checkout of the file (git status clean, HEAD c3271af9), the same command passed. This matches the author's reported result.
  • Determinism of the new test. The test abandons the write with timeout(Duration::ZERO, …). The scoped local adapter writes through object_store's local backend, which runs the write as a blocking task. The first poll therefore returns pending, so the test does not depend on how fast the disk is.
  • GQT. This contract cannot be expressed in GQT. It is about admission and lock custody, not query rows, counts or errors. The in-source admission tests are the correct owner, as the earlier review said.
  • cargo fmt --all --check and scripts/check-agents-md.sh pass. I did not run scripts/check-docs.py (this box lacks its pinned Python packages), clippy, typos, or the S3 and Azure suites.
  • CI at c3271af9: 28 passed, 4 skipped (tag and release jobs).
  • Merge check. git merge-tree against main 2bfb3f35 is clean.
  • Code inspection. I checked the claims in the reply against code: the Azure no-op in with_io_scope, the early return in release_converged_deployment, the settlement steps in release_settled, and the absence of a watchdog in the CLI and cluster crates. I searched docs/ and skills/ for statements that a direct apply keeps or retains its lock. O4 and O5 are what remains. The user guide (docs/user/clusters/index.md:76-82) is correct.

Liability

This commit adds no state, API or invariant. It adds one test (about 30 lines) to the existing owner. It removes a gap: before this commit, nothing tested the only check the direct apply depends on. The O1 deferral keeps one known cost: two release call sites with different Azure checks until #1082. The doc comment now records what each caller must prove, which limits that cost.

…pply releases its lock

Answer the review of c3271af (ModernRelay#1360):
- O4: docs/dev/architecture.md no longer says the CLI returns a delivery
  failure with a retained lock; the lock follows the apply's outcome, so a
  converged apply has still released it (never on Azure).
- O5: SKILL.md (cluster deployment), references/commands.md and
  references/server-policy.md now say a converged direct apply keeps its
  lock on Azure, matching SKILL.md:204, references/cluster.md and the
  user guide.
@ragnorc

ragnorc commented Nov 12, 2026

Copy link
Copy Markdown
Contributor Author

Reply to the review of c3271af9 (review 5872788053). I pushed 7178b6b3, one commit, to the same branch. It changes four documentation files and no code. I did not merge main. GitHub reports the branch mergeable, and git merge-tree with main at 1715b88a is clean.

O4: valid, fixed. docs/dev/architecture.md:229-233 now says that the CLI returns the delivery failure with its original ID and root. It also says that the lock follows the outcome of the apply, so a converged apply has released it (never on Azure). The test app_tests::direct_deployment_delivery_failure_retains_one_result_and_original_identity asserts status.lock_id.is_none() after the converged apply (app_tests.rs:1701-1703). The error it returns carries the custody line from commands/cluster/mod.rs:325, not a held lock. I added the Azure exception to the suggested wording, because it applies here too.

O5: valid, fixed. Each of the three passages now names the Azure exception:

  • skills/omnigraph/SKILL.md:324-326: "releases its admission lock once its writes have settled (never on Azure)".
  • skills/omnigraph/references/commands.md:505-506: "Each direct writer except a converged cluster apply (never on Azure) keeps its admission lock".
  • skills/omnigraph/references/server-policy.md:26-28: the writers that keep the lock now include "any cluster apply on Azure".

They now agree with SKILL.md:204, references/cluster.md:74-79 and docs/user/clusters/index.md:76-82. The reason is the same as for O2. On Azure, ClusterStore::with_io_scope returns the store without a scope, so release_converged_deployment returns early and keeps the lock. I searched docs/ (without release notes and RFCs), skills/, changelog.d/ and the CLI source with git grep for lines that name both "converge" and "lock". No other passage leaves out Azure. The example at references/cluster.md:64 uses a local root, so it does not change.

Gates: scripts/check-docs.py (503 files), scripts/check-agents-md.sh, cargo fmt --all --check and typos on the four files pass. cargo dev-nextest -E 'package(omnigraph-skill) | (package(omnigraph-cli) & test(/skill/))': 11 passed, 0 failed. I did not run the Rust suites again, because no code changed.

CI at 7178b6b3: 28 checks pass and 4 are skipped (release and tag jobs), run 43263925535. GitHub reports the branch mergeable.

The description now lists the architecture guide under "Docs and skill" and has a Response run: line for this run.

@ragnorc ragnorc left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review of 7178b6b

Scope. This review covers the delta since my review of c3271af9 (review 5872788053): one commit, 7178b6b3, which answers O4 and O5. I do not repeat earlier findings. The design verdict from the earlier reviews stands: the fix is at the right layer, and it reuses the settled-release proof that a clean server shutdown uses.

What changed. The commit edits four documentation files and no code: docs/dev/architecture.md, skills/omnigraph/SKILL.md, skills/omnigraph/references/commands.md and skills/omnigraph/references/server-policy.md. Each now says that a converged direct apply releases its lock, except on Azure.

Verdict. Ready to merge. O4 and O5 are resolved. No blocking findings and no new optional findings.

Replies to the earlier items

  • O4: resolved. docs/dev/architecture.md:229-233 no longer says the delivery-failure path returns a "retained lock". It now says the lock follows the outcome of the apply. I checked this against the code. The CLI lets the core call finish before it returns the delivery failure, and that core call is apply_direct_bundle, which calls release_converged_deployment for a converged result (crates/omnigraph-cluster/src/deployment/execution.rs:1709). That function returns early without a storage scope, which is the Azure case (execution.rs:1725-1727). The test that owns this path asserts status.lock_id.is_none() after the converged apply (crates/omnigraph-cli/src/app_tests.rs:1701-1703). If settlement fails, the core call returns an error that names the held lock, so the sentence stays true.
  • O5: resolved. The three passages now name the Azure exception: SKILL.md:324-326, references/commands.md:505-506 and references/server-policy.md:26-28. They agree with SKILL.md:204, references/cluster.md:74-79, docs/user/clusters/index.md:76-82, the custody line (crates/omnigraph-cli/src/commands/cluster/mod.rs:325) and the changelog note.

Blocking

None.

Optional

None new.

Validation

All checks ran at exact head 7178b6b3, in a clean detached worktree.

  • Search for stale lock text. I searched docs/ (without release notes, RFCs and specs) and skills/ for "retain(s/ed) the lock", "keep(s) its lock", "release(s) its lock" and every force-unlock line. Each statement now agrees with the code. docs/user/operations/s3-cluster-roots.md:18-23 is about S3 only, so it correctly omits Azure. docs/dev/testing.md:299 is about a failed owned write, which still keeps the lock.
  • scripts/check-docs.py with the pinned packages from scripts/requirements-docs.txt: "Documentation OK (503 Markdown files checked)".
  • scripts/check-agents-md.sh: OK.
  • cargo dev-nextest -E 'package(omnigraph-skill) | (package(omnigraph-cli) & test(/skill/))' (debug, RUST_MIN_STACK=16777216): 11 passed, 0 failed.
  • CI at 7178b6b3: 28 passed, 4 skipped (release and tag jobs), run 43263925535.
  • Merge check. git merge-tree against main 1715b88a is clean. It is also clean with #1322, #1344 and #1349, which touch the same skill or developer-guide files. #952, #1175, #1235 and #1343 conflict with this branch only in the files where they already conflict with main.
  • GQT and regression proof. Not applicable to this delta: it changes no behavior. The regression proof for the code change is in the earlier reviews.
  • Not run: the Rust suites for the cluster crate, clippy, typos, and the S3 and Azure suites. No code changed since c3271af9, where I ran the cluster suite.

Liability

This commit adds no state, API, invariant or test. It removes a small documentation liability: the skill that omnigraph skill export ships no longer tells an agent on Azure that a converged apply released its lock. The remaining known cost is unchanged from my earlier review: two release call sites with different Azure checks until #1082.

@ragnorc

ragnorc commented Nov 13, 2026

Copy link
Copy Markdown
Contributor Author

Reply to the review of 7178b6b3 (review 5880539030). I agree that O4 and O5 are resolved and that nothing is blocking. I made no change, so the head stays at 7178b6b3.

O4 and O5: agreed. I read the cited lines at 7178b6b3. A converged result goes to release_converged_deployment (crates/omnigraph-cluster/src/deployment/execution.rs:1708-1709). That function returns without a release when the admission has no storage scope, which is the Azure case (execution.rs:1725-1727). crates/omnigraph-cli/src/app_tests.rs:1703 asserts that status.lock_id is none after the converged apply.

Main: main is now b1d569bc, 12 commits after this branch's merge base. git merge-tree --write-tree origin/main 7178b6b3 is clean. The only file that both sides change is docs/dev/testing.md, where #1365 edits an unrelated table row.

CI at 7178b6b3: 28 passed, 4 skipped (run 43263925535). GitHub reports the pull request as mergeable and clean.

Conflict in skills/omnigraph/SKILL.md: main removed the repair command
(ModernRelay#1372) on the same line where this branch says a converging direct
cluster apply releases the admission lock. Keep this branch's lock text
and drop repair from the maintenance command list.
@ragnorc

ragnorc commented Nov 14, 2026

Copy link
Copy Markdown
Contributor Author

The merge conflict with main is resolved in 1d879b17, a merge of main 9d0cbb7d into this branch. There was no rebase and no force-push. No review or comment was waiting for an answer.

Conflict: one line in skills/omnigraph/SKILL.md. Main removed the repair command in #1372 (35b046dd), on the same line where this branch says a converging direct cluster apply releases the admission lock. I kept this branch's lock text and removed repair from its list of maintenance commands, as main did. git grep finds no other place where this branch brings back a mention of repair. All other files merged without conflict.

Local checks on 1d879b17:

  • cargo dev-build-tests builds.
  • cargo dev-nextest -E 'package(omnigraph-cluster) | test(/cluster|issue_1359|repair|lock/)': 418 of 428 pass. The 10 failures also fail on main 9d0cbb7d on this machine. Eight are server-start timeouts in the system journeys. Two are symlink checks (bounded_reads_reject_special_files_and_symlinked_ancestors_without_blocking, managed_lifecycle_capture_and_retry_records_refuse_symlinks_without_reading_targets). These failures come from this machine's environment, not from this change. CI runs these tests and they pass there.
  • cargo fmt --all --check, scripts/check-docs.py and scripts/check-release-note.py pass.

CI at 1d879b17: 27 passed, 4 skipped, including Test Workspace, Lint (clippy) and CI Gate (run 43537820604), DST (run 43537820485) and GQ Logic Tests (run 43537820600). GitHub reports the pull request as mergeable and clean.

Conflict in skills/omnigraph/references/commands.md: main added the graph
shallow-clone and --stats text (ModernRelay#1416) in the paragraph where this branch
says a converged direct cluster apply releases its admission lock. Keep
main's new text and this branch's lock sentence.
@ragnorc

ragnorc commented Nov 15, 2026

Copy link
Copy Markdown
Contributor Author

The merge conflict with main is resolved in 7e14cc39, a merge of main 4f7c2076 into this branch. There was no rebase and no force-push. No review or comment was waiting for an answer.

Conflict: one paragraph in skills/omnigraph/references/commands.md. Main added the text about graph shallow clones (--base) and --stats in #1416 (4f7c2076). This branch changed the next sentence to say that a converged direct cluster apply releases its admission lock (never on Azure). I kept main's new text and this branch's sentence about the lock. All other files merged without conflict. I read the lock passages that main's #1408 (ca337ff9) changed in the skill and the guides. None of them says again that a converged apply keeps the lock.

Local checks on 7e14cc39:

  • cargo dev-build-tests builds.
  • cargo dev-nextest -E 'package(omnigraph-cluster) | test(/cluster|issue_1359|lock/)': 418 of 428 pass. The 10 failures also fail on main 4f7c2076 on this machine: eight server-start timeouts in the system journeys and two symlink checks. They are the same 10 as in the last merge. CI runs these tests and they pass there.
  • cargo fmt --all --check, scripts/check-docs.py, scripts/check-agents-md.sh and scripts/check-release-note.py pass.

CI at 7e14cc39: 27 passed, 4 skipped, including Test Workspace, Lint (clippy) and CI Gate (run 43775822436), DST (run 43775822427) and GQ Logic Tests (run 43775822365). GitHub reports the pull request as mergeable and clean.

Conflict in docs/user/operations/s3-cluster-roots.md: main added the
paragraph saying a direct write or maintenance command is judged by the
graph's applied policy (ModernRelay#1423), just before the admission-lock paragraph
this branch rewrote. Keep main's new paragraph and this branch's lock text.
@ragnorc

ragnorc commented Nov 17, 2026

Copy link
Copy Markdown
Contributor Author

I resolved the merge conflict with main in b3376293, which merges main 54ced5d3 into this branch. I did not rebase or force-push. No review or comment was waiting for an answer.

The conflict was in one place, docs/user/operations/s3-cluster-roots.md. Main's #1423 added a paragraph before the admission-lock paragraph. It says that the graph's applied policy judges a direct write or maintenance command, so --as must name an actor the policy grants. This branch had rewritten the lock paragraph to say that a converged cluster apply releases the lock. I kept main's new paragraph and then this branch's lock text.

All other files merged without a conflict. #1423 also changed admission.rs (it added graph_policy) and the CLI cluster test support. That change only adds code. The test helper unlock_cluster_fixture still returns early when no lock file exists, so it works with an apply that releases its lock.

Local checks on b3376293:

  • cargo dev-nextest -E 'package(omnigraph-cluster) | test(/cluster|issue_1359|lock|policy|direct/)': 620 of 639 pass. The 19 failures are 17 server-start timeouts and 2 symlink checks. The same 19 tests also fail on main 54ced5d3 on this 2-CPU machine, so they come from the machine and not from this change.
  • cargo fmt --all --check, scripts/check-docs.py (534 files), scripts/check-agents-md.sh and scripts/check-release-note.py pass.

CI at b3376293: 27 passed and 4 skipped (the tag-release jobs). This includes Test Workspace, Lint (clippy) and CI Gate (run 44010962261), DST (run 44010962111) and GQ Logic Tests (run 44010962275). GitHub reports the pull request as mergeable and clean. Nothing needs a decision.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

cluster apply on a direct root keeps its lock, so a first local server needs force-unlock

1 participant