Repository navigation
Conversation
A direct cluster apply kept its admission lock after it succeeded, so the first local server refused to start until cluster force-unlock released it. The apply now runs its control and graph writes in one storage scope. When the deployment completes converged, it releases the lock through the same settlement check a clean server shutdown uses: the scope closes, every accepted request settles, and none is uncertain. A failed, outstanding or unconverged deployment, an uncertain scope, and Azure keep the lock as before. Closes ModernRelay#1359
The RustFS journeys still force-unlocked after a converged direct apply, which now leaves no lock on S3 either.
…journey Azure keeps its stopped-process boundary, so a converged direct apply there still retains the lock; only S3 releases it.
ragnorc
left a comment
There was a problem hiding this comment.
Review of cdda4e0
Why. After a direct cluster apply, the first server refused to start until the operator ran cluster force-unlock (#1359).
What. A direct apply now runs all its writes in one StorageIoScope. When the deployment ends Complete with converged: true, it releases its lock. It uses the same settlement check a clean server shutdown uses: close the scope, wait for idle, refuse if a request is uncertain. Azure, failed, outstanding and unconverged results keep the lock as before.
How it fits. The fix is at the right layer. apply_direct_bundle owns the deployment's admission, so it is the one place that knows both the logical outcome (converged) and the storage outcome (the scope). It adds no new state and no new API. It reuses release_after_settlement → ClusterStore::release_settled, which already closes and drains a tracked scope and refuses cluster_admission_io_uncertain. I checked the claim that every write of the deployment is in the scope: the lock and ledger writes use store.adapter, which with_io_scope rebinds, and every graph open and create in deployment/execution.rs goes through store.graph_storage(...), which uses self.io_scope(). No Omnigraph::open(uri) or unscoped storage_for_uri call remains in non-test cluster code. bootstrap_ledger runs before the scope, but it holds and releases its own lock after awaited control writes, as on main.
A good side effect: S3 direct applies now use the single-attempt client for conditional and mutable keys. On main, a hidden retry of a conditional ledger or lock write could not prove what happened to the first attempt. Now such a failure ends the apply with the lock kept. That is the scope's documented contract, and the server already runs this way.
Verdict. The design is correct and I found no correctness defect. One small blocking item: two skill statements still contradict the new behavior (B1). The rest is optional.
Blocking
B1. Two skill statements still say a direct apply keeps its lock. AGENTS.md asks that every skills/omnigraph statement about a changed behavior is reconciled in the same change, because omnigraph skill export ships it with the binary.
skills/omnigraph/SKILL.md:204: "Each direct writer on the root takes the admission lock and keeps it; once that writer has stopped and its I/O has settled, release its exactlock_id... before the next writer or the server starts." This line is about the inlines3://root, where a converged apply now releases.skills/omnigraph/references/cluster.md:74-75: "Direct apply creates graphs, not rows ... It retains its exact admission lock after completion." This contradicts lines 46-48 of the same file, which the PR updated.
Also stale, but a developer guide: docs/dev/testing.md:216 says direct-apply delivery tests inspect the "retained lock". direct_deployment_delivery_failure_retains_one_result_and_original_identity now asserts release and then models a later owner's lock.
Optional
O1. Give "settle, then release" one owner. The rule now lives in two places with two different checks:
- Server (
crates/omnigraph-server/src/lib.rs:4322-4345): testscanonical_root().starts_with("az://")for Azure, closes and drains the scope, checksis_uncertain()itself, waits withwait_for_exclusive_owner(), then callsrelease_after_settlement. - Direct apply (
execution.rs:1721-1738): testsio_scope().is_none()for Azure, and relies on the uncertainty check insideClusterStore::release_settled. It does not wait for the exclusive owner; a live clone givescluster_admission_in_useat once. That is fail-closed and no clone exists on this path today.
The PR names maintenance (#1082) and reconciliation as next users. A single ClusterAdmission method (for example one that returns Released or RetainedUntracked, or an error) would hold the Azure rule, the scope check and the exclusive-owner wait in one place, and both callers would use it. Also, the doc comment on release_after_settlement (admission.rs:390-399) still says "This method does not establish that proof". With a tracked scope it now does establish the storage half. Please say what it checks and what the caller must still prove (the logical outcome).
O2. Report what happened to the lock. The custody line is printed before execution (report_id), so it can only say "released when the deployment converges and its writes settle, else retained". Two cases read wrong:
- On Azure a converged apply keeps the lock, but nothing printed after completion says so. The operator sees
converged: trueand the line above, and can conclude the lock was released. - If release fails, the CLI prints
{"ok": false, "diagnostics": [...]}for a deployment that converged. The message does name the deployment and the lock, which is good, but a script that readsoksees a failed apply.
Consider one line after completion ("Admission lock released" or "Admission lock retained: "), or a lock field in the result. Then the output is the one source of truth for the next step.
O3. Pin the uncertainty refusal with a test. The uncertainty branch in ClusterStore::release_settled (store.rs:1250-1262) had no test anywhere. The server checks uncertainty before it calls this, so the direct apply is the first caller that depends on it. I checked it with a temporary test in the admission.rs tests module, and then removed the test:
#[tokio::test]
async fn review_probe_uncertain_scope_keeps_the_lock() {
let dir = state_fixture(2, false);
let root = dir.path().to_str().unwrap();
let store = crate::store::ClusterStore::for_storage_root(root).unwrap()
.with_io_scope(omnigraph_storage::StorageIoScope::new()).unwrap();
let owner = acquire_with_store(&store, ClusterAdmissionPurpose::Deployment)
.await.unwrap().unwrap();
let graph = dir.path().join("graphs/knowledge.omni");
let adapter = owner.store().graph_storage(graph.to_str().unwrap()).unwrap();
let payload = "x".repeat(64 << 20);
let write = adapter.write_text(graph.join("_refs/probe.json").to_str().unwrap(), &payload);
let _ = tokio::time::timeout(std::time::Duration::ZERO, write).await; // abandon a mutable-key write
let error = owner.release_after_settlement().await.unwrap_err();
assert_eq!(error.code, "cluster_admission_io_uncertain");
assert!(dir.path().join("__cluster/lock.json").exists());
}Result at cdda4e0: uncertain=true pending=0, release refused with cluster_admission_io_uncertain, and the lock file stayed. It passes. A test like this would stop a later edit from removing the check inside release_settled because "the caller already checked", as the server does.
Validation
All at exact head cdda4e0, in a clean detached worktree. Debug profile, cargo dev-nextest feature selection, RUST_MIN_STACK=16777216, rustc 1.97.1, 2-core box.
cargo dev-nextest -E 'package(omnigraph-cluster) | (package(omnigraph-cli) & (test(/issue_1359/) | test(direct_deployment_delivery_failure) | test(cluster_e2e_offline_deployment_has_root_only_receipts_and_explicit_unlock) | test(cluster_force_unlock_wrong_id_exits_nonzero)))': 183 passed, 0 failed. That is 179 cluster tests, includingtests/failpoints.rsandtests/identity_recovery.rs, plus the 4 CLI tests.- Regression proof. I reverted only
crates/omnigraph-cluster/src/deployment/execution.rsandcrates/omnigraph-cli/src/commands/cluster/mod.rsto the merge base 825213b and kept the PR's tests:deployment::tests::bootstrap_and_add_graph_use_one_v2_protocol_without_reset_or_unlock_issue_1359failed attests.rs:66:left: Some("01M7715MRJFBF4EE9Z7KBQ1PWQ"),right: None. This is the predicted reason: the converged apply kept its lock.cli_cluster::cluster_apply_bootstraps_v2_and_releases_its_lock_issue_1359failed atcli_cluster.rs:3106:__cluster/lock.jsonstill exists.- With the files restored (
git statusclean), both pass.
- GQT. This contract cannot be expressed in GQT. It is about cluster admission and lock custody between processes, not query rows, counts or errors. The owners are the cluster's in-source deployment tests and the CLI's
cli_cluster, as the PR uses. - CI at cdda4e0: 28 passed, 4 skipped (tag/release jobs). Clippy, fmt, docs and the S3 journey ran there. I did not run them locally.
- Code inspection. Every graph open and create in
deployment/execution.rsusesstore.graph_storage. Lock and ledger writes use the reboundstore.adapter. Leases are server-only (lease.rs:1-4), so the new scope adds no lease fence to a direct apply. - Merge check.
git merge-treeis clean against main ad41b29. Against the other open PRs, #1360 adds no conflicts: #1318, #1341, #1345, #1346, #1348, #1349 and #1357 merge clean, and #1175, #1184, #1217, #1234, #1235 and #1316 conflict only in files this PR does not touch. #1217 changes the S3 lease and takeover. It does not interact with this release path, because the released marker is still written by exact-version CAS. - Not run: the S3 and Azure suites (no bucket or container here), and a fault injected into the apply itself (see O3 for the store-level check).
typoswas not run.
Liability
Removes: a manual step from every first-run guide, and a scripted force-unlock that parsed cluster status output. Adds: one more release call site with its own Azure test, and a breaking change for scripts that still force-unlock after apply (the PR says this; state_lock_missing is a clear failure). No new state, API or invariant. After five more changes of this kind (maintenance per #1082, load, reconciliation), the risk is five copies of "if scoped, release, map the error". O1 removes that risk.
…in the uncertain release Answer the review of cdda4e0 (ModernRelay#1360): - B1: SKILL.md and references/cluster.md no longer say a converged direct apply keeps its lock; docs/dev/testing.md says the delivery tests inspect the released lock. - O1: release_after_settlement's doc comment says what it proves with a tracked scope and what the caller must still prove. - O2: the custody line and changelog note say Azure never releases. - O3: admission test that an in-flight write abandoned in the scope makes the release refuse cluster_admission_io_uncertain and keep the lock.
|
Reply to the review of B1: valid, fixed. Both skill statements contradicted the change, and so did the testing guide.
O1: partly taken.
O2: partly taken.
O3: taken. Gates (debug,
I did not run the S3 or Azure suites locally; CI ran RustFS and Azurite. |
ragnorc
left a comment
There was a problem hiding this comment.
Review of c3271af
Scope. This review covers the delta since my review of cdda4e0c: one commit, c3271af9, which answers B1 and O1–O3. I do not repeat the earlier findings. The design verdict from that review stands: the fix is at the right layer and reuses the settled-release proof the server uses.
What changed. The commit edits two skill passages and the testing guide (B1), rewrites the doc comment on ClusterAdmission::release_after_settlement (O1), adds "(never on Azure)" to the custody line and the changelog note (O2), and adds admission::tests::uncertain_scope_keeps_the_lock_through_a_settled_release_issue_1359 (O3). It changes no production logic.
Verdict. Ready. B1 is resolved. No blocking findings. Two optional documentation items remain, both one-line edits.
Replies to the earlier items
- B1: resolved.
skills/omnigraph/SKILL.md:204andskills/omnigraph/references/cluster.md:74-79now agree with lines 46-48 ofcluster.mdand withdocs/user/clusters/index.md:76-82.docs/dev/testing.md:216now names the released lock. - O1: I accept the deferral. I checked the reason:
watchdogappears only incrates/omnigraph-server/src/lib.rs, not inomnigraph-cli/srcoromnigraph-cluster/src.wait_for_exclusive_owner(admission.rs:371-377) tells the caller to keep its own deadline. A shared method that waits would therefore need a deadline parameter, or it would turn a leaked clone into a hung CLI. Adding that method together with the next caller (#1082) is a reasonable time to do it. The new doc comment (admission.rs:390-401) is accurate:ClusterStore::release_settled(store.rs:1250-1263) closes the scope, waits for idle and refusescluster_admission_io_uncertain, which is what the comment says. - O2: I accept both parts.
ClusterStore::with_io_scopereturns the store unscoped on Azure (store.rs:191-193), sorelease_converged_deploymentreturns atio_scope().is_none()(execution.rs:1725-1727) and keeps the lock. The custody line now says so. Keepingok: falsewhen release fails is the safer exit for a script. - O3: resolved. The test is in the existing admission owner, as asked. See Validation for its before/after proof.
Blocking
None.
Optional
O4. One developer-guide sentence still says the lock is retained after a delivery failure. docs/dev/architecture.md:231: "the CLI captures any diagnostic delivery failure, lets that one core call finish, then returns the failure with its original ID, root and retained lock." The PR does not touch this file. The test that owns this path, app_tests::direct_deployment_delivery_failure_retains_one_result_and_original_identity, now asserts status.lock_id.is_none() after the converged apply (app_tests.rs:1701-1703). Suggested wording: "…returns the failure with its original ID and root; a converged apply has still released its lock."
O5. Three skill passages leave out the Azure exception. SKILL.md:204, references/cluster.md:74-79, the custody line, the changelog note and the user guide all say a converged apply never releases on Azure. These three do not:
skills/omnigraph/SKILL.md:324-329(Cluster deployment): "A direct apply that converges releases its admission lock once its writes have settled".skills/omnigraph/references/commands.md:505-506: "Each direct writer except a convergedcluster applykeeps its admission lock".skills/omnigraph/references/server-policy.md:24-28: the list of writers that keep the lock names "acluster applythat fails or does not converge" but not a converged apply on Azure.
An agent that reads one of these on Azure can skip the unlock step. The server then refuses with state_lock_held and names the lock, so this fails closed. It is still the kind of stale skill text AGENTS.md asks to reconcile in the same change, because omnigraph skill export ships it. Adding "(never on Azure)" to each passage makes all of them match.
Validation
All checks ran at exact head c3271af9, in a clean detached worktree. Debug profile, cargo dev-nextest feature selection, RUST_MIN_STACK=16777216, rustc 1.97.1, 2-core box.
cargo dev-nextest -E 'package(omnigraph-cluster) | package(omnigraph-skill) | (package(omnigraph-cli) & (test(/issue_1359/) | test(direct_deployment_delivery_failure) | test(cluster_e2e_offline_deployment_has_root_only_receipts_and_explicit_unlock) | test(cluster_force_unlock_wrong_id_exits_nonzero) | test(/skill_export/)))': 188 passed, 0 failed. That is 180 cluster tests (179 before, plus the new one), 1 skill test and 7 CLI tests.- Proof for the new test (O3). I changed only
crates/omnigraph-cluster/src/store.rs:1257toif false && scope.is_uncertain() {and rancargo dev-nextest -E 'test(=admission::tests::uncertain_scope_keeps_the_lock_through_a_settled_release_issue_1359)'. It failed atadmission.rs:880:called Result::unwrap_err() on an Ok value: (). That is the predicted reason: without the check, the release succeeds while a write is uncertain. Aftergit checkoutof the file (git statusclean, HEADc3271af9), the same command passed. This matches the author's reported result. - Determinism of the new test. The test abandons the write with
timeout(Duration::ZERO, …). The scoped local adapter writes throughobject_store's local backend, which runs the write as a blocking task. The first poll therefore returns pending, so the test does not depend on how fast the disk is. - GQT. This contract cannot be expressed in GQT. It is about admission and lock custody, not query rows, counts or errors. The in-source admission tests are the correct owner, as the earlier review said.
cargo fmt --all --checkandscripts/check-agents-md.shpass. I did not runscripts/check-docs.py(this box lacks its pinned Python packages), clippy,typos, or the S3 and Azure suites.- CI at
c3271af9: 28 passed, 4 skipped (tag and release jobs). - Merge check.
git merge-treeagainst main2bfb3f35is clean. - Code inspection. I checked the claims in the reply against code: the Azure no-op in
with_io_scope, the early return inrelease_converged_deployment, the settlement steps inrelease_settled, and the absence of a watchdog in the CLI and cluster crates. I searcheddocs/andskills/for statements that a direct apply keeps or retains its lock. O4 and O5 are what remains. The user guide (docs/user/clusters/index.md:76-82) is correct.
Liability
This commit adds no state, API or invariant. It adds one test (about 30 lines) to the existing owner. It removes a gap: before this commit, nothing tested the only check the direct apply depends on. The O1 deferral keeps one known cost: two release call sites with different Azure checks until #1082. The doc comment now records what each caller must prove, which limits that cost.
…pply releases its lock Answer the review of c3271af (ModernRelay#1360): - O4: docs/dev/architecture.md no longer says the CLI returns a delivery failure with a retained lock; the lock follows the apply's outcome, so a converged apply has still released it (never on Azure). - O5: SKILL.md (cluster deployment), references/commands.md and references/server-policy.md now say a converged direct apply keeps its lock on Azure, matching SKILL.md:204, references/cluster.md and the user guide.
|
Reply to the review of O4: valid, fixed. O5: valid, fixed. Each of the three passages now names the Azure exception:
They now agree with Gates: CI at The description now lists the architecture guide under "Docs and skill" and has a |
ragnorc
left a comment
There was a problem hiding this comment.
Review of 7178b6b
Scope. This review covers the delta since my review of c3271af9 (review 5872788053): one commit, 7178b6b3, which answers O4 and O5. I do not repeat earlier findings. The design verdict from the earlier reviews stands: the fix is at the right layer, and it reuses the settled-release proof that a clean server shutdown uses.
What changed. The commit edits four documentation files and no code: docs/dev/architecture.md, skills/omnigraph/SKILL.md, skills/omnigraph/references/commands.md and skills/omnigraph/references/server-policy.md. Each now says that a converged direct apply releases its lock, except on Azure.
Verdict. Ready to merge. O4 and O5 are resolved. No blocking findings and no new optional findings.
Replies to the earlier items
- O4: resolved.
docs/dev/architecture.md:229-233no longer says the delivery-failure path returns a "retained lock". It now says the lock follows the outcome of the apply. I checked this against the code. The CLI lets the core call finish before it returns the delivery failure, and that core call isapply_direct_bundle, which callsrelease_converged_deploymentfor a converged result (crates/omnigraph-cluster/src/deployment/execution.rs:1709). That function returns early without a storage scope, which is the Azure case (execution.rs:1725-1727). The test that owns this path assertsstatus.lock_id.is_none()after the converged apply (crates/omnigraph-cli/src/app_tests.rs:1701-1703). If settlement fails, the core call returns an error that names the held lock, so the sentence stays true. - O5: resolved. The three passages now name the Azure exception:
SKILL.md:324-326,references/commands.md:505-506andreferences/server-policy.md:26-28. They agree withSKILL.md:204,references/cluster.md:74-79,docs/user/clusters/index.md:76-82, the custody line (crates/omnigraph-cli/src/commands/cluster/mod.rs:325) and the changelog note.
Blocking
None.
Optional
None new.
Validation
All checks ran at exact head 7178b6b3, in a clean detached worktree.
- Search for stale lock text. I searched
docs/(without release notes, RFCs and specs) andskills/for "retain(s/ed) the lock", "keep(s) its lock", "release(s) its lock" and everyforce-unlockline. Each statement now agrees with the code.docs/user/operations/s3-cluster-roots.md:18-23is about S3 only, so it correctly omits Azure.docs/dev/testing.md:299is about a failed owned write, which still keeps the lock. scripts/check-docs.pywith the pinned packages fromscripts/requirements-docs.txt: "Documentation OK (503 Markdown files checked)".scripts/check-agents-md.sh: OK.cargo dev-nextest -E 'package(omnigraph-skill) | (package(omnigraph-cli) & test(/skill/))'(debug,RUST_MIN_STACK=16777216): 11 passed, 0 failed.- CI at
7178b6b3: 28 passed, 4 skipped (release and tag jobs), run 43263925535. - Merge check.
git merge-treeagainst main1715b88ais clean. It is also clean with #1322, #1344 and #1349, which touch the same skill or developer-guide files. #952, #1175, #1235 and #1343 conflict with this branch only in the files where they already conflict with main. - GQT and regression proof. Not applicable to this delta: it changes no behavior. The regression proof for the code change is in the earlier reviews.
- Not run: the Rust suites for the cluster crate, clippy,
typos, and the S3 and Azure suites. No code changed sincec3271af9, where I ran the cluster suite.
Liability
This commit adds no state, API, invariant or test. It removes a small documentation liability: the skill that omnigraph skill export ships no longer tells an agent on Azure that a converged apply released its lock. The remaining known cost is unchanged from my earlier review: two release call sites with different Azure checks until #1082.
|
Reply to the review of O4 and O5: agreed. I read the cited lines at Main: CI at |
Conflict in skills/omnigraph/SKILL.md: main removed the repair command (ModernRelay#1372) on the same line where this branch says a converging direct cluster apply releases the admission lock. Keep this branch's lock text and drop repair from the maintenance command list.
|
The merge conflict with main is resolved in Conflict: one line in Local checks on
CI at |
Conflict in skills/omnigraph/references/commands.md: main added the graph shallow-clone and --stats text (ModernRelay#1416) in the paragraph where this branch says a converged direct cluster apply releases its admission lock. Keep main's new text and this branch's lock sentence.
|
The merge conflict with main is resolved in Conflict: one paragraph in Local checks on
CI at |
Conflict in docs/user/operations/s3-cluster-roots.md: main added the paragraph saying a direct write or maintenance command is judged by the graph's applied policy (ModernRelay#1423), just before the admission-lock paragraph this branch rewrote. Keep main's new paragraph and this branch's lock text.
|
I resolved the merge conflict with main in The conflict was in one place, All other files merged without a conflict. #1423 also changed Local checks on
CI at |
Summary
A direct
cluster applythat converges now releases its cluster lock, so the first local server starts withoutcluster force-unlock. It uses the same check a clean server shutdown uses (#899): all of the apply's writes run in one storage scope, the scope closes, every request it accepted settles, and none is uncertain. Every other case keeps the lock as before.Closes #1359
Reproduction
On main
825213be, with the issue's steps (a fresh local root, one graph):The released 0.15.0 CLI shows the same
lock_idafter a converged apply. Verdict: reproduced.The failing test on main is
deployment::tests::bootstrap_and_add_graph_use_one_v2_protocol_without_reset_or_unlock_issue_1359(cluster crate). It fails at its first new assertion withleft: Some("01M76WWPM0R1E109VGKW4Y7DKR"),right: None.Root cause
apply_direct_bundle(crates/omnigraph-cluster/src/deployment/execution.rs) released the lock only when no effect had started. Once an apply created a graph or changed a schema, it returned with the lock held, because it had no proof that its writes had settled. The server has that proof:admit_serving_snapshotgives its store aStorageIoScope, and clean shutdown (crates/omnigraph-server/src/lib.rs) closes the scope, waits for idle, refuses on uncertainty and then callsrelease_after_settlement. The direct apply built its store without a scope, so it could not make the same check.Change
apply_direct_bundlegives its store a freshStorageIoScopeafterbootstrap_ledger, before admission. Graph opens and creation go throughstore.graph_storage, and ledger writes go through the store adapter, so every write of the deployment is in that scope.Completeresult withconverged: truecallsrelease_after_settlement. That call already closes the scope, waits for idle and returnscluster_admission_io_uncertainwhen a request is uncertain; it also refuses while another admission handle is alive.with_io_scopeis a no-op) keeps the lock, as the server does on Azure.Outstandingresult, an unconverged result, or an error keeps the lock, unchanged.Decisions
--release-lockflag. The issue offered both. The default is the only one that removes the step from every first-run guide, and it adds no API surface. The proof already exists and is what servers use to release on local storage and S3.apply --deployment-id --writers-stopped) keep their locks. They could use the same mechanism later.Before and after
After the change, the same steps:
After a clean SIGTERM,
lock_idis stillNone.Tests
bootstrap_and_add_graph_use_one_v2_protocol_without_reset_or_unlock_issue_1359. Bootstrap and a second apply that adds a graph now run with no manual unlock between them. The test asserts no lock after each converged apply, then admits a serving owner.cli_cluster::cluster_apply_bootstraps_v2_and_releases_its_lock_issue_1359(was..._retains_exact_owner): the CLI apply prints its custody line, then leaves nolock.json, andcluster statusreports a nulllock_id.tests/failpoints.rsthat is the newassert_releasedhelper.unlock_offlinestays strict after failed or reconciled applies, so those tests still prove failures keep the lock.offline_deployment_captures_schema_queries_and_retains_original_lookup,app_tests::direct_deployment_delivery_failure_retains_one_result_and_original_identity,cli_cluster_e2e::cluster_e2e_offline_deployment_has_root_only_receipts_and_explicit_unlock) now assert release first. They then model a later writer's kept lock: an admission dropped without release, orwrite_cluster_lock.unlock_storage_fixturetolerates a released lock.admission::tests::uncertain_scope_keeps_the_lock_through_a_settled_release_issue_1359: a write abandoned in flight makesrelease_after_settlementrefusecluster_admission_io_uncertainand keep the lock. The direct apply relies on that check inside the release, since it checks nothing before the call.cluster_e2e_s3_direct_root_deploys_unlocks_and_maintains_issue_1163and the sharedobject_storage_offline_deploymentins3_cluster.rsnow expect the lock to be released on S3. On the first push, RustFS showed the release on real S3: the old assertion found nolock_id. Azure runs the same journey and still expects the lock to be kept, because the apply has no tracked storage scope there.Local results on a 2-core box:
omnigraph-cluster: all 179 tests pass, including the failpoints and identity-recovery suites.omnigraph-serverandomnigraph-skill: all pass.cargo dev-clippy,cargo fmt --all --checkandscripts/check-docs.pypass.cdda4e0c: all 28 checks pass, including Test Workspace, RustFS S3 Integration (default and failpoints), Azurite Azure Integration and Storage Upgrade Compatibility.omnigraph-cli, 70 tests fail locally, and they fail the same way with the fix reverted. 68 are the support harness's 30-second server-readiness timeout ("server did not become ready or report its bound address", empty stderr). The other two aremanaged_lifecycle_capture_and_retry_records_refuse_symlinks_without_reading_targetsandsystem_local::cluster_server_boot_ignores_local_config_in_cwd("could not findCargo.toml"). CI is the evidence for those.Docs and skill
The user guides (
deployment.mdwriter topology,clusters/index.mddirect bootstrap,operations/s3-cluster-roots.md), the developer guides (control-plane.mdconcurrency, the invariants support boundary, thearchitecture.mdCLI delivery paragraph) andskills/omnigraph(SKILL.md,cluster.md,commands.md,server-policy.md) now say a converged direct apply releases its lock, and that Azure keeps it. The testing guide says the direct-apply delivery tests inspect the released lock. The CLI's custody line says the lock is never released on Azure. A changelog note is inchangelog.d/direct-apply-releases-lock.changed.md.Risks
force-unlock "$(cluster status ... | sed ...)") now finds no lock and fails withstate_lock_missing. Omnigraph Studio's first-run screen (spec 029) shows that step; it should drop it once this ships.release_settled), and the CLI system journeys this box cannot spawn.Workflow run: run-wf-reproduce-modernrelay-omnigraph-issue-20261112T0312Z-10668a
Response run: run-wf-respond-modernrelay-omnigraph-pr-20261112T0922Z-606fa2
Response run: run-wf-respond-modernrelay-omnigraph-pr-20261112T1637Z-d25653
Response run: run-wf-respond-modernrelay-omnigraph-pr-20261113T0002Z-0f999c
Response run: run-wf-respond-modernrelay-omnigraph-pr-20261114T0027Z-d64a8f
Response run: run-wf-respond-modernrelay-omnigraph-pr-20261115T2312Z-366c63
Response run: run-wf-respond-modernrelay-omnigraph-pr-20261117T0307Z-f021ec