You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
docs: operations/server.md doesn't say a restart needs cluster force-unlock #983
On 0.13.0 a server keeps the cluster admission lock after a clean SIGTERM. The log says so:
v2 cluster admission retained after shutdown; establish prior graph/control I/O quiescence before exact-ID force-unlock … lock_id=…
The next start then refuses:
[state_lock_held] __cluster/lock.json: cluster state lock already exists (lock id …); run `omnigraph cluster force-unlock …` only after confirming no cluster operation is active
A start that fails after taking the lock (for example, the port is already in use) leaves the lock behind too.
This is only explained under "Writer topology" in deployment.md. The "Start a server" section of operations/server.md has the start command and nothing else, so anyone following that page hits the error on their first restart.
Suggested fix: a short "Restarting" note in server.md:
the exact-ID unlock: omnigraph cluster force-unlock <LOCK_ID> --config .;
where the ID comes from: __cluster/lock.json, or the shutdown log line;
the same for a failed start.
#899 would make a clean shutdown release the lock. The note is still needed for 0.13.0 and for unclean exits.
Triage: reproduced on 0.13.0, partially reproduced on current main
Summary
A mutation-capable server holds the cluster's exclusive writer lock, __cluster/lock.json. When the server process stops and keeps that lock, the next omnigraph-server --cluster … start refuses with state_lock_held. The operator must then run an exact-ID cluster force-unlock. docs/user/operations/server.md ("Start a server") does not say this. The procedure is only in deployment.md ("Writer topology") and clusters/index.md.
0.13.0 (latest release): every exit keeps the lock. This includes a clean SIGTERM (exit 0) and a failed start. The report is correct.
Current main:feat(server): release native admission after settled clean shutdown #899 is merged (merge commit ff6fe551, 2026-10-14), not only closed. It is not released yet; 0.13.0 is still the latest release. On main, a clean SIGTERM on a local or S3 root releases the lock, and the next start works. A start that fails after it takes the lock still keeps the lock. Examples are a port that is already in use and a missing --unauthenticated. server.md on main still has no restart note.
So the docs gap is real on both versions. The content of the note must be different for 0.13.0 and for the next release.
Verdict
Step
0.13.0 release
main 4d7536d68b1db6da34bf9cff22a90d9fdd77bed4
Clean SIGTERM (exit 0), then the same start command
Clean SIGTERM, then restart: reproduced on 0.13.0, not reproduced on main. Failed start: reproduced on both. Missing note in server.md: reproduced on both.
Environment. Linux x86_64. Local file cluster root. OMNIGRAPH_HOME set to an empty directory. The server ran with --unauthenticated (except where noted) and --bind 127.0.0.1:<port>.
0.13.0: omnigraph-linux-x86_64.tar.gz from the v0.13.0 release. Its sha256 0123daa6…7a30 matches the published .sha256. omnigraph version prints omnigraph 0.13.0 / internal-schema 14 (serves v14 to v14).
main: cargo build --locked -p omnigraph-server -p omnigraph-cli (dev profile) in a clean worktree at 4d7536d68b1db6da34bf9cff22a90d9fdd77bed4. omnigraph version prints omnigraph 0.14.0 / internal-schema 16 (serves v14 to v16) / http-contract 0.13.
Method. One script runs the same steps on both binaries. The bundle has one graph (node Person { name: String @key }). The steps are: direct cluster apply, exact-ID unlock of the apply lock, start, SIGTERM, start again, then a failed start on a busy port, then SIGKILL.
0.13.0 output (excerpt)
$ omnigraph cluster apply --config . --json
Admission lock: 01M54N1J43K95V9GSHB0FVAT1V; retain until prior work is quiescent, then exact-ID force-unlock
$ omnigraph cluster force-unlock 01M54N1J43K95V9GSHB0FVAT1V --config .
cluster force-unlock: removed lock (id=01M54N1J43K95V9GSHB0FVAT1V, operation=deployment, ...)
### start, then clean SIGTERM
readyz: 200
server exit status after SIGTERM: 0
WARN omnigraph_server: v2 cluster admission retained after shutdown; establish prior graph/control I/O quiescence before exact-ID force-unlock root=file:///tmp/r983-013/c lock_id=01M54N1JYAPB4X47TJ21WH6V0F
$ cat __cluster/lock.json
{ "version": 1, "lock_id": "01M54N1JYAPB4X47TJ21WH6V0F", "operation": "serve", ... }
### restart with the same command
server exit status: 1
[state_lock_held] __cluster/lock.json: cluster state lock already exists (lock id 01M54N1JYAPB4X47TJ21WH6V0F); run `omnigraph cluster force-unlock 01M54N1JYAPB4X47TJ21WH6V0F` only after confirming no cluster operation is active
### unlock with the ID from the shutdown log line
$ omnigraph cluster force-unlock 01M54N1JYAPB4X47TJ21WH6V0F --config .
cluster force-unlock: removed lock (id=01M54N1JYAPB4X47TJ21WH6V0F, operation=serve, pid=5659, ...)
readyz: 200
### failed start: port already in use
$ omnigraph-server --cluster . --bind 127.0.0.1:18983 --unauthenticated
Error:
0: Address already in use (os error 98)
[exit 1]
$ cat __cluster/lock.json
{ "version": 1, "lock_id": "01M54N1QYA7XSNMEV81A67RRM5", "operation": "serve", ... }
(next start: [state_lock_held] ... lock id 01M54N1QYA7XSNMEV81A67RRM5)
$ omnigraph --cluster file:///tmp/r983-013/c cluster force-unlock 01M54N1QYA7XSNMEV81A67RRM5
{ "unlocked": true, "lock_id": "01M54N1QYA7XSNMEV81A67RRM5" }
### failed start: no credentials and no --unauthenticated
0: server has no bearer tokens and no policy file configured. This is a fully open server — pass `--unauthenticated` ...
cluster status: locked=True lock_id=01M54N8K8GGEARJHR1B6BP3HZH lock_operation=serve
### start, then clean SIGTERM
readyz: 200
server exit status after SIGTERM: 0
INFO omnigraph_server: shutdown signal received
__cluster/lock.json: absent
### restart with the same command
readyz: 200
server exit status after SIGTERM: 0
__cluster/lock.json: absent
### failed start: port already in use
$ omnigraph-server --cluster . --bind 127.0.0.1:18984 --unauthenticated
Error:
0: Address already in use (os error 98)
[exit 1]
$ cat __cluster/lock.json
{ "version": 1, "lock_id": "01M54N4TKW8FY40MKPNK1F6VQ5", "operation": "serve", ... }
### restart after the failed start
[state_lock_held] __cluster/lock.json: cluster state lock already exists (lock id 01M54N4TKW8FY40MKPNK1F6VQ5); run `omnigraph cluster force-unlock 01M54N4TKW8FY40MKPNK1F6VQ5` only after confirming no cluster operation is active
### failed start: no credentials and no --unauthenticated
0: server has no bearer tokens and no policy file configured. ...
cluster status: locked=True lock_id=01M54N80FGBEYP2Y2N9A6H62PX lock_operation=serve
### failed start: OMNIGRAPH_SERVER_BEARER_TOKENS_JSON='{not json'
cluster status: locked=True lock_id=01M54N80HADFYEBNXKACYDAJH5 lock_operation=serve
### SIGKILL
server exit status after SIGKILL: 137
[state_lock_held] ... (lock id 01M54N4X997GZ6G7A2PB25MNQV)
$ omnigraph cluster status --config . --json
"state_observations": { "locked": true, "lock_id": "01M54N4X997GZ6G7A2PB25MNQV", "lock_operation": "serve", "lock_pid": 6630, ... }
The shutdown log line is a 0.13.0-only lock-ID source. It appears only on a clean exit. It does not appear for a failed start on either version. Main removed it (0.13.0 lib.rs#L2663-L2667). On main, the lock-keeping paths log no lock ID.
__cluster/lock.json is a correct source for local roots only. On S3, main keeps the key after a clean release and writes a "released" marker into it (state_lock.rs#L200-L215, #L254-L265). The raw file then exists, but it is not a held lock. cluster status and cluster observe ignore a released marker: they report locked: false and no lock_id (types.rs#L124-L133). These two sources work on both versions and on every backend. The state_lock_held message itself also prints the ID.
There are more failed-start triggers than a busy port. A missing --unauthenticated (with no tokens) and a malformed token JSON also keep the lock. A first-time operator can easily make the first mistake.
The suggested command is correct on both versions: omnigraph cluster force-unlock <LOCK_ID> --config <dir> (default --config .). The root-addressed form omnigraph --cluster <root-uri> cluster force-unlock <LOCK_ID> also works and is what clusters/index.md shows. The hint in state_lock_held omits the address (store.rs#L1356). From another directory it fails safely with cluster_config_read_error cluster.yaml.
The first start after a direct cluster apply also needs an unlock, because apply keeps its own lock. clusters/index.md already documents this, so it is not part of this gap.
Regression proof on main (local only, not pushed)
GQT cannot express this contract: it is a process-lifecycle and storage-lock property, not rows or errors from a query. The docs checker (scripts/check-docs.py) checks links and locations, not content. The existing owner is crates/omnigraph-server/tests/boot_settings.rs. It owns server startup refusals and #899's clean-restart regression (owned_shutdown::disconnected_write_and_shutdown_share_ownership_v2_finish). I added one test there. It boots the owner's v2 fixture through the production load_server_settings + serve. Then it checks that a start refused before any graph opens leaves no lock.
Baseline: v2_settings_capture_holds_admission_across_drop_and_refuses_another_server and owned_shutdown::disconnected_write_and_shutdown_share_ownership_v2_finish: 2 passed; 0 failed.
New test on unmodified main: 0 passed; 1 failed. Both cases keep the lock:
starts refused before any graph open retained admission:
occupied-listener: refused with `Address already in use (os error 98)`; next start: the cluster at '/tmp/.tmpVSEzs6' is not ready to serve:
[state_lock_held] __cluster/lock.json: cluster state lock already exists (lock id 01M54NYNZ1E28EJG9F91C1CQNB); run `omnigraph cluster force-unlock 01M54NYNZ1E28EJG9F91C1CQNB` only after confirming no cluster operation is active
open-server-without-opt-in: refused with `server has no bearer tokens and no policy file configured. ...`; next start: the cluster at '/tmp/.tmpDUZJxY' is not ready to serve:
[state_lock_held] __cluster/lock.json: cluster state lock already exists (lock id 01M54NYNZQXW4XBDRSBKCE8QVX); ...
test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 42 filtered out
Important: this test describes the proposed contract (fix B below). It is not a currently documented one. The control-plane guide says today that "Only unique admission, successful startup and unpoisoned settlement permit final release" (control-plane.md#L233). So the failure shows the current behaviour precisely. It is the regression test for fix B, not proof of a code bug against the written contract. The docs fix A closes this issue without it.
Test source (rustfmt clean; Clippy not run)
/// A start refused before any graph opens has no accepted work to settle, so/// the next ordinary start must not need `cluster force-unlock` (issue #983).#[tokio::test(flavor = "multi_thread")]asyncfnv2_server_start_refused_before_graph_open_releases_admission_issue_983(){letmut retained = Vec::new();for case in["occupied-listener","open-server-without-opt-in"]{let temp = converged_cluster_dir("").await;
omnigraph_cluster::upgrade_deployment_ledger(&format!("file://{}", temp.path().display()),true,&omnigraph_cluster::DeploymentCaller::storage_owner(None),).await.unwrap();let occupied = std::net::TcpListener::bind("127.0.0.1:0").unwrap();letmut settings = omnigraph_server::load_server_settings(Some(&temp.path().to_path_buf()),Some(if case == "occupied-listener"{
occupied.local_addr().unwrap().to_string()}else{"127.0.0.1:0".into()}),
case == "occupied-listener",false,).await.unwrap();
settings.shutdown_grace = std::time::Duration::from_secs(5);let refusal = omnigraph_server::serve(settings).await.unwrap_err();let lock = temp.path().join("__cluster/lock.json");if lock.exists(){let refusal = refusal.to_string();
retained.push(format!("{case}: refused with `{}`; next start: {}",
refusal.lines().next().unwrap_or_default(),
cluster_settings(temp.path()).await.unwrap_err()));continue;}cluster_settings(temp.path()).await.unwrap().cluster_admission.unwrap().release_after_settlement().await.unwrap();}assert!(
retained.is_empty(),"starts refused before any graph open retained admission:\n{}",
retained.join("\n"));}
Root cause
The lock is kept on purpose by default. Dropping an admission or returning from writable work keeps the persisted lock (control-plane.md#L190-L199). A test pins this (boot_settings.rs#L45-L87). A crash or cancellation therefore never looks like a settled owner. This is correct and must stay.
0.13.0 has no release path for a server. Shutdown only logs the WARN line with the lock ID (0.13.0 lib.rs#L2663-L2667).
Main releases only at the end of a successful serve. The release block runs after HTTP drain and graph startup finish. It closes the storage scope, refuses uncertain I/O, waits for unique ownership and calls release_after_settlement (lib.rs#L3007-L3032). Azure is skipped on purpose.
Process-local checks run after admission and return with ?.load_server_settings* in main.rs takes the admission first. Read-only settings refusals in that phase already release it (settings.rs#L47-L61, #L471-L481). But serve_config then runs more checks that need no storage: token source and load (lib.rs#L2864-L2866), session defaults (#L2867-L2868), the auth classification (#L2886-L2890) and the listener bind (#L2946). Each one returns early and skips the release block. Graph opens start only after the bind (#L2971-L2980). So these failures have no graph or control effect to settle. The lock is kept only because of where the checks are in the code, not because of any data risk.
A. Docs, now (closes this issue). Add a short "Stop and restart" subsection under "Start a server" in server.md. Write it for both versions and link to "Writer topology" for the rules. Proposed text:
### Stop and restart
A mutation-capable server holds the cluster's exclusive writer lock. Stop it
with SIGTERM.
- From the release after 0.13.0, a clean stop (exit 0) on a local or S3 root
releases the lock. Start the next server with the same command.
- These exits keep the lock: exit 2 (grace deadline or uncertain work), SIGKILL,
a crash, every exit on an Azure root, and a start that fails after it took
the lock (for example `Address already in use`, or no tokens and no
`--unauthenticated`). In 0.13.0, every exit keeps the lock, including a clean
stop.
The next start then refuses with `state_lock_held` and shows the lock ID. To
read the ID without changing anything:
```bash
omnigraph cluster status --config ./company-brain --json # state_observations.lock_id```
Make sure that the previous process has exited and that its writes are
complete. Then remove that exact lock and start again:
```bash
omnigraph cluster force-unlock <LOCK_ID> --config ./company-brain
# or by root: omnigraph --cluster s3://company-data/omnigraph/company-brain cluster force-unlock <LOCK_ID>```
Do not delete `__cluster/lock.json` yourself. On S3 the file stays after a clean
release. See [writer topology](../deployment.md#writer-topology).
Why this shape: cluster status is the one ID source that works on 0.13.0 and main, on every backend, and after a failed start. The 0.13.0 log line and the raw file each fail one of those cases. Naming the version inline is the per-section marking that #985 asks for. In the same change, or at release, update the skill text in point 6 so that it agrees with the release that ships it. If fix B lands, remove the failed-start bullet for those checks.
B. Code, as a separate change: run process-local preflight before admission (correct by design). Move every check that needs no cluster storage in front of the admission acquire in main.rs / load_server_settings*. These checks are token source and load, ProcessDefaults::from_env, classify_server_runtime_state and TcpListener::bind. Pass the bound listener into serve. The rule becomes "no fallible process-local check after the writer lock is taken". One placement enforces this rule, and new checks get it automatically. With --bootstrap-handoff the same ordering is also correct: the check fails before the claim, so the claimed lock that must never disappear is never created. The local test above is its regression test.
Tradeoffs: the socket is bound during settings load. Connections queue in the kernel backlog and get no response until axum::serve starts. Today this happens too, but for a shorter time. Probes time out instead of getting "connection refused". Update the deployment.md sentence "The listener opens after configuration and authorization validation". The token load (for example Secrets Manager) runs before the lock. It is read-only, so it is safe.
Liability: low. It is one ordering rule in one place. It adds no new release path and does not change the protocol or storage format.
Alternatives set aside:
Release in serve_config's early-error paths (like release_settings_refusal). This works, but each new check after admission must remember to call it. It also needs a flag so that bootstrap-claimed locks stay held, and it must drop state clones before Arc::try_unwrap. More ongoing liability than B.
Release on Drop. Rejected. Drop-retains is pinned on purpose, so that cancellation and abandonment never look settled.
Expire the lock by lease, age or PID, or add an "unlock stale" server flag. Rejected by the invariants: process-local locks are not distributed fencing, and lock age or a stopped PID is not proof of quiescence (deployment.md#L305-L307).
Unlock without an ID. Rejected. The exact ID stops an operator from removing a successor's lock.
C. Small diagnostics, optional. Log the lock ID when main keeps admission (the Azure, uncertain and failed-start paths), as 0.13.0 did on clean exit. Add the address to the state_lock_held hint (--config <dir> or --cluster <root>).
Quality traded: fix A keeps the operator burden for unclean exits. This is deliberate, because robustness (never unlock without proof) has priority over convenience.
Lance upstream
Lance does not cause or fix this issue. OmniGraph pins Lance =11.0.0 (Cargo.toml). The latest Lance release is v13.0.0, and the newest pre-release is v14.0.0-beta.19. __cluster/lock.json is an OmniGraph control object. OmniGraph's storage adapter writes it (local create-if-absent, S3 conditional put). It is not a Lance dataset or a Lance commit.
There is one indirect dependency. Main's clean release requires settled Lance I/O with no hidden retries. Scoped storage forces client_max_retries=0 and lance_aimd_max_retries=0 (lance_access/scoped.rs#L45-L73). Check these at the next Lance upgrade:
lance#9913 (merged 2026-10-14, after v13.0.0) moves multipart-complete retries to the HTTP layer. The loop appears to be bounded by the same max_retries value that OmniGraph sets to 0. I did not confirm the option mapping.
lance#9996 (open) fixes a possible hang in ObjectWriter::poll_write. Such a hang would end in exit 2 with the lock kept. I did not see it here.
lance#10033 (open) adds conditional multipart manifest uploads. That is a new multipart lifetime, which the storage scope must count.
lance#10008 (open) is about which storage options are honoured on reads. It is not relevant to the lock.
Activity
ragnorc commentedon Oct 17, 2026
Triage: reproduced on 0.13.0, partially reproduced on current main
Summary
A mutation-capable server holds the cluster's exclusive writer lock,
__cluster/lock.json. When the server process stops and keeps that lock, the nextomnigraph-server --cluster …start refuses withstate_lock_held. The operator must then run an exact-IDcluster force-unlock.docs/user/operations/server.md("Start a server") does not say this. The procedure is only indeployment.md("Writer topology") andclusters/index.md.ff6fe551, 2026-10-14), not only closed. It is not released yet; 0.13.0 is still the latest release. On main, a clean SIGTERM on a local or S3 root releases the lock, and the next start works. A start that fails after it takes the lock still keeps the lock. Examples are a port that is already in use and a missing--unauthenticated.server.mdon main still has no restart note.So the docs gap is real on both versions. The content of the note must be different for 0.13.0 and for the next release.
Verdict
4d7536d68b1db6da34bf9cff22a90d9fdd77bed4state_lock_heldAddress already in use (os error 98)--unauthenticatedOMNIGRAPH_SERVER_BEARER_TOKENS_JSONcluster force-unlock <LOCK_ID> --config .omnigraph --cluster file://<root> cluster force-unlock <LOCK_ID>lock_id=Clean SIGTERM, then restart: reproduced on 0.13.0, not reproduced on main. Failed start: reproduced on both. Missing note in
server.md: reproduced on both.Environment. Linux x86_64. Local file cluster root.
OMNIGRAPH_HOMEset to an empty directory. The server ran with--unauthenticated(except where noted) and--bind 127.0.0.1:<port>.omnigraph-linux-x86_64.tar.gzfrom the v0.13.0 release. Its sha2560123daa6…7a30matches the published.sha256.omnigraph versionprintsomnigraph 0.13.0/internal-schema 14 (serves v14 to v14).cargo build --locked -p omnigraph-server -p omnigraph-cli(dev profile) in a clean worktree at4d7536d68b1db6da34bf9cff22a90d9fdd77bed4.omnigraph versionprintsomnigraph 0.14.0/internal-schema 16 (serves v14 to v16)/http-contract 0.13.Method. One script runs the same steps on both binaries. The bundle has one graph (
node Person { name: String @key }). The steps are: directcluster apply, exact-ID unlock of the apply lock, start, SIGTERM, start again, then a failed start on a busy port, then SIGKILL.0.13.0 output (excerpt)
main 4d7536d output (excerpt)
Differences from the report:
lib.rs#L2663-L2667). On main, the lock-keeping paths log no lock ID.__cluster/lock.jsonis a correct source for local roots only. On S3, main keeps the key after a clean release and writes a "released" marker into it (state_lock.rs#L200-L215,#L254-L265). The raw file then exists, but it is not a held lock.cluster statusandcluster observeignore a released marker: they reportlocked: falseand nolock_id(types.rs#L124-L133). These two sources work on both versions and on every backend. Thestate_lock_heldmessage itself also prints the ID.--unauthenticated(with no tokens) and a malformed token JSON also keep the lock. A first-time operator can easily make the first mistake.omnigraph cluster force-unlock <LOCK_ID> --config <dir>(default--config .). The root-addressed formomnigraph --cluster <root-uri> cluster force-unlock <LOCK_ID>also works and is whatclusters/index.mdshows. The hint instate_lock_heldomits the address (store.rs#L1356). From another directory it fails safely withcluster_config_read_error cluster.yaml.cluster applyalso needs an unlock, because apply keeps its own lock.clusters/index.mdalready documents this, so it is not part of this gap.Regression proof on main (local only, not pushed)
GQT cannot express this contract: it is a process-lifecycle and storage-lock property, not rows or errors from a query. The docs checker (
scripts/check-docs.py) checks links and locations, not content. The existing owner iscrates/omnigraph-server/tests/boot_settings.rs. It owns server startup refusals and #899's clean-restart regression (owned_shutdown::disconnected_write_and_shutdown_share_ownership_v2_finish). I added one test there. It boots the owner's v2 fixture through the productionload_server_settings+serve. Then it checks that a start refused before any graph opens leaves no lock.v2_settings_capture_holds_admission_across_drop_and_refuses_another_serverandowned_shutdown::disconnected_write_and_shutdown_share_ownership_v2_finish:2 passed; 0 failed.0 passed; 1 failed. Both cases keep the lock:Command:
cargo test --locked -p omnigraph-server --test boot_settings -- --exact v2_server_start_refused_before_graph_open_releases_admission_issue_983Important: this test describes the proposed contract (fix B below). It is not a currently documented one. The control-plane guide says today that "Only unique admission, successful startup and unpoisoned settlement permit final release" (
control-plane.md#L233). So the failure shows the current behaviour precisely. It is the regression test for fix B, not proof of a code bug against the written contract. The docs fix A closes this issue without it.Test source (rustfmt clean; Clippy not run)
Root cause
control-plane.md#L190-L199). A test pins this (boot_settings.rs#L45-L87). A crash or cancellation therefore never looks like a settled owner. This is correct and must stay.lib.rs#L2663-L2667).serve. The release block runs after HTTP drain and graph startup finish. It closes the storage scope, refuses uncertain I/O, waits for unique ownership and callsrelease_after_settlement(lib.rs#L3007-L3032). Azure is skipped on purpose.?.load_server_settings*inmain.rstakes the admission first. Read-only settings refusals in that phase already release it (settings.rs#L47-L61,#L471-L481). Butserve_configthen runs more checks that need no storage: token source and load (lib.rs#L2864-L2866), session defaults (#L2867-L2868), the auth classification (#L2886-L2890) and the listener bind (#L2946). Each one returns early and skips the release block. Graph opens start only after the bind (#L2971-L2980). So these failures have no graph or control effect to settle. The lock is kept only because of where the checks are in the code, not because of any data risk.server.mdhas the start command only (server.md#L13-L44). The rules are indeployment.md(#L105-L122,#L300-L315). On main these describe feat(server): release native admission after settled clean shutdown #899's unreleased behaviour, and they do not mention failed starts.skills/omnigraph/references/cluster.mdsays a server keeps the lock "even after a clean exit" (#L78-L83,#L283).server-policy.mdsays the same (#L16-L20). This is correct for 0.13.0. It becomes wrong when feat(server): release native admission after settled clean shutdown #899 ships, and the CLI embeds this skill.Recommended fix
A. Docs, now (closes this issue). Add a short "Stop and restart" subsection under "Start a server" in
server.md. Write it for both versions and link to "Writer topology" for the rules. Proposed text:Why this shape:
cluster statusis the one ID source that works on 0.13.0 and main, on every backend, and after a failed start. The 0.13.0 log line and the raw file each fail one of those cases. Naming the version inline is the per-section marking that #985 asks for. In the same change, or at release, update the skill text in point 6 so that it agrees with the release that ships it. If fix B lands, remove the failed-start bullet for those checks.B. Code, as a separate change: run process-local preflight before admission (correct by design). Move every check that needs no cluster storage in front of the admission acquire in
main.rs/load_server_settings*. These checks are token source and load,ProcessDefaults::from_env,classify_server_runtime_stateandTcpListener::bind. Pass the bound listener intoserve. The rule becomes "no fallible process-local check after the writer lock is taken". One placement enforces this rule, and new checks get it automatically. With--bootstrap-handoffthe same ordering is also correct: the check fails before the claim, so the claimed lock that must never disappear is never created. The local test above is its regression test.axum::servestarts. Today this happens too, but for a shorter time. Probes time out instead of getting "connection refused". Update thedeployment.mdsentence "The listener opens after configuration and authorization validation". The token load (for example Secrets Manager) runs before the lock. It is read-only, so it is safe.Alternatives set aside:
serve_config's early-error paths (likerelease_settings_refusal). This works, but each new check after admission must remember to call it. It also needs a flag so that bootstrap-claimed locks stay held, and it must drop state clones beforeArc::try_unwrap. More ongoing liability than B.Drop. Rejected. Drop-retains is pinned on purpose, so that cancellation and abandonment never look settled.deployment.md#L305-L307).C. Small diagnostics, optional. Log the lock ID when main keeps admission (the Azure, uncertain and failed-start paths), as 0.13.0 did on clean exit. Add the address to the
state_lock_heldhint (--config <dir>or--cluster <root>).Quality traded: fix A keeps the operator burden for unclean exits. This is deliberate, because robustness (never unlock without proof) has priority over convenience.
Lance upstream
Lance does not cause or fix this issue. OmniGraph pins Lance
=11.0.0(Cargo.toml). The latest Lance release is v13.0.0, and the newest pre-release is v14.0.0-beta.19.__cluster/lock.jsonis an OmniGraph control object. OmniGraph's storage adapter writes it (local create-if-absent, S3 conditional put). It is not a Lance dataset or a Lance commit.There is one indirect dependency. Main's clean release requires settled Lance I/O with no hidden retries. Scoped storage forces
client_max_retries=0andlance_aimd_max_retries=0(lance_access/scoped.rs#L45-L73). Check these at the next Lance upgrade:max_retriesvalue that OmniGraph sets to 0. I did not confirm the option mapping.ObjectWriter::poll_write. Such a hang would end in exit 2 with the lock kept. I did not see it here.I read the complete Lance object store, observability and transactions pages for this check.
Related work
changelog.d/native-clean-restart.changed.md. It did not changeserver.md, the failed-start paths or the skill.CONFLICTING, one commit from 2026-10-07, before feat(server): release native admission after settled clean shutdown #899): it rewritesdeployment.mdand the skill to say "Even a clean exit retains the cluster lock". That is correct for 0.13.0 but wrong for main. Its onlyserver.mdchange is aPOST /cluster/planroute row. It does not fix this issue, and this issue does not supersede it. A rebase of Refresh skills and operator guides for 0.13 #892 must keep feat(server): release native admission after settled clean shutdown #899's clean-release text, or mark it by version.control-plane.md#L223-L242(clean native shutdown), the server runtime RFCdocs/rfcs/2026-09-29-server-runtime-and-online-deployment.md, andinvariants.md#L178-L182.Supersession
None. No open issue or PR covers the
server.mdrestart note or the failed-start lock. This issue stays open.Limitations
boot_settingsbinary on it. The 0.13.0 malformed-token case was not run.