Skip to content

docs: operations/server.md doesn't say a restart needs cluster force-unlock #983

Open

Description

@pronskiy

On 0.13.0 a server keeps the cluster admission lock after a clean SIGTERM. The log says so:

v2 cluster admission retained after shutdown; establish prior graph/control I/O quiescence before exact-ID force-unlock … lock_id=…

The next start then refuses:

[state_lock_held] __cluster/lock.json: cluster state lock already exists (lock id …); run `omnigraph cluster force-unlock …` only after confirming no cluster operation is active

A start that fails after taking the lock (for example, the port is already in use) leaves the lock behind too.

This is only explained under "Writer topology" in deployment.md. The "Start a server" section of operations/server.md has the start command and nothing else, so anyone following that page hits the error on their first restart.

Suggested fix: a short "Restarting" note in server.md:

  • the exact-ID unlock: omnigraph cluster force-unlock <LOCK_ID> --config .;
  • where the ID comes from: __cluster/lock.json, or the shutdown log line;
  • the same for a failed start.

#899 would make a clean shutdown release the lock. The note is still needed for 0.13.0 and for unclean exits.

Activity

  1. ragnorc commented on Oct 17, 2026

    @ragnorc
    Contributor

    Triage: reproduced on 0.13.0, partially reproduced on current main

    Summary

    A mutation-capable server holds the cluster's exclusive writer lock, __cluster/lock.json. When the server process stops and keeps that lock, the next omnigraph-server --cluster … start refuses with state_lock_held. The operator must then run an exact-ID cluster force-unlock. docs/user/operations/server.md ("Start a server") does not say this. The procedure is only in deployment.md ("Writer topology") and clusters/index.md.

    • 0.13.0 (latest release): every exit keeps the lock. This includes a clean SIGTERM (exit 0) and a failed start. The report is correct.
    • Current main: feat(server): release native admission after settled clean shutdown #899 is merged (merge commit ff6fe551, 2026-10-14), not only closed. It is not released yet; 0.13.0 is still the latest release. On main, a clean SIGTERM on a local or S3 root releases the lock, and the next start works. A start that fails after it takes the lock still keeps the lock. Examples are a port that is already in use and a missing --unauthenticated. server.md on main still has no restart note.

    So the docs gap is real on both versions. The content of the note must be different for 0.13.0 and for the next release.

    Verdict

    Step 0.13.0 release main 4d7536d68b1db6da34bf9cff22a90d9fdd77bed4
    Clean SIGTERM (exit 0), then the same start command refused: state_lock_held starts normally (lock released)
    Start fails: Address already in use (os error 98) lock kept lock kept
    Start fails: no tokens and no --unauthenticated lock kept lock kept
    Start fails: malformed OMNIGRAPH_SERVER_BEARER_TOKENS_JSON not run lock kept
    SIGKILL (exit 137) lock kept lock kept
    cluster force-unlock <LOCK_ID> --config . works works
    omnigraph --cluster file://<root> cluster force-unlock <LOCK_ID> works works
    Shutdown log line with lock_id= present on a clean exit only removed

    Clean SIGTERM, then restart: reproduced on 0.13.0, not reproduced on main. Failed start: reproduced on both. Missing note in server.md: reproduced on both.

    Environment. Linux x86_64. Local file cluster root. OMNIGRAPH_HOME set to an empty directory. The server ran with --unauthenticated (except where noted) and --bind 127.0.0.1:<port>.

    • 0.13.0: omnigraph-linux-x86_64.tar.gz from the v0.13.0 release. Its sha256 0123daa6…7a30 matches the published .sha256. omnigraph version prints omnigraph 0.13.0 / internal-schema 14 (serves v14 to v14).
    • main: cargo build --locked -p omnigraph-server -p omnigraph-cli (dev profile) in a clean worktree at 4d7536d68b1db6da34bf9cff22a90d9fdd77bed4. omnigraph version prints omnigraph 0.14.0 / internal-schema 16 (serves v14 to v16) / http-contract 0.13.

    Method. One script runs the same steps on both binaries. The bundle has one graph (node Person { name: String @key }). The steps are: direct cluster apply, exact-ID unlock of the apply lock, start, SIGTERM, start again, then a failed start on a busy port, then SIGKILL.

    0.13.0 output (excerpt)
    $ omnigraph cluster apply --config . --json
    Admission lock: 01M54N1J43K95V9GSHB0FVAT1V; retain until prior work is quiescent, then exact-ID force-unlock
    $ omnigraph cluster force-unlock 01M54N1J43K95V9GSHB0FVAT1V --config .
    cluster force-unlock: removed lock (id=01M54N1J43K95V9GSHB0FVAT1V, operation=deployment, ...)
    
    ### start, then clean SIGTERM
    readyz: 200
    server exit status after SIGTERM: 0
    WARN omnigraph_server: v2 cluster admission retained after shutdown; establish prior graph/control I/O quiescence before exact-ID force-unlock root=file:///tmp/r983-013/c lock_id=01M54N1JYAPB4X47TJ21WH6V0F
    $ cat __cluster/lock.json
    { "version": 1, "lock_id": "01M54N1JYAPB4X47TJ21WH6V0F", "operation": "serve", ... }
    
    ### restart with the same command
    server exit status: 1
    [state_lock_held] __cluster/lock.json: cluster state lock already exists (lock id 01M54N1JYAPB4X47TJ21WH6V0F); run `omnigraph cluster force-unlock 01M54N1JYAPB4X47TJ21WH6V0F` only after confirming no cluster operation is active
    
    ### unlock with the ID from the shutdown log line
    $ omnigraph cluster force-unlock 01M54N1JYAPB4X47TJ21WH6V0F --config .
    cluster force-unlock: removed lock (id=01M54N1JYAPB4X47TJ21WH6V0F, operation=serve, pid=5659, ...)
    readyz: 200
    
    ### failed start: port already in use
    $ omnigraph-server --cluster . --bind 127.0.0.1:18983 --unauthenticated
    Error:
       0: Address already in use (os error 98)
    [exit 1]
    $ cat __cluster/lock.json
    { "version": 1, "lock_id": "01M54N1QYA7XSNMEV81A67RRM5", "operation": "serve", ... }
    (next start: [state_lock_held] ... lock id 01M54N1QYA7XSNMEV81A67RRM5)
    
    $ omnigraph --cluster file:///tmp/r983-013/c cluster force-unlock 01M54N1QYA7XSNMEV81A67RRM5
    { "unlocked": true, "lock_id": "01M54N1QYA7XSNMEV81A67RRM5" }
    
    ### failed start: no credentials and no --unauthenticated
       0: server has no bearer tokens and no policy file configured. This is a fully open server — pass `--unauthenticated` ...
    cluster status: locked=True lock_id=01M54N8K8GGEARJHR1B6BP3HZH lock_operation=serve
    
    main 4d7536d output (excerpt)
    ### start, then clean SIGTERM
    readyz: 200
    server exit status after SIGTERM: 0
    INFO omnigraph_server: shutdown signal received
    __cluster/lock.json: absent
    
    ### restart with the same command
    readyz: 200
    server exit status after SIGTERM: 0
    __cluster/lock.json: absent
    
    ### failed start: port already in use
    $ omnigraph-server --cluster . --bind 127.0.0.1:18984 --unauthenticated
    Error:
       0: Address already in use (os error 98)
    [exit 1]
    $ cat __cluster/lock.json
    { "version": 1, "lock_id": "01M54N4TKW8FY40MKPNK1F6VQ5", "operation": "serve", ... }
    
    ### restart after the failed start
    [state_lock_held] __cluster/lock.json: cluster state lock already exists (lock id 01M54N4TKW8FY40MKPNK1F6VQ5); run `omnigraph cluster force-unlock 01M54N4TKW8FY40MKPNK1F6VQ5` only after confirming no cluster operation is active
    
    ### failed start: no credentials and no --unauthenticated
       0: server has no bearer tokens and no policy file configured. ...
    cluster status: locked=True lock_id=01M54N80FGBEYP2Y2N9A6H62PX lock_operation=serve
    
    ### failed start: OMNIGRAPH_SERVER_BEARER_TOKENS_JSON='{not json'
    cluster status: locked=True lock_id=01M54N80HADFYEBNXKACYDAJH5 lock_operation=serve
    
    ### SIGKILL
    server exit status after SIGKILL: 137
    [state_lock_held] ... (lock id 01M54N4X997GZ6G7A2PB25MNQV)
    $ omnigraph cluster status --config . --json
      "state_observations": { "locked": true, "lock_id": "01M54N4X997GZ6G7A2PB25MNQV", "lock_operation": "serve", "lock_pid": 6630, ... }
    

    Differences from the report:

    1. feat(server): release native admission after settled clean shutdown #899 is merged. On main a clean SIGTERM releases the lock (local and S3). Azure still keeps it. feat(server): release native admission after settled clean shutdown #899 is not in a release yet.
    2. The shutdown log line is a 0.13.0-only lock-ID source. It appears only on a clean exit. It does not appear for a failed start on either version. Main removed it (0.13.0 lib.rs#L2663-L2667). On main, the lock-keeping paths log no lock ID.
    3. __cluster/lock.json is a correct source for local roots only. On S3, main keeps the key after a clean release and writes a "released" marker into it (state_lock.rs#L200-L215, #L254-L265). The raw file then exists, but it is not a held lock. cluster status and cluster observe ignore a released marker: they report locked: false and no lock_id (types.rs#L124-L133). These two sources work on both versions and on every backend. The state_lock_held message itself also prints the ID.
    4. There are more failed-start triggers than a busy port. A missing --unauthenticated (with no tokens) and a malformed token JSON also keep the lock. A first-time operator can easily make the first mistake.
    5. The suggested command is correct on both versions: omnigraph cluster force-unlock <LOCK_ID> --config <dir> (default --config .). The root-addressed form omnigraph --cluster <root-uri> cluster force-unlock <LOCK_ID> also works and is what clusters/index.md shows. The hint in state_lock_held omits the address (store.rs#L1356). From another directory it fails safely with cluster_config_read_error cluster.yaml.
    6. The first start after a direct cluster apply also needs an unlock, because apply keeps its own lock. clusters/index.md already documents this, so it is not part of this gap.

    Regression proof on main (local only, not pushed)

    GQT cannot express this contract: it is a process-lifecycle and storage-lock property, not rows or errors from a query. The docs checker (scripts/check-docs.py) checks links and locations, not content. The existing owner is crates/omnigraph-server/tests/boot_settings.rs. It owns server startup refusals and #899's clean-restart regression (owned_shutdown::disconnected_write_and_shutdown_share_ownership_v2_finish). I added one test there. It boots the owner's v2 fixture through the production load_server_settings + serve. Then it checks that a start refused before any graph opens leaves no lock.

    • Baseline: v2_settings_capture_holds_admission_across_drop_and_refuses_another_server and owned_shutdown::disconnected_write_and_shutdown_share_ownership_v2_finish: 2 passed; 0 failed.
    • New test on unmodified main: 0 passed; 1 failed. Both cases keep the lock:
    starts refused before any graph open retained admission:
    occupied-listener: refused with `Address already in use (os error 98)`; next start: the cluster at '/tmp/.tmpVSEzs6' is not ready to serve:
      [state_lock_held] __cluster/lock.json: cluster state lock already exists (lock id 01M54NYNZ1E28EJG9F91C1CQNB); run `omnigraph cluster force-unlock 01M54NYNZ1E28EJG9F91C1CQNB` only after confirming no cluster operation is active
    open-server-without-opt-in: refused with `server has no bearer tokens and no policy file configured. ...`; next start: the cluster at '/tmp/.tmpDUZJxY' is not ready to serve:
      [state_lock_held] __cluster/lock.json: cluster state lock already exists (lock id 01M54NYNZQXW4XBDRSBKCE8QVX); ...
    test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 42 filtered out
    

    Command: cargo test --locked -p omnigraph-server --test boot_settings -- --exact v2_server_start_refused_before_graph_open_releases_admission_issue_983

    Important: this test describes the proposed contract (fix B below). It is not a currently documented one. The control-plane guide says today that "Only unique admission, successful startup and unpoisoned settlement permit final release" (control-plane.md#L233). So the failure shows the current behaviour precisely. It is the regression test for fix B, not proof of a code bug against the written contract. The docs fix A closes this issue without it.

    Test source (rustfmt clean; Clippy not run)
    /// A start refused before any graph opens has no accepted work to settle, so
    /// the next ordinary start must not need `cluster force-unlock` (issue #983).
    #[tokio::test(flavor = "multi_thread")]
    async fn v2_server_start_refused_before_graph_open_releases_admission_issue_983() {
        let mut retained = Vec::new();
        for case in ["occupied-listener", "open-server-without-opt-in"] {
            let temp = converged_cluster_dir("").await;
            omnigraph_cluster::upgrade_deployment_ledger(
                &format!("file://{}", temp.path().display()),
                true,
                &omnigraph_cluster::DeploymentCaller::storage_owner(None),
            )
            .await
            .unwrap();
            let occupied = std::net::TcpListener::bind("127.0.0.1:0").unwrap();
            let mut settings = omnigraph_server::load_server_settings(
                Some(&temp.path().to_path_buf()),
                Some(if case == "occupied-listener" {
                    occupied.local_addr().unwrap().to_string()
                } else {
                    "127.0.0.1:0".into()
                }),
                case == "occupied-listener",
                false,
            )
            .await
            .unwrap();
            settings.shutdown_grace = std::time::Duration::from_secs(5);
            let refusal = omnigraph_server::serve(settings).await.unwrap_err();
            let lock = temp.path().join("__cluster/lock.json");
            if lock.exists() {
                let refusal = refusal.to_string();
                retained.push(format!(
                    "{case}: refused with `{}`; next start: {}",
                    refusal.lines().next().unwrap_or_default(),
                    cluster_settings(temp.path()).await.unwrap_err()
                ));
                continue;
            }
            cluster_settings(temp.path())
                .await
                .unwrap()
                .cluster_admission
                .unwrap()
                .release_after_settlement()
                .await
                .unwrap();
        }
        assert!(
            retained.is_empty(),
            "starts refused before any graph open retained admission:\n{}",
            retained.join("\n")
        );
    }

    Root cause

    1. The lock is kept on purpose by default. Dropping an admission or returning from writable work keeps the persisted lock (control-plane.md#L190-L199). A test pins this (boot_settings.rs#L45-L87). A crash or cancellation therefore never looks like a settled owner. This is correct and must stay.
    2. 0.13.0 has no release path for a server. Shutdown only logs the WARN line with the lock ID (0.13.0 lib.rs#L2663-L2667).
    3. Main releases only at the end of a successful serve. The release block runs after HTTP drain and graph startup finish. It closes the storage scope, refuses uncertain I/O, waits for unique ownership and calls release_after_settlement (lib.rs#L3007-L3032). Azure is skipped on purpose.
    4. Process-local checks run after admission and return with ?. load_server_settings* in main.rs takes the admission first. Read-only settings refusals in that phase already release it (settings.rs#L47-L61, #L471-L481). But serve_config then runs more checks that need no storage: token source and load (lib.rs#L2864-L2866), session defaults (#L2867-L2868), the auth classification (#L2886-L2890) and the listener bind (#L2946). Each one returns early and skips the release block. Graph opens start only after the bind (#L2971-L2980). So these failures have no graph or control effect to settle. The lock is kept only because of where the checks are in the code, not because of any data risk.
    5. The operator page does not say this. "Start a server" in server.md has the start command only (server.md#L13-L44). The rules are in deployment.md (#L105-L122, #L300-L315). On main these describe feat(server): release native admission after settled clean shutdown #899's unreleased behaviour, and they do not mention failed starts.
    6. The skill on main is stale in the other direction. skills/omnigraph/references/cluster.md says a server keeps the lock "even after a clean exit" (#L78-L83, #L283). server-policy.md says the same (#L16-L20). This is correct for 0.13.0. It becomes wrong when feat(server): release native admission after settled clean shutdown #899 ships, and the CLI embeds this skill.

    Recommended fix

    A. Docs, now (closes this issue). Add a short "Stop and restart" subsection under "Start a server" in server.md. Write it for both versions and link to "Writer topology" for the rules. Proposed text:

    ### Stop and restart
    
    A mutation-capable server holds the cluster's exclusive writer lock. Stop it
    with SIGTERM.
    
    - From the release after 0.13.0, a clean stop (exit 0) on a local or S3 root
      releases the lock. Start the next server with the same command.
    - These exits keep the lock: exit 2 (grace deadline or uncertain work), SIGKILL,
      a crash, every exit on an Azure root, and a start that fails after it took
      the lock (for example `Address already in use`, or no tokens and no
      `--unauthenticated`). In 0.13.0, every exit keeps the lock, including a clean
      stop.
    
    The next start then refuses with `state_lock_held` and shows the lock ID. To
    read the ID without changing anything:
    
    ```bash
    omnigraph cluster status --config ./company-brain --json   # state_observations.lock_id
    ```
    
    Make sure that the previous process has exited and that its writes are
    complete. Then remove that exact lock and start again:
    
    ```bash
    omnigraph cluster force-unlock <LOCK_ID> --config ./company-brain
    # or by root: omnigraph --cluster s3://company-data/omnigraph/company-brain cluster force-unlock <LOCK_ID>
    ```
    
    Do not delete `__cluster/lock.json` yourself. On S3 the file stays after a clean
    release. See [writer topology](../deployment.md#writer-topology).

    Why this shape: cluster status is the one ID source that works on 0.13.0 and main, on every backend, and after a failed start. The 0.13.0 log line and the raw file each fail one of those cases. Naming the version inline is the per-section marking that #985 asks for. In the same change, or at release, update the skill text in point 6 so that it agrees with the release that ships it. If fix B lands, remove the failed-start bullet for those checks.

    B. Code, as a separate change: run process-local preflight before admission (correct by design). Move every check that needs no cluster storage in front of the admission acquire in main.rs / load_server_settings*. These checks are token source and load, ProcessDefaults::from_env, classify_server_runtime_state and TcpListener::bind. Pass the bound listener into serve. The rule becomes "no fallible process-local check after the writer lock is taken". One placement enforces this rule, and new checks get it automatically. With --bootstrap-handoff the same ordering is also correct: the check fails before the claim, so the claimed lock that must never disappear is never created. The local test above is its regression test.

    • Tradeoffs: the socket is bound during settings load. Connections queue in the kernel backlog and get no response until axum::serve starts. Today this happens too, but for a shorter time. Probes time out instead of getting "connection refused". Update the deployment.md sentence "The listener opens after configuration and authorization validation". The token load (for example Secrets Manager) runs before the lock. It is read-only, so it is safe.
    • Liability: low. It is one ordering rule in one place. It adds no new release path and does not change the protocol or storage format.

    Alternatives set aside:

    • Release in serve_config's early-error paths (like release_settings_refusal). This works, but each new check after admission must remember to call it. It also needs a flag so that bootstrap-claimed locks stay held, and it must drop state clones before Arc::try_unwrap. More ongoing liability than B.
    • Release on Drop. Rejected. Drop-retains is pinned on purpose, so that cancellation and abandonment never look settled.
    • Expire the lock by lease, age or PID, or add an "unlock stale" server flag. Rejected by the invariants: process-local locks are not distributed fencing, and lock age or a stopped PID is not proof of quiescence (deployment.md#L305-L307).
    • Unlock without an ID. Rejected. The exact ID stops an operator from removing a successor's lock.

    C. Small diagnostics, optional. Log the lock ID when main keeps admission (the Azure, uncertain and failed-start paths), as 0.13.0 did on clean exit. Add the address to the state_lock_held hint (--config <dir> or --cluster <root>).

    Quality traded: fix A keeps the operator burden for unclean exits. This is deliberate, because robustness (never unlock without proof) has priority over convenience.

    Lance upstream

    Lance does not cause or fix this issue. OmniGraph pins Lance =11.0.0 (Cargo.toml). The latest Lance release is v13.0.0, and the newest pre-release is v14.0.0-beta.19. __cluster/lock.json is an OmniGraph control object. OmniGraph's storage adapter writes it (local create-if-absent, S3 conditional put). It is not a Lance dataset or a Lance commit.

    There is one indirect dependency. Main's clean release requires settled Lance I/O with no hidden retries. Scoped storage forces client_max_retries=0 and lance_aimd_max_retries=0 (lance_access/scoped.rs#L45-L73). Check these at the next Lance upgrade:

    • lance#9913 (merged 2026-10-14, after v13.0.0) moves multipart-complete retries to the HTTP layer. The loop appears to be bounded by the same max_retries value that OmniGraph sets to 0. I did not confirm the option mapping.
    • lance#9996 (open) fixes a possible hang in ObjectWriter::poll_write. Such a hang would end in exit 2 with the lock kept. I did not see it here.
    • lance#10033 (open) adds conditional multipart manifest uploads. That is a new multipart lifetime, which the storage scope must count.
    • lance#10008 (open) is about which storage options are honoured on reads. It is not relevant to the lock.

    I read the complete Lance object store, observability and transactions pages for this check.

    Related work

    Supersession

    None. No open issue or PR covers the server.md restart note or the failed-start lock. This issue stays open.

    Limitations

    • I ran only local file roots. I did not run S3 or Azure (no credentials here). The S3 released-marker and Azure statements come from code and feat(server): release native admission after settled clean shutdown #899's own evidence.
    • main was a dev-profile build, not a release build.
    • The new test ran only by name. I did not run Clippy or the full boot_settings binary on it. The 0.13.0 malformed-token case was not run.
    • The test describes the proposed contract of fix B, not a current documented contract.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    needs-triageAwaiting maintainer triage

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions