Skip to content

fix(cluster): refuse actor-less recovery under graph policy before admission - #1438

Open
ragnorc wants to merge 5 commits into
ModernRelay:mainfrom
ragnorc:fix/og-reconcile-without-actor
Open

ragnorc wants to merge 5 commits into
ModernRelay:mainfrom
ragnorc:fix/og-reconcile-without-actor

Conversation

@ragnorc

@ragnorc ragnorc commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Summary

The documented recovery of an outstanding deployment,
omnigraph cluster apply --target <root> --deployment-id <ID> --writers-stopped,
failed on a graph with an applied policy when no --as was given and the
deployment's schema change had started. It reported deployment_outcome_unknown
and kept a new admission lock, although nothing ran. It now refuses with
policy_denied, names --as, and does so before it takes the lock, so nothing
changes and no force-unlock is needed before the retry.

Findings queue item: og-stress-reconcile-without-actor-reports-unknown
(found by the stress workflow; not filed on GitHub). Related: #1414 / #1423.

Reproduction

On main 54ced5d3f573a45bfbe717dd2099cd1fa3b06220, the existing owner test
identity_recovery::offline_recovery_reauthorizes_executor_and_preserves_original_and_fence_authors
pinned the reported behaviour. With its two actor-less steps changed to the new
contract (commit 8ae92610), it fails on main:

assertion `left == right` failed: Diagnostic { code: "deployment_outcome_unknown", ...,
  message: "could not prepare original settlement evidence under admission 01M7M4BMBZ62NWV7Y91PPQCKE2:
  policy: policy denied action 'schema_apply': a policy is installed and the request names no actor" }
  left: "deployment_outcome_unknown"
 right: "policy_denied"

The test reaches the reported state without S3: a deployment interrupted after
Started (cluster seam DEPLOYMENT_AFTER_STARTED) on a graph with a policy, the
dead owner's lock removed, then reconcile_deployment as
DeploymentCaller::storage_owner(None) (what the CLI passes without --as). The
stress report reached the same state by killing a served apply on an S3 root.
#1423 already replaced the engine's "almost certainly a bug" wording; the wrong
code and the retained lock remained.

Root cause

reconcile_deployment_in_store (crates/omnigraph-cluster/src/deployment/execution.rs)
takes admission first. In the GraphDeploymentState::Started schema arm it checks
graph actions only for DeploymentCaller::AuthenticatedIdentity, then calls
prepare_schema_settlement_as(original, caller.actor()). The engine's
prepared::authorize enforces schema_apply and refuses a policy-bound call
that names no actor; the arm maps every error to deployment_outcome_unknown.
A graph deletion has the same need (authorize_graph_deletion already refuses
policy_denied without an actor), but also only after admission.

Change

  • authorize_recovery (since 8084e5ac, after review): one function makes the
    complete decision recovery makes for each outstanding graph. A deletion needs
    what forward deletion needs (authorize_graph_deletion: read and
    schema_apply, an actor under policy). Settling a started schema change needs
    read and schema_apply for an identity, and, for a storage owner on a graph
    with an applied policy, an actor that policy permits schema_apply on main,
    the same request the engine checks when it settles. It runs before admission
    and again under admission against the re-read policy, in place of the loop's
    former checks. So every caller the policy refuses, not only a missing actor,
    gets policy_denied with no lock.
  • The engine still enforces its own gate when recovery runs.
  • The deletion refusal without an actor now names --as.
  • Docs (docs/user/clusters/index.md, "Inspect and recover a deployment") and the
    skill (references/cluster.md) say when --as is needed and which actions
    (deletion also needs read); release note in
    changelog.d/reconcile-requires-actor-under-policy.fixed.md.

Decisions

  • Refuse, not bypass. Letting a storage owner reconcile with no actor would
    contradict fix(cli)!: direct writes to a cluster graph obey its applied policy #1423, which made direct writes obey the graph's policy and refuse a
    missing actor. The settlement fence is authored by the recovery actor, so the
    policy has someone to judge.
  • Before admission. A refusal before any effect should not leave a lock, like
    the existing policy_denied for an unpermitted identity in the same test. A
    refusal after admission (another writer changing the ledger between the reads)
    keeps the lock, as every recovery exit after admission does; the doc comment
    of authorize_recovery says so (since de2d9307).
  • The full decision, not presence only. The first version only required an
    actor before admission. Review showed a named actor the policy refuses still
    took the lock. The engine checks schema_apply with target branch main, and
    AppliedPolicies::check_graph(actor, graph, SchemaApply) makes the same
    request on the same policy bundle that open_graph installs, so the cluster
    can decide before admission. A storage owner still needs no read: the engine
    does not check it, and requiring it would refuse recoveries that work today.
  • Not changed: forward cluster apply without --as on a policy graph, which has
    the same identity-only preflight shape. Out of scope here.

Tests

  • identity_recovery::offline_recovery_reauthorizes_executor_and_preserves_original_and_fence_authors:
    three refused callers (storage owner with no actor, storage owner --as principal:outsider, identity principal:manager with config_manage but no
    graph action), before and after a settlement fence is persisted, now require
    policy_denied (a message naming --as when no actor is given), no
    __cluster/lock.json, an unchanged ledger and unchanged graph bytes, and the
    persisted fence kept. The later recovery with an actor still adopts the fence.
    A new final phase interrupts a graph deletion after Started; the same three
    callers are refused before admission with no byte changed, and a permitted
    actor then completes the deletion.
  • Since de2d9307: a storage owner --as principal:applier, whose only graph
    action is schema_apply, settles a second started schema change and is
    refused for the pending deletion. This pins the two rules apart: with read
    dropped from the deletion check, or added to storage-owner settlement, the
    test fails; the earlier rows pass under both changes.
  • Before/after for the review change: with execution.rs from 6f563d75, the
    extended test fails at --as principal:outsider with deployment_outcome_unknown
    (policy denied action 'schema_apply' for unknown actor 'principal:outsider');
    at 8084e5ac it passes.

Local gates (Linux x86_64, debug)

At de2d9307: cargo dev-nextest -E 'package(omnigraph-cluster)' (failpoints)
180/180, cargo fmt --all --check, and
cargo clippy -p omnigraph-cluster --all-targets --features failpoints -- -D warnings.
The change is a test row and a doc comment. The gates below ran at 8084e5ac:

  • cargo dev-nextest -E 'package(omnigraph-cluster)' (failpoints): 180/180.
  • CLI cli_cluster* and skill tests plus omnigraph-skill: 105/106. The failure,
    cli_cluster::managed_lifecycle_capture_and_retry_records_refuse_symlinks_without_reading_targets,
    fails the same way on unmodified main on this box (it runs as root).
  • cargo fmt --all --check, cargo dev-clippy (workspace, -D warnings), scripts/check-docs.py,
    scripts/check-release-note.py with this title. typos 1.50.1 ran on the changed files at 6f563d75; it is not installed on the box that made 8084e5ac, so CI owns it there.
  • GQT and DST not run locally: GQT does not depend on omnigraph-cluster, and DST
    uses only its lease protocol. CI runs both.

Risks and what was not verified

  • A storage-owner recovery with no actor on a policy graph that used to run
    (none should have: the engine refused it) is now refused earlier. A recovery of
    graph creation or of a not-started query-only change is unaffected. A deletion
    without an actor was already refused policy_denied; it is now refused before
    the lock is taken.
  • A named recovery caller the policy refuses (a storage owner whose --as lacks
    schema_apply, an identity without graph read or schema_apply, a deletion
    caller without read) was refused after admission and kept the lock; it is now
    refused before admission. A permitted caller's recovery is unchanged.
  • Not verified: the stress report's S3 + process-kill journey end to end, and Azure.
  • Lance not involved.

Workflow run: run-wf-reproduce-modernrelay-omnigraph-issue-20261117T0632Z-631735
Response run: run-wf-respond-modernrelay-omnigraph-pr-20261117T0747Z-978ecf
Response run: run-wf-respond-modernrelay-omnigraph-pr-20261117T1002Z-d9202d
Response run: run-wf-respond-modernrelay-omnigraph-pr-20261117T1222Z-8b9507
Response run: run-wf-respond-modernrelay-omnigraph-pr-20261117T1327Z-eaa5a3


Devin Review

…ecovery

A storage-owner reconciliation of a started schema change on a graph with
an applied policy names no actor. It fails on main with
deployment_outcome_unknown and retains a fresh admission lock, although
nothing ran.
…mission

Reconciling an outstanding deployment as a storage owner with no actor
failed when a started schema change or a deletion touched a graph with an
applied policy: the engine refused the settlement for lack of an actor,
the refusal was reported as deployment_outcome_unknown, and the attempt
kept a new admission lock. Nothing had run.

Recovery now checks this before it takes admission and refuses with
policy_denied, naming --as. The documented recovery command and the skill
say when --as is needed.
@ragnorc
ragnorc requested a review from aaltshuler as a code owner November 17, 2026 07:25

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔍 Devin Review: 2 flags

Not posted on this PR by your GitHub settings — view them in Devin Review. (Configure)

Devin Review

@ragnorc ragnorc left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review of #1438 at 6f563d75

Recommendation: revise before merge. The change is correct for the case it names. But the same defect remains for a named recovery caller that the policy refuses. One small change closes the whole class (B1).

What the change does

A direct cluster apply --deployment-id … --writers-stopped with no --as, on a graph with an applied policy, used to take the admission lock and then fail inside the engine. The cluster reported deployment_outcome_unknown and kept the lock, although nothing ran. Now require_recovery_actor refuses this case with policy_denied before admission. So no lock is taken and nothing changes. The docs, the skill and a release note say when --as is necessary.

Impact: an operator who follows the documented recovery gets a true refusal and a usable remedy, not a false "unknown outcome" and a lock to remove by hand.

Blocking

B1. The fix covers one cell of the class. A named caller that the policy refuses still gets a lock, and a storage owner still gets deployment_outcome_unknown.

The violated invariant: a policy refusal that is known before any effect must be reported as a refusal, before admission, with no lock. Recovery breaks it for every caller the policy refuses, not only for a caller with no actor. With this PR:

Caller Policy Result at 6f563d75
storage owner, no --as graph policy installed policy_denied, no lock (fixed)
storage owner, --as principal:outsider outsider has no schema_apply deployment_outcome_unknown, lock held
identity principal:manager has config_manage, no graph read policy_denied, lock held
storage owner, --as an unpermitted actor, pending deletion — policy_denied from authorize_graph_deletion after admission, lock held (code reading, execution.rs:2972)

I showed rows 2 and 3 with a temporary test in tests/identity_recovery.rs. It used the same DEPLOYMENT_AFTER_STARTED setup as the PR test, then called reconcile_deployment with each caller:

REVIEW storage owner --as principal:outsider: code=deployment_outcome_unknown lock_held=true
  message=could not prepare original settlement evidence under admission 01M7M7FAB4E99KM26T269RD1BC:
  policy: policy denied: policy denied action 'schema_apply' for unknown actor 'principal:outsider'
REVIEW identity principal:manager: code=policy_denied lock_held=true
  message=policy denied action 'read' for unknown actor 'principal:manager'

Row 2 is the reported symptom again (wrong code and a held lock). The only difference is that the operator typed a wrong --as. An operator who reads the new docs and passes --as with an actor that lacks schema_apply gets the same misleading result this PR removes.

Cause: the pre-admission check tests only that an actor is present. The actual decision stays in three places after admission: authorize_graph_deletion (:2972), the inline identity checks (:3114), and the engine's prepared::authorize through prepare_schema_settlement_as (:3126). The rule "policy installed and no actor refuses" is now in three places too: Omnigraph::enforce, authorize_graph_deletion and require_recovery_actor.

The PR's decision "presence only for storage owners" says that checking read up front would refuse recoveries that work today. That is correct for read. But the engine's decision for a storage owner is schema_apply with target branch main (prepared.rs:214-219). AppliedPolicies::check_graph(actor, graph, SchemaApply) evaluates the same policy engine with the same scope (authorization.rs:376-397). And open_graph installs the same policy.graph(graph) bundle that the preflight reads. So the cluster can make the complete decision before admission, not only the presence part.

Proposed fix (smaller and by construction): replace require_recovery_actor with one authorize_recovery(policy, caller, pending) that makes, for each pending graph, the exact decision the loop makes now:

  • deletion (any state): authorize_graph_deletion (read + schema_apply, actor required under policy);
  • started schema change, no create: identity → read + schema_apply; storage owner → if the graph has a policy, an actor plus schema_apply on main;
  • other arms: nothing.

Call it once with preflight_policy before admission. Call it again with the re-read policy after admission, in place of the call at :2972 and the inline checks at :3114-3115. The engine keeps its own gate (invariant 10), so this adds no bypass. The post-admission call matters only if the policy changes between the two reads, and the policy_digests comparison already refuses that as deployment_base_changed.

Liability: this removes one presence-only predicate and the inline identity checks, and it gives recovery one owner for "who may settle this graph". After five more recovery arms, each arm adds its rule in one function that runs before the lock. With the current shape, each new arm needs a presence rule before admission and a decision after it, and those two can drift. Please extend the temporary test above (rows 2-3, plus a deletion row) in identity_recovery.rs as the regression. Update the docs sentence too: deletion needs read as well as schema_apply.

Optional

  • O1. The docs and skill say "an actor that policy permits schema_apply". For a deletion, authorize_graph_deletion also requires read. Say "read and schema_apply" for deletion, or name both in one sentence.
  • O2. After B1, the deployment_outcome_unknown mapping of an engine Policy error from prepare_schema_settlement_as (:3126-3133) remains for the race only. prepare authorizes before it writes anything, so a typed policy_denied there would also be true. The lock is already held at that point, so this is about the code only. It is not needed if B1 lands.

Context checked

  • #1434 (one rule for policy-bundle selection) keeps AppliedPolicies::graph as the single source that the preflight and open_graph both read. No conflict.
  • #1360 releases the lock after a converged forward apply. Recovery still keeps admission after it completes. No overlap with this refusal path.
  • Forward cluster apply without --as prepares the schema before acceptance, and that read-only preflight refusal releases the lock. So it is not in this class, and leaving it out of scope is correct.

Validation

  • Clean detached worktree at 6f563d75 (merge base 54ced5d3).
  • cargo dev-nextest -E 'package(omnigraph-cluster)' (failpoints): 180/180 PR tests pass. cargo fmt --all --check passes.
  • Before/after: with crates/omnigraph-cluster/src/deployment/execution.rs from 54ced5d3 and the PR's test, offline_recovery_reauthorizes_executor_and_preserves_original_and_fence_authors fails at identity_recovery.rs:318 with deployment_outcome_unknown … a policy is installed and the request names no actor (the predicted reason). At 6f563d75 it passes.
  • Temporary test review_temp_named_unpermitted_recovery_refuses_before_admission (rows 2-3 above) fails the same way before and after the PR. I removed it and restored the worktree. Nothing pushed.
  • GQT does not apply: GQT does not depend on omnigraph-cluster, and this contract is the recovery CLI/ledger boundary. The cluster's own integration test is the correct owner.
  • CI at 6f563d75: 28 pass, 4 skipped.
  • Not verified: the S3 process-kill journey, Azure. No Lance surface.

No Decision needed threads on this PR.

Review of ModernRelay#1438 (B1): the pre-admission check only tested that an actor was
present, so a caller the policy refuses still took the lock: a storage owner
whose --as lacks schema_apply got deployment_outcome_unknown, and an identity
without graph read got policy_denied after admission.

authorize_recovery now makes the complete per-graph decision that recovery
made inside its loop: a deletion needs what forward deletion needs (read and
schema_apply, an actor under policy); settling a started schema change needs
read and schema_apply for an identity, and an actor with schema_apply on main
for a storage owner on a graph with a policy, the same request the engine
checks when it settles. It runs before admission and again under admission
against the re-read policy, replacing the loop's checks. The engine keeps its
own gate.

The deletion refusal now names --as. The recovery test covers both refused
named callers before and after a persisted fence, and a pending deletion.
@ragnorc

ragnorc commented Nov 17, 2026

Copy link
Copy Markdown
Contributor Author

Reply to the review of 6f563d75 (review 5934724871).

B1: valid, fixed in 8084e5ac. I checked the claim before changing code. The engine's enforce sends schema_apply with target branch main to the policy that open_graph installs (policies.graph(graph)). AppliedPolicies::check_graph(actor, graph, SchemaApply) sends the same request to the same bundle. So the cluster can make the whole decision before admission.

  • require_recovery_actor is replaced by authorize_recovery(policy, caller, pending). For each outstanding graph it makes the decision the loop made: a deletion calls authorize_graph_deletion (read and schema_apply); a started schema change (not a creation) needs read and schema_apply for an identity, and an actor with schema_apply on main for a storage owner on a graph with a policy; other arms need nothing.
  • It runs before admission with preflight_policy, and again after admission with the re-read policy, right after the policy_digests check. The call in the deletion arm and the inline identity checks in the Started arm are removed. The engine keeps its own gate.
  • The deletion refusal with no actor now names --as. Before, it said only "graph deletion requires an actor when policy is installed".

Regression: identity_recovery::offline_recovery_reauthorizes_executor_and_preserves_original_and_fence_authors now checks three refused callers: a storage owner with no actor, a storage owner --as principal:outsider, and identity principal:manager (config_manage only). It checks them before and after the settlement fence, and in a new pending-deletion phase (interrupted after Started), after which a permitted actor completes the deletion. Each refusal must be policy_denied, with no __cluster/lock.json, an unchanged ledger and unchanged bytes.

  • Before: with execution.rs from 6f563d75, the test fails at the outsider row: deployment_outcome_unknown … policy denied action 'schema_apply' for unknown actor 'principal:outsider'. The test stops at that first failing row.
  • After: passes at 8084e5ac.

O1: valid, fixed in 8084e5ac. The docs and the skill now say a deletion needs read and schema_apply, and settling a schema change needs schema_apply. The release note now covers every refused caller, not only a missing --as.

O2: not changed. As you say, after B1 that mapping is reached only if the engine's policy differs from the cluster's check, and the policy_digests comparison already refuses that as deployment_base_changed. The lock is held at that point either way, so the code would change but the operator would see nothing new. I kept this PR to the recovery-authorization change.

Devin review (5934521191): its two flags are not posted on GitHub, and its page needs a login. There is nothing here to answer. If they are posted, a later run will judge them.

Gates at 8084e5ac (Linux x86_64, debug):

  • cargo dev-nextest -E 'package(omnigraph-cluster)' (failpoints): 180/180.
  • CLI cli_cluster* and skill tests plus omnigraph-skill: 105/106. The failure, cli_cluster::managed_lifecycle_capture_and_retry_records_refuse_symlinks_without_reading_targets, also fails on unmodified main on this box (it runs as root).
  • cargo fmt --all --check, cargo dev-clippy, scripts/check-docs.py, scripts/check-release-note.py.
  • CI on 8084e5ac: 27 pass, 4 skipped, none failing.

Not verified: the S3 process-kill journey and Azure. PR description updated (Change, Decisions, Tests, Risks) with a Response run: line.

@ragnorc ragnorc left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review of #1438 at 8084e5ac

Recommendation: ready to merge. No blocking defect in the delta. B1 and O1 of review 5934724871 are fixed by design. Two optional points follow.

This review covers the delta 6f563d75..8084e5ac (one commit, 8084e5ac, +161 −50).

Why, what, how, impact

  • Why. At 6f563d75 the preflight checked only that an actor was present. A named caller that the policy refuses still took the lock and ended as deployment_outcome_unknown.
  • What. authorize_recovery(policy, caller, pending) replaces require_recovery_actor. It is the one decision for each outstanding graph. A deletion goes through authorize_graph_deletion (read and schema_apply). A started schema change that is not a creation needs read and schema_apply for an identity. For a storage owner on a graph with an applied policy, it needs an actor with schema_apply on main. Nothing else acts as the caller.
  • How. The function runs before admission against preflight_policy (execution.rs:2956). It runs again under admission against the re-read policy (execution.rs:2986). The check in the deletion arm and the inline identity check in the Started arm are gone.
  • Impact. Each caller that the policy refuses now gets policy_denied with no lock and no change. This includes a missing --as, an unpermitted --as and an identity without graph rights. Before, the checks ran graph by graph inside the effect loop. Now every graph is decided before the first effect, so a refusal can no longer stop recovery between two graphs.

What I checked

  • The cluster decision matches the engine gate. The engine settles through prepared::authorize (crates/omnigraph/src/db/omnigraph/schema_apply/prepared.rs:214). It is called from prepare_schema_settlement and settle_prepared_schema (settlement.rs:110, :278). It sends schema_apply with ResourceScope::TargetBranch("main"), so the branch is None and the target branch is main. AppliedPolicies::check_graph(actor, graph, SchemaApply) (authorization.rs:376) sends the same pair. open_graph installs policies.graph(graph) with with_policy, so both sides use the same bundle. The storage-owner rule therefore predicts the engine's answer exactly. The rule also holds when a settlement is already persisted (execution.rs:3140), because settle_prepared_schema_as checks again.
  • Coverage of the arms. The arms that act as the caller are a deletion (any state, as before), Started without create (settlement), and Started with create. The last one settles a creation and runs no graph action as the caller. authorize_recovery selects exactly the first two. NotStarted work without a create or an intent only opens the graph for catalog settlement.
  • Liability. The change removes two inline checks and one partial preflight, and adds one function. That is one place to decide who may recover. It keeps a second copy of the engine's gate, by design, so the refusal can come before admission. The doc comment names the scope that it mirrors (schema_apply on main). If the engine's settlement scope changes, this copy must change with it. The outsider row of the regression test would show such drift, because the engine refusal maps to deployment_outcome_unknown. After five more recovery arms, each one adds one branch to this single function, not a check inside the loop.

Optional

O1. The check under admission refuses while it holds the lock. The second authorize_recovery (execution.rs:2986) runs before any effect. But an error there drops the admission without release_refused_preflight, so __cluster/lock.json stays. Forward apply releases the lock on the same kind of refusal (execution.rs:1697). The deployment_base_changed refusal just before it behaves the same way, and both can happen only if the outstanding record changes between the two reads while writers are stopped. So this is not a defect today. If you keep the second call as a guard, either release the lock as forward apply does, or state in the doc comment that a refusal at that point keeps the lock. The user docs now say that a refusal "refuses … before it takes the lock and changes nothing".

O2. No regression row separates the deletion rule from the settlement rule. None of the three refused callers holds any graph action. If a later change dropped read from the deletion check, or added read to storage-owner settlement, all three rows would still pass. One more caller would pin the difference: a storage owner --as an actor with schema_apply but not read. Recovery must refuse it for the pending deletion and admit it for the started schema change.

Previous findings

  • B1: fixed (see above).
  • O1 (docs): fixed. docs/user/clusters/index.md and skills/omnigraph/references/cluster.md now say that a deletion needs read and schema_apply and that settlement needs schema_apply. The release note covers every refused caller.
  • O2 (mapping of a Policy error after admission): not changed, with reasons in the author's reply. I agree. After this change the path needs the engine and cluster policies to differ, and the policy_digests check refuses that first.

There are no Decision needed threads on this PR.

Validation

  • Clean detached worktree at 8084e5ac (merge base 54ced5d3).
  • At the head: cargo dev-nextest -E 'package(omnigraph-cluster)' (with failpoints) passes 180/180, and identity_recovery passes 2/2. cargo fmt --all --check passes.
  • Before/after: with execution.rs from 6f563d75 and the new test, offline_recovery_reauthorizes_executor_and_preserves_original_and_fence_authors fails at identity_recovery.rs:542 for the outsider row. It ends as deployment_outcome_unknown with "policy denied action 'schema_apply' for unknown actor 'principal:outsider'", which is the predicted reason. At 8084e5ac it passes. The test stops at its first failing row, so the before run does not reach the deletion phase. I checked that phase at the head only.
  • CI at 8084e5ac: 27 pass, 4 skipped, none failing.
  • GQT does not apply. The contract is cluster recovery authorization, and GQT does not reach omnigraph-cluster. The owner is crates/omnigraph-cluster/tests/identity_recovery.rs.
  • No Lance surface in the delta.
  • Not verified: the S3 process-kill journey and Azure.
  • I restored the temporary edits and removed the worktree. Nothing was pushed.

A storage owner --as an actor with schema_apply but not read settles a
started schema change and is refused for a pending deletion, so a change
that drops read from the deletion rule or adds it to settlement fails the
test. The doc comment of authorize_recovery says that its check under
admission keeps the lock, as every recovery exit after admission does.
@ragnorc

ragnorc commented Nov 17, 2026

Copy link
Copy Markdown
Contributor Author

Reply to the review of 8084e5ac (review 5936264444). Pushed de2d9307: one test row and one doc comment. No behavior change.

O1: partly valid. Doc comment added; the lock is not released.
The fact holds. If the second authorize_recovery refuses, the admission is dropped, and Drop for ClusterAdmission (admission.rs:472) does not release the lock. But releasing it there would break recovery's own rule. Recovery keeps its lock on every exit after admission, success included: the end of reconcile_deployment_in_store drops the admission on purpose ("Retain admission until explicit operator quiescence/unlock"), and the user docs say "Recovery retains a new admission lock". Forward apply is different, because it releases on success too (execution.rs:1699). So I took your second option. The doc comment of authorize_recovery now says that the check under admission can refuse only if the outstanding record changed after the first check (a changed policy is refused first, as deployment_base_changed), and that this refusal keeps the lock, as every recovery exit after admission does. The user-doc sentence is about a policy refusal, which the first check decides before the lock, so I left it unchanged.

O2: valid. Fixed.
identity_recovery::offline_recovery_reauthorizes_executor_and_preserves_original_and_fence_authors has a new caller: a storage owner --as principal:applier, whose only graph action is schema_apply. It settles a second started schema change (the real engine settlement runs, and recovery_executors names the applier). It is refused before admission for the pending deletion.

  • Drop read from authorize_graph_deletion: the test fails at the deletion row, because the recovery runs (identity_recovery.rs:604, unwrap_err on Ok).
  • Add read to storage-owner settlement: the test fails at the new schema row with policy denied action 'read' on branch 'main' for actor 'principal:applier'.
  • With the test file from 8084e5ac, both changes pass, as you said. So the new row is what pins the difference.

Gates at de2d9307 (Linux x86_64, debug): cargo dev-nextest -E 'package(omnigraph-cluster)' (failpoints) 180/180; cargo fmt --all --check; cargo clippy -p omnigraph-cluster --all-targets --features failpoints -- -D warnings. CI: 25 pass and 4 skipped on de2d9307, including Test Workspace and Storage Upgrade Compatibility. Two fail, and neither is from this change. Dependency Guard (cargo deny) (job) refuses three unmaintained advisories that the RustSec database added at 10:14Z today: RUSTSEC-2026-0543 (chrono), RUSTSEC-2026-0542 (humantime) and RUSTSEC-2026-0555 (chrono-tz). Cargo.lock is unchanged, the advisories give no safe upgrade, and the unrelated gq3/phase-0-front-end run fails the same way (job). CI Gate fails only on cargo_deny. I did not change deny.toml. That needs a maintainer to fix the dependencies or accept the advisories on main, and then this check can be rerun.

No decision is needed on this pull request.

@ragnorc ragnorc left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review of #1438 at de2d9307

Recommendation: ready to merge. No blocking defect in the delta, and no new optional point. Both optional points of review 5936264444 are closed.

This review covers the delta 8084e5ac..de2d9307 (one commit, de2d9307, +70 −3: one doc comment and one test row). It replies to the author's comment.

Why, what, how, impact

  • Why. Review 5936264444 asked for two things. O1: say what happens to the lock when the check under admission refuses. O2: add a test row that tells the deletion rule apart from the settlement rule.
  • What. The doc comment of authorize_recovery (execution.rs:2347) now says that the check under admission can refuse only if the outstanding record changed after the first check, and that this refusal keeps the lock. identity_recovery.rs adds a storage owner --as principal:applier. The policy gives that actor schema_apply and nothing else.
  • How. The test starts a schema change (email: String? on Person), stops it after Started with DEPLOYMENT_AFTER_STARTED, and recovers it as the applier. The real engine settlement runs, and recovery_executors names the applier. Then the same caller is refused before admission for the pending deletion.
  • Impact. No behavior change. The difference between the two rules is now pinned by a test.

What I checked

  • O1, the doc comment is true. lookup_policies (execution.rs:130) builds the policy from state.applied_revision. Under admission, the code compares the pending record's result_revision, resource_digests, schema_contracts and policy_digests with the re-read state before the second authorize_recovery (execution.rs:2976-2990). So a changed policy is refused first, as deployment_base_changed. With the same applied revision the policy is the same. The second check can then give a different answer only if the outstanding record's graph entries changed.
  • O1, keeping the lock is correct here. I agree with the author. Recovery keeps its admission on every exit after admission, success included (execution.rs:3199, "Retain admission until explicit operator quiescence/unlock"). Forward apply is different and releases on a refused preflight (execution.rs:1697). The user doc (docs/user/clusters/index.md:276-282) talks about a policy refusal, and the first check makes that decision before the lock. So the doc sentence stays true. The doc comment was the right place for the note.
  • O2, the row is a real settlement. The knowledge graph exists, so the entry is Started without create, which is the arm that acts as the caller. The test passes only if the engine's own gate (prepared::authorize, schema_apply on target branch main) admits the applier. The policy rule uses target_branch_scope: any, so it matches that check. The new policy group does not change the earlier rows, because the applier is not in operators and is not one of the refused callers.
  • Liability. The change adds only test setup and four lines of doc comment. The test function is long now (one function, several phases), but each phase adds one row to the existing owner. It does not add a parallel fixture.

Previous findings

  • O1 (lock kept by the check under admission): closed by the doc comment, with the author's reasons.
  • O2 (no row tells the deletion rule from the settlement rule): fixed.
  • Everything from review 5934724871 stays as review 5936264444 recorded it.

There are no Decision needed threads on this PR.

Validation

  • Clean detached worktree at de2d9307, merge base 54ced5d3.
  • At the head: cargo dev-nextest -E 'binary(identity_recovery)' (with failpoints) passes 2/2.
  • Mutation checks with the head test file. Each mutation fails for the predicted reason:
    • Remove read from authorize_graph_deletion: offline_recovery_reauthorizes_executor_and_preserves_original_and_fence_authors fails at identity_recovery.rs:604. unwrap_err gets Ok, because the applier deletes the graph.
    • Add read to the storage-owner arm of authorize_recovery: the same test fails at identity_recovery.rs:495 with policy_denied, "policy denied action 'read' on branch 'main' for actor 'principal:applier'".
  • Before: with the test file from 8084e5ac, each of the two mutations passes 2/2. So the new row is what catches them.
  • CI at de2d9307: 25 pass, 4 skipped, 2 fail. Dependency Guard (cargo deny) fails on RustSec advisories that were published today (RUSTSEC-2026-0542, -0543, -0555). Cargo.lock does not change in this PR, and #1316 fails the same check. CI Gate fails only because of that check. The last CI run on main (25b51992, 07:45Z) passed before the advisories were published, so this is not yet confirmed on main. It does not come from this PR.
  • GQT does not apply. The contract is cluster recovery authorization, and GQT does not reach omnigraph-cluster. The owner is crates/omnigraph-cluster/tests/identity_recovery.rs.
  • No Lance surface in the delta.
  • Not verified: the S3 process-kill journey and Azure.
  • I restored the temporary edits and removed the worktree. Nothing was pushed.

@ragnorc

ragnorc commented Nov 17, 2026

Copy link
Copy Markdown
Contributor Author

Reply to the review of de2d9307 (review 5938318261). No new commit. The review has no finding, so there is nothing to fix or decline.

CI: the Dependency Guard (cargo deny) failure now also happens on main. I ran cargo deny --locked check (cargo-deny 0.21.0) in a clean worktree of main 25b51992. It fails with advisories FAILED, bans ok, licenses ok, sources ok. The errors are the same three unmaintained advisories: RUSTSEC-2026-0543 (chrono), RUSTSEC-2026-0542 (humantime) and RUSTSEC-2026-0555 (chrono-tz). No CI run on main shows it yet, because the last main CI run (07:45Z) and the nightly guard (04:02Z) ran before RustSec published these advisories at 10:14Z. The failure does not come from this pull request, and I did not change deny.toml. A maintainer must replace these dependencies or accept the advisories on main. Then the check can run again on this pull request.

No decision is needed on this pull request.

@ragnorc

ragnorc commented Nov 17, 2026

Copy link
Copy Markdown
Contributor Author

CI now passes on d5c8f808. That commit merges main 5a1ec458 into this branch. It changes nothing else.

The Dependency Guard (cargo deny) failure is gone. Main 5a1ec458 (#1439) added the three advisories (RUSTSEC-2026-0542, -0543 and -0555) to deny.toml, so this branch needed only the merge. The merge had no conflicts.

  • Local: cargo deny --locked check (cargo-deny 0.21.0) gives advisories ok, bans ok, licenses ok, sources ok. cargo dev-nextest -E 'package(omnigraph-cluster)': 182 passed, because main's new policy_binding.rs touches the same crate.
  • CI on d5c8f808: 27 passed, 4 skipped, none failed. Dependency Guard, CI Gate.

No review finding is open and no decision is needed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant