Repository navigation
Conversation
ragnorc
left a comment
There was a problem hiding this comment.
Review of 5a4bdcb (draft RFC: trust snapshots and credential refresh)
Why. Issue #434: today the server reads static bearer tokens once, at boot. When an operator adds a token, it gets 401 until a restart. When an operator removes a token, the server keeps accepting it.
What. One TrustHolder type for each credential surface. Each source declares its freshness policy (boot-only, or polled). One installation function. Detection by content digest, not by stat. The authentication mode is fixed at boot. Static file refresh is opt-in.
How. A poller re-reads the source, short-circuits on the raw digest, validates the content, and swaps in a new generation with ArcSwap. Each request binds one snapshot when it is admitted.
Impact. Operators can onboard and offboard users from a mounted Secret without a restart. In later phases, OIDC and data-token trust move onto the same mechanism.
Verdict. The direction is correct, and I agree with the main decisions: one holder with per-source policy, polling instead of SIGHUP or a new endpoint, digest instead of stat, a boot-fixed mode, and the rejection of tokens in the ledger. Three items block acceptance. Each one is a stated fact or rule in the RFC that the code or a later phase contradicts.
Blocking
B1. The boot-fixed mode is based on a premise that is false. The RFC says (lines 186–191, 349–350) that the mode comes from "what was configured", and that "the server already refuses to boot with a configured source that yields no tokens", so "no existing deployment observes a difference". But the classifier receives token content, not configuration: classify_server_runtime_state(!tokens.is_empty() || data_token_trust.is_some() || oidc_identity_trust.is_some(), …) (lib.rs:4069-4073). I ran the server with OMNIGRAPH_SERVER_BEARER_TOKENS_FILE pointing to a file that holds {}:
- With no flag and no policy, startup is refused ("server has no bearer tokens and no policy file configured"). The refused start leaves
__cluster/lock.jsonbehind (see #983). - With
--unauthenticated, the server boots Open.GET /graphs/knowledge/snapshotwith noAuthorizationheader returns 200.
So a configured source that holds no tokens is not refused today. The RFC must decide what "configured" means, and then state the consequences:
- Is the mode input "a static source is configured" (for example,
_FILEis set) or "the static set is non-empty"? If the input is configuration, then the{}+--unauthenticateddeployment changes behavior (it is either refused or becomes DefaultDeny), and that is a compatibility note. If the input stays content, then a server that boots Open with an empty file and has refresh on never starts to require tokens when tokens are added. In that case, startup must refuse a refresh interval on a server that booted Open. - Boot with a valid empty file. The RFC installs a valid empty set at refresh (lines 142–145), but boot refuses the same state. Suppose the last user is offboarded and the pod then restarts for any reason. The pod goes into a crash loop, and each refused start leaves the lock behind. That is the outage and the #983 interaction that the RFC sets out to remove. The RFC's own principle (the mode comes from configuration, not from content) says that boot should accept an empty file when the source is configured. Please decide this case explicitly. One rule for one state is the goal.
The single owner the RFC proposes is correct: an immutable mode value derived from configuration. Make the classifier take that value as its input too. Then boot, requires_bearer_auth, has_credential_source, and the /mcp router gate (lib.rs:3936) all read the same fact. Today four places derive it from live content.
B2. The refused-digest short-circuit is not sound when validation depends on the clock, and phase 3 depends on the clock. Step 2 (lines 250–253) and "equal bytes cannot validate to different sets" (line 279) are correct for static tokens. They are not correct for OIDC. Snapshot::parse refuses generated_at > now + 30 and expires_at <= now (oidc_identity.rs:158-170). If a writer's clock is more than 30 s ahead, its snapshot is refused at time t and is valid at t+31 with the same bytes. Today refresh_at retries every 5 s and installs the snapshot when it becomes valid. Under step 2, the bytes are recorded in refused and skipped until they change. The OIDC phase claims "existing cases pass unchanged", but this is a behavior change that the RFC does not state. Data-token claims are also checked against now (exp, iat, data_tokens.rs:248-249). The simplest fix: use the refused digest only to deduplicate the warning, and continue to validate refused bytes on each tick. The cost is one parse of at most 256 KiB every few seconds, and only while the source is invalid. Another fix: the surface's validator declares that its result is a pure function of the bytes, and only such refusals are skipped. In both cases, the general rule must not claim a property that one of its three instances does not have.
B3. The phase-1 refusal rule does not cover the AWS source. resolve_token_source selects Secrets Manager whenever _AWS_SECRET is set, and then it ignores _FILE and _TOKEN (auth.rs:73-95). With _AWS_SECRET + _FILE + _REFRESH_SECONDS, the rule "a value with no _FILE configured refuses startup" (lines 111–113) passes, but phase 1 has nothing to poll. State the rule against the selected source: in phase 1, refuse an interval unless the selected source is a FileSource. This also makes the startup line static credential refresh enabled name the source that is actually polled.
Optional
- O1. Serialize installation instead of a CAS retry. Step 6 retries a lost compare-and-swap once and then refuses it. If that refusal records the digest in
refused, as line 271 does for the earlier steps, a lost race puts valid bytes into the skip set. A per-holder mutex aroundrefresh, withArcSwapkept only for readers, makes "one place where a trust set changes" true by construction. It also removes the retry path, and it covers later triggers (signal, endpoint) and the concurrent embedders that the OIDC CAS comment mentions (oidc_identity.rs:276-281). - O2. Align with #1323 and #1324. #1323 already sends the AWS payload through
validate_bearer_tokensand reserves theprincipal:prefix, so step 3 should cite that change and not present it as a new fix. #1324 adds a permission and regular-file check before the token file is read. Step 1 adopts the OIDC reader, which has no mode check. Use one reader for the static file at boot and at each poll, with #1324's check included. A file that becomes group-writable after boot is then refused on the next poll and keeps the last good set. Two readers would drift. With #1324, the Kubernetes notes should also recommenddefaultMode: 0400. - O3. OIDC expiry at admission.
check_atenforcesexpires_atper request (oidc_identity.rs:325-333). The genericInstalled<S>has no place for it. Say that per-request validity is a method ofS, so that the holder stays independent of time and OIDC does not keep a parallel check. #1224 (scope ceilings) adds to the same admission path. - O4. Line 151 says
GET /access. The route is per graph:/graphs/{graph_id}/access(handlers.rs:837, nested under the graph prefix). Line 392 has the correct form. - O5. "Audited" (lines 143, 268) is an
infolog line. The server has no audit sink. Say "logged", or name the sink, so that operators do not expect a durable record. - O6. Read failures (a missing file, a permission error, an AWS throttle) have no content digest. Say how their warning is deduplicated, for example once per error class until the next successful read.
Tradeoffs and liability
The design removes a dead hook (supports_refresh, which nothing calls), and it removes the risk of three freshness mechanisms on one AppState. It adds one generic type, one environment variable, one background task for each polled surface, and one mode field. After five more sources (Vault, the Kubernetes API, delegated credentials), each source is one TrustSource implementation with a declared Freshness. That is the right shape. The remaining liability is the abstraction boundary: the three surfaces have different policy gates (revision monotonicity, binding, expiry at admission). B2 and O3 are the two places where the generic holder currently claims more than its instances guarantee. Fix them in the holder contract, and the abstraction removes liability instead of adding it.
The "configured credential" concept is now derived from content in four places (the classifier input, requires_bearer_auth, has_credential_source, and the MCP gate). B1's single mode value is the abstraction that makes the invariant hold by construction. I recommend it, with the classifier as its first reader.
Fit with other open work: #1322 (bind --unauthenticated to loopback) touches the Open case in B1. #1210 keeps credentials out of the graph, which agrees with the rejected ledger alternative.
Validation
- Code claims, checked at the PR head
5a4bdcbc(merge base0065a69f). Current main62779133has no changes tosettings.rs,auth.rsoroidc_identity.rs, and none to the auth lines oflib.rs. Confirmed: the immutablebearer_tokensfield and its single load inserve_config. Thesupports_refreshhook exists and nothing calls it.requires_bearer_authreads the live set. The AWS parser skips the duplicate-token check._TOKEN+_FILEcombine.read_bearer_tokens_filehas no bound. The OIDC reader is bounded, opens the path again (so it follows the..dataswap), and checks the opened file is regular. The poller usesSkip,spawn_blockingandWeak. The test owners the RFC names exist (auth.rsenv_or_file_source_does_not_support_refresh,oidc_identity_tests.rs,boot_settings.rsmod owned_shutdown). The doc sentences cited (server.md "Choose one static token source", clusters/index.md, runtime RFC, RFC 0053) are present. - Process evidence for B1, with the debug
omnigraph/omnigraph-serverbuilt from maine750a6be, whose token-loading and classifier code equals the merge base. I applied a one-graph local cluster, released its lock, and settokens.json={}(mode 600). With no flag: exit 1 with the "no bearer tokens" refusal, andcluster statusthen shows a held lock (01M763A816J1XECB902GT2629D). With--unauthenticated: therunning with --unauthenticatedwarning,GET /graphs/knowledge/snapshotwith no token → 200, and SIGTERM → exit 0 with the lock released. - B2 is from code inspection (
Snapshot::parseagainstnow). I did not run it, because no holder implementation exists. - Regression test. Not applicable: this PR changes no behavior. The RFC is correct that GQT cannot express timers, process environment or file swaps, and it selects the cheapest Rust owners.
- Upstream. The Kubernetes Secret page confirms that a
subPathmount gets no updates, and that the delay is "kubelet sync period + cache propagation delay". The AWS quota page lists GetSecretValue at 10,000 per second per Region, shared by the account, so the 60 s floor is about cost and noise, not quota. No Lance surface is touched, so no Lance page applies. - Docs and CI.
python3 scripts/check-docs.py: Documentation OK (491 files).typosis not installed here. CI on5a4bdcbc: 17 passed, 14 skipped, none failed. - Limitation. This review covers a design document. Phases 2–4 are judged against the current OIDC and data-token code, not against an implementation.
Draft RFC for issue ModernRelay#434: one trust holder per credential surface, one installation function, freshness as a property of the source, a boot-fixed authentication mode, and opt-in polling of the static token file as the first phase.
B1 mode input is configuration (classifier, requires_bearer_auth, has_credential_source, /mcp mount read one value; two boot states of a configured empty source change, recorded under compatibility). B2 refused digest no longer skips validation (clock-dependent OIDC and data-token checks); it deduplicates the warning. B3 phase-1 interval refusal against the selected source. O1 mutex-serialized install. O2 one reader with ModernRelay#1324's check; cite ModernRelay#1323. O3 per-request validity on the set. O4-O6 route path, logged not audited, content-less failure dedup.
5a4bdcb to
bad6067
Compare
|
Revised in B1 — mode input is configuration. The RFC now says what "configured" means ( B2 — refused digest no longer skips validation. Step 2 keeps the raw-digest short-circuit only against the installed content (sound: equal bytes cannot install anything else) and says explicitly why refused bytes are re-validated every tick: B3 — refusal against the selected source. The phase-1 rule now reads: an interval refuses startup unless O1 — Still open for acceptance: the one unresolved question (whether an invalid static file should ever time out its last-good set). Everything else from the review is in the body; the decision log records the revision. |
…y, AWS interval guidance
Phase 1 of the RFC "Trust snapshots and credential refresh".
The static bearer tokens are held as a trust snapshot: a
TrustHolder<StaticTokens> whose refresh function is the one place the
set changes. It reads the source through one bounded reader (the path
reopened, the regular-file and permission check, a 256 KiB bound),
short-circuits on bytes equal to the installed ones, validates through
the shared validator, hashes and zeroizes the plaintext, and installs a
new generation only when the canonical digest changes. A refusal keeps
the installed set and is warned about once per refused content or once
per error class until the next successful read.
OMNIGRAPH_SERVER_BEARER_TOKENS_REFRESH_SECONDS (1 to 3600) polls the
token file on an interval; unset or 0 keeps the boot-only contract. An
interval refuses startup unless the selected source is the file.
The authentication mode is now classified from what is configured: a
set static token variable is a credential source whatever it holds. The
mode is one immutable field on AppState that requires_bearer_auth,
has_credential_source and the /mcp mount read, so no refresh can change
whether a credential is required.
BREAKING CHANGE: a configured static source that holds no tokens (a
token file or JSON object of {}) now boots DefaultDeny, or
PolicyEnabled with a policy, instead of being refused; with
--unauthenticated it no longer runs Open. Unset the variable to run an
open server. TokenSource::supports_refresh is removed.
Set implementation: in-progress, record the phase-1 commit and the text the code changed.
A panic inside a refresh tick was discarded with the JoinError. Warn with the source and a fixed class, without the panic payload; the installed set stays in force and the next tick reads again.
This PR carries the RFC "Trust snapshots and credential refresh" (
docs/rfcs/2026-11-11-trust-snapshots-and-credential-refresh.md) and its phase 1 implementation, for #434.RFC
Decides live credential refresh for
omnigraph-server, the decision the runtime RFC deferred:TrustHolderper credential surface, and one installation function that is the only place a trust set changes. Triggers call it and nothing else.Polledin 1–3600 s, and Secrets Manager will bePolledwith a 60 s floor (phase 2).stat. A raw-byte digest gives the short-circuit; a canonical digest of the validated set decides whether a change is a change.OMNIGRAPH_SERVER_BEARER_TOKENS_REFRESH_SECONDS, off by default. No new HTTP surface and no SIGHUP handler.Phase 1 (this PR)
crates/omnigraph-server/src/trust.rs:TrustHolder<S: TrustSet>(ArcSwap<Installed<S>>, the installer mutex guardingwarned,Freshness,TrustSource),refresh(read → raw short-circuit → validate → hash and zeroize → canonical digest → install), and the poller (tokiointerval,MissedTickBehavior::Skip,spawn_blocking, holds aWeak).auth.rs:StaticTokens(constant-timeadmit),FileSource(BootOnlyorPolled1–3600 s),EnvSource(BootOnlyonly), and source selection. An interval refuses startup unless_FILEis set and_AWS_SECRETis not, and the refusal names the selected source.supports_refresh()is removed. Secrets Manager stays boot-only through itsTokenSource.settings.rs: one bounded reader for the token file at boot and at every poll (reopens the path; fix(server)!: refuse a bearer-tokens file other accounts can write #1324's regular-file and permission check; 256 KiB bound; O_NONBLOCK), and one in-place validator that compares tokens by hash.lib.rs:AppStateholds the static tokens in the holder; the immutablebearer_tokensfield is gone. The classifier's input is now "a static source is configured", and its result is one immutable mode onAppStatethatrequires_bearer_auth,has_credential_sourceand the/mcpmount read.changelog.d/configured-token-source-requires-auth.breaking.md): a configured source holding no tokens ({}) now bootsDefaultDeny/PolicyEnabledinstead of being refused, and with--unauthenticatedit no longer runsOpen. The remedy is to unset the variable.docs/user/operations/bearer-tokens.md(sources, refresh contract, Kubernetes), summarized in and linked fromserver.mdanddeployment.md. Both pages were at the docs checker's 350-line cap, which is why the detail lives on its own page. Alsoclusters/index.md,policy.md,mcp.md, and the skill'sserver-policy.md.OIDC and data-token trust do not move onto the holder in this phase.
Tests
auth.rsholder tests callrefreshdirectly: refusals keep the set and recordwarned; unchanged bytes do nothing; a cosmetic rewrite bumps no generation; a same-size, same-mtime rotation is installed; a..dataswap is installed;{}is installed; intervals are a property of the source; refusals name the selected source; warnings are logged once per class until a successful read.settings.rs: the classifier takes configuration; an empty refreshed set keeps bearer authentication and/mcp.boot_settings.rsowned_shutdown::static_token_file_refresh_issue_434: the real server with a 1 s interval and kubelet-style..dataswaps, probingGET /graphs/owned/access.Checks run locally
cargo fmt --all --check: OKcargo clippy -p omnigraph-server --all-targets --locked -- -D warnings -W clippy::dbg_macro: OKcargo test -p omnigraph-server --locked: all pass (lib 243, boot_settings 44, integration 294, multi_graph 8, operation_routes 4)cargo check -p omnigraph-server --features aws --locked: OKpython3 scripts/check-docs.py,typos(v1.50.1),bash scripts/check-agents-md.sh,scripts/check-release-note.py,scripts/check-dependency-sources.py: OKzeroize1.9.0 was already inCargo.lock(Apache-2.0 OR MIT); the server now depends on it directly, so the lockfile gains one edge.