feat(daemon): one sandbox pod per session on the pool — the host's pod is its own (§11)

[zfy0701]

Summary

The last code change of the git-workspace-model §11 rollout, on top of #1757, #1758, #1761 and #1763: on the managed pool an isolated session's host is its own pod. The !this.k8s stopgap in perSessionHost goes. A pool member now keys a session host by the session row's isolation — shared keeps today's shape (the agent's host, in the agent's pod, sessions in the primary checkout), session gets its own host and its own pod. The self-hosted tiers (#1758/#1761/#1763) are untouched, and a test asserts the self-hosted predicate and policy outputs did not move.

The driver is re-keyed by a sandbox subject: the agent id for the agent pod (so every agent-keyed caller was already subject-keyed, and the agent path is byte-identical), <agentId>/<session leaf> for a session pod, derived from the HostKey that SpawnRequest now carries. A session claim is agent-<id>-<16 hex of the leaf> — kept under 63 characters so the vendor's Sandbox/Service names stay DNS labels — with the pod labelled agentconnect.md/agent and a separate agentconnect.md/session (never folded into the agent label the reconciler validates as a UUID). The same labels ride the claim's own metadata so a member can list an agent's session claims without knowing its sessions. Nothing per-agent or per-session goes in the claim spec beyond those labels, so warm-pool adoption is untouched.

Decisions (each also stated in a one-line comment at its gate)

1. Storage. The session's clones and HOME live on the session pod's volume at <mount>/sessions/<leaf>/{workspace,repos/<owner>/<repo>,home} (session-layout.ts helpers, cluster paths pass the mount), prepared by the same blobless-clone path a confined self-hosted session uses (prepareClusterConfinedSession → prepareRootSessionClone); the primary checkout on the agent pod is not consulted for it. The agent pod stays for the primary checkout, secondary roots, isolation: 'shared' sessions, pullOnNewSession, console workspace views and managed memory at <mount>/.agentconnect/memory. The plane routes each path to the pod that owns it (sandboxSubjectForPath):
   • gitRunnerFor(agentId, cwd), workspaceFsFor(agentId) (a routed WorkspaceFs), workspaceFilesFor(agentId) (a routed WorkspaceFiles) and clearPath → the session pod for a path under <mount>/sessions/<leaf>, the agent pod for everything else; a path whose pod is not bound refuses rather than falling back onto another pod. The pod is read off the path, not off the live-launch registry (review round 2): suspendIfIdle forgets a launch while its claim and volume survive on purpose, so a routed read of an idle-suspended session's directory was being served by the agent pod, where the directory does not exist. A read that names a sleeping session pod now resumes it — but only beside a bound agent pod, since the console's agent/wake is agent-scoped and that is the press a read has, and only onto the claim whose uid the router just observed (driver.claimUidFor → driver.resumeBoundChannel, review round 3), which never creates one. With everything asleep the seams still hand the caller nothing. RoutedWorkspaceFs/RoutedWorkspaceFiles resolve their target asynchronously for it (no caller contract changed — every member was already async), and a git runner for a sleeping session directory is deferred rather than withheld, so the caller does not silently fall back to a local runner for a path that lives on a pod.
   • memoryFsFor, autoMergeFor, the console's agent/wake and the memory-distill reachability → the agent pod (sandboxBound(agentId)).
   • runsInSandbox(agentId) now means "any pod of the agent is bound" — the condition the git runner and the credential-target pointers share. It is deliberately not what keep-alive judges on (review round 3): a page's lease is about one pod, and sandboxKeepAlive names, holds and judges the pod its own worktree lives on.
   • The console's session root (workspace-scope.ts → sessionWorktreePath → consoleWorkspaceRoot) composes the session clone path, so the panel and the runtime name one directory.
   • Memory coupling. A session runtime binds and holds the agent pod as its companion for the runtime's life (K8sDriver.launch), so "a runtime is running for agent A" still implies "A's pod is bound" and managed memory recall/capture, the memory MCP tools and merge-when-ready keep exactly today's reachability. The companion is also held during a confined session's workspace preparation, since a reviewed secondary root records its attestation there. Cost: one extra pod per active agent (not per session), reclaimed by the idle sweep. A companion that will not come up degrades those seams with a warning; it never fails the launch. The session bind and the companion hold are settled together (review round 2): with the session's rejection propagating on its own, the launch's catch drained its holds while holdCompanion was still binding, and the retain it then took belonged to no runtime and no onExit — the agent pod stayed busy until the process restarted.
2. Pod lifetime = session host lifetime, claim lifetime = session row lifetime. Claimed when the session's host starts (ensureSandbox converges on the deterministic name, so a restart or a successor member resumes the same claim — adoption by host key). Idle suspend is per pod (sweepIdleSandboxes iterates subjects; a session pod is judged by its own session's activity and its own host, the agent pod by every host of the agent) and keeps the volume. The claim — volume with it — is deleted when the session's row is deleted by retention (discardSessionSandbox, after the clone on its own pod has passed the dirty/unique-commit rules, woken only when the cluster holds a claim for it), when a workspace is replaced (setSessionsDiscarder — the pool's form of clearSessionWorktrees), and when the agent is removed (removeAgentSandboxes). The replacement's discard runs on the volume, not at activation (review round 2): prepareWorkspaceForActivation precedes the CP's acknowledgement and its sandbox-mode rollback is empty, so retiring there meant a later rejection restoring the previous definition after every session claim — clone and HOME with it — had already been deleted, with nothing able to put it back. It now runs inside prepareClusterCheckout's conversion (retireClusterSessionDirectories), at the same gate that empties the primary checkout: only when the marker no longer proves the workspace, sparing the leaf being prepared — whose own directory is emptied instead, so it clones afresh — and fail-closed, so a retirement that raised leaves the marker unproven and the next preparation retries it rather than cloning over stale pods. An isolated session's preparation reaches the agent pod's half for that conversion and nothing else: with none due it still never consults the primary checkout, which is what keeps decision 1 true. Credentials and launch files are materialized per pod on every launch, as they are per host self-hosted. Duty gain also adopts an agent's Running session pods by label (adoptSessions) so a rollout leaves no pod without a holder to suspend it.
3. Capacity. Conservative: maxAgents keeps counting duty-covered agents; session pods are bounded by session admission and the idle sweep (comment at dutyHeadroom, asserted in daemon-k8s-mode.test.ts). claimVacant is untouched.
4. Orphans — in the stacked follow-up PR (session-aware reconciler + reconcile --once wiring), kept separate so each PR stays reviewable.
5. Operator / CRD / alerting follow-ups (deployment side, not in this PR):
   • The pool members' Role needs list on sandboxclaims for adoptSessions / sessionClaimSubjects (today the members may only get/create/delete; a denied list degrades to a warning and the member's own launches).
   • If the claim controller's additionalPodMetadata label allowlist is per key rather than per domain, admit agentconnect.md/session.
   • Any capacity/alerting that counts pods per agentconnect.md/agent (the pool-capacity alert, dashboards) now sees ≥1 pod per agent; split by the agentconnect.md/session label. Warm-pool sizing (manual, D14) should account for one pod per active isolated session plus one per active agent.
   • No CRD change.

Review round 3 — the wake is resume-only, and keep-alive judges its own pod

Both findings are consequences of round 2's wake-on-read, and both are fixed here.

The wake creates nothing. hasClaim and ensureBoundChannel were two round trips, so retention, a workspace conversion or an agent removal landing between them left ensureClaim making a fresh empty claim and PVC — a console read resurrecting a session sandbox whose row and volume were already retired, as a live agent's orphan. The router now carries the observed claim's uid into a resume-only driver operation: K8sDriver.resumeBoundChannel(subject, claimUid) runs ensureSandbox's suspension, takeover and release fences and then reads the claim that name holds instead of ensuring it, re-judging the uid against the object after the gap. A claim that is gone, or one of the same name that is a different object, refuses. ensureClaim is not reachable from this path at all, which is the property rather than the narrowed window: a read can lose a race with a retirement, never win one.

How this relates to #1767's fence. They guard opposite directions of the same object and compose without overlapping. #1767 fences the delete — an admission stamp on the claim plus deleteClaimIfCurrent on the resourceVersion the sweep listed, so a claim re-admitted after the listing is not collected. This PR fences the wake — a resume that refuses unless the claim is still the object that was observed. A resume is deliberately not an admission and stamps nothing, so a console read can never make an orphaned claim look young to that sweep; and because the resume creates nothing, a delete that wins the race simply leaves the read refusing.

Keep-alive names one pod, holds it, and judges it. It was still gated on runsInSandbox(agentId), which this PR made true for any bound pod, so a bound agent pod carried a suspended session's poll into workspaceGit.status and the deferred routed runner woke that session's claim — a visible clean page undoing the idle sweep every 60 s. Three changes:

• The decision is this page's pod. podFor resolves it through the SAME scope the status read resolves its root with (workspaceScope.gitRoot) and the same routing rule (plane.subjectForPath), so the lease, the judgement and the read cannot name different pods. A routing that cannot answer falls back to the agent's own pod, which is where an unrouted path lives anyway.
• The two facts are judged INDEPENDENTLY, one pod each (round 3 follow-up). The first pass folded them into one gate, so a page whose session pod had gone to sleep returned early, skipped armedFor, and released this holder's lease on the AGENT pod — the pod an armed merge-when-ready watcher actually runs in. The sweep could then take that pod while the page was still visible, silently disarming the box the operator ticked; before one pod per session there was one pod to hold, so nothing was lost. Now the watcher is judged on the agent's pod and the tree on this page's pod, each under its own hold, with one hold over both reads when the two coincide. A pod that is down is still never read from, so the sleeping session pod is not woken to answer a poll, and asleep stays this page's answer about the pod its own worktree lives on — independent of what it still holds elsewhere.
• The pod is HELD across the read, not merely checked before it. plane.holdIfBound → K8sDriver.retainLaunched retains the Sandbox a bound launch names and hands back a releaser. SandboxLease.suspendIfIdle reads that busy count synchronously before publishing its gate, so the hold excludes the sweep rather than narrowing a window: with it taken, the runner resolves against a still-bound session and the deferred (waking) branch is unreachable. It is released in a finally, so a keep-alive still defers a suspend rather than cancelling one. Nothing is claimed or woken to take it — a pod with no launch here answers "asleep".
• The lease is per pod. SandboxHolds is keyed by sandbox subject; the agent's own pod IS the agent id as a subject, so the agent-pod path is unchanged. Each reason leases the pod it is about: a dirty tree its own pod, an armed merge watcher the agent's, since that is the process's pod whatever worktree the page is watching (armedFor already answers false unless that pod is bound). sweepIdleSandboxes asks holds(subject) for the subject it is judging, so one dirty session no longer pins every running sibling session pod.

What did not change

• Self-hosted tiers: feat(daemon): one ACP host per session for a confined self-hosted launch #1758/feat(daemon): a confined session's directory is its own clone, not a worktree of the primary #1761/feat(daemon): a confined session's runtime HOME lives under its own directory #1763 behavior is byte-identical (daemon-session-hosts.test.ts asserts the predicate ignores isolation off the pool and the local policy directory is unchanged).
• isolation: 'shared' on the pool: one pod per agent, sessions in the primary checkout.
• The warm-pool adoption property (claim spec carries only warmPoolRef and pod labels).
• Wire: shim/hello still carries the real agent id (the pod's skill handler checks it); the subject rides SpawnRecord.subject.

Migration note

Legacy <mount>/worktrees/<sid> session worktrees on an agent pod are no longer prepared or addressed; a pre-existing isolated session's next turn clones fresh onto its own pod, and retention still removes the legacy worktree on the agent pod (removeSessionWorktree(…, 'worktrees')). The pool learns a session's isolation from the requests that reach the member (sessionIsolation); a session-shaped host key the pool does not know as isolated (a dream, a model-session host) stays in the agent pod.

Verification

Unit level only — no live cluster was exercised.

• packages/daemon/test/k8s-session-pods.test.ts (new): claim keyed per session host with agent+session labels and no per-session spec; two sessions → two pods, agent pod untouched; companion hold and release on runtime exit (both suspend as idle); suspending one session pod leaves its sibling and the agent pod; restart survival (a second member converges on the same claim, creating nothing); label-based adoption; agent removal deletes every pod; single-session retirement; agent path unchanged. Mutation-checked: a claim name that ignores the session leaf fails 6 of 9 cases.
  • Round 2 adds: the pod a path lives on is read off the path (sandboxSubjectForPath, including the neighbours a prefix match would swallow and a mount that is not the default); a companion that bound after the session bind failed is released, not leaked; a companion that alone cannot come up still degrades. Mutation-checked: Promise.all again ⇒ 1 red; a companion rejection failing the launch ⇒ 3 red; routing any second path segment to a session pod ⇒ 1 red; a prefix rather than a segment match ⇒ 1 red; ignoring the mount ⇒ 3 red across two files.
  • Round 3 adds: a resume onto the observed claim binds without a create, and refuses once retention deleted that claim or replaced it with a different object of the same name; retainLaunched makes suspendIfIdle answer busy for that pod alone, is idempotent, and answers nothing (claiming nothing) for a pod this member holds no launch for. Mutation-checked: the resume calling ensureSandbox ⇒ 3 red across two files; a fence on "a claim exists" rather than on its uid ⇒ 1 red; a hold that retains nothing ⇒ 1 red.
• sandbox-keep-alive.test.ts (round 3): an idle-suspended session pod is neither read nor woken however up the agent pod is; a watcher armed in a bound agent pod keeps its lease there while this page's session pod sleeps, still without reading the sleeping one; both pods asleep drops every lease on both; a dirty tree leases its own pod and neither the agent's nor a sibling's; an armed watcher leases the agent pod while the dirty tree leases the session pod, in one poll; the pod is held for the whole status read and released after it, including when the read throws; an unroutable page falls back to the agent pod. Mutation-checked: falling back to "any pod of the agent is bound" ⇒ 2 red; the dirty-tree lease back on the agent key ⇒ 2 red; releasing the hold before the read instead of after ⇒ 1 red; this page's sleeping pod short-circuiting the agent pod's watcher lease (the regression itself) ⇒ 3 red.
• k8s-sandbox-lease.test.ts: the two-key suspendIfIdle case — a session pod mid-suspend neither blocks nor is blocked by its siblings or the agent pod. Mutation-checked: an agent-keyed suspension gate fails it.
• k8s-runtime-plane.test.ts: a session host binds its own pod; <mount>/sessions/<leaf> layout in pod coordinates; path routing (session directory → session pod, everything else refuses while the agent pod is down; memory/auto-merge stay the agent pod's). Round 2 adds: a read of an idle-suspended session's directory waits for that session's pod and refuses when its channel does not arrive, while the agent pod — bound throughout — is never asked and still answers its own paths, and no new claim is made; and with nothing bound the read refuses without waking anything. Mutation-checked: gating the wake on the live-launch registry ⇒ 1 red Round 3 adds: a read whose session claim is deleted between the router's observation and the resume's re-read refuses, creates nothing, leaves the agent pod's claim and launch untouched, and the agent pod still answers its own paths. Mutation-checked: the wake going back to ensureBoundChannel ⇒ 1 red.
• cluster-workspace-prepare.test.ts: the pool's isolated tier is the blobless clone under sessions/<leaf>/workspace (and repos/<owner>/<repo> for secondaries), reviews fetch into the clone, retention judges the clone, console root composition, a retired secondary root no longer holds the session's copy. Round 2 adds: an activation rejected before its preparation ran retires nothing, and neither do the restored definition's next turns; the conversion retires the others and spares the leaf it is preparing, emptying that leaf's directory and re-cloning from the new repository, after which a sibling session retires nobody; a shared session names no leaf to spare; a retirement that raised fails the preparation closed and is retried by the next one. Mutation-checked: the discard back at activation ⇒ 3 red; sparing no leaf ⇒ 1 red; swallowing the retirement error ⇒ 1 red; the confined branch skipping the agent pod's half ⇒ 1 red; that branch running the whole half ⇒ 3 red.
• daemon-k8s-mode.test.ts: isolated → session host key and pod subject, shared → agent host key; dream/model-session keys stay in the agent pod; capacity counts agents. Round 3 adds: the idle sweep judges the hold per POD — a lease on one session pod leaves the agent pod and a sibling session pod suspendable. Mutation-checked: the sweep asking holds(agentId) ⇒ 1 red.
• daemon-session-hosts.test.ts: self-hosted predicate and policy unchanged.
• pnpm --filter @agentconnect.md/daemon exec tsc -p tsconfig.typecheck.json --noEmit, eslint and prettier on the touched files; the k8s, shim, cluster, workspace and daemon-pool suites listed above pass. The full daemon suite was not run (known to OOM on this machine).

Design docs: docs/designs/k8s-daemon-pool.md §4 and docs/designs/git-workspace-model.md §11 carry the pool paragraph. Round 3 updates the §4 keep-alive and wake paragraphs: the sweep skips a POD rather than an agent, the pod keep-alive judges for the tree is the page's own and is held across its read while the watcher is judged independently on the agent's, and a read RESUMES a sleeping session pod onto the claim it observed rather than waking whatever the name then holds.

🤖 Generated with Claude Code

[agentconnect-md-test]

Requesting changes for three regressions introduced by the per-session pod lifecycle:

• Workspace replacement deletes every retained session claim/PVC before the configuration activation is committed, so a later activation rejection restores the old definition after its sessions’ clones and HOME have already been lost.
• Once an idle session pod is suspended, console workspace reads can no longer address or wake its retained volume; the existing agent/wake path starts only the agent pod and path routing knows only currently launched session subjects.
• A failed session-pod bind can leave the concurrently started companion agent pod permanently retained, preventing idle suspension until the daemon restarts.

The subject identity, per-pod binding, and normal success/exit paths otherwise look coherent. The orphan-collection limitation is explicitly scoped to the stated stacked follow-up and is not treated as blocking here. git diff --check passes. I could not run the targeted daemon tests or typecheck because this isolated checkout has no installed dependencies and Corepack could not resolve registry.npmjs.org, including after the required network escalation; that is an environment limitation, not a project failure.

_{sent by review-bot (Codex · gpt-5.6-sol) · open in session}

[zfy0701]

@agentconnect-md/review-bot review

[zfy0701]

@agentconnect-md/review-bot review

[zfy0701]

@agentconnect-md/review-bot review

[agentconnect-md-test]

The three findings from the previous round are resolved: session-volume retirement now runs at the fail-closed conversion gate, path routing no longer depends on the live-launch registry, and the companion attempt settles before launch holds are drained.

Two newly exposed subject-lifecycle issues remain. First, the read wake performs a separate hasClaim check before calling a path that may create the claim, so retention or workspace conversion can delete the claim in between and the read recreates an empty session volume. Second, the existing keep-alive flow remains agent-scoped: with runsInSandbox now true for any bound pod, a bound agent pod lets a clean session-page poll run Git status, whose deferred runner wakes that session pod after the idle sweep suspended it; the same agent-wide hold also pins sibling pods.

git diff --check passes, and the current GitHub Build, Check, unit, integration, sandbox, PostgreSQL-store, and Windows jobs pass. Local focused tests were not run because this review checkout has no installed dependencies; Corepack itself was verified with a task-scoped writable cache.

_{sent by review-bot (Codex · gpt-5.6-sol) · open in session}

[zfy0701]

@agentconnect-md/review-bot review

[agentconnect-md-test]

The resume-only path now carries the observed claim UID through a non-creating driver operation, and keep-alive routing/leases are correctly subject-keyed for dirty worktrees. One blocking edge remains in the split keep-alive flow: once a clean session pod suspends, the next poll also drops the still-valid auto-merge-armed lease on the agent pod, so that pod can be swept and the armed watcher silently lost while the page remains visible.

_{sent by review-bot (Codex · gpt-5.6-sol) · open in session}

[agentconnect-md-test]

The delivered head is unchanged from the previous review, and the remaining P2 still applies. When an isolated session is clean, its own pod receives no persistent lease and may suspend while the visible page keeps the agent pod alive for auto-merge-armed. The next keep-alive poll then takes the early !release branch for the sleeping session pod, skips armedFor, and releases that page holder's agent-pod lease. The agent pod can subsequently be swept and the armed merge watcher silently lost even though the page remains visible. The agent-pod watcher reason needs to be evaluated and held independently of the session-pod status path; Git status should remain skipped so the sleeping session pod is not woken.

_{sent by review-bot (Codex · gpt-5.6-sol) · open in session}

[zfy0701]

@agentconnect-md/review-bot review

[agentconnect-md-test]

The follow-up fixes the remaining keep-alive regression. Watcher state is now judged under a hold on the agent pod independently from the dirty-tree read on the session pod, so an idle-suspended session remains untouched while an armed merge watcher continues to lease its own pod. The same-pod path correctly shares one hold, missing pods clear only their own stale leases, and the response can accurately report an asleep worktree pod alongside an active watcher lease. Together with the earlier UID-fenced, non-creating resume path and subject-keyed sweep, I found no remaining blocking issue in this revision.

_{sent by review-bot (Codex · gpt-5.6-sol) · open in session}

[agentconnect-md-test]

Re-reviewed the unchanged 518135d0 revision. The observed-UID resume path remains non-creating, dirty-tree routing and idle holds remain scoped to the owning sandbox subject, and the follow-up independently preserves the agent-pod lease for an armed merge watcher while skipping reads of a sleeping session pod. I found no blocking issue; the remaining CI jobs are still in progress.

_{sent by review-bot (Codex · gpt-5.6-sol) · open in session}