Skip to content

app-server client never marks itself unusable after child exit — two unbounded hangs in the broker dispatch path (1.0.4) #665

Description

@90stm

Summary

In plugin version 1.0.4, when the spawned codex app-server child dies, the client in scripts/lib/app-server.mjs never marks itself unusable — handleExit() rejects the requests already in pending and clears the map, but nothing anchors future or non-RPC waits to the connection's death. This produces two distinct unbounded hangs in the broker dispatch path (scripts/app-server-broker.mjs), which on our machine silently lost 3 of 5 dispatches over two days while codex exec (which does not use the broker) worked perfectly, masking the failure.

Defect 1 — requests issued after child death hang forever

After the child exits, handleExit() has already cleared pending. A subsequent request() writes to a dead pipe and registers a pending entry no code path can ever reject or resolve. The caller hangs unboundedly (we observed dispatch workers parked 24+ minutes at 0.0% CPU, no children).

Compounding it: the broker keeps its Unix socket listening with a dead child behind it, and endpoint readiness is judged by socket-connect success alone — so every subsequent client reuses the broken broker indefinitely.

Repro: start the broker, let it spawn its app-server child, kill the child, then issue any request (e.g. thread/start) through the broker. It never resolves, never rejects, no timeout, no log.

Defect 2 — child death during the notification wait hangs forever

captureTurn in scripts/lib/codex.mjs awaits state.completion — a promise whose rejectCompletion is created (~line 302 in 1.0.4) but never called anywhere in the codebase. The RPC for review/start / turn/start resolves when the turn starts; the real work (8–9 minutes for a review) is awaited on the notification stream via state.completion, settled only by turn/completed (or the inferred-completion timer, which arms only after a final answer is seen). If the app-server dies in that window, handleExit() rejects only the — by then empty — pending map, and nothing rejects state.completion: the dispatcher hangs forever. Since nearly all of a review's wall-clock lives in this window, this is the dominant hang in practice.

Repro: start a review, wait until the review/start RPC has returned and notifications are flowing, then kill -9 the app-server child. The dispatcher never settles.

Root cause, generalized

handleExit marks the connection finished, but no consumer wait is anchored to it. Any wait that only an expected message can settle becomes a permanent hang when the peer dies.

Suggested fix (what we patched locally; verified over 24/24 dispatches after 2/5 before)

  1. Derive a single rejecting promise from exitPromise (e.g. connectionLoss, with a no-op .catch attached so it can't become an unhandled rejection), and race every unbounded consumer wait against it — the promise returned by request(), and captureTurn's state.completion. This adds no timeout to the notification stream, so long-running reviews are unaffected.
  2. Guard request()/notify() admission on closed || exitResolved.
    Caveat: do not implement this by setting closed = true inside handleExithandleExit is also reachable with the child still alive (a JSONL parse failure at ~line 126), and close()'s early return on this.closed then skips readline.close(), stdin.end(), and the SIGTERM timer, leaking the child process. We hit this on our first attempt; it is reproducible. Keep two flags: closed = "close() ran" (gates teardown), exitResolved = "connection finished" (gates admission).
  3. In the broker: exit the process when the app-server client dies, so the socket disappears and ensureBrokerSession respawns a clean broker. Without this, fix 2 turns the infinite hang into infinite errors, because endpoint readiness passes on socket connect alone.

Happy to provide our full local patches or test harness details if useful.

Environment

  • macOS (darwin 25.x), Node 23.11 (note: Node's default unhandled-rejection behavior is fatal, relevant to any timer/race added around these paths)
  • codex-plugin-cc 1.0.4 (Claude Code plugin)
  • Codex CLI current as of 2026-08-21

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions