feat(overseer): query_open_loops — "what am I forgetting?" lens

docs/plans/2026-07-31-overseer-open-loops-implementation.md

@@ -0,0 +1,85 @@
+# Overseer "what am I forgetting?" — cold open-loops lens (implementation)
+
+> **Branch:** `feat/overseer-open-loops` (stacked on `feat/overseer-text-converse`).
+> **Design + evidence:** [`2026-07-31-overseer-forgotten-open-loops-lens.md`](./2026-07-31-overseer-forgotten-open-loops-lens.md)
+> (validated live against the 27B on `:3006`). This doc = what landed.
+
+The lens is the **neglect axis** — "what have I abandoned?" — orthogonal to the urgency axis
+("what needs me now?", `query_inbox` + `explain_priority`). It is self-populating from the
+`AGENT_NOTIFY_SUMMARY` each worker turn already emits; **no operator triage required**.
+
+## What landed (build path steps 1 + 2)
+
+### Step 2 — `query_open_loops` read-only tool
+
+An 8th read-only Overseer tool. Definition:
+
+> **A cold open loop** = a session whose *latest* status-bearing worker event is NOT `done`
+> (`needs_decision` / `needs_review` / `blocked` / `failed` / `stalled`) and was never closed by
+> a later `completed`.
+
+- **Substrate:** raw `events`, not the coalesced inbox (the design showed events surface ~10 real
+  forgotten decisions that never became inbox items). One indexed query takes the latest
+  status-bearing worker event **per session** (`progress` is excluded — a progress ping does not
+  close an operator-owed decision; `completed` is the only closer).
+- **Strong filter:** `status != done`. A no-op `action` ("none"/"complete"/"n/a"/…) is nulled but
+  the loop still surfaces — action text is a *tiebreak*, not the filter (per the spec correction
+  that killed 108 Tier-B false positives).
+- **Buckets:** `waiting_on_you` (needs_decision / needs_review — the operator owes a decision) is
+  presented **before** `half_finished` (blocked / failed / stalled). Each bucket is **coldest-first**.
+- **Args:** `{ minAgeMs?, bucket?, project?, limit? }` — `minAgeMs` is the "went cold" knob (default 0,
+  raise it to focus on genuinely stale threads).
+- **Returns:** `{ openLoops: [{ sessionId, name, project, flavor, status, eventType, eventId, action,
+  summary, lastTs, ageMs, ageDays, bucket }], counts: { total, waitingOnYou, halfFinished } }`.
+  The brain-facing projection thins this to `{ id, name, project, status, action, what, ageDays, bucket }`.
+
+### System-prompt changes (converse/entity layer)
+
+- **Two questions, two axes** section: urgency (`query_inbox`, priority-ordered) vs neglect
+  (`query_open_loops`, age-ordered). Tells the brain which tool answers "what am I forgetting?".
+- **Priority direction fix:** priority is **lower-is-higher** (1 = most important). This corrects the
+  27B's live mistake of calling p50 "highest".
+
+## Step 1 — zero-code weekly digest (works today, no substrate change)
+
+`query_open_loops` makes the "what am I forgetting?" converse prompt reliable. A scheduled weekly
+digest can send this to `POST /api/overseer/converse` and post the reply once (never an interrupt):
+
+```
+What have I forgotten or abandoned? Use query_open_loops (minAgeMs = 3 days). Lead with the
+"Waiting on You" bucket — decisions I owe — then half-finished work. For each, one line: what it is,
+how many days cold, and the concrete next step (skip ones with no real next step). Do not rank by
+priority; this is about neglect, not urgency. Keep it to the top ~15.
+```
+
+This doubles as a **triage bootstrap**: surface ~15 cold loops, operator dispositions them, and the
+inbox disposition loop the spec always wanted becomes tractable.
+
+## Ingest-peer handoff items (PR #99 → this layer)
+
+- **H1 (done):** system-prompt rule — when the operator asks about a *specific* inbox item, the brain
+  first calls `explain_priority` then `query_events{sessionId}` to pull the rest of that session's
+  recorded activity as salience (capability already existed; `sessionId` is an accepted arg).
+- **H2 (done):** a two-level `detail: 'lean' | 'full'` knob (default `lean`) on every context tool
+  (`query_events` / `query_inbox` / `get_session_state` / `get_session_recent_output` /
+  `get_worker_health` / `list_active_workers` / `query_open_loops`), threaded into
+  `projectToolResultForBrain(tool, result, detail)`. Coverage gap closed: `get_session_state`,
+  `get_session_recent_output` (raw terminal text capped at 280 chars in lean — was a token bomb), and
+  `get_worker_health` (signal trail dropped in lean) now have lean projections. `full` returns the raw
+  rows, still bounded by `limit`/`n` and the outer char clamp. Deliberately NOT a token-budget engine —
+  two levels + good defaults.
+- **H3 (deferred):** `query_session_actions` reader over `inbox_operator_actions` — deferred until
+  disposition volume justifies it (~0 today). ~30-line add when wanted.
+- **H4 (confirmed):** `query_open_loops` spans **all non-deleted sessions** (active AND archived). It
+  reads only the events table and never filters on `session.active`; deleted sessions drop out because
+  `deleteSession` detaches their events (`related_session_id = NULL`).
+
+## Not in this branch (follow-ups)
+
+- **Archiving hygiene (step 3):** aggressive session archive + sweeping legacy `stale`
+  ("No agent output for 30 minutes") rows that predate the fix which stopped writing them
+  (`checkStaleSessions` already returns `[]`; those legacy rows are `source_kind=system` so
+  `query_open_loops` — worker-only — already excludes them from the lens, but they still bloat
+  Session Logs). Coordinate the sweep with the inbox-ingest lane.
+- **Dependency:** inbox PR-title + priority-band fix (PR #99) lands on the urgency axis, independent
+  of this lens.

hub/src/overseer/converse.ts

@@ -124,8 +124,10 @@ export async function runOverseerConverse(params: {
             try {
                 const result = runOverseerTool(overseer, name, args)
                 toolTrace.push({ tool: name, args, ok: true })
-                const lean = projectToolResultForBrain(name, result)
-                resultLines.push(`${name}(${argsRaw}) => ${clampToolResult(JSON.stringify(lean ?? null))}`)
+                // The brain opts into 'full' per call when it needs depth; default lean.
+                const detail = args.detail === 'full' ? 'full' : 'lean'
+                const projected = projectToolResultForBrain(name, result, detail)
+                resultLines.push(`${name}(${argsRaw}) => ${clampToolResult(JSON.stringify(projected ?? null))}`)
             } catch (error) {
                 const msg = error instanceof Error ? error.message : String(error)
                 toolTrace.push({ tool: name, args, ok: false, error: msg })

hub/src/overseer/runOverseerTool.ts

@@ -34,6 +34,8 @@ export function runOverseerTool(overseer: OverseerEntity, tool: OverseerToolName
         }
         case 'list_active_workers':
             return { workers: overseer.listActiveWorkers(overseerToolArgsSchemas.list_active_workers.parse(args)) }
+        case 'query_open_loops':
+            return overseer.queryOpenLoops(overseerToolArgsSchemas.query_open_loops.parse(args))
         default: {
             const exhaustive: never = tool
             throw new Error(`Unknown overseer tool: ${String(exhaustive)}`)

hub/src/overseer/toolProjection.test.ts

@@ -67,8 +67,52 @@ describe('projectToolResultForBrain', () => {
         expect(JSON.stringify(lean)).not.toContain('flavor')
     })
 
-    it('passes un-projected tools through untouched', () => {
-        const state = { state: { sessionId: 'x', observedState: 'idle' } }
-        expect(projectToolResultForBrain('get_session_state', state)).toBe(state)
+    it('thins open loops to id/name/project/status/action/what/ageDays/bucket', () => {
+        const raw = {
+            counts: { total: 2, waitingOnYou: 1, halfFinished: 1 },
+            openLoops: [
+                { sessionId: 'a', name: 'peer-a', project: 'web', flavor: 'cursor', status: 'needs_decision', eventType: 'needs_decision', eventId: 5, action: 'choose target', summary: 'peer-a needs_decision', lastTs: 111, ageMs: 999, ageDays: 10, bucket: 'waiting_on_you' }
+            ]
+        }
+        const lean = projectToolResultForBrain('query_open_loops', raw) as { counts: unknown; openLoops: unknown[] }
+        expect(lean.counts).toEqual({ total: 2, waitingOnYou: 1, halfFinished: 1 })
+        expect(lean.openLoops[0]).toEqual({ id: 'a', name: 'peer-a', project: 'web', status: 'needs_decision', action: 'choose target', what: 'peer-a needs_decision', ageDays: 10, bucket: 'waiting_on_you' })
+        expect(JSON.stringify(lean)).not.toContain('eventId')
+        expect(JSON.stringify(lean)).not.toContain('lastTs')
+    })
+
+    it('thins session state to observed/reported essentials', () => {
+        const raw = { state: { sessionId: 'sess-a', name: 'peer-a', project: 'web', flavor: 'codex', active: true, thinking: false, observedState: 'idle', workerReportedState: 'blocked', lastActivityAt: 999, silenceMs: 1200, lastToolCallAgeMs: 500, pendingRequestCount: 1 } }
+        const lean = projectToolResultForBrain('get_session_state', raw) as { state: Record<string, unknown> }
+        expect(lean.state).toEqual({ id: 'sess-a', name: 'peer-a', project: 'web', observed: 'idle', reported: 'blocked', silenceMs: 1200, pending: 1 })
+        expect(JSON.stringify(lean)).not.toContain('lastToolCallAgeMs')
+    })
+
+    it('caps raw transcript chunk text (token bomb) in lean mode', () => {
+        const long = 'x'.repeat(1000)
+        const raw = { chunks: [{ messageId: 'm1', role: 'worker', text: long, createdAt: 5 }] }
+        const lean = projectToolResultForBrain('get_session_recent_output', raw) as { total: number; chunks: Array<{ text: string }> }
+        expect(lean.total).toBe(1)
+        expect(lean.chunks[0]!.text.length).toBeLessThan(300)
+        expect(lean.chunks[0]!.text.endsWith('…')).toBe(true)
+    })
+
+    it('thins worker health and drops the verbose signal trail', () => {
+        const raw = { health: { sessionId: 'sess-a', name: 'peer-a', project: 'web', flavor: 'codex', reportedState: 'blocked', observedState: 'stale', inferredState: 'blocked', inferredConfidence: 0.9, signals: ['a', 'b', 'c'], lastActivityAt: 1, silenceMs: 60000, pendingRequestCount: 0 } }
+        const lean = projectToolResultForBrain('get_worker_health', raw) as { health: Record<string, unknown> }
+        expect(lean.health).toEqual({ id: 'sess-a', name: 'peer-a', project: 'web', reported: 'blocked', observed: 'stale', inferred: 'blocked', confidence: 0.9, silenceMs: 60000, pending: 0 })
+        expect(JSON.stringify(lean)).not.toContain('signals')
+    })
+
+    it('detail:full returns the raw rows untouched', () => {
+        const raw = { chunks: [{ messageId: 'm1', role: 'worker', text: 'x'.repeat(1000), createdAt: 5 }] }
+        expect(projectToolResultForBrain('get_session_recent_output', raw, 'full')).toBe(raw)
+        const inbox = { items: [{ id: 1, title: 't', status: 'new', priority: 5, reasonForPriority: 'r' }] }
+        expect(projectToolResultForBrain('query_inbox', inbox, 'full')).toBe(inbox)
+    })
+
+    it('passes un-projected tools (explain_priority) through untouched', () => {
+        const explanation = { explanation: { inboxItemId: 1, title: 'x' } }
+        expect(projectToolResultForBrain('explain_priority', explanation)).toBe(explanation)
     })
 })

hub/src/overseer/toolProjection.ts

@@ -11,12 +11,28 @@ import type { OverseerToolName } from '@hapi/protocol'
  *
  * Projection is applied ONLY on the converse path (brain-facing). The HTTP tool
  * endpoint and debug panels still get the full rows.
+ *
+ * The `detail` knob is a deliberately TWO-LEVEL switch (no token-budget engine):
+ *  - `lean` (default) — the cheap shapes below (~20x smaller).
+ *  - `full` — the raw richer rows, still bounded by the tool's `limit`/`n` arg and
+ *    by the outer MAX_TOOL_RESULT_CHARS clamp in the converse loop.
+ * The brain opts into `full` per call when it genuinely needs depth.
  */
 
+export type ToolResultDetail = 'lean' | 'full'
+
+/** Cap raw transcript chunk text in lean mode (raw terminal output is a token bomb). */
+const LEAN_CHUNK_TEXT_MAX = 280
+
 function isObj(value: unknown): value is Record<string, unknown> {
     return typeof value === 'object' && value !== null
 }
 
+function truncate(text: unknown, max: number): unknown {
+    if (typeof text !== 'string') return text
+    return text.length > max ? `${text.slice(0, max)}…` : text
+}
+
 /**
  * Inbox item → the minimum for triage:
  *  - `id`       — to reference it (explain_priority, follow-ups)
@@ -64,13 +80,84 @@ function projectWorker(worker: unknown): Record<string, unknown> {
     }
 }
 
-export function projectToolResultForBrain(tool: OverseerToolName, result: unknown): unknown {
+/** Session state → the observed/reported essentials; drops raw activity timestamps. */
+function projectSessionState(state: unknown): Record<string, unknown> {
+    const o = isObj(state) ? state : {}
+    return {
+        id: o.sessionId,
+        name: o.name,
+        project: o.project,
+        observed: o.observedState,
+        reported: o.workerReportedState,
+        silenceMs: o.silenceMs,
+        pending: o.pendingRequestCount
+    }
+}
+
+/** Transcript chunk → role + capped text (raw terminal output would blow the window). */
+function projectChunk(chunk: unknown): Record<string, unknown> {
+    const o = isObj(chunk) ? chunk : {}
+    return { role: o.role, at: o.createdAt, text: truncate(o.text, LEAN_CHUNK_TEXT_MAX) }
+}
+
+/** Worker health → the three states + confidence; drops the verbose signal trail (full only). */
+function projectWorkerHealth(health: unknown): Record<string, unknown> {
+    const o = isObj(health) ? health : {}
+    return {
+        id: o.sessionId,
+        name: o.name,
+        project: o.project,
+        reported: o.reportedState,
+        observed: o.observedState,
+        inferred: o.inferredState,
+        confidence: o.inferredConfidence,
+        silenceMs: o.silenceMs,
+        pending: o.pendingRequestCount
+    }
+}
+
+/** Open loop → the minimum for the "what am I forgetting?" answer. */
+function projectOpenLoop(loop: unknown): Record<string, unknown> {
+    const o = isObj(loop) ? loop : {}
+    return {
+        id: o.sessionId,
+        name: o.name,
+        project: o.project,
+        status: o.status,
+        action: o.action,
+        what: o.summary,
+        ageDays: o.ageDays,
+        bucket: o.bucket
+    }
+}
+
+export function projectToolResultForBrain(
+    tool: OverseerToolName,
+    result: unknown,
+    detail: ToolResultDetail = 'lean'
+): unknown {
+    // `full` returns the raw rows — still bounded by the tool's limit/n arg and
+    // the converse loop's outer char clamp. No thinning applied.
+    if (detail === 'full') return result
+
     if (tool === 'query_events' && isObj(result) && Array.isArray(result.events)) {
         return { total: result.events.length, events: result.events.map(projectEvent) }
     }
     if (tool === 'list_active_workers' && isObj(result) && Array.isArray(result.workers)) {
         return { total: result.workers.length, workers: result.workers.map(projectWorker) }
     }
+    if (tool === 'query_open_loops' && isObj(result) && Array.isArray(result.openLoops)) {
+        return { counts: result.counts, openLoops: result.openLoops.map(projectOpenLoop) }
+    }
+    if (tool === 'get_session_state' && isObj(result) && 'state' in result) {
+        return { state: result.state == null ? null : projectSessionState(result.state) }
+    }
+    if (tool === 'get_session_recent_output' && isObj(result) && Array.isArray(result.chunks)) {
+        return { total: result.chunks.length, chunks: result.chunks.map(projectChunk) }
+    }
+    if (tool === 'get_worker_health' && isObj(result) && 'health' in result) {
+        return { health: result.health == null ? null : projectWorkerHealth(result.health) }
+    }
     if (tool === 'query_inbox' && isObj(result) && Array.isArray(result.items)) {
         // The raw result is {items, candidates, surfaced, held} — four arrays that
         // repeat the same rows (a big part of the ~75k-token bloat). We keep only

hub/src/store/eventStore.ts

@@ -6,6 +6,7 @@ import {
     insertSystemEvent,
     listSystemEvents,
     queryEvents,
+    queryLatestWorkerStatusPerSession,
     repointSessionEvents,
     type InsertSystemEventInput,
     type ListSystemEventsOptions,
@@ -30,6 +31,11 @@ export class EventStore {
         return queryEvents(this.db, options)
     }
 
+    /** Latest status-bearing worker event per session (cold-open-loops substrate). */
+    latestWorkerStatusPerSession(limit?: number): StoredSystemEvent[] {
+        return queryLatestWorkerStatusPerSession(this.db, limit)
+    }
+
     getById(id: number): StoredSystemEvent | null {
         return getSystemEventById(this.db, id)
     }

hub/src/store/events.ts

@@ -294,6 +294,34 @@ export function queryEvents(db: Database, options: QueryEventsOptions = {}): Sto
     return rows.map(mapRow)
 }
 
+/**
+ * Latest status-bearing worker event per session — the substrate for the
+ * cold-open-loops lens. "Status-bearing" = the notify-derived types that either
+ * open a loop (needs_decision/needs_review/blocked/failed/stale) or close it
+ * (completed); `progress` is intentionally excluded so a progress ping does not
+ * mask an unanswered decision. The caller decides open vs closed by inspecting
+ * the returned event's type. Bounded and index-friendly (one row per session).
+ */
+export function queryLatestWorkerStatusPerSession(db: Database, limit = 500): StoredSystemEvent[] {
+    const cap = Math.min(Math.max(limit, 1), 2000)
+    const rows = db.prepare(`
+        SELECT e.* FROM events e
+        JOIN (
+            SELECT related_session_id AS sid, MAX(id) AS max_id
+            FROM events
+            WHERE source_kind = 'worker'
+              AND related_session_id IS NOT NULL
+              AND event_type IN (
+                  'needs_decision', 'needs_review', 'blocked', 'failed', 'stale', 'completed'
+              )
+            GROUP BY related_session_id
+        ) latest ON e.id = latest.max_id
+        ORDER BY e.ts ASC
+        LIMIT ?
+    `).all(cap) as SystemEventRow[]
+    return rows.map(mapRow)
+}
+
 export function insertEventLink(
     db: Database,
     input: {