feat(overseer): opt-in LLM summary fallback (#90)

docs/plans/2026-07-24-overseer-summary-emission.md

@@ -1,7 +1,8 @@
 # Overseer summary emission (Half B) — 2026-07-24
 
 Status: Piece 1 live; Piece 2 hub text-synth **removed**; Piece 3 in flight
-(Claude remote + Codex app-server only; Grok/OpenCode deferred to #89)
+(Claude remote + Codex app-server only; Grok/OpenCode deferred to #89).
+Option A LLM fallback implemented (default OFF, #90).
 Owner: feat/overseer-summary-emit (peer of 🔁overseer prep)
 Scope: FORK-ONLY. Never upstream. The whole overseer feature is fork-private.
 
@@ -133,7 +134,9 @@ ordinary, useful project config, never as surveillance:
 - Grok / OpenCode first-turn prepend: title + skill-lookup only. Session-summary
   for those flavors is [#89](https://github.com/heavygee/hapi/issues/89).
 - kimi + generic ACP / pi: also #89.
-- Better LLM / oneshot-agent fallback: [#90](https://github.com/heavygee/hapi/issues/90), default off.
+- Better LLM / oneshot-agent fallback: Option A implemented behind
+  `HAPI_OVERSEER_LLM_FALLBACK` (default off); see § Better fallback / #90.
+  Option B oneshot agent remains out of scope.
 
 ## Better fallback (opt-in — tracked #90)
 
@@ -146,7 +149,7 @@ and is a real cost tax - so it must be **opt-in**, clearly labeled, and rare
 
 ### Gate: rarity first, quality never second
 
-Do **not** ship a better fallback until primary emission is good enough that
+Do **not** enable a better fallback until primary emission is good enough that
 fallback is a thin residue - target **well under 5% of turns** (5% is already
 generous). Measure emit vs missing-line ratio fleet-wide after Piece 3 is
 live; only then enable LLM fallback.
@@ -156,18 +159,34 @@ feed the **full last-turn assistant content** (no input-char truncation that
 would make the summary worse than the agent would have written). Rarity is the
 cost control; accuracy is non-negotiable on the rare path.
 
-### Option A — raw OpenAI-compatible completions call
+### Option A — raw OpenAI-compatible completions call (implemented)
 
-Hub (or a tiny side worker) POSTs the full last assistant turn text to an
-operator-configured base URL (`/v1/chat/completions` or `/v1/responses`) with a
-fixed prompt: "emit exactly one AGENT_NOTIFY_SUMMARY JSON line." Local (Ollama /
-vLLM / gateway) or remote (OpenAI) - same wire format.
+Hub POSTs the full last assistant turn text to an operator-configured base URL
+(`/v1/chat/completions` or `/v1/responses`) with a fixed prompt: "emit exactly
+one AGENT_NOTIFY_SUMMARY JSON line." Local (Ollama / vLLM / gateway) or remote
+(OpenAI) - same wire format.
 
-- Pros: cheap to wire, no session surface, easy to bill/attribute as
-  `provenance: hub-llm-fallback`.
-- Cons: large turns = large prompt tokens (accepted when rare); operator must
-  provision a key/URL; prefer Chat Completions for local-gateway compatibility,
-  Responses for OpenAI-native - support both behind one adapter.
+**Enable (default OFF — never surprise usage):**
+
+```bash
+export HAPI_OVERSEER_LLM_FALLBACK=1
+export HAPI_OVERSEER_LLM_BASE_URL=http://127.0.0.1:11434/v1   # include /v1
+export HAPI_OVERSEER_LLM_MODEL=llama3.3
+# optional:
+export HAPI_OVERSEER_LLM_API_KEY=ollama                       # Bearer token; empty OK for local
+export HAPI_OVERSEER_LLM_API=chat-completions                # or: responses
+export HAPI_OVERSEER_LLM_TIMEOUT_MS=30000
+```
+
+Prefer `chat-completions` for local-gateway compatibility; use `responses` for
+OpenAI-native. Failures / non-compliant model output produce **no** Session Log
+row (no first-line heuristic). Session-end may still write `completed_fallback`
+if the session completes without a later successful notify/LLM row. Events are marked
+`provenance: hub-llm-fallback ...` with `payload.synthesis = "llm-fallback"`,
+`attentionCandidate = 0` (Session Log only — not inbox / voice).
+
+**Cost warning:** every missed primary emit becomes a full-turn prompt. Enable
+only after the rarity gate, or accept the bill deliberately.
 
 ### Option B — out-of-band oneshot agent
 
@@ -180,6 +199,7 @@ in Session Log / inbox so the operator never wonders "wtf usage is this."
   multi-step retrieval if needed.
 - Cons: heavier; looks like a phantom session if not carefully labeled; higher
   cost variance; more moving parts.
+  **Out of scope for #90** — revisit only if Option A proves insufficient.
 
 ### Shared requirements (either option)
 
@@ -192,7 +212,7 @@ in Session Log / inbox so the operator never wonders "wtf usage is this."
   primary turn lacked a contract - never pretend the primary agent said it.
 - **Kill-criterion:** if opt-in users report surprise usage, the toggle and
   provenance labels failed - fix UX before expanding defaults. If fallback
-  summaries are worse than the heuristic first-line, do not ship.
+  summaries are worse than a primary `AGENT_NOTIFY_SUMMARY` emit, do not ship.
 
 Prefer **Option A** as the first better-fallback ship: smaller blast radius,
 easier to reason about cost, no phantom sessions.

hub/README.md

@@ -42,6 +42,19 @@ See `src/configuration.ts` for all options.
 - `HAPI_RELAY_FORCE_TCP` - Force TCP relay mode (true/1).
 - `VAPID_SUBJECT` - Contact email/URL for Web Push.
 
+### Optional (Overseer LLM fallback — fork, default OFF)
+
+Only when primary agents omit `AGENT_NOTIFY_SUMMARY`. Costs a full-turn LLM call
+per miss — enable after miss rate is rare (~<5%). See
+`docs/plans/2026-07-24-overseer-summary-emission.md`.
+
+- `HAPI_OVERSEER_LLM_FALLBACK` - `1`/`true` to enable (default: off).
+- `HAPI_OVERSEER_LLM_BASE_URL` - OpenAI-compatible base including `/v1` (required when enabled).
+- `HAPI_OVERSEER_LLM_MODEL` - Model id (required when enabled).
+- `HAPI_OVERSEER_LLM_API_KEY` - Bearer token (optional for local gateways).
+- `HAPI_OVERSEER_LLM_API` - `chat-completions` (default) or `responses`.
+- `HAPI_OVERSEER_LLM_TIMEOUT_MS` - Request timeout (default: 30000).
+
 ## Running
 
 Binary (single executable):

hub/src/configuration.ts

@@ -21,6 +21,8 @@
  * - VAPID_SUBJECT: Contact email or URL for Web Push (defaults to mailto:admin@hapi.run)
  * - HAPI_HOME: Data directory (default: ~/.hapi)
  * - DB_PATH: SQLite database path (default: {HAPI_HOME}/hapi.db)
+ * - HAPI_OVERSEER_LLM_FALLBACK: Opt-in hub LLM summary fallback when AGENT_NOTIFY_SUMMARY is missing (default: off)
+ * - HAPI_OVERSEER_LLM_BASE_URL / _MODEL / _API_KEY / _API / _TIMEOUT_MS: OpenAI-compatible endpoint for that fallback
  */
 
 import { existsSync, mkdirSync } from 'node:fs'

hub/src/sync/overseerEventRecorder.fallback.test.ts

@@ -34,12 +34,12 @@ function agentText(message: string) {
 }
 
 describe('OverseerEventRecorder — no hub text synth', () => {
-    it('does not synthesize from assistant text without AGENT_NOTIFY_SUMMARY', () => {
+    it('does not synthesize from assistant text without AGENT_NOTIFY_SUMMARY', async () => {
         const store = new Store(':memory:')
         const recorder = new OverseerEventRecorder(store.events, store.inbox)
         const session = store.sessions.getOrCreateSession('cur', { flavor: 'cursor', path: '/tmp', host: 'local' }, null, 'default')
 
-        const event = recorder.onAgentMessage(
+        const event = await recorder.onAgentMessage(
             toSessionSnapshot(makeSession(session.id, 'cursor'), session.tag),
             'msg-fb',
             agentText('Refactored the parser and added tests.\n\nMore detail here.'),
@@ -51,64 +51,30 @@ describe('OverseerEventRecorder — no hub text synth', () => {
         expect(store.inbox.count()).toBe(0)
     })
 
-    it('still records a real AGENT_NOTIFY_SUMMARY', () => {
+    it('still records a real AGENT_NOTIFY_SUMMARY', async () => {
         const store = new Store(':memory:')
         const recorder = new OverseerEventRecorder(store.events, store.inbox)
         const session = store.sessions.getOrCreateSession('cur2', { flavor: 'cursor', path: '/tmp', host: 'local' }, null, 'default')
 
-        const event = recorder.onAgentMessage(
+        const event = await recorder.onAgentMessage(
             toSessionSnapshot(makeSession(session.id, 'cursor'), session.tag),
             'msg-real',
             agentText('All done.\nAGENT_NOTIFY_SUMMARY {"version":1,"status":"done","action":"Review PR","summary":"Shipped"}'),
             Date.now()
         )
 
         expect(event?.provenance).toBe('AGENT_NOTIFY_SUMMARY')
-        expect(store.events.list({ eventType: 'progress' })).toHaveLength(0)
         expect(store.events.count()).toBe(1)
     })
 
-    it('still records malformed notify as validation_error', () => {
-        const store = new Store(':memory:')
-        const recorder = new OverseerEventRecorder(store.events, store.inbox)
-        const session = store.sessions.getOrCreateSession('cur3', { flavor: 'cursor', path: '/tmp', host: 'local' }, null, 'default')
-
-        const event = recorder.onAgentMessage(
-            toSessionSnapshot(makeSession(session.id, 'cursor'), session.tag),
-            'msg-bad',
-            agentText('Working.\nAGENT_NOTIFY_SUMMARY {not valid json'),
-            Date.now()
-        )
-
-        expect(event?.eventType).toBe('validation_error')
-        expect(store.events.list({ eventType: 'progress' })).toHaveLength(0)
-    })
-
-    it('ignores user messages', () => {
-        const store = new Store(':memory:')
-        const recorder = new OverseerEventRecorder(store.events, store.inbox)
-        const session = store.sessions.getOrCreateSession('cur6', { flavor: 'cursor', path: '/tmp', host: 'local' }, null, 'default')
-
-        const userContent = { role: 'user', content: { type: 'text', text: 'do the thing' } }
-        const event = recorder.onAgentMessage(
-            toSessionSnapshot(makeSession(session.id, 'cursor'), session.tag),
-            'msg-user',
-            userContent,
-            Date.now()
-        )
-
-        expect(event).toBeNull()
-        expect(store.events.count()).toBe(0)
-    })
-
-    it('does not synth mid-turn ACP text flushes (multiple messages, no notify)', () => {
+    it('does not synth mid-turn ACP text flushes', async () => {
         const store = new Store(':memory:')
         const recorder = new OverseerEventRecorder(store.events, store.inbox)
         const live = store.sessions.getOrCreateSession('cur7', { flavor: 'cursor', path: '/tmp', host: 'local' }, null, 'default')
         const snapshot = toSessionSnapshot(makeSession(live.id, 'cursor'), live.tag)
 
-        expect(recorder.onAgentMessage(snapshot, 'msg-mid-1', agentText('Pulling the last hour of events.'), Date.now())).toBeNull()
-        expect(recorder.onAgentMessage(snapshot, 'msg-mid-2', agentText('Found something important.'), Date.now())).toBeNull()
+        expect(await recorder.onAgentMessage(snapshot, 'msg-mid-1', agentText('Pulling events.'), Date.now(), { thinking: true })).toBeNull()
+        expect(await recorder.onAgentMessage(snapshot, 'msg-mid-2', agentText('Found something.'), Date.now(), { thinking: true })).toBeNull()
         expect(store.events.count()).toBe(0)
     })
 })