Skip to content

engineering-notes.md

Latest commit

 

History

History
156 lines (126 loc) · 48.1 KB

engineering-notes.md

File metadata and controls

156 lines (126 loc) · 48.1 KB
Raw

Relay engineering notes

The running engineering log for the project: design decisions, bugs found and how they were fixed, and known limitations in full detail. The short, user-facing README is ../README.md. Paths below are relative to the repository root.

A grounded AI assistant over your own Google Workspace — Drive/Docs, Gmail, Calendar, and Sheets indexed into a RAG pipeline you can ask natural-language questions against, every answer cited back to its real source or refused rather than guessed.

Full spec: PRD.md. Current status: M1, M2, and M5 complete, M3 done for both push-capable surfaces — OAuth, Drive/Docs+PDF ingestion (with real chunking), Gmail ingestion + polling, Calendar ingestion + push sync, Sheets ingestion (structured, per-row facts riding on Drive's own push channel), the chat pipeline, background job processing, a unified auto-sync toggle covering every mechanism, a 23-question ground-truth + 7-case adversarial eval suite spanning all four surfaces, and an observability dashboard (FR-20: query volume, refusal rate, per-surface sync lag, latency) are implemented and verified against real Google/QStash infrastructure with real personal data. The Apps Script companion (M4) is the only milestone still ahead (see PRD §9).

Why this exists

Built specifically to close a gap found while tailoring resumes against real job postings: hands-on Google Workspace integration, named down to the API level in one posting, was the single most recurring missing skill across ~20 applications. See PRD §1.

Why hand-rolled OAuth

The OAuth2 authorization-code + PKCE flow (lib/google/oauth.ts, lib/crypto.ts) is written against Google's raw endpoints directly, not through google-auth-library's OAuth2Client.generateAuthUrl()/.getToken() convenience methods. That's deliberate — the whole point of this project is proving this flow is understood and can be written by hand, not delegated to a library or (elsewhere in this portfolio) to n8n's Google nodes. See PRD §6.3.

The googleapis SDK is still used in lib/google/client.ts to actually call the Drive/Docs/Gmail/Calendar/Sheets APIs once an access token is already held — that's just a typed API client, it doesn't hide any OAuth mechanics, so using it there isn't a contradiction.

Stack

Next.js (App Router) · Drizzle ORM + Postgres · Qdrant (vector store) · Hugging Face Inference API (embeddings + reranking) · Groq (generation) — same core stack as this portfolio's Mimo project, reused deliberately rather than introducing new infra alongside a new integration surface.

Setup

Local infrastructure (Postgres, Qdrant, encryption/state secrets) is already provisioned — dedicated relay-postgres and relay-qdrant Docker containers (their own volumes, not shared with any other project on this machine), with a fresh AES-256-GCM key, HMAC state secret, and Qdrant API key generated straight into .env.local. What's left needs your own accounts in a browser:

./scripts/setup.sh

This walks you through, in order:

  1. Google Cloud — create a project, enable the Drive/Docs/Gmail/Calendar/Sheets APIs.
  2. Google Cloud — configure the OAuth consent screen (External, Testing mode, the exact 5 scopes lib/google/scopes.ts requests, and your own account added as a test user).
  3. Google Cloud — create the OAuth Client ID and capture GOOGLE_CLIENT_ID/GOOGLE_CLIENT_SECRET.
  4. Hugging Face — an Inference API token (embeddings + reranking).
  5. Groq — an API key (generation).

Every value it captures is written straight to .env.local (already gitignored — see .gitignore). Because this stays in Google's Testing publishing status (deliberate, see PRD §6.4/§8), you'll hit an "unverified app" warning on first connect — click through it, it's expected for an app only you are a test user of.

Background jobs (ingestion) run through a local QStash emulator when QSTASH_DEV=true (see .env.example) — instrumentation.ts tries to auto-start it itself, but its auto-installer has a confirmed bug on Windows/Git Bash (tar misreads a C:\... path as a remote-host spec and fails to extract). On Windows, start it manually first, in its own terminal, and leave it running:

npx @upstash/qstash-cli@latest dev

Then, in another terminal:

npm install
npm run db:push
npm run dev

Open http://localhost:3000/connections, connect Drive, click Sync Drive/Docs now, then ask a question at http://localhost:3000/chat. (/ is now the public site — see below.)

No accounts to click through, or setting up on a machine without the local Postgres/Qdrant already running? See .env.example for every variable this app reads and where it comes from — scripts/setup.sh is a convenience, not the only way to fill it in.

Architecture

Browser → Next.js (App Router pages + API routes)
        → hand-rolled OAuth2/PKCE against Google's raw endpoints (lib/google/oauth.ts)
        → googleapis SDK for Drive/Docs/Gmail/Calendar/Sheets calls (lib/google/client.ts)
        → Drizzle → Postgres (provenance, connections, chat, observability)
        → Qdrant (vectors) + Hugging Face (embed/rerank) + Groq (generation)

See PRD.md §6 for the full component breakdown, request flow, and the reasoning behind each non-obvious choice (why not n8n, why Gmail polls instead of pushing, why v1 has no public demo).

Repo layout

Path What it is
lib/db/schema.ts Drizzle schema — connections, sync state, documents/chunks (citation provenance), chat, observability.
lib/crypto.ts AES-256-GCM token encryption, PKCE, and signed OAuth state — vetted primitives (Node crypto), hand-rolled orchestration.
lib/google/oauth.ts The hand-rolled OAuth2 flow itself — see "Why hand-rolled OAuth" above.
lib/google/tokens.ts Access-token cache + refresh with a single-flight guard (FR-3).
lib/ingest/ Per-surface ingestion adapters behind a common interface (drive.ts M1, Docs + PDF; gmail.ts M2, opt-in-scoped per FR-6; calendar.ts M2, primary calendar by default per FR-7; sheets.ts M2, one chunk per row per FR-8's "structured facts, not a flattened blob"). chunk.ts does fixed-size chunking with overlap for the three prose surfaces; pdf.ts extracts PDF text via pdfjs-dist.
lib/rag/ The grounded RAG pipeline: embed → retrieve → rerank → confidence-gate → generate-or-refuse (pipeline.ts), mirroring Mimo's proven design.
app/api/webhooks/drive/route.ts, app/api/webhooks/calendar/route.ts Google push-notification receivers for Drive and Calendar changes (FR-10), channel-token verified, each enqueues the real sync work rather than running it inline.
lib/qstash.ts Upstash QStash client — publishes background jobs so ingestion doesn't run in the request/response cycle.
app/api/jobs/backfill/route.ts, app/api/jobs/drive-sync/route.ts, app/api/jobs/calendar-sync/route.ts Signature-verified QStash job handlers that do the actual ingestion work published by app/api/ingest/backfill/route.ts and the two webhook receivers, respectively. drive-sync also triggers Sheets' own syncChanges afterward (FR-12 — a Sheet is a Drive file, no channel of its own).
lib/google/watch.ts Registers/renews the Drive changes.watch and Calendar events.watch push-notification channels (FR-10).
app/api/jobs/renew-watch-channels/route.ts, scripts/register-qstash-schedules.mts Scheduled renewal worker for both Drive's and Calendar's watch channels, and the one-time script that registers both QStash Schedules this project uses (watch renewal every 6h, Gmail polling every 5m).
app/api/jobs/gmail-sync/route.ts Scheduled Gmail polling worker (FR-11) — history.list since the last cursor, for every connection with a Gmail scope configured.
app/api/ingest/gmail/labels/route.ts Lists a connection's real Gmail labels (id + name) so the connections UI can offer a picker instead of a raw label-ID text field.
lib/sync-lock.ts Durable per-(connection, surface) sync lock (surface_sync.sync_started_at) — used by every ingestion job both as a dedup guard against at-least-once delivery and as the signal the connections UI polls to show a real "Syncing…" state.
instrumentation.ts Auto-starts the local QStash dev emulator (QSTASH_DEV=true) before any request is handled — see Setup's Windows caveat.
lib/backfill.ts Shared "publish a backfill job to QStash" helper — used by both the manual sync endpoint and the auto-sync toggle so they kick off work identically.
app/api/connections/[connectionId]/auto-sync/route.ts The auto-sync toggle — one switch per connection covering every surface's own freshness mechanism (Drive/Calendar: push channel; Gmail: polling).
lib/text.ts, lib/google/api-errors.ts Small helpers shared between gmail.ts and calendar.ts (HTML stripping for description/body fields; extracting an HTTP status out of googleapis' inconsistent error shapes) — extracted once Calendar needed the same logic Gmail already had, rather than a second copy.
app/api/observability/route.ts, app/(app)/observability/page.tsx The observability dashboard (FR-20) — query volume, refusal rate, latency percentiles, and per-surface sync lag, all computed from query_logs/surface_sync that were already being recorded since day one.
app/(marketing)/, components/marketing/ The public site: landing page (/), How it works, Security, Get started and Design guidelines. A separate route group with its own layout, so the product pages under app/(app)/ (chat, connections, observability) keep the app shell.
PRD.md The full product spec this was built from.

Known limitations (M1)

  • The Qdrant client connects over plain http://localhost with an API key — Qdrant itself warns about this ("Api key is used with unsecure connection"). Acceptable for local dev where the traffic never leaves the machine; a real deployment needs TLS in front of Qdrant (e.g. a reverse proxy) so the key isn't sent in the clear even on the wire between app and DB.
  • The access-token cache in lib/google/tokens.ts is in-process, so it only single-flights within one server instance — not a correctness issue at this project's scale, just not shared across serverless instances (see its own inline comment).
  • Sheets rows have a positional, not content-based, sourceId (spreadsheetId:tab:rowIndex) — deleting a row shifts every later row up by one, so persistChunk correctly re-embeds the shifted content under its new position's old sourceId, but the sheet's true last row's old slot becomes an orphan until something else touches it, and a citation can briefly point at a shifted row rather than the one originally cited. Documented in lib/ingest/sheets.ts, not discovered later — there's no reliable stable key across arbitrary spreadsheets to do better here for v1.
  • Calendar's backfill window defaults to -90/+180 days from now (lib/ingest/calendar.ts's DEFAULT_PAST_DAYS/DEFAULT_FUTURE_DAYS) — Calendar has no cheap equivalent of Drive's getStartPageToken(), so bounding the window keeps a long-lived calendar's history of recurring-meeting instances from meaning years of events get ingested. Deliberate default, not a hard limit.
  • Calendar's incremental sync (via syncToken) is not bounded to that same window once established — Google's own API rejects combining timeMin/timeMax with a syncToken on later calls, so once the watermark exists, ongoing sync tracks changes across the whole calendar, not just the original backfill range. Confirmed from Google's own docs, not assumed; a new event created far outside the original window still gets caught by an ongoing sync, which is the right behavior even though it's asymmetric with the bounded initial import.
  • Cancelled/deleted Calendar events (and deleted Drive files, deleted Gmail messages) don't retract their already-ingested chunks — the adapters correctly stop re-ingesting them, but nothing proactively deletes what's already in Postgres/Qdrant. Consistent, pre-existing v1 limitation across all three surfaces, not something newly introduced for Calendar.
  • Gmail's history.list polling only honors the labelId half of a configured scope, not the date range — a "before" bound only ever meant "stop the initial backfill there," since an ongoing poll is inherently forward-looking and history.list has no date filter at all. Documented in lib/ingest/gmail.ts, not a silent gap.
  • Gmail freshness is 5-minute polling, not real push notifications — deliberate for now (PRD §6/§11: a Pub/Sub topic would force a future user through GCP project setup just to connect Gmail). Revisit once this is actually deployed to a real public HTTPS URL: real Gmail push notifications (users.watch() + Google Cloud Pub/Sub) become genuinely viable at that point, either push-subscription (needs the same public URL Drive's changes.watch needs) or pull-subscription (no public URL needed, app polls Pub/Sub instead of Gmail directly — near-instant without hitting Gmail's own per-user quota). Pub/Sub's free tier (10 GB/month throughput) comfortably covers a single real inbox's notification volume, though GCP still requires a billing account linked to enable the API at all. Explicitly deferred rather than built now, since there's no public URL yet to verify a real push end-to-end against.
  • A full Gmail re-backfill still re-fetches every message (messages.get) to check its content hash, even when nothing changed — the hash check only skips the expensive re-embed step, not the Gmail API call itself. Confirmed in practice: re-running backfill on an unchanged ~211-message inbox still took a few minutes. Fine at this project's scale; a real optimization would check messages.list's own cheap metadata (or just the historyId delta) before ever calling messages.get.
  • googleConnections.autoSyncEnabled is the one unified "keep this connection fresh automatically" switch, covering both mechanisms transparently: ON registers Drive's push channel (via its own initial backfill) and flags this connection for Gmail's polling job; OFF genuinely stops Drive's channel with Google (channels.stop) rather than just ignoring future notifications, and the polling job (app/api/jobs/gmail-sync/route.ts) filters on this flag via a join, not just "does a scope exist." Verified live: toggled on (both surfaces started syncing immediately, confirmed via the polling status from earlier), toggled off (flag flips, a manually-triggered poll afterward confirmed zero rows processed — lastSyncedAt genuinely unchanged, not just "nothing new to sync"). Manual "Sync now" clicks work regardless of this flag either way — it only gates the automatic paths.
  • Calendar ingestion is an interface stub — lib/ingest/calendar.ts implements FR-7 (backfill via events.list, singleEvents: true so recurring meetings expand into individually citable occurrences) and gets real push notifications via events.watch (FR-10) — the same mechanism as Drive, no Pub/Sub needed, unlike Gmail. lib/google/watch.ts's renewal-lookup was generalized from Drive-only to any surface (findChannelsDueForRenewal(surface)) and the renewal worker renamed from renew-drive-watch to renew-watch-channels to cover both. The auto-sync toggle and connections UI were extended to include Calendar alongside Drive/Gmail. Verified against this author's real Google Calendar end-to-end: backfilled a real event (an actual job-interview booking), got a correctly cited answer to "what is my Pivotly screening about and when" pointing at Calendar's own real htmlLink, registered a genuine events.watch channel through a temporary tunnel with the correct ~24h expiration, forced it near-expiry and confirmed the generalized renewal job replaced it, then stopped the real channel and reverted the tunnel-only env changes when done. Chunk counts matched exactly between Postgres and Qdrant throughout (3622/3622 after the Calendar backfill).
  • Gmail backfill/sync is sequential per-message (one messages.get + one embed call at a time) — confirmed in practice against a real, active inbox: 211 messages over about a week's date range took roughly 5 minutes end to end through the QStash job. Fine at this project's real personal-inbox scale; a much larger backfill would want batching/parallelism.
  • Sheets ingestion is an interface stub — lib/ingest/sheets.ts implements FR-8 (one IngestedChunk per row, header cells supplying field labels, so "Name: John Doe, Status: Active" reads as a real fact rather than a floating fragment) and FR-12 (no channel of its own — app/api/jobs/drive-sync/route.ts triggers Sheets' own independent syncChanges after handling Drive's own sync, only once Sheets has been backfilled at least once). Verified against this author's real Drive: backfilled 4 real spreadsheets (4153 rows total, one with over 3,000), asked a structured question spanning multiple rows ("what information does the Pipeline Runs spreadsheet track") and got a correctly synthesized, multi-row-cited answer with real range-anchored citation URLs, then directly triggered the drive-sync job and confirmed it really did cascade into establishing Sheets' own sync watermark as a side effect — proving the FR-12 coupling works, not just that it compiles. Chunk counts matched exactly throughout (7775/7775). One genuinely surprising finding, confirmed rather than assumed: none of this needed the dedicated spreadsheets.readonly scope at all — drive.readonly alone was sufficient for the Sheets API's own read calls in practice, the same way it already was for reading Docs content. The separate scope stays requested (lib/google/scopes.ts) for FR-2's least-privilege intent and explicit-consent clarity, not because it turned out to be technically required.
  • instrumentation.ts's automatic QStash dev-server startup doesn't work on Windows/Git Bash — confirmed bug, not a maybe: the SDK shells out to tar with a raw C:\... path, and tar interprets the drive letter + colon as a remote-host spec (host:path syntax) and fails to extract. Works fine as documented for a real deployment (QSTASH_DEV unset) and should work on Linux/Mac for local dev; on Windows, run npx @upstash/qstash-cli@latest dev manually first (see Setup above) rather than relying on the auto-start.

Fixed since first written:

  • A real bug, not a hypothetical one: that ~5-minute Gmail backfill got delivered twice by QStash — the request log showed two separate "POST /api/jobs/backfill 200 in 5.0min" completions back to back, almost certainly QStash retrying after its own delivery timeout while the first attempt was still genuinely in progress server-side. Both ran the same backfill concurrently, doubling the real Gmail API call rate, and tripped Gmail's own "Units per minute per user" quota — surfaced as a real last_error on the connections page, not just a theoretical risk. First fixed with an in-process dedup guard, then upgraded to a durable one (lib/sync-lock.ts, backed by surface_sync.sync_started_at) once it became clear the same signal was needed for the connections UI anyway — see the next entry. claimSyncLock/releaseSyncLock are used by all three ingestion job handlers (backfill, drive-sync, gmail-sync), not just the one that originally broke. Also added 429 retry-with-backoff around every Gmail API call in lib/ingest/gmail.ts as defense in depth, independent of the dedup fix. Verified for real afterward: triggered a fresh sync through a temporary ngrok tunnel, confirmed a single clean execution (not a pair), the error cleared, and — as a bonus — real new mail had arrived during testing, so this also ended up being the first live proof that the history.list incremental "new message" path (not just the initial cursor-establishment call) actually works, catching and correctly ingesting 5 genuinely new messages.
  • The connections UI showed no sync-in-progress state at all — the backfill/sync request returns a 202 almost instantly (it just enqueues a job), so the page's "busy" flag cleared right away even though the real work was still running in the background for minutes. Fixed by exposing surface_sync.sync_started_at (the same column the dedup lock above uses) through /api/connections as a syncing boolean per surface; the connections page now polls every 3s while anything is syncing (stops once nothing is), shows "Syncing…" on the relevant button and in the per-surface status list, and disables that surface's own button without blocking the other one. Verified live: watched syncing flip true immediately after triggering a Drive backfill and back to false once it genuinely finished, and directly tested claimSyncLock/releaseSyncLock in isolation (claim succeeds → second claim while held correctly fails → succeeds again after release).
  • Ingestion stored a whole document as a single chunk with no size cap — real fixed-size chunking with overlap now runs at ingest time (lib/ingest/chunk.ts), splitting each document into ~1000-char pieces before embedding. This is what previously crashed the HuggingFace reranker (tensor-size mismatch) and blew past Groq's 8,000 TPM limit; MAX_CHARS_PER_TEXT/MAX_CHARS_PER_SOURCE in lib/rag/rerank.ts/lib/rag/generate.ts stay in place as safety nets against a pathological chunk, not as the primary truncation mechanism anymore. Re-running this project's own eval suite (npm run eval) after the change showed no regression in retrieval accuracy or refusal calibration.
  • PDF text extraction isn't implemented yet — implemented via pdfjs-dist directly (lib/ingest/pdf.ts), wired into both Drive backfill and incremental sync in lib/ingest/drive.ts. Verified against this author's own real resume PDFs in Drive: correctly extracted, chunked, and cited in a live query. Required excluding pdfjs-dist from Turbopack's server bundle (serverExternalPackages in next.config.ts) — bundled, its Node worker fallback tries to dynamically import a file Turbopack has relocated and fails at runtime.
  • Fixed in passing while rewriting chunk persistence: re-syncing a changed document deleted its old chunk rows from Postgres but never told Qdrant to drop the vectors those rows pointed to, silently leaving orphaned points behind on every edit. persistChunk now deletes the stale Qdrant points before writing the new ones.
  • Ingestion/backfill runs synchronously in the request cycle — app/api/ingest/backfill/route.ts now just validates and publishes a job to Upstash QStash (lib/qstash.ts); the real work happens in the signature-verified app/api/jobs/backfill/route.ts. The Drive push-webhook receiver (app/api/webhooks/drive/route.ts) got the same treatment — it verifies the channel token fast, then enqueues app/api/jobs/drive-sync/route.ts to do the actual sync, so a real Google notification gets a fast ack regardless of how much changed. Verified end-to-end against real cloud QStash (not the local dev emulator) via a temporary ngrok tunnel: publish → signature-verified delivery → real ingestion → surface_sync updated, and confirmed both job endpoints reject a request with no valid Upstash-Signature header (403) before doing any work.
  • Nothing registers a Drive files.watch()/changes.watch() push-notification channel — lib/google/watch.ts does, using changes.watch (not files.watch, which only watches one file's metadata, not the whole change feed). Registration runs automatically at the end of a successful Drive backfill job; a QStash Schedule (scripts/register-qstash-schedules.mts, run once against a real deployment) hits a renewal worker (app/api/jobs/renew-drive-watch/route.ts) every 6 hours to renew any channel within 2 hours of expiring. Verified end-to-end against real Google infrastructure via a temporary ngrok tunnel: registered a real channel, received Google's own "sync" handshake notification back through the receiver, forced an expiring channel and confirmed the renewal job replaced it with a fresh one, then formally stopped the channel and deleted the schedule when done (both tied to the temporary tunnel, not meant to outlive it). Two things worth knowing, found only by actually testing this rather than trusting docs or memory: omitting expiration in the changes.watch request silently capped the channel to 1 hour instead of the assumed ~24h — request it explicitly; and despite what's sometimes true for other Google push-notification setups, this did not require verifying the webhook domain in Search Console — an arbitrary ngrok address worked with zero domain-ownership proof.
  • Gmail ingestion is an interface stub — lib/ingest/gmail.ts implements FR-6 (backfill via messages.list/get) and FR-11 (incremental sync via history.list polling, since Gmail push notifications would force a future multi-tenant user through GCP Pub/Sub setup just to connect an inbox). FR-6's opt-in scoping is enforced in code, not just documented: validateGmailScope throws rather than defaulting to "index everything" when no label/date range is configured, checked both synchronously at the API route (fast 400) and again inside the job itself. The connections UI (app/connections/page.tsx) offers a real label picker (backed by app/api/ingest/gmail/labels) plus after/before date inputs. Verified against this author's real, active Gmail inbox end-to-end through a temporary ngrok tunnel: backfilled 211 real messages (label INBOX, one week back) with correct subject/from/body extraction (including HTML-to-text fallback for messages with no plain-text part), queried real content and got a correctly cited answer pointing at a real Gmail deep link, then ran the polling worker for real and confirmed it established a genuine historyId watermark with zero errors. Chunk counts matched exactly between Postgres and Qdrant (1955/1955) after the run.
  • The eval suite (tests/eval/) stayed Drive-only long after Gmail/Calendar/Sheets ingestion landed, so PRD §5's success criteria were never actually being checked against three of the four real surfaces. Extended ground-truth.json from 12 to 23 real cases (Gmail, Calendar, and Sheets questions built from this author's own real ingested content, plus two new real scope-boundary refusal tests — a Gmail question scoped to the Sent folder, a Calendar question outside the backfill window — replacing one stale case that assumed Gmail wasn't ingested yet) and adversarial.json from 6 to 7 (a Gmail-specific case per PRD §7's own explicit ask: an injection hidden inside an HTML comment in a forwarded/quoted email body, the highest-risk pattern FR-14 itself names). seed-adversarial.mts now reads a per-case surface field instead of hardcoding "drive". report.mts gained a per-surface breakdown table, since one aggregate number could hide a surface that's actually dragging. Running the extended suite surfaced two real bugs neither of the prior two eval runs (Drive-only) could have caught:
    • A genuine false refusal on real, correctly-ingested content ("Who is interviewing me for the Pivotly screening?" scored 0.0482 against the then-0.05 threshold). Checked the real score distribution rather than guessing: genuine out-of-scope refusals scored 0.00037 and 0.0025, while every genuinely answerable case scored at least 0.048 — a real, wide gap. Moved CONFIDENCE_THRESHOLD (lib/rag/pipeline.ts) to 0.02, ~8x above the highest confirmed-irrelevant score and comfortably below every confirmed-relevant one, rather than resting on the original single-pair estimate.
    • The generation model silently substitutes typographic Unicode punctuation for plain ASCII inside content it's supposed to reproduce verbatim — a non-breaking hyphen (U+2011) in a real billing account number, a narrow no-break space (U+202F) between a number and its unit, even inside a person's name ("Keith Rufo"). A real fidelity problem, not cosmetic: a user copying that "identical-looking" account number elsewhere would get a string that doesn't byte-match the original. A system-prompt instruction alone didn't fully close it (kept anyway — it likely helps some of the time), so added normalizeUnicodePunctuation() in lib/rag/generate.ts to correct it server-side, the same "don't rely on prompt compliance for something enforceable in code" reasoning this file's own citation-marker fallback already uses. Verified live, byte-by-byte: re-queried both previously-broken cases and confirmed plain ASCII (0x20, 0x2d) in the response instead of the Unicode substitutes, then re-ran the full ground-truth suite and confirmed both cases now fully pass with zero regressions elsewhere. Final report: 100% citation accuracy, 100% correct refusals, 100% adversarial resistance across all 4 surfaces; Drive's 63% full-keyword-match rate is the pre-existing baseline (legitimate paraphrasing on 3 questions, unrelated to this pass), not a regression.
  • No observability dashboard — app/observability/page.tsx + app/api/observability/route.ts implement FR-20 (query volume, refusal rate, latency, per-surface sync lag) by aggregating query_logs and surface_sync, both of which were already being written on every real query/sync since day one — this was a read-only UI on top of existing data, not new instrumentation. Verified against 166 real query_logs rows accumulated over this session's own testing: correct totals (166 total / 88 last 24h), a real refusal rate (23%), real latency percentiles (p50 4.2s / p95 13.7s, consistent with what was observed live throughout this session), and correct per-surface lag for all four surfaces.
  • Two real, user-visible bugs existed across every page — Drive, chat, connections, this new dashboard — that no amount of API-level testing all session could have caught, because nobody had actually looked at a rendered page in a real browser until the Claude-in-Chrome extension got connected partway through this pass. Both fixed and re-verified live in the browser, not just typechecked:
    • Every page was unreadable in dark mode. The default Next.js template's globals.css flips to a dark background under prefers-color-scheme: dark, but every page hardcoded literal text-black/border-black/bg-black Tailwind classes rather than theme-aware tokens — low-opacity black text (labels, timestamps, hints) became nearly invisible against the dark background. Fixed by switching every page (app/page.tsx, app/connections/page.tsx, app/observability/page.tsx) to Tailwind v4's auto-generated text-foreground/border-foreground/bg-foreground utilities, which already existed from globals.css's own --color-foreground/--color-background theme tokens — nothing new to configure, the app just wasn't using what was already there. Verified with real before/after screenshots in an actual dark-mode browser session.
    • The chat UI rendered the model's Markdown output as a raw string. The model formats answers with real Markdown fairly often — bold (**text**), bullet lists — visible throughout this project's own eval transcripts all session, but app/page.tsx printed m.content as plain text, so a user saw literal **asterisks** and - list markers instead of actual formatting. Fixed with react-markdown, scoped to assistant messages only (a user's own typed input has no reason to be parsed as Markdown), with custom component overrides to match the existing chat bubble's text sizing rather than pulling in a typography plugin for what's really just bold + lists + links. Verified live: a real multi-field Sheets answer that previously would have shown a wall of **run_id**, **timestamp**, ... now renders as a proper bulleted list with real bold field names.
  • A repo-local design skill (.skills/design/skill.md, an "Autonomous Art Director" ruleset referenced from CLAUDE.md/AGENTS.md) was applied across all three pages, in two passes. The skill's own default target is AWWWARDS-level marketing sites, which doesn't fit an internal power-user tool — its own "Dashboards & Data Visualization" module points at the "Developer/Terminal Dashboard (Linear/Vercel style)" archetype instead for exactly this kind of product, and its harmony clause says to fuse with existing constraints rather than override them, so that's the archetype actually applied both times.
    • First pass: lucide-react + framer-motion were added for real icons and restrained motion (message fade-in, an active-nav indicator, a spinner-driven SyncStatus component, a spring-animated Toggle replacing the native checkbox); app/globals.css moved from pure-black-on-white tokens to a tinted neutral scale plus one restrained accent color (60-30-10 rule, Law #37 semantic locking), and dark mode is hand-picked rather than inverted, with elevation coming from lighter surface shades instead of shadows (Law #38/#54). Also fixed a real, unrelated bug found while touching this file: body hardcoded font-family: Arial, silently overriding the Geist variable fonts layout.tsx was already loading via next/font — Geist had never actually been rendering.
    • Second pass, after direct feedback that the result still read as generic AI output — specifically, no real interaction patterns for the actual data, and a sidebar that wasn't asked for. Three concrete changes: (1) components/Shell.tsx's sidebar became a sticky header with a pill-shaped segmented nav (backdrop-blur, active-tab indicator) — still one cohesive shell instead of three independent pages with duplicated headers (fixes the skill's "Component Stacking Syndrome" tell), just not a sidebar; (2) added real components/ui/Modal.tsx and components/ui/Drawer.tsx primitives (Law #69: modals for confirmations, drawers for per-item detail/workflows) and used them for actual, previously-missing interactions — Connections' four-surface-plus-Gmail-scope-form block (previously always expanded inline on every account card, real "Card Soup") now lives behind a per-surface tile that opens a drawer; disconnecting an account now requires confirming in a modal instead of firing immediately (a real destructive-action gap, not just a style nit); Observability's per-surface row and recent-query row both open a drawer/modal with the same data laid out for actual reading instead of a single truncated line; the chat page's citations went from bare links to pill buttons that open a modal showing the real retrieved grounding snippet (Citation.snippet, a field the RAG pipeline had been computing and sending since M1 but the UI never rendered) instead of only linking out; (3) fixed two real bugs the interactive pass surfaced: a long sync-error message inside a flex row was overflowing instead of truncating (missing min-width:0 on the flex child — text-overflow:ellipsis silently does nothing without it) — fixed in SyncStatus.tsx with a detail prop that also lets the same component show the full, wrapped error inside a drawer instead of clipping it there too; and closing a drawer/modal was clearing its backing state immediately, so the close animation animated out a blank panel — fixed by separating "what's selected" from "is it open" (selectedSurface/surfaceDrawerOpen, etc.), the render-time state-adjustment pattern from the React docs rather than an effect, so a real, populated panel animates out instead of an empty one.
    • Third pass, after direct feedback that the whole thing still "looks too AI generated" — specifically the indigo-on-near-black palette (the single most overused default in AI-generated Tailwind UI, shared by shadcn's and v0's own defaults) and a single sans-serif doing every job. Rebuilt around a "Warm Terminal" identity, offered as one of three concrete named directions (with real token/font previews) rather than guessed at again: warm near-black/terracotta tokens replace the indigo scale (app/globals.css, dark-first now — light is the secondary fallback, not the primary design, matching this being a real dev tool rather than a light-first B2B dashboard); JetBrains_Mono (app/layout.tsx) is now paired with Geist Sans specifically for data — ids, timestamps, counts, technical status — so data reads as visually distinct from prose instead of one typeface doing both jobs (Law #21/#58, real font-pairing tension); every rounded-lg/rounded-md/rounded-xl/rounded-full control became a uniform 4px rounded (sharp, not the soft/pill defaults); the header nav's pill-shaped segmented control and the connections page's iOS-style green/gray pill toggle — that exact toggle is itself a named AI tell (#143) — were replaced with a sharp bracketed [on]/[off] mono switch; and all soft shadow-xl elevation on Modal/Drawer was dropped for flat hairline borders only, with the header's backdrop-blur removed too (Anti-AI Tell #81, blur/glow effects). The "Relay" wordmark also moved from a generic icon+text lockup to pure lowercase mono typography (relay_) per Law #29 (logotypes over generic icon marks).
    • Fourth pass — a specific feature request, not a "make it less AI" note this time: enclose the chat in a CLI-terminal-style window, add a real online/health signal in its title bar (hover reveals per-surface status), and add conversation-starter pills. app/page.tsx now wraps the chat in a bordered terminal "window" (title bar with mono relay — chat label and three status-colored dots standing in for the traffic-light window controls, sized to the warm-terminal palette rather than literal macOS red/yellow/green) instead of a full-bleed page. components/ConnectionStatus.tsx (new) is a real health readout, not a decorative always-green badge — it fetches the same /api/connections data the Connections page uses and reduces it to one of four honest states (online/syncing/issues/offline); hovering it (after a deliberate 300ms delay, Law #90, so it doesn't flash on every incidental mouse pass) reveals each of the four surfaces' real sync status via the same SyncStatus component the Connections/Observability drawers already use — no duplicated status logic. Four starter pills (one per connected surface: Calendar, Gmail, Sheets, Drive/Docs) sit in the empty state and fire the query immediately on click (sendMessage was refactored into a shared send(text) so both the form submit and a pill click go through the identical path) rather than just populating the input.
    • Verified live in the connected browser against this author's real data (not placeholders) after all four passes: the terminal window chrome, window-control dots, and title bar render correctly; hovering the status indicator showed a genuine live "issues" state (the real Gmail quota error from earlier, truncating correctly in the compact popover row) alongside Drive/Calendar/Sheets correctly showing healthy with real timestamps; and clicking a starter pill ("What do my spreadsheets track?") fired a real end-to-end query that returned a correctly-cited multi-spreadsheet answer (two real workbooks, "Breen Design — AWS Server Info & Software Report" and "Pipeline Runs") — confirming the pill-click path works identically to typing and submitting by hand. Zero console errors throughout.
    • Fifth pass — animation and interaction density, plus a direct "the chatbox has too much text, add more interaction" note. Added a global page-transition (components/Shell.tsx now wraps {children} in AnimatePresence/motion.div keyed on the route, so switching Chat/Connections/Observability fades+slides instead of hard-cutting) and a small blinking terminal-cursor animation on the relay_ wordmark. On the chatbox specifically: the empty state dropped its explanatory paragraph entirely (the four starter pills demonstrate the capability directly, so the sentence was purely redundant) and the pills now stagger in on mount with a hover lift; the always-visible "Refused — low confidence retrieval" sentence became a compact refused badge with the explanation moved into a new components/ui/Tooltip.tsx (reused immediately on the Connections page too, for the auto-sync explanation — same "short label, hover for the sentence" pattern rather than duplicating a bespoke tooltip per page); and assistant messages gained a hover-revealed copy-to-clipboard button whose icon flips to a checkmark for 1.5s as its own feedback (Law #235) instead of a separate toast. Connections and Observability got the same interaction pass requested ("apply this idea to other pages as well"): the connected-accounts list and per-surface tiles stagger in and lift slightly on hover, and Observability's stat-card sections, per-surface rows, and recent-query rows do the same — list items now visibly animate in sequentially (Law #81) instead of popping in all at once.
    • Verified live: route transitions fade smoothly between all three pages, the wordmark cursor blinks, the chat empty state is visibly less busy with pills staggering into place, the copy button appears on hovering an assistant message, and the Connections auto-sync tooltip and Observability's staggered stat cards/rows all render correctly against this author's real data. Zero console errors, tsc/eslint clean throughout.
    • Sixth pass — a proper dashboard for Observability, following a real reference (a "Progress Metric Card" KPI pattern: headline figure + trend badge + a data-driven chart) rather than guessing at a layout. app/api/observability/route.ts gained real backing data for it, not just a chart on top of existing numbers: a 14-day daily bucketing pass over the same query_logs rows already being fetched (count, refusal rate, avg latency per day — zero-query days stay in the series as real zeros, not dropped, so the x-axis is an honest continuous timeline) and a real week-over-week trends calculation (this week's total/average vs. the prior week's) — a 14-day window rather than the reference's 30, since this is a single-user dev app where a 30-day window would be mostly empty days. recharts (the same charting library the reference itself uses) renders three components/ui/MetricCard.tsx cards — Query volume, Refusal rate, Latency — each pairing a headline number with a trend arrow (colored by whether an increase is actually good for that specific metric: more queries is neutral-positive, more refusals or latency is not, so the same "up" arrow is green on one card and red on another) and a real gradient-filled area chart with an exact-value tooltip on hover, replacing the old flat Stat grid. Kept the Warm Terminal identity rather than copying the reference's visual style directly — terracotta gradient instead of the reference's green, monospace figures, sharp 4px corners, no card shadow/blur — and reused the existing real numbers (24h/7d/total, p50/p95/avg) as each card's footer row instead of inventing new ones.
    • Verified live against real accumulated data: the three cards render actual 14-day trend lines (a visible spike matching a real 91-query day from earlier testing), hovering the chart shows the exact value and date in a tooltip ("91 queries · Sep 18", confirmed against the raw API response), and all three cards correctly show a new badge instead of a fabricated percentage when the prior 7-day window has no data to compare against (avoiding a div-by-zero-shaped lie). Per-surface sync lag and recent-queries sections below are unchanged and still work. tsc/eslint clean, zero console errors.
    • Seventh pass — a real time-range control for the trend charts, and a proper search/date filter for Recent Queries (a static top-20 list had no way to reach anything older, and no way to find a specific past query at all). app/api/observability/route.ts's trend window generalized from a hardcoded 14 days to a real days query param (7/14/30, a 7D/14D/30D segmented control in the page header) — bucketing now runs over 2×days so the trend badge always compares a full period against an equally long prior period, not just a fixed 7-vs-7 regardless of the selected range, and only the more recent half is actually plotted. Recent Queries became a real searchable log viewer instead of a static slice: a new app/api/observability/queries/route.ts runs an actual ILIKE + date-range (gte/lte on createdAt) query against query_logs directly, so search reaches the full history, not just whatever 20 rows happened to already be on the page. components/RecentQueriesPanel.tsx debounces the search input (300ms, same reasoning as the tooltip delay elsewhere) and re-queries on every filter change, with a "clear" affordance that only appears once a filter is actually active (Law #232) and a distinct empty state ("No queries match these filters" vs. "No queries logged yet").
    • Verified live: switching the range control to 30D visibly re-shapes the same real spike into a wider window and the headline/footer numbers update; searching "pipeline" returned real matches going back to 9/18 — including rows well outside the old top-20 window that would have been invisible before — and the "clear" button correctly appeared only once the search was active. tsc/eslint clean, zero console errors.
  • A public site for the project, and two real bugs it surfaced. app/(marketing)/ adds a landing page and four more pages (How it works, Security, Get started, Design), built on the same Warm Terminal identity. The product moved into an app/(app)/ route group: chat is now at /chat (it was /), while /connections and /observability keep their URLs, and the OAuth callback still redirects to /connections. The pages hold to the numbers in the code rather than marketing copy: models, chunk size, the 12-candidate / top-4 retrieval, the 0.02 gate, the eval results (with what each metric actually measures, which corrected my first draft of the wording), and a Security page that states plainly that the app has no login of its own and that passages are sent to Hugging Face and Groq. The Design page is the UI/UX guidelines: live tokens with WCAG contrast computed from the token values, real components, a motion lab, and an accessibility section that lists the known gaps. The public demo uses labelled sample data, never content from a real workspace. Verified by driving Chrome with puppeteer-core (scrolling so reveals fire, clicking the demo tabs, accordion and modal, phone width): zero console errors, no horizontal overflow on mobile. Two bugs were found on the way: (1) useReducedMotion() differs between server and client, so the demo hydrated different markup for reduced-motion visitors and threw a hydration error, fixed with a useSyncExternalStore hook that hydrates with a server snapshot; (2) the older, bigger one: globals.css had an unlayered * { border-color: var(--border) } that out-ranked every Tailwind border-colour utility, so border-danger, border-accent, hover:border-accent/40 and focus-within:border-accent had silently never applied anywhere in the app, now moved into @layer base. Also corrected stale model names (reranker and generator) in .env.example and scripts/setup.sh.
  • Hosting the public site separately from the app. Deploy the repo to Vercel with NEXT_PUBLIC_SITE_ONLY=1 (and optionally NEXT_PUBLIC_REPO_URL=<your public repo>) and only the marketing pages are served; proxy.ts answers 404 for /chat, /connections, /observability and every /api/* route, since those have no login and need a database. The header's button becomes "Get started". Self-hosted installs leave the variable unset and get the whole app. Verified against a production build: marketing routes 200, product and API routes 404. Testing it also exposed a real bug, next build failed because lib/ingest/pdf.ts used require.resolve at import time (Turbopack rewrites it to a number); it now builds the fonts path from the project root.
  • Public-repo cleanup and a deployable public site. MIT LICENSE; the site links to the repo (lib/site.ts); Open Graph/Twitter metadata with a generated, type-only share image, canonical URLs, sitemap.xml and robots.txt. The first Vercel build failed with DATABASE_URL is not set because lib/db/index.ts connected at import and every QStash job route called verifySignatureAppRouter at module scope, and next build imports each route. Both are now lazy (db connects on first use, verifiedJob() in lib/qstash.ts builds the verifier on the first request, still failing closed if the keys are missing), and a build with no .env.local at all now passes, which is exactly what the site-only deployment has.