Skip to content

feat(analytics): name setup failures and model-less chats so the activation ladder can read them - #640

Merged
danyaberezun merged 1 commit into
mainfrom
analytics-funnel-improvements
Oct 5, 2026
Merged

danyaberezun merged 1 commit into
mainfrom
analytics-funnel-improvements

Conversation

@rsolmano

Copy link
Copy Markdown
Collaborator

Problem

The Growth & Retention dashboard shows ~65% of packaged installs never send a message, and on clean cohorts 48% boot the host and are never observed again. The consent-gated setup events meant to explain that were nearly useless: project_open failures all reported reason=unknown (the host threw plain Errors, so the existing not_git enum value was never populated), the native folder picker was unobserved, and a chat created with no usable model was indistinguishable from a configured custom provider — pi substitutes a {provider:"unknown"} placeholder model when none is given, so the if (created.model) guard in trackChatStarted never suppressed anything and such chats landed in the custom bucket.

Approach

Three closed-vocabulary additions, all host-mediated, none widening the privacy boundary (no paths, error text, or ids):

  • Feature modules name their failures, the host maps the code. openProject throws CodedError("NOT_GIT") / CodedError("ALREADY_OPEN") — the same shared/codedError pattern pr and git already use — and failureReason() maps codes to the closed reasons. Nothing is ever derived from error text; every untyped failure stays unknown.
  • The picker outcome is observed at the handler (dialog.selectDirectory), the only place the host sees it; a null path is cancelled.
  • The none chat bucket is decided by the host's own new-chat resolution (defaults.model === null), not by sniffing pi's placeholder. For a resolved chat the created session's model is reported, because a session_start extension can call pi.setModel() during creation (regression-tested).

Reviewed in pair with gpt-6-astra; its must-fix on the extension case is what drove the defaults.model ? created.model : null wiring.

Changes

  • packages/contracts: WsErrorCode gains NOT_GIT and ALREADY_OPEN (UI receives named codes on project.open rejections; errorText()-based flows are unchanged).
  • packages/server/src/projects: openProject throws the two coded errors.
  • packages/server/src/analytics: AnalyticsFailureReason gains already_open; SetupAction gains directory_pick; NONE_BUCKET exported.
  • packages/server/src/host: failureReason() maps NOT_GIT/ALREADY_OPEN; dialog.selectDirectory wrapped in observeSetupAction("directory_pick", …, directoryPickOutcome); trackChatStarted emits provider/model = none, auth_method = unknown for model-less chats; both call sites pass defaults.model ? created.model : null.
  • Tests: typed project.open rejection + picker succeeded/failed/cancelled with no path in the payload; model-less chat → none; startup extension switching the model → reported as the switched model.
  • Specs: analytics, host, projects, contracts updated for the new vocabulary and the none rationale.

Dashboard side (no code, already live on PostHog 962459): install tiles filtered to build ≠ source, channel = stable; annotations for the v0.1.2–v0.1.3 app_installed gap; new HogQL tiles (first-boot activation ladder, os×build matched cohorts, readiness, setup outcome×reason, first-chat provider→send, first user-run outcome). Existing chat tiles should add provider != none once this ships.

Screenshots

Not applicable — no frontend changes.

Checklist

  • Fast gates pass: bun run lint, bun run typecheck, bun run test — lint/typecheck/deps/boundaries/seams via the pre-commit hook on the final tree; bun run test → 1338 pass / 0 fail (18 packages); affected server tests re-run after rebase → 106 pass; bun run check:spec-surface → OK
  • E2E suite passes for app-affecting changes (bun run e2e, or bun run e2e:full when touching agent behavior) — bun run e2e on the rebased tree → 465 passed, 8/8 shards. (A first post-rebase run had two e2e/layout.spec.ts failures — frontend group reorder / window placement, no web files in this diff — that passed on --last-failed and on the full re-run; noting it as a possible flake under 8-lane load after feat(web): replace the Resources popover with a trigger, dock and inspector #631.)
  • Before/after screenshots are included for frontend changes, or marked not applicable
  • Relevant SPEC.md / top-level specs updated to reflect any boundary, contract, or behavior change
  • I have read the Contributing guide and agree to the Code of Conduct

…vation ladder can read them

Dashboard analysis showed 48% of packaged installs boot and are never observed
again, while the consent-gated setup events reported almost every failure as
`unknown`. Three closed-vocabulary additions make the pre-chat part of the
funnel legible without widening the privacy boundary:

- `openProject` throws `CodedError("NOT_GIT")` / `CodedError("ALREADY_OPEN")`
  (new `WsErrorCode`s, so the UI gets named codes too); `failureReason()` maps
  them to `not_git` / new `already_open` instead of `unknown`. Reasons are
  never derived from error text.
- `dialog.selectDirectory` is observed as a `directory_pick` setup action
  (succeeded / cancelled on a null path / failed); the path never leaves the
  host.
- `chat_started` reports `provider`/`model` = `none` for a chat created with no
  usable model. pi substitutes a `{provider:"unknown"}` placeholder model when
  none is given, so the previous `if (created.model)` guard never suppressed
  anything and such chats were hidden inside `custom`. The host's own new-chat
  resolution decides `none`; otherwise the created session's actual model is
  reported, because a `session_start` extension may switch it during creation.

Specs updated: analytics, host, projects, contracts.

@jetbrains-air jetbrains-air Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved — ready to merge.

@danyaberezun
danyaberezun enabled auto-merge October 5, 2026 21:43
@danyaberezun
danyaberezun added this pull request to the merge queue Oct 5, 2026
Merged via the queue into main with commit 646900e Oct 5, 2026
10 of 14 checks passed
@danyaberezun
danyaberezun deleted the analytics-funnel-improvements branch October 5, 2026 22:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants