Repository navigation
feat(analytics): name setup failures and model-less chats so the activation ladder can read them - #640
Merged
Conversation
…vation ladder can read them
Dashboard analysis showed 48% of packaged installs boot and are never observed
again, while the consent-gated setup events reported almost every failure as
`unknown`. Three closed-vocabulary additions make the pre-chat part of the
funnel legible without widening the privacy boundary:
- `openProject` throws `CodedError("NOT_GIT")` / `CodedError("ALREADY_OPEN")`
(new `WsErrorCode`s, so the UI gets named codes too); `failureReason()` maps
them to `not_git` / new `already_open` instead of `unknown`. Reasons are
never derived from error text.
- `dialog.selectDirectory` is observed as a `directory_pick` setup action
(succeeded / cancelled on a null path / failed); the path never leaves the
host.
- `chat_started` reports `provider`/`model` = `none` for a chat created with no
usable model. pi substitutes a `{provider:"unknown"}` placeholder model when
none is given, so the previous `if (created.model)` guard never suppressed
anything and such chats were hidden inside `custom`. The host's own new-chat
resolution decides `none`; otherwise the created session's actual model is
reported, because a `session_start` extension may switch it during creation.
Specs updated: analytics, host, projects, contracts.
Rinat S (rsolmano)
requested review from
Olga Lavrichenko (OLavrik),
Rustam Sadykov (SBOne-Kenobi) and
danyaberezun
as code owners
October 5, 2026 19:43
danyaberezun
approved these changes
Oct 5, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The Growth & Retention dashboard shows ~65% of packaged installs never send a message, and on clean cohorts 48% boot the host and are never observed again. The consent-gated setup events meant to explain that were nearly useless:
project_openfailures all reportedreason=unknown(the host threw plainErrors, so the existingnot_gitenum value was never populated), the native folder picker was unobserved, and a chat created with no usable model was indistinguishable from a configured custom provider — pi substitutes a{provider:"unknown"}placeholder model when none is given, so theif (created.model)guard intrackChatStartednever suppressed anything and such chats landed in thecustombucket.Approach
Three closed-vocabulary additions, all host-mediated, none widening the privacy boundary (no paths, error text, or ids):
openProjectthrowsCodedError("NOT_GIT")/CodedError("ALREADY_OPEN")— the sameshared/codedErrorpatternprandgitalready use — andfailureReason()maps codes to the closed reasons. Nothing is ever derived from error text; every untyped failure staysunknown.dialog.selectDirectory), the only place the host sees it; a null path iscancelled.nonechat bucket is decided by the host's own new-chat resolution (defaults.model === null), not by sniffing pi's placeholder. For a resolved chat the created session's model is reported, because asession_startextension can callpi.setModel()during creation (regression-tested).Reviewed in pair with gpt-6-astra; its must-fix on the extension case is what drove the
defaults.model ? created.model : nullwiring.Changes
packages/contracts:WsErrorCodegainsNOT_GITandALREADY_OPEN(UI receives named codes onproject.openrejections;errorText()-based flows are unchanged).packages/server/src/projects:openProjectthrows the two coded errors.packages/server/src/analytics:AnalyticsFailureReasongainsalready_open;SetupActiongainsdirectory_pick;NONE_BUCKETexported.packages/server/src/host:failureReason()mapsNOT_GIT/ALREADY_OPEN;dialog.selectDirectorywrapped inobserveSetupAction("directory_pick", …, directoryPickOutcome);trackChatStartedemitsprovider/model = none, auth_method = unknownfor model-less chats; both call sites passdefaults.model ? created.model : null.project.openrejection + picker succeeded/failed/cancelled with no path in the payload; model-less chat →none; startup extension switching the model → reported as the switched model.nonerationale.Dashboard side (no code, already live on PostHog 962459): install tiles filtered to
build ≠ source,channel = stable; annotations for the v0.1.2–v0.1.3app_installedgap; new HogQL tiles (first-boot activation ladder, os×build matched cohorts, readiness, setup outcome×reason, first-chat provider→send, first user-run outcome). Existing chat tiles should addprovider != noneonce this ships.Screenshots
Not applicable — no frontend changes.
Checklist
bun run lint,bun run typecheck,bun run test— lint/typecheck/deps/boundaries/seams via the pre-commit hook on the final tree;bun run test→ 1338 pass / 0 fail (18 packages); affected server tests re-run after rebase → 106 pass;bun run check:spec-surface→ OKbun run e2e, orbun run e2e:fullwhen touching agent behavior) —bun run e2eon the rebased tree → 465 passed, 8/8 shards. (A first post-rebase run had twoe2e/layout.spec.tsfailures — frontend group reorder / window placement, no web files in this diff — that passed on--last-failedand on the full re-run; noting it as a possible flake under 8-lane load after feat(web): replace the Resources popover with a trigger, dock and inspector #631.)SPEC.md/ top-level specs updated to reflect any boundary, contract, or behavior change