Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions packages/contracts/SPEC.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,8 @@ of the host.
- **Owns:** the wire — entity types, the `pi` event/message types (re-exported), the WS method & channel
registries, and the protocol version. Including **`WsErrorCode`** — the closed set of failures the *host
names* (`WsResponse.errorCode`, today `UNKNOWN_COMMIT`, `PUSH_AUTH_FAILED`,
`NOT_GIT` and `ALREADY_OPEN` (the two `project.open` refusals: a folder that is not a repository, and a
folder already owned by a workspace),
`RESOURCE_UNAVAILABLE`, and `SUBAGENT_TRANSCRIPT_NOT_FOUND` — the latter is
`subagent.getTranscript`'s **permanent** miss, the
signal that stops the transcript dialog's polling. A known child whose first transcript file is not
Expand Down
2 changes: 2 additions & 0 deletions packages/contracts/src/wsProtocol.ts
Original file line number Diff line number Diff line change
Expand Up @@ -841,6 +841,8 @@ export type WsClientMessage = WsRequest | WsAck | WsResume;
export type WsErrorCode =
| "UNKNOWN_COMMIT"
| "PUSH_AUTH_FAILED"
| "NOT_GIT"
| "ALREADY_OPEN"
| "SUBAGENT_TRANSCRIPT_NOT_FOUND"
| "RESOURCE_UNAVAILABLE"
| "STALE_VIEW"
Expand Down
10 changes: 7 additions & 3 deletions packages/server/src/analytics/SPEC.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,8 +31,12 @@ properties and is emitted before `app_started` once when a non-CI/non-test binar
claims the shared installation marker. Initialization constructs the basic sink before claiming; once claimed,
the state is installed before enqueue. A sink-construction failure therefore leaves the marker available, while
a later delivery failure does not clear or retry it. Existing `{ id }` records emit on their first eligible packaged boot. Launch means
host boot, not UI readiness. Chats can be empty;
sends count after `ackSend`, exclude TODO-control nudges, and do not prove successful execution. Login
host boot, not UI readiness. Chats can be empty; a chat the host had to create without any usable model
reports `provider` and `model` as `none` with `auth_method: unknown`; the host's own model resolution
decides that case rather than pi's placeholder model (which would otherwise bucket as `custom`), while
a resolved chat reports the model the created session actually holds (a startup extension may switch it).
"Created a chat but had nothing to send with" thus stays distinguishable from a configured custom provider.
Sends count after `ackSend`, exclude TODO-control nudges, and do not prove successful execution. Login
requires correlated success, or the existing applied Central connection action. `auth_method` is a closed
`api_key | subscription | oauth | central | other | unknown` category, never credentials or account/plan
identities. It describes the observed authentication path, not billing entitlement. Host captures send
Expand All @@ -46,7 +50,7 @@ bounded browser-derived strings, never resource identities or arbitrary product
| Event | Signal |
| --- | --- |
| `setup_state_observed` | Provider/model/project readiness (`yes/no/unknown`); first current observation and changes, not every poll. |
| `setup_action_finished` | Explicit setup operation, outcome and fixed failure category; automatic Default provisioning is excluded. |
| `setup_action_finished` | Explicit setup operation, outcome and fixed failure category; automatic Default provisioning is excluded. `directory_pick` is the native folder picker (succeeded / cancelled / failed, never the path); `project_open` reports the typed `not_git` / `already_open` reasons from the projects module's coded errors, and every untyped failure stays `unknown` — reasons are never derived from error text. |
| `agent_run_started` | Work-cycle origin, workspace kind and catalog-bucketed provider/model. |
| `agent_run_settled` | Final outcome, elapsed-time/retry/compaction buckets; only `agent_settled`, never attempt-level `agent_end`. |
| `task_completed` | Nonempty task-group completion transition after artifact reconciliation, change evidence and whether verification was recorded. |
Expand Down
3 changes: 3 additions & 0 deletions packages/server/src/analytics/events.ts
Original file line number Diff line number Diff line change
Expand Up @@ -32,11 +32,13 @@ export type AnalyticsFailureReason =
| "network"
| "permission"
| "not_git"
| "already_open"
| "unsupported"
| "unknown"
| "none";
export type SetupAction =
| "provider_connect"
| "directory_pick"
| "project_open"
| "project_init"
| "worktree_create"
Expand Down Expand Up @@ -168,6 +170,7 @@ export function bucketCount(count: number): AnalyticsCountBucket {
}

export const CUSTOM_BUCKET = "custom";
export const NONE_BUCKET = "none";

let catalog: Map<string, ReadonlySet<string>> | null = null;

Expand Down
1 change: 1 addition & 0 deletions packages/server/src/analytics/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@ export {
bucketDuration,
bucketProvider,
bucketProviderModel,
NONE_BUCKET,
} from "./events";
export {
type AnalyticsOptions,
Expand Down
8 changes: 6 additions & 2 deletions packages/server/src/host/SPEC.md
Original file line number Diff line number Diff line change
Expand Up @@ -115,7 +115,9 @@ channel fan-out, and the process-boot wrapper both launchers share.
reconstruct pre-consent work; campaign-only enrichment is inactive while off and restored from its strict
server record when on without retrying a consumed claim.
`shutdownAnalytics()` remains a best-effort drain in `stop()` and awaited by graceful shutdown;
every capture site lives here, including the existing basic events: `chat_started` in `session.create`, `message_sent` (via the
every capture site lives here, including the existing basic events: `chat_started` in `session.create` (the host's new-chat
resolution decides `none` for a chat created with no usable model instead of pi's placeholder; otherwise
the created session's actual model is reported), `message_sent` (via the
local `trackSend(mode, text)`) after an **accepted** `session.prompt`/`session.steer`/`session.followUp`
(`prompt`/`steer`/`follow_up`; skipped when contracts' `isControlMessage(text)` — the client's TODO
wake-nudge rides the same methods and is not a user message; `session.answerQuestion` is a tool reply,
Expand All @@ -133,7 +135,9 @@ channel fan-out, and the process-boot wrapper both launchers share.
[[submodule-server-analytics]], with transient consent-scoped correlation and task-artifact reconciliation.
Host alone mediates these events; analytics initialization emits the packaged-install lifecycle event,
while no provider-change capture exists.
Setup observes existing read results, never triggers provider work; only explicit setup mutations count.
Setup observes existing read results, never triggers provider work; only explicit setup mutations count
(`dialog.selectDirectory` is one: `directory_pick` with a null path as `cancelled`; `project.open` maps the
projects module's `NOT_GIT` / `ALREADY_OPEN` coded errors to the closed reasons, everything else `unknown`).
Run timing starts at canonical `agent_start`, with local send intent recorded before calling pi (not
after `ackSend`); unproven provenance stays unknown and retries remain one cycle until `agent_settled`.
Accepted queued origins survive send resolution until canonical cycle observation; rejected sends do not
Expand Down
13 changes: 7 additions & 6 deletions packages/server/src/host/authAnalytics.ts
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@ import {
bucketProvider,
bucketProviderModel,
type LoginMethod,
NONE_BUCKET,
type ProviderAnalyticsProperties,
track,
} from "../analytics";
Expand Down Expand Up @@ -118,10 +119,10 @@ export function trackChatStarted(created: {
sessionId: string;
model: Pick<WireModel, "provider" | "id"> | null;
}): void {
if (created.model) {
track({
name: "chat_started",
params: sessionProviderAnalytics(created.sessionId, created.model),
});
}
track({
name: "chat_started",
params: created.model
? sessionProviderAnalytics(created.sessionId, created.model)
: { provider: NONE_BUCKET, model: NONE_BUCKET, auth_method: "unknown" },
});
}
52 changes: 51 additions & 1 deletion packages/server/src/host/authAttribution.test.ts
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
import { afterEach, beforeEach, expect, test } from "bun:test";
import { mkdtempSync, rmSync } from "node:fs";
import { mkdirSync, mkdtempSync, rmSync, writeFileSync } from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { createProvider, envApiKeyAuth, InMemoryCredentialStore } from "@earendil-works/pi-ai";
Expand Down Expand Up @@ -255,6 +255,56 @@ test("unmodified builtin providers report preconfigured stored keys without an i
expect(properties("message_sent")).toEqual([]);
});

test("a chat created without any usable model still counts, bucketed as none", async () => {
const bare = await ModelRuntime.create({
credentials: new InMemoryCredentialStore(),
modelsPath: null,
allowModelNetwork: false,
refreshOnCreate: false,
});
configurePiRuntime(bare);
const created = (await handleRequest(
"session.create",
{ workspaceId },
context,
)) as WsResult<"session.create">;
sessionIds.push(created.sessionId);
await shutdownAnalytics();
expect(properties("chat_started")).toMatchObject([
{ provider: "none", model: "none", auth_method: "unknown" },
]);
});

test("a startup extension that switches the model is reported as the model the chat actually got", async () => {
const alternate = await addFauxProvider(runtime, credentials, `${PRIVATE}-provider`, "oauth");
const extensionPath = join(directory, "pi", "extensions", "switch-on-start.ts");
mkdirSync(join(directory, "pi", "extensions"), { recursive: true });
writeFileSync(
extensionPath,
[
'import type { ExtensionAPI } from "@earendil-works/pi-coding-agent";',
"export default function (pi: ExtensionAPI) {",
'\tpi.on("session_start", async (_event, ctx) => {',
`\t\tconst model = ctx.modelRegistry.find(${JSON.stringify(`${PRIVATE}-provider`)}, ${JSON.stringify(`${PRIVATE}-model`)});`,
"\t\tif (model) await pi.setModel(model);",
"\t});",
"}",
"",
].join("\n"),
);
const created = (await handleRequest(
"session.create",
{ workspaceId, model: toWireModel(faux.getModel()) },
context,
)) as WsResult<"session.create">;
sessionIds.push(created.sessionId);
expect(created.model?.provider).toBe(alternate.getModel().provider);
await shutdownAnalytics();
expect(properties("chat_started")).toMatchObject([
{ provider: "custom", model: "custom", auth_method: "oauth" },
]);
});

test("model changes affect later sends but not provider/auth captured before a delayed acknowledgement", async () => {
const alternate = await addFauxProvider(runtime, credentials, `${PRIVATE}-provider`, "oauth");
const sessionId = await createChat();
Expand Down
14 changes: 11 additions & 3 deletions packages/server/src/host/handlers.ts
Original file line number Diff line number Diff line change
Expand Up @@ -201,6 +201,7 @@ import {
additionalCapture,
captureAdditional,
centralConnectOutcome,
directoryPickOutcome,
observePrAction,
observeSetupAction,
observeSetupRead,
Expand Down Expand Up @@ -362,7 +363,10 @@ async function sendToFileChat(
...(defaults.model ? { model: defaults.model } : {}),
thinkingLevel: defaults.thinkingLevel,
});
trackChatStarted(created);
trackChatStarted({
sessionId: created.sessionId,
model: defaults.model ? created.model : null,
});
await markCommentsSent(workspaceId, ids, created.sessionId);
fireReviewPrompt(workspaceId, ids, created.sessionId, pkg);
return { ...created, reused: false };
Expand Down Expand Up @@ -508,7 +512,8 @@ const handlers: Record<string, Handler> = {
},
),
),
"dialog.selectDirectory": () => selectDirectory(),
"dialog.selectDirectory": () =>
observeSetupAction("directory_pick", selectDirectory, directoryPickOutcome),
"fs.readDir": (params) => {
const p = params as { workspaceId: string; path: string };
void ensureWatch(p.workspaceId);
Expand Down Expand Up @@ -792,7 +797,10 @@ const handlers: Record<string, Handler> = {
thinkingLevel: defaults.thinkingLevel,
});
if (p.model && created.model) noteRecentModel(created.model);
trackChatStarted(created);
trackChatStarted({
sessionId: created.sessionId,
model: defaults.model ? created.model : null,
});
return created;
},
"session.prompt": (params, ctx) => {
Expand Down
43 changes: 43 additions & 0 deletions packages/server/src/host/productAnalytics.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,7 @@ import {
applyAdditionalAnalyticsSettings,
captureAdditional,
centralConnectOutcome,
directoryPickOutcome,
failureReason,
initialAdditionalAnalyticsEnabled,
observeCurrentSetup,
Expand Down Expand Up @@ -349,6 +350,48 @@ test("observational sink and projection failures cannot change a successful feat
expect(failureReason(new Error("PUSH_AUTH_FAILED secret"))).toBe("unknown");
});

test("project opening names its typed failures; the folder picker reports selection, cancellation and failure without paths", async () => {
const ctx = { clientKey: "private-client" };
const plain = join(dataDir, "plain-folder");
mkdirSync(plain);
await expect(handleRequest("project.open", { path: plain }, ctx)).rejects.toMatchObject({
code: "NOT_GIT",
});
expect(failureReason(new CodedError("ALREADY_OPEN", "/private/worktree"))).toBe("already_open");
const savedPick = process.env.THINKRAIL_PICK_DIR;
try {
process.env.THINKRAIL_PICK_DIR = "/private/picked";
expect(await handleRequest("dialog.selectDirectory", {}, ctx)).toEqual({
path: "/private/picked",
});
const directive = join(dataDir, "picker-directive");
writeFileSync(directive, "error: private picker failure");
process.env.THINKRAIL_PICK_DIR = directive;
await expect(handleRequest("dialog.selectDirectory", {}, ctx)).rejects.toThrow(
"private picker failure",
);
} finally {
if (savedPick === undefined) delete process.env.THINKRAIL_PICK_DIR;
else process.env.THINKRAIL_PICK_DIR = savedPick;
}
await observeSetupAction("directory_pick", () => ({ path: null }), directoryPickOutcome);
const events = await captured("setup_action_finished");
expect(
events.map((event) => [
event.properties.action,
event.properties.outcome,
event.properties.reason,
]),
).toEqual([
["project_open", "failed", "not_git"],
["directory_pick", "succeeded", "none"],
["directory_pick", "failed", "unknown"],
["directory_pick", "cancelled", "none"],
]);
expect(JSON.stringify(sent)).not.toContain("private");
expect(JSON.stringify(sent)).not.toContain("plain-folder");
});

function git(root: string, ...args: string[]): void {
const result = Bun.spawnSync(["git", "-C", root, ...args], { stdout: "ignore", stderr: "pipe" });
if (!result.success) throw new Error(`git fixture failed: ${args[0]}`);
Expand Down
18 changes: 16 additions & 2 deletions packages/server/src/host/productAnalytics.ts
Original file line number Diff line number Diff line change
Expand Up @@ -58,8 +58,16 @@ export function captureAdditional(
}

export function failureReason(error: unknown): FailureReason {
if (errorCodeOf(error) === "PUSH_AUTH_FAILED") return "auth";
return "unknown";
switch (errorCodeOf(error)) {
case "PUSH_AUTH_FAILED":
return "auth";
case "NOT_GIT":
return "not_git";
case "ALREADY_OPEN":
return "already_open";
default:
return "unknown";
}
}

export function providerAvailability(
Expand Down Expand Up @@ -202,6 +210,12 @@ export function centralConnectOutcome(
};
}

export function directoryPickOutcome(selected: {
path: string | null;
}): Pick<SetupResult, "outcome" | "reason"> {
return { outcome: selected.path === null ? "cancelled" : "succeeded", reason: "none" };
}

export async function observePrAction(
operation: () => Promise<OpenPrResult>,
): Promise<OpenPrResult> {
Expand Down
8 changes: 5 additions & 3 deletions packages/server/src/projects/SPEC.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,15 +20,16 @@ bootstrap it into one so it can be opened.
or mutating it: host-absolute paths pass, exact `~` and `~/…` expand against the host account, and every
other relative path is rejected rather than interpreted against the host process cwd. The same resolved
path feeds `openProject`, `inspectProjectPath`, and `initProject`, so their classifications and actions
cannot disagree. It then validates a path is a repo (`git rev-parse --show-toplevel`), dedupes by root,
cannot disagree. It then validates a path is a repo (`git rev-parse --show-toplevel`; a non-repo rejects
with the named `NOT_GIT` code), dedupes by root,
and assigns a stable unique readable `slug`; `getProjects` returns all known records with slug backfill,
`listProjects` returns open records by `lastOpened`, and `listRecentProjects` returns open + closed records
by `lastOpened`. A persisted
optional **`Project.closed: true`** is the entire membership state: absence means open, so existing
records migrate as open. **`openProject`** finds a known root even when closed, clears `closed`, bumps
`lastOpened`, preserves its id, persists, and publishes the full snapshot; **`closeProject`** marks that
same record closed and publishes it without deleting the project, repository, workspace records, or
live runtimes. **One cwd, one ThinkRail identity:** `openProject` rejects a root already held as some
live runtimes. **One cwd, one ThinkRail identity:** `openProject` rejects (named `ALREADY_OPEN`) a root already held as some
workspace's `worktreePath` — pi keys chat transcripts by *directory*, so a second identity on an owned
folder would serve that workspace's chats as its own and have them purged when either side is archived.
Compared **canonically** (a managed worktree's stored path is composed, `--show-toplevel` answers
Expand All @@ -53,6 +54,7 @@ bootstrap it into one so it can be opened.
`getProjects`, `setProjectPublisher`, `inspectProjectPath`, `initProject`, `setProjectTrust`,
`setProjectSkillEnabled`, `setProjectGroupEnabled`, `acknowledgeProjectSkills`.
- **Allowed deps:** `persistence`; the `git` sub-module (shared `git()` runner, which now owns the
environment its children spawn under — this module passes none); `contracts` (`Project`, `ProjectPathStatus`); Node/Bun.
environment its children spawn under — this module passes none); `contracts` (`Project`, `ProjectPathStatus`);
`shared/codedError` (the two named open failures); Node/Bun.
- **Forbidden:** `host`; sibling features other than `git` (`workspaces` depends on `projects`, never the
reverse).
8 changes: 6 additions & 2 deletions packages/server/src/projects/projects.ts
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@ import { rmSync, statSync } from "node:fs";
import { homedir } from "node:os";
import { basename, isAbsolute, join, resolve } from "node:path";
import type { Project, ProjectPathStatus } from "@thinkrail/contracts";
import { CodedError } from "@thinkrail/shared/codedError";
import { canonicalPath, git } from "../git";
import { loadProjects, loadWorkspaces, saveProjects } from "../persistence";

Expand Down Expand Up @@ -71,7 +72,7 @@ export function getProjects(): Project[] {
export function openProject(inputPath: string): Project {
const path = resolveProjectPath(inputPath);
const root = gitToplevel(path);
if (!root) throw new Error(`Not a git repository: ${path}`);
if (!root) throw new CodedError("NOT_GIT", `Not a git repository: ${path}`);

const projects = getProjects();
const existing = projects.find((p) => p.path === root);
Expand All @@ -85,7 +86,10 @@ export function openProject(inputPath: string): Project {

const wanted = canonicalPath(root);
if (loadWorkspaces().some((ws) => canonicalPath(ws.worktreePath) === wanted))
throw new Error(`This folder is already open in ThinkRail as a workspace: ${root}`);
throw new CodedError(
"ALREADY_OPEN",
`This folder is already open in ThinkRail as a workspace: ${root}`,
);

const taken = new Set(projects.map((p) => p.slug));
const project: Project = {
Expand Down
Loading