Repository navigation
Trajectory evaluation, CI, unit tests, MCP eval server, measured palette + cold start - #1
Closed
ashiksharonm wants to merge 13 commits into
Closed
ashiksharonm wants to merge 13 commits into
ashiksharonm wants to merge 13 commits into
Conversation
Time-machine debugging for agents: every run (model turns + MCP tool calls) is recorded as a replayable trace. Fork any step — edit a tool result, reroll a model turn, or rewrite the prompt — re-execute the future, and compare timelines side by side. - Node 24 + Express + node:sqlite trace store - Real MCP toolchain (stdio server + client, 4 deterministic demo tools) - Claude API agent loop (claude-opus-4-8) with simulated fallback mode - React + Vite UI: dashboard, trace timeline, fork dialog, compare view - 29-check smoke suite, Dockerfile, Render blueprint
Added live URL to README and updated formatting.
Covers buildMessages (parallel tool groups, error flags, no-prompt guard), recording, and every fork edit type: tool_result (whole parallel group kept, parent never mutated, ids re-keyed), reroll truncation, prompt edit, and validation errors. Root 'npm test' now runs unit tests; the HTTP smoke suite moves to 'npm run smoke'.
Pure scoring over recorded steps: exact / in_order / any_order trajectory matching (parallel tool groups treated as unordered), tool-call precision, recall and F1 via maximum bipartite matching with exact/subset/ignore argument matching, efficiency and budget checks, tool errors, retries, redundant calls, forbidden tools, and final-answer outcome checks. diffEvaluations labels each metric improved/regressed between two timelines. Covered by 21 unit tests.
npm run eval records 7 scenarios on the simulated agent through the real MCP tools server (3 of them forks: edited tool result, reroll, prompt edit), scores them, and writes eval/results/scenarios.json + summary.md. Two negative controls (exact-mode mismatch, an unknown-city agent bug) must fail; the script exits non-zero if any verdict disagrees with its expectation. Also adds npm run eval:trace for scoring exported traces offline, and a recorded simulated trace fixture with tests.
POST /api/runs/:id/evaluate scores a run against a spec, another run's trajectory (referenceRunId), or reference-free. POST /api/compare scores two timelines under one spec and returns the per-metric diff. GET /api/eval/scenarios lists the shipped scenario specs. Specs are validated with zod; regex outcome checks are rejected over HTTP to avoid ReDoS on the public demo. Smoke suite grows to 33 checks.
server/src/mcp/rewind-server.ts is a stdio MCP server over a Rewind instance's HTTP API with list_runs, get_run, fork_run, evaluate_run, compare_runs and list_eval_scenarios. scripts/mcp-smoke.mjs drives it with the official SDK client; CI runs it plus an eval reproducibility check.
Scores both timelines under one spec (A's own tool calls by default, or any shipped scenario, or hand-edited JSON) and shows each metric for A and B with its delta and an improved/regressed verdict, plus the pass flip and per-run findings (missing/unexpected calls, failed outcome checks). Screenshot captured with headless Chromium against a local server.
scripts/palette-check.mjs reads the :root tokens and writes
docs/palette-report.{json,md}: WCAG 2.x contrast for every text/status token
(4.5:1) and chart token (3:1) on both surfaces, plus CIE76 separation of
the chart series and verdict colors under simulated protanopia,
deuteranopia and tritanopia (Machado 2009). --critical failed 4.5:1 as
text (3.62:1) and is lightened to #ea5a5a (5.07:1). CI fails if the
report drifts from the stylesheet.
Pitch, eval panel screenshot, mermaid architecture, eval spec and metrics, scenario results table (from eval/results), MCP server config, measured Render cold start (raw output committed), test matrix, measured palette report, and an honest limitations section.
The simulated agent used toLocaleString() with the host locale, so the same run produced '3,16,800 JPY' on an en-IN machine and '316,800 JPY' on CI — traces were not reproducible across hosts. Caught by the CI eval reproducibility check. Regenerates eval results and the trace fixture.
ashiksharonm
force-pushed
the
improve/2026-10-audit
branch
from
October 11, 2026 08:51
72ab62e to
a2bcc2b
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Turns Rewind from "fork and compare agent runs" into "fork, compare, and grade agent runs", and fixes the audit findings (no CI, no unit tests, unbacked palette claim, undocumented cold start, no trace fixture).
buildMessages, parent never mutated, parallel tool groups, reroll truncation, every edit type, validation).server/src/eval/): exact / in_order / any_order matching (parallel groups unordered), tool-call precision/recall/F1 with exact/subset/ignore argument matching (max bipartite matching), efficiency + budgets, tool errors / retries / redundant calls, forbidden tools, outcome checks, and a timeline diff with improved/regressed verdicts + pass flip.npm run eval: 7 deterministic scenarios on the simulated agent via the real MCP tools server (3 forks, 2 negative controls), committed toeval/results/;npm run eval:tracescores exported traces offline; recorded simulated trace fixture.POST /api/runs/:id/evaluate,POST /api/compare,GET /api/eval/scenarios(zod-validated; regex checks rejected over HTTP to avoid ReDoS).server/src/mcp/rewind-server.ts: Rewind as an MCP server (list_runs,get_run,fork_run,evaluate_run,compare_runs,list_eval_scenarios).scripts/palette-check.mjscomputes WCAG contrast + CVD separation →docs/palette-report.md.--criticalfailed 4.5:1 as text (3.62:1) and was lightened to#ea5a5a(5.07:1).Before / after
npm testnpm run smokenpm run smoke:mcp.github/workflows/ci.ymleval/results/summary.md--criticalfix; 21/22 before)docs/palette-report.mddocs/render-cold-start.txtReproduce
Test output (local, Apple M5, macOS, Node 24.18)
npm test: 34 passed (engine 11, trajectory 21, fixture 2)npm run smoke: 33 passed, 0 failednpm run smoke:mcp: 4/4npm run eval: 7/7 verdicts as expected; output byte-identical across two runsNot done / limitations
ANTHROPIC_API_KEYwas available. Every eval number comes from the scripted simulated agent and describes the evaluator, not an LLM.docs/compare.pngpredates the eval panel (the panel has its own screenshot).