Skip to content

Report each run's API token usage and estimated cost - #168

Merged
dadachi merged 1 commit into
mainfrom
feature/167-run-cost
Oct 18, 2026
Merged

dadachi merged 1 commit into
mainfrom
feature/167-run-cost

Conversation

@dadachi

@dadachi dadachi commented Oct 18, 2026

Copy link
Copy Markdown
Contributor

Closes #167.

Each run now reports what it cost, instead of leaving the user to filter tmp/trace/ by timestamp.

What changed

  • src/usage.ts: a per-run ledger (AsyncLocalStorage around dispatch(), so concurrent MCP calls and the smoke tests' many dispatches don't mix). The planner and each judge call record their usage; the repair agent records the Agent SDK's total_cost_usd. Priced exactly like scripts/cost.sh (list rates from OPUS_PRICE_PER_MTOK; calls served by a refusal-fallback model are counted in fallbackCalls and priced at the same rates).
  • report.json gets a cost block (estimatedUsd, calls, byAgent.{planner,judge,repair} with tokens and USD, pricing). Absent when the run made no API call (stub mode).
  • HTML report: an "API cost (estimate)" table and the rounded total in the header.
  • CLI: cost: ≈ $0.2197 (25 API calls: planner $0.0273, judge $0.1924) after the overall: line; the same line goes to tmp/trace/dispatch.log.
  • MCP: comes with report in structuredContent, no extra field.
  • Docs: README Runtime/Cost (per-run cost, the 2026-10-18 Console reconciliation, --tests adds ~2.5 min and no API cost since the suites run in parallel), docs/validation-report.md (section + data model), docs/PLUGIN-GUIDE.md, and the generate-app skill (report shape; quotes the cost in its headline).

Tests (npm test: 206 pass; build + typecheck:tests clean)

  • Pricing uses the Orange Baton run that matched the Console export token-for-token: planner $0.0273 + judge $0.1924 = $0.2197, with the hand calculation in the test.
  • Fallback counting and repair SDK cost taken as-is; concurrent ledgers stay separate and nothing is recorded outside one; stub dispatch has no cost; report/HTML/CLI line carry it.
  • Seen red: zeroing the cache-read rate, sharing one ledger across runs, and dropping the fallback count each fail the new tests.

Run for real (local build, --visual=0, barbershop spec): report.json cost = planner 2,722 in / 849 out = $0.0279, identical to scripts/cost.sh on that run's trace; the CLI line and the HTML section show it. The judge path wasn't exercised live (no simulator booted at the time); it's the same recordMessageUsage call next to the existing usage: trace line.

🤖 Generated with Claude Code

https://claude.ai/code/session_016UPsTupJqJ5X1NK2gZEN5c

Planner, judge and repair calls are collected per run in memory
(src/usage.ts, AsyncLocalStorage around dispatch) and written to
report.json as `cost`, shown in the HTML report, the CLI's closing
summary and the dispatch trace, and returned by the MCP tool with the
report. Priced like scripts/cost.sh: list rates from src/models.ts,
repair at the Agent SDK's own total_cost_usd.

README, the report design doc, the plugin guide and the generate-app
skill describe the new block; the skill quotes it in its headline.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016UPsTupJqJ5X1NK2gZEN5c
@dadachi
dadachi merged commit 3a03416 into main Oct 18, 2026
1 check passed
@dadachi dadachi mentioned this pull request Oct 18, 2026
@dadachi
dadachi deleted the feature/167-run-cost branch October 18, 2026 09:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Report each run's API token usage and estimated cost

1 participant