Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 5 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -131,7 +131,7 @@ End-to-end wall-clock per run, measured from `report.json`'s `meta.durationMs` o
| `1` | + full `xcodebuild build` + `./gradlew assembleDebug` + home-screen capture + Stage 1 vision judge | ~2.5 min (barbershop-queue · 2026-05-24) | ~$0.13 (derived) |
| `2` | + Rails server boot + scripted-CRUD walk via `mobile-mcp` + Stage 2 vision judge | ~7–8 min (sentova / sushi / task-tracker · 2026-05-23; orange-baton 7m 12s · 2026-10-17) | ~$0.22 (orange-baton, measured 2026-10-17) |

Cold builds, first-run cocoapods/gradle dependency resolution, or unbooted simulators add a one-time minute or two. `--tests` adds a few minutes and no API cost: the three suites run in parallel, so the same orange-baton VISUAL=2 run took 9m 52s with them (2026-10-18), about 2.5 min more. The self-repair loop (opt-in, hard-capped at 5 iterations) can extend a failing run substantially — budget for it if you set `NATIVEAPPTEMPLATE_REPAIR=on`.
Cold builds, first-run cocoapods/gradle dependency resolution, or unbooted simulators add a one-time minute or two. `--tests` adds a few minutes and no API cost: the three suites run in parallel, so the same orange-baton VISUAL=2 run took 9m 52s with them (2026-10-18), about 2.5 min more; a 0.4.2 re-run with `--domain=orangebaton.com` and `--rename-ja` took 9m 06s (Rails 24 s, Android 23 s and iOS 85 s of tests). The self-repair loop (opt-in, hard-capped at 5 iterations) can extend a failing run substantially — budget for it if you set `NATIVEAPPTEMPLATE_REPAIR=on`.

**Cost.** The agent makes real `claude-opus-5-5` API calls ($4 / $20 per million input / output tokens; 5-minute cache writes $5, cache reads $0.40; rates in [`src/models.ts`](./src/models.ts)). How the numbers above were made, on 2026-10-15:

Expand All @@ -140,13 +140,14 @@ Cold builds, first-run cocoapods/gradle dependency resolution, or unbooted simul
- **Derived, per run:** VISUAL=1 = planner + Stage 1 on iOS and Android ≈ $0.13; VISUAL=2 adds Stage 2 on both ≈ $0.24. A judge retry after a transient render failure adds ~$0.05 per retried platform.
- **Measured, per run (2026-10-17):** a passing paid-edition VISUAL=2 run of the Orange Baton spec (see [Validation report](#validation-report)) cost **$0.223** — planner 2,761 in / 874 out = $0.029, plus 24 judge calls (Stage 1 + Stage 2 on iOS and Android) = 3,456 in / 2,976 out / 17,320 cache-write / 86,600 cache-read = $0.195. That's in line with the derived ~$0.24.
- **Checked against billing (2026-10-18):** three Orange Baton runs, as `tmp/trace/` logged them, summed to 17,835 in / 10,235 out / 47,584 cache-write / 237,920 cache-read tokens, ≈ $0.61 — the Claude Console's usage export for that key and day showed exactly the same token counts. Another VISUAL=2 run that day came to the same 24 judge calls, $0.2197 in all.
- **Reported per run (0.4.2, 2026-10-18):** a passing VISUAL=2 + `--tests` run of the same spec, with `--domain=orangebaton.com` and `--rename-ja Shop=案件 --rename-ja ItemTag=バトン`, recorded **$0.2163** in its own `report.json` — planner 2,761 in / 820 out = $0.0274, plus 24 judge calls = 3,456 in / 2,692 out / 17,320 cache-write / 86,600 cache-read = $0.1889. Same input and cache token counts as the trace-based measurement above; only the output tokens differ.
- **Self-repair** (`NATIVEAPPTEMPLATE_REPAIR`): a toy fix converged in 3 turns for ~$0.10 (measured). A real Compose/Hilt build fix can take many more turns (up to 40); estimate ~$0.10–$1 per attempt, at most 5 attempts per run. Re-validation after a repair does not re-run the judge.

**Check your own spend:** every run reports its own estimate. The CLI's closing summary prints it after the verdict:

```
overall: PASS
cost: ≈ $0.2197 (25 API calls: planner $0.0273, judge $0.1924)
cost: ≈ $0.2163 (25 API calls: planner $0.0274, judge $0.1889)
```

`report.json` has the per-agent breakdown under `cost` (calls, input / output / cache-write / cache-read tokens, USD; repair is the Agent SDK's own `total_cost_usd`), the HTML report shows it as an "API cost (estimate)" table, and the MCP tool returns it with the report. It is priced at the list rates above; for billed amounts, see usage per API key in the Claude Console, and set a workspace spend cap (see [Security](#security)) as a backstop — the agent exits non-zero on validation failure, but doesn't gate on spend itself. To total several runs, `npm --prefix <path-to-this-repo> run cost` (or `bash <path-to-this-repo>/scripts/cost.sh [trace_dir]`) adds up the per-call lines in `tmp/trace/` of the directory you ran the agent from; those logs append across runs, so clear `tmp/trace/` first to cost a single one.
Expand Down Expand Up @@ -293,9 +294,9 @@ See [`docs/SPEC.md`](./docs/SPEC.md) for the full design.

Every run writes a report of the validation results to the output directory:

[![Example validation report (paid edition, VISUAL=2): a self-contained HTML page showing the overall PASS verdict, a platform×layer matrix, per-layer detail, embedded iOS + Android home-screen captures with the vision judge's rationales, the Stage 2 walk-through filmstrip, the contract-parity check, and the domain rename plan](./docs/images/validation-report.png)](./docs/images/validation-report.png)
[![Example validation report (paid edition, VISUAL=2): a self-contained HTML page showing the overall PASS verdict, a platform×layer matrix, per-layer detail, formatter and test-suite results, embedded iOS + Android home-screen captures with the vision judge's rationales, the Stage 2 walk-through filmstrip, the contract-parity check, the API cost table, the domain rename plan and the app identifiers](./docs/images/validation-report.png)](./docs/images/validation-report.png)

*Example report from a paid-edition `NATIVEAPPTEMPLATE_VISUAL=2` run of `"a job hand-off tracker for small field-service companies: each project always has exactly one open baton held by one teammate; finishing it means choosing the next stage and the next holder"` with `--project-name="Orange Baton" --rename Shop=Project --rename ItemTag=Baton --rename Shopkeeper=Teammate` — all three layers green, Stage 2 walked 46/46 steps on iOS and 48/48 on Android, 7m 12s, $0.22. Click to view full size; the real artifact is a live, self-contained HTML file you open in a browser.*
*Example report from a paid-edition `NATIVEAPPTEMPLATE_VISUAL=2` run of `"a job hand-off tracker for small field-service companies: each project always has exactly one open baton held by one teammate; finishing it means choosing the next stage and the next holder"` with `--project-name="Orange Baton" --domain=orangebaton.com --rename Shop=Project --rename ItemTag=Baton --rename Shopkeeper=Teammate --rename-ja Shop=案件 --rename-ja ItemTag=バトン --tests` (agent 0.4.2) — all three layers green, Stage 2 walked 46/46 steps on iOS and 48/48 on Android, all three test suites passed, 9m 06s, $0.22. Click to view full size; the real artifact is a live, self-contained HTML file you open in a browser.*

- **`out/<slug>/validation-report.html`** — a self-contained HTML report (screenshots base64-embedded, no external assets, no JavaScript) you can open in a browser, attach to a PR, or drop into a demo. It shows the overall verdict — **PASS**, **FAIL**, or **PARTIAL** (nothing failed, but Layer 3 didn't judge the UI, as in a default `NATIVEAPPTEMPLATE_VISUAL=0` run) — a platform×layer matrix, Layer 1 leftover-token findings, Layer 2 build commands + `stderr`, Layer 3 home-screen screenshots with the vision judge's per-criterion rationales (plus the Stage 2 filmstrip when `NATIVEAPPTEMPLATE_VISUAL=2`), the reviewer's contract diff, the run's API cost, and the domain rename plan.
- **`out/<slug>/report.json`** — the same data, machine-readable, for CI gating or programmatic use. The full schema lives in [`docs/validation-report.md`](./docs/validation-report.md).
Expand Down
Binary file modified docs/images/validation-report.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
2 changes: 1 addition & 1 deletion docs/validation-report.md
Original file line number Diff line number Diff line change
Expand Up @@ -193,7 +193,7 @@ Reuse the social-preview palette for brand coherence (`docs/social-preview.svg`)

- Always write `out/<slug>/report.json` + `validation-report.html`.
- Print the report path as a `file://` URL (clickable in terminals) alongside the existing summary lines.
- When the run made API calls, print `cost: ≈ $0.2197 (25 API calls: planner $0.0273, judge $0.1924)` after the `overall:` line.
- When the run made API calls, print `cost: ≈ $0.2163 (25 API calls: planner $0.0274, judge $0.1889)` after the `overall:` line.

Flags:

Expand Down
Loading