feat(batch): --auto-restart supervisor and clawbench-batch-watch alerting

[vaibhavdabas16]

Advances #160. Meets three of its five acceptance criteria — --auto-restart N, the clawbench-eval-watch skeleton (named clawbench-batch-watch here, matching the clawbench-batch family), and docs/operations.md. The two remaining are real-run tests (Test 1 / Test 2) that need a multi-hour batch and are not something I can do from here; the doc states Test 2's semantics explicitly.

--auto-restart N

The key design point: the batch process cannot retry its own death. An OOM-killed container, a SIGKILL, an unhandled exception — the process is gone. So this is not a retry loop inside async_main; it's a supervisor that spawns clawbench-batch as a child and, on non-zero exit, waits and re-invokes it with --resume into the same batch directory, up to N times.

clawbench-batch --models m --cases-suite v2 --all-cases --harness hermes --auto-restart 3

• The batch directory is created before the first attempt, so every attempt — including the first — resumes into one place. --resume can be passed explicitly to supervise an existing batch.
• Ctrl-C is never retried. The child shares the supervisor's process group and receives it too; the supervisor waits for it to wind down (so containers aren't orphaned) and exits 130.
• Final exit status is the last attempt's: 0 only when an attempt finished with every job terminal.
• Lives in a new runner/supervise.py; batch.py gains only the two flags and a hand-off in main(). It does not touch the resume logic, so there's no overlap with fix(batch): resume from run-meta.json, not from log-file existence #329.

One honest dependency: today's --resume treats any job with a log file as done, including errored ones (#297). So after a batch that completed with infra errors, a restart currently resumes-and-skips them rather than retrying. Once #329 lands, restart automatically retries failed jobs — that's the "network blip" case in #160 — with no change needed here. For the abort case (process died mid-run) it works correctly today.

clawbench-batch-watch

clawbench-batch-watch test-output/batch-20260912-081500 --pid <batch pid> --heartbeat-every 10

Polls a batch directory and posts to a webhook on abort (last task, error tail from the newest log, elapsed), completion (pass/total, wall-clock), and optionally a heartbeat every N tasks. It reads only artifacts the batch already writes — run-meta.json, batch-logs/*.log, batch-summary.json — so it needs nothing from the process.

• Abort detection: --pid gone without a summary, or --stall-timeout (default 45 min) with no artifact activity.
• Config in ~/.config/clawbench/notify.toml (webhook_url, interval_s, heartbeat_every, stall_timeout_s), flags override. tomllib is stdlib on 3.11+, and the POST uses urllib — no new dependency (the issue mentions requests, but the repo doesn't have it).
• The payload carries both text and content, so one webhook shape serves Slack and Discord.
• A trap worth knowing about: the pid liveness probe is deliberately not os.kill(pid, 0). On Windows, os.kill with any signal other than the Ctrl events calls TerminateProcess with that value as the exit code — the standard POSIX liveness idiom would kill the batch it was watching. The Windows path uses OpenProcess/GetExitCodeProcess instead, and there's a test that probes a live and a dead pid.

Testing

tests/test_batch_resilience.py — 16 cases: the supervisor's restart-until-success loop (same --resume dir on every attempt, sleeps only between retries), the N+1 attempt bound, no-restart-on-interrupt, flag stripping so the child never recurses, and an end-to-end check that clawbench-batch --auto-restart hands off to the supervisor rather than running in-process; the watcher's snapshot counts, quiet-while-healthy, completion, pid-gone abort with error tail, stall abort, heartbeat-without-repeat, pid probe, TOML/flag precedence, and --once. clawbench.runner.watch is added to the CLI help-module test.

Full suite: 292 passed, 10 skipped locally; ruff and pyright clean on the new files.