Make Bowu Bench replays faithful and isolated

[areshand]

Summary

• preserve the supervisor-authenticated direct user request separately from evaluator-owned instructions for both ops and conversation replays
• replace the legacy whole-session ops-plan benchmark with faithful request replays bound to the nearest preceding direct request and completed source answer
• provide pseudonymized captured tool results as local mock evidence, with secret-bearing output removed and no production access or fixture edits allowed
• enforce a read-only Codex sandbox for the single-agent ops baseline and retain the production multiagent lifecycle for comparison
• add the named 36-row bowu_bench adapter, local-outside-repo run paths, and optional post-run LLM semantic judging
• write all private component and combined manifests as mode 0600

Validation

• python3 -m unittest discover -s tests -p 'test_*.py' (113 passed)
• bash tests/run.sh (passed; existing compiler warnings only)
• BOWU_BENCH_SPLIT=all ... python3 -m evaluation.cli --adapter bowu_bench --selftest (all 36 rows passed good/bad scorer checks)
• regenerated canonical local manifest: 24 ops rows + 12 conversation rows, no empty requests/references/mock evidence, no duplicate IDs
• git diff --check

Live trace evidence

Before replacing the legacy ops-plan contract, an isolated rerun of trace-21806ac212af proved the direct Secrets Manager request reached plan alignment and caused the unrelated plan to be rejected. The revised replay contract removes that evaluator-induced conflict: roles now receive the authentic bounded conversation, answer the actual task from local mock evidence, and are scored for completion, isolation, routing, and semantic correctness.

Private Bowu Bench data remains under /Users/bowu/projects/traces and is not committed. A full live model replay still requires explicit approval for disclosure of the pseudonymized prompts, bounded history, mock evidence, and reference answers to the selected model provider.