Problem
Generated Harbor tasks use network_mode = "public" and run stock agents with shell access. The instruction says to use only the browser, but an agent can still call a site API with curl, Python, Node, or another direct client. That can invalidate benchmark results.
Required scope
- Define the browser-only policy and an explicit egress allowlist for sites, model endpoints, judge, PurelyMail, and browser providers.
- Enforce the policy at the container/tool boundary, not only in
instruction.md.
- Detect and preserve evidence of blocked direct requests or disallowed tools.
- Emit
failure_category = "policy_violation" without conflating it with an agent mismatch.
Acceptance criteria
- Direct API/HTTP submission attempts are blocked or recorded as policy violations.
- Legitimate CDP/browser flows continue to work.
- Harbor and native result summaries can separate policy violations from agent failures.
This is HAR-002 in docs/codebase-evaluation-2026-09.md and is related to #331. @Perry2004 please review the policy boundary before implementation.
Problem
Generated Harbor tasks use
network_mode = "public"and run stock agents with shell access. The instruction says to use only the browser, but an agent can still call a site API withcurl, Python, Node, or another direct client. That can invalidate benchmark results.Required scope
instruction.md.failure_category = "policy_violation"without conflating it with an agent mismatch.Acceptance criteria
This is
HAR-002indocs/codebase-evaluation-2026-09.mdand is related to #331. @Perry2004 please review the policy boundary before implementation.