Skip to content

[Harbor] Enforce browser-only execution policy instead of relying on prompt text #352

Description

@reacher-z

Problem

Generated Harbor tasks use network_mode = "public" and run stock agents with shell access. The instruction says to use only the browser, but an agent can still call a site API with curl, Python, Node, or another direct client. That can invalidate benchmark results.

Required scope

  • Define the browser-only policy and an explicit egress allowlist for sites, model endpoints, judge, PurelyMail, and browser providers.
  • Enforce the policy at the container/tool boundary, not only in instruction.md.
  • Detect and preserve evidence of blocked direct requests or disallowed tools.
  • Emit failure_category = "policy_violation" without conflating it with an agent mismatch.

Acceptance criteria

  • Direct API/HTTP submission attempts are blocked or recorded as policy violations.
  • Legitimate CDP/browser flows continue to work.
  • Harbor and native result summaries can separate policy violations from agent failures.

This is HAR-002 in docs/codebase-evaluation-2026-09.md and is related to #331. @Perry2004 please review the policy boundary before implementation.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

enhancementNew feature or requesthelp wantedExtra attention is needed

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions