fix(harbor): make generated tasks safe on remote sandboxes (e2b)

CHANGELOG.md

@@ -8,6 +8,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/).
 
 ## [Unreleased]
 ### Added
+- Documented running the generated Harbor dataset on remote sandboxes (`harbor run -e e2b`), including what the tasks assume about the sandbox, how credentials are passed, and the per-trial resource floor.
 - Added `scripts/export_openeval.py`, an additive script exporting a batch's `rescore-summary.json` as an [EvalPort](https://github.com/adhabnr-ux/evalport) `ResultSet` Thanks to [@adhabnr-ux](https://github.com/adhabnr-ux).
 - Added a `--browser-runtime kernel` mode to the Harbor adapter that runs each task against one Kernel cloud browser, exposing only a credential-free CDP bridge to the agent, and finalizes the replay and deletes the browser during verification.
 
@@ -18,6 +19,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/).
 - Changed the default Harbor version to `0.22.0`.
 
 ### Fixed
+- Harbor task setup now waits for the same condition as the step healthcheck — runtime server up, request interceptor armed, and CDP live — instead of accepting any 200 from `/api/status`. The server answers before its CDP handler attaches, so a task could previously start with interception inactive and silently fail to score Stage 1.
+- `start-runtime.sh` polls for the runtime server and Chromium's CDP endpoint instead of sleeping a fixed number of seconds, so a slower remote sandbox no longer fails a trial on provisioning latency.
 - Host-timeout container termination now uses the lazy container-engine resolver.
 - Added host-side container and batch-job timeouts so a wedged run cannot stall a batch indefinitely.
 - Fixed a judge-provider outage (or an unparseable judge reply) being recorded as an agent failure. `run.py` now exits 3 instead of 1 when the judge never renders a verdict, `batch.py` gives it its own `judge_inconclusive` bucket in `batch-summary.json` instead of folding it into `failed`, and `clawbench-rescore` now retries a cached `match: null` verdict even without `--force`.

docs/harbor.md

@@ -136,6 +136,35 @@ Generated tasks register a pinned Playwright MCP package (`@playwright/mcp@0.0.7
 
 Export `KERNEL_API_KEY` (and optionally `KERNEL_BASE_URL` for non-production gateways) before `harbor run`; no extra flags are needed.
 
+## Remote sandboxes (`-e e2b`)
+
+Harbor can run each trial on a remote sandbox provider instead of the local container daemon:
+
+```bash
+uvx --from harbor==0.22.0 harbor run -p ./harbor-datasets/clawbench-v2 \
+  -e e2b \
+  -a hermes -m deepseek/deepseek-v4-flash \
+  --env-file .env \
+  --ve CLAWBENCH_JUDGE_BASE_URL="$CLAWBENCH_JUDGE_BASE_URL" \
+  --ve CLAWBENCH_JUDGE_API_KEY="$CLAWBENCH_JUDGE_API_KEY" \
+  --ve CLAWBENCH_JUDGE_MODEL="${CLAWBENCH_JUDGE_MODEL:-deepseek-v4-pro}" \
+  --ve CLAWBENCH_JUDGE_API_TYPE="${CLAWBENCH_JUDGE_API_TYPE:-openai-completions}"
+```
+
+The generated tasks are written to be provider-agnostic, so `-e daytona` and `-e modal` take the same shape.
+
+**What the generated task assumes about its sandbox.** Nothing host-specific:
+
+- **No bind mounts outside the task directory.** Everything the runtime needs is baked into the environment image or written under `/data`, `/tmp/clawbench-run`, and `/my-info` at setup time.
+- **No local X11 or GPU.** Chromium runs headful under Xvfb with SwiftShader (`--use-gl=angle --use-angle=swiftshader`), and `--disable-dev-shm-usage` keeps it off a small `/dev/shm`.
+- **Readiness is polled, never slept for.** `start-runtime.sh` waits for the runtime server and Chromium's CDP endpoint to actually answer before continuing, and the step's setup script waits for the same condition Harbor's healthcheck checks — runtime server up, **request interceptor armed**, CDP live. A 200 from `/api/status` alone is not readiness: the server answers before the CDP handler attaches, and a task that starts in that window runs with interception inactive. Override the budgets with `CLAWBENCH_RUNTIME_WAIT_TIMEOUT_S` (default 60s, per component) and `CLAWBENCH_RUNTIME_READY_TIMEOUT_S` (default 180s, overall) if your provider provisions slowly.
+
+**Credentials never live in the dataset.** Every secret in `task.toml` is an environment reference (`${PURELY_MAIL_API_KEY}`, `${CLAWBENCH_JUDGE_API_KEY}`, `${KERNEL_API_KEY}`), resolved at run time from `--env-file` and `--ve`. Pass them that way and nothing sensitive is committed or shipped to the provider's image registry.
+
+**Resource floor.** Budget the same **1 CPU core and ~2 GB RAM per concurrent trial** as a local run — a remote sandbox still runs a full Chromium. Providers bill per sandbox-second, so `--timeout-multiplier` and per-task `time_limit` translate directly into cost.
+
+> **Not yet verified end-to-end.** The e2b smoke run in [#331](https://github.com/TIGER-AI-Lab/ClawBench/issues/331) is still open; this documents what the generated tasks require, not a passing run.
+
 ## Making it fast
 
 A full V2 sweep is 129 containerized browser sessions, each capped by the task's `time_limit`. Serial, that is a very long night. What actually moves the needle, in order:

src/clawbench/eval/harbor_adapter.py

@@ -131,6 +131,25 @@ def copy_environment(env_dir: Path) -> None:
             chmod_executable(script)
 
 
+def runtime_ready_command(cdp_url: str) -> str:
+    """Shell test for "the ClawBench runtime is ready to score this task".
+
+    Ready means three things at once: the runtime server answers, its request
+    interceptor is armed, and the browser's CDP endpoint is live. A 200 from
+    ``/api/status`` alone is not enough — the server starts answering before
+    the CDP handler has attached, so a task that begins in that window runs
+    with interception inactive and silently cannot score Stage 1.
+
+    Harbor's per-step healthcheck and the step's own setup script both use
+    this, so the two can never drift apart.
+    """
+    return (
+        "curl -sf http://127.0.0.1:7878/api/status "
+        + "| grep -q '\"eval_interceptor_ready\":true' "
+        + f"&& curl -sf {cdp_url}/json/version >/dev/null"
+    )
+
+
 def playwright_mcp_server(cdp_url: str) -> dict[str, Any]:
     return {
         "name": "playwright",
@@ -207,11 +226,8 @@ def task_toml(
             + '\nCLAWBENCH_RECORDING_MODE = "provider-download"'
         )
         mcp_servers = mcp_servers_toml([playwright_mcp_server(REMOTE_BRIDGE_CDP_URL)])
-    healthcheck_command = (
-        "curl -sf http://127.0.0.1:7878/api/status | grep -q '"
-        + '\\"eval_interceptor_ready\\":true'
-        + f"' && curl -sf {cdp_url}/json/version >/dev/null"
-    )
+    # The shell form is escaped once for TOML; setup.sh takes it verbatim.
+    escaped_healthcheck = json.dumps(runtime_ready_command(cdp_url))
     return (
         f"""schema_version = "1.3"
 source = "clawbench-v2"
@@ -260,7 +276,7 @@ def task_toml(
 timeout_sec = 300.0
 
 [steps.healthcheck]
-command = "{healthcheck_command}"
+command = {escaped_healthcheck}
 interval_sec = 2.0
 timeout_sec = 5.0
 start_period_sec = 2.0
@@ -273,15 +289,9 @@ def task_toml(
 
 def setup_script(browser_runtime: str = "local") -> str:
     kernel_setup = ""
-    readiness = (
-        "  if curl -sf http://127.0.0.1:7878/api/status >/dev/null \\\n"
-        "    && curl -sf http://127.0.0.1:9223/json/version >/dev/null; then\n"
-    )
+    cdp_url = REMOTE_BRIDGE_CDP_URL if browser_runtime == "kernel" else LOCAL_CDP_URL
+    readiness = f"  if {runtime_ready_command(cdp_url)}; then\n"
     if browser_runtime == "kernel":
-        readiness = (
-            "  if curl -sf http://127.0.0.1:7878/api/status >/dev/null \\\n"
-            "    && curl -sf http://127.0.0.1:9223/json/version >/dev/null; then\n"
-        )
         kernel_setup = (
             "# Create the Kernel browser and replay before the runtime server"
             " starts so it can bridge the provider CDP endpoint.\n"
@@ -310,15 +320,20 @@ def setup_script(browser_runtime: str = "local") -> str:
 
 {kernel_setup}/app/src/harbor/start-runtime.sh
 
-for _ in $(seq 1 60); do
+# Remote sandboxes (e2b, daytona, modal) cold-start slower than a local
+# container daemon, so this waits well past the local worst case rather than
+# failing a trial on provisioning latency.
+for _ in $(seq 1 "${{CLAWBENCH_RUNTIME_READY_TIMEOUT_S:-180}}"); do
 {readiness}    rm -f /app/setup.sh
     trap - EXIT
     exit 0
   fi
   sleep 1
 done
 
-echo "ClawBench Harbor runtime did not become ready" >&2
+echo "ClawBench Harbor runtime did not become ready: \
+runtime server, request interceptor, or CDP endpoint never came up" >&2
+tail -n 40 /tmp/clawbench-run/runtime-server.log >&2 || true
 exit 1
 """
 

src/clawbench/runtime/harbor/start-runtime.sh

@@ -3,6 +3,24 @@ set -euo pipefail
 
 mkdir -p /data /tmp/clawbench-run
 
+# Remote sandboxes (e2b, daytona, modal) start slower and less predictably
+# than a local container daemon, so every step below polls for the thing it
+# needs instead of sleeping a fixed number of seconds and hoping.
+WAIT_TIMEOUT_S="${CLAWBENCH_RUNTIME_WAIT_TIMEOUT_S:-60}"
+
+wait_for_url() {
+  local url=$1 label=$2 elapsed=0
+  while [ "$elapsed" -lt "$WAIT_TIMEOUT_S" ]; do
+    if curl -sf "$url" >/dev/null 2>&1; then
+      return 0
+    fi
+    sleep 1
+    elapsed=$((elapsed + 1))
+  done
+  echo "timed out after ${WAIT_TIMEOUT_S}s waiting for $label ($url)" >&2
+  return 1
+}
+
 if [ -f /tmp/clawbench-run/runtime.pid ] && kill -0 "$(cat /tmp/clawbench-run/runtime.pid)" 2>/dev/null; then
   echo "ClawBench Harbor runtime is already running."
   exit 0
@@ -20,13 +38,19 @@ if [ "$REMOTE_MODE" = false ]; then
   export DISPLAY="${DISPLAY:-:99}"
   Xvfb "$DISPLAY" -screen 0 1920x1080x24 >/tmp/clawbench-run/xvfb.log 2>&1 &
   echo "$!" > /tmp/clawbench-run/xvfb.pid
-  sleep 1
+  # Xvfb serves no HTTP endpoint to poll; wait for its socket to appear.
+  for _ in $(seq 1 "$WAIT_TIMEOUT_S"); do
+    if [ -e "/tmp/.X11-unix/X${DISPLAY#:}" ]; then
+      break
+    fi
+    sleep 1
+  done
 fi
 
 cd /app/src/runtime-server
 uv run --no-sync uvicorn server:app --host 0.0.0.0 --port 7878 >/tmp/clawbench-run/runtime-server.log 2>&1 &
 echo "$!" > /tmp/clawbench-run/runtime-server.pid
-sleep 1
+wait_for_url http://127.0.0.1:7878/api/status "runtime server"
 
 if [ "$REMOTE_MODE" = true ]; then
   # Keep the agent-facing CDP endpoint identical across browser runtimes.
@@ -129,13 +153,12 @@ LOAD_EXTS="/app/src/chrome-extension"
   about:blank >/tmp/clawbench-run/chrome.log 2>&1 &
 echo "$!" > /tmp/clawbench-run/chrome.pid
 
-sleep 2
+wait_for_url http://127.0.0.1:9222/json/version "Chromium CDP"
 socat TCP-LISTEN:9223,fork,reuseaddr,bind=0.0.0.0 TCP:127.0.0.1:9222 >/tmp/clawbench-run/socat.log 2>&1 &
 echo "$!" > /tmp/clawbench-run/socat.pid
 
 x11vnc -display "$DISPLAY" -nopw -shared -forever -rfbport 5900 -xkb >/tmp/clawbench-run/x11vnc.log 2>&1 &
 echo "$!" > /tmp/clawbench-run/x11vnc.pid
-sleep 1
 
 /opt/novnc/utils/novnc_proxy --vnc localhost:5900 --listen 6080 >/tmp/clawbench-run/novnc.log 2>&1 &
 echo "$!" > /tmp/clawbench-run/novnc.pid

tests/test_harbor_adapter.py

@@ -2,20 +2,25 @@
 
 import json
 import os
+import re
 import subprocess
 import sys
 import textwrap
 import tomllib
 from pathlib import Path
 
 from clawbench.eval.harbor_adapter import (
+    LOCAL_CDP_URL,
+    REMOTE_BRIDGE_CDP_URL,
     discover_cases,
+    runtime_ready_command,
     sanitize_task_name,
     unique_output_name,
     write_harbor_task,
 )
 from clawbench.runtime.harbor import verify as harbor_verify
 from clawbench.runtime.harbor.verify import write_reward
+from clawbench.utils.paths import RUNTIME_ROOT
 
 SRC_ROOT = Path(__file__).resolve().parents[1] / "src"
 
@@ -285,3 +290,104 @@ def test_harbor_verifier_omits_unknown_judge_match_metric(tmp_path: Path) -> Non
 
     reward = json.loads((tmp_path / "reward.json").read_text())
     assert reward == {"reward": 0.0, "intercepted": 0.0}
+
+
+def test_healthcheck_and_setup_share_one_readiness_contract(tmp_path: Path) -> None:
+    """A 200 from /api/status is not readiness: the interceptor must be armed.
+
+    The server answers before the CDP handler attaches, so a task that starts
+    in that window runs with interception inactive and cannot score Stage 1.
+    """
+    out = write_harbor_task(
+        task_dir=_write_case(tmp_path / "v2", "v2-047-x", _task()),
+        task=_task(),
+        output_root=tmp_path / "out",
+        output_name="v2-047-x",
+        org="clawbench",
+        dataset_name="v2",
+    )
+
+    config = tomllib.loads((out / "task.toml").read_text())
+    healthcheck = config["steps"][0]["healthcheck"]["command"]
+    setup = (out / "steps" / "run" / "workdir" / "setup.sh").read_text()
+
+    assert healthcheck == runtime_ready_command(LOCAL_CDP_URL)
+    assert healthcheck in setup
+    # Unescaped in the shell script, escaped exactly once in the TOML source.
+    assert "'\"eval_interceptor_ready\":true'" in setup
+    assert '\\"eval_interceptor_ready\\":true' in (out / "task.toml").read_text()
+
+
+def test_setup_waits_long_enough_for_a_remote_sandbox(tmp_path: Path) -> None:
+    out = write_harbor_task(
+        task_dir=_write_case(tmp_path / "v2", "v2-047-x", _task()),
+        task=_task(),
+        output_root=tmp_path / "out",
+        output_name="v2-047-x",
+        org="clawbench",
+        dataset_name="v2",
+    )
+
+    setup = (out / "steps" / "run" / "workdir" / "setup.sh").read_text()
+
+    assert "CLAWBENCH_RUNTIME_READY_TIMEOUT_S:-180" in setup
+    # A timeout must say what failed rather than exiting silently.
+    assert "runtime-server.log" in setup
+
+
+def test_kernel_setup_uses_the_bridge_readiness_contract(tmp_path: Path) -> None:
+    out = write_harbor_task(
+        task_dir=_write_case(tmp_path / "v2", "v2-047-x", _task()),
+        task=_task(),
+        output_root=tmp_path / "out",
+        output_name="v2-047-x",
+        org="clawbench",
+        dataset_name="v2",
+        browser_runtime="kernel",
+    )
+
+    config = tomllib.loads((out / "task.toml").read_text())
+    setup = (out / "steps" / "run" / "workdir" / "setup.sh").read_text()
+
+    assert config["steps"][0]["healthcheck"]["command"] in setup
+    assert runtime_ready_command(REMOTE_BRIDGE_CDP_URL) in setup
+    assert "kernel-browser.py start" in setup
+
+
+def test_runtime_startup_polls_instead_of_sleeping() -> None:
+    """Fixed sleeps encode local daemon timing; remote sandboxes are slower."""
+    script = (RUNTIME_ROOT / "harbor" / "start-runtime.sh").read_text()
+
+    assert "wait_for_url http://127.0.0.1:7878/api/status" in script
+    assert "wait_for_url http://127.0.0.1:9222/json/version" in script
+    assert "CLAWBENCH_RUNTIME_WAIT_TIMEOUT_S" in script
+    # No bare `sleep <n>` left standing in for a readiness check; the only
+    # sleeps remaining are the one-second steps inside the polling loops.
+    assert re.findall(r"^sleep \d+$", script, re.MULTILINE) == []
+
+
+def test_generated_task_carries_no_host_paths_or_baked_secrets(tmp_path: Path) -> None:
+    """A remote sandbox has none of the host's filesystem, and no .env."""
+    out = write_harbor_task(
+        task_dir=_write_case(tmp_path / "v2", "v2-047-x", _task()),
+        task=_task(),
+        output_root=tmp_path / "out",
+        output_name="v2-047-x",
+        org="clawbench",
+        dataset_name="v2",
+    )
+
+    config = tomllib.loads((out / "task.toml").read_text())
+    env = config["environment"]["env"]
+
+    # Every credential is a variable reference resolved by --ve/--env-file at
+    # run time, never a literal baked into the committed dataset.
+    for key, value in env.items():
+        if "KEY" in key or "MAIL" in key:
+            assert value.startswith("${"), f"{key} is not an env reference: {value}"
+
+    setup = (out / "steps" / "run" / "workdir" / "setup.sh").read_text()
+    test = (out / "steps" / "run" / "tests" / "test.sh").read_text()
+    for script in (setup, test):
+        assert str(tmp_path) not in script
+        assert "/host" not in script