From f9b3d6e53f4e9e0fea994db30356de3ae25fe1a1 Mon Sep 17 00:00:00 2001 From: Jessie Liu Date: Wed, 29 Jul 2026 11:28:09 -0700 Subject: [PATCH 1/6] Add agent harness documentation MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Documents the agent layer: the harnesses under test, how a harness is chosen independently of the model it drives, and the capability surface a harness grants an agent. - docs/components/agents.md — the registered harnesses, harness vs model, and the AGENT_* / BENCH_* configuration. - docs/how-to/add-an-agent-harness.md — the steps to plug in a new harness. - .agents/references/harness-capabilities.md — the shared capability map that the review and cleanup skills consult. The harness table adds a row for the antigravity harness, which registers under that key here but was absent from the table. Signed-off-by: Jessie Liu --- .agents/references/harness-capabilities.md | 40 ++++++ docs/components/agents.md | 129 +++++++++++++++++ docs/how-to/add-an-agent-harness.md | 157 +++++++++++++++++++++ 3 files changed, 326 insertions(+) create mode 100644 .agents/references/harness-capabilities.md create mode 100644 docs/components/agents.md create mode 100644 docs/how-to/add-an-agent-harness.md diff --git a/.agents/references/harness-capabilities.md b/.agents/references/harness-capabilities.md new file mode 100644 index 000000000..e42e8ea10 --- /dev/null +++ b/.agents/references/harness-capabilities.md @@ -0,0 +1,40 @@ +# Harness capabilities map + +The eval skills and references are **agent-agnostic**: they describe what they +need as generic capabilities ("spawn a sub-agent", "run a command detached", +"schedule a wakeup", "keep durable state", "use an isolated worktree", "ask the +operator", "emit a heartbeat"). This file is the **one place** that maps those +capabilities to concrete tools per harness. + +Skills should express needs generically and consult this map; **degrade +gracefully when a capability is absent** — every row has a generic fallback that +works on a bare harness with nothing but a shell. + +| Capability | Claude Code | Antigravity | Codex | Generic fallback | +|---|---|---|---|---| +| **Spawn a sub-agent** | `Agent` (`subagent_type`) | `invoke_subagent` / `define_subagent` | sub-agent via `codex exec` | run the work inline yourself in one shell | +| **Cheap vs strong model tier** | Haiku / Sonnet / Opus | `/models` (Flash / Pro) | Codex mini / standard | one model for everything; just spend it sparingly | +| **Background / detached run** | `run_in_background` | `manage_task` / `manage_subagents` | shell job (`codex exec` async) | `nohup … &` and poll a file/marker | +| **Scheduled wakeup / timer** | `ScheduleWakeup` | `schedule` | shell cron / `at` | `sleep` between checks, or re-poll each turn | +| **Durable state** | task list | Artifacts / `write_to_file` | a file in the repo | a plain notes file on disk | +| **Isolated worktree** | `EnterWorktree` | `run_command` + `git worktree` | `git worktree` | `git worktree add` + a branch | +| **Ask the operator** | `AskUserQuestion` | `ask_question` | prompt the user | ask in chat | +| **Heartbeat / keepalive** | progress line, no early "done" | progress line, no early "done" | periodic re-check + status line | print a `still working: …` line each tick | + +Notes: + +- **Claude Code** and **Antigravity** rows are seeded from the run-parallel-evals + §2 harness-portability table; each cell names **one primitive** — assume the + harness chains the supporting calls (args, follow-ups, file reads) from it. +- **Codex**: Codex CLI / `codex exec`; background via a shell job; state in files; + isolation via `git worktree`; prompt the user to clarify; keepalive via periodic + re-check. No native scheduler — re-poll on a `sleep`/cron. +- **Antigravity tool set** (confirmed from a live instance): files `view_file` · + `list_dir` · `grep_search` · `write_to_file` · `replace_file_content` · + `multi_replace_file_content`; exec `run_command` · `ask_permission` · + `list_permissions`; web `search_web` · `read_url_content`; subagents/background + `invoke_subagent` · `define_subagent` · `manage_subagents` · `send_message` · + `manage_task` · `schedule`; interaction `ask_question` · `generate_image`. +- The **runner host** holds the durable run state (`RESUME_STAMP` under + `~/matrix-runs//`), so even a bare harness — one shell, no sub-agents, no + scheduler — can drive and re-attach to a run by polling files. diff --git a/docs/components/agents.md b/docs/components/agents.md new file mode 100644 index 000000000..ab8d34417 --- /dev/null +++ b/docs/components/agents.md @@ -0,0 +1,129 @@ +# Agents + +An **agent harness** is the thing under test. It drives one AI agent against one +task prompt and hands back a typed result the rest of the benchmark can score. +Everything in this layer lives under `devops_bench/agents/`. + +The base class is `AgentHarness` (`devops_bench/agents/base.py`). It owns two +concerns so subclasses never have to: the base `run()` stamps wall-clock +**latency** onto every result, and it wraps the agent in a **safety net** — any +crash inside the agent is caught and turned into an errored result, so one faulty +agent never aborts the whole benchmark. Subclasses implement a single method, +`_execute()`, which does the provider-specific work and returns an `AgentResult`. + +```text +agent.run(prompt) -> AgentResult # base: latency + safety net + └─ agent._execute(prompt) # subclass: build invocation, parse, return +``` + +## Supported harnesses + +Four harnesses ship today. Each self-registers under a canonical key. + +| Key | Wraps | How it runs | Capabilities | +| --- | --- | --- | --- | +| `gemini` | The Google **Gemini CLI** binary | Headless subprocess; trajectory parsed from `--output-format stream-json` on stdout | MCP, skills, rules, allowed-tools | +| `openclaw` | The **Openclaw Agent CLI** | `openclaw agent --local` with per-run isolated state/config; trajectory via `openclaw sessions export-trajectory` | MCP, skills, rules | +| `antigravity` | The **Antigravity CLI** (`agy` binary) | Headless subprocess that keeps the real `HOME` so cached OAuth/ADC credentials work; trajectory parsed from the transcript JSONL it writes, token usage read from the conversation DB | MCP, skills, rules | +| `api` | **In-process** model call | Calls `get_model(provider, model)` and runs a model-agnostic MCP tool-use loop (`max_turns`, default 50) | MCP (spawns a stdio server), skills (served as tools), rules (system instruction) | + +> `oc` is just a shorthand alias for the `openclaw` CLI; this doc uses `openclaw` throughout. + +> [!NOTE] +> The `gemini` key names the CLI **harness** — the program that drives the agent. +> It is not the gemini **model**. You can run the gemini *model* through the `api` +> harness, or run a non-gemini model through the `gemini` CLI, because the harness +> and the model are chosen independently (see [Harness vs model](#harness-vs-model)). +> The alias `gemini-cli` also resolves to `gemini`, and is the default agent type. + +## Harness vs model + +A harness does **not** hardcode a model. It reads `AGENT_PROVIDER` and +`AGENT_MODEL` from its config and maps them onto whatever it drives. + +Every harness resolves `AGENT_PROVIDER` through one shared contract +(`devops_bench/core/model_providers.py`), so the same `AGENT_*` config behaves +identically across them. The `api` harness uses it to pick the adapter family and +backend for `get_model(provider, model)` and runs the tool-use loop in-process. +The CLI harnesses (`gemini`, `openclaw`) use it to route `AGENT_API_KEY` onto the +binary's provider-specific env var(s) and pass the model through: the Gemini CLI +gets `GEMINI_MODEL`, and openclaw gets a `--model provider/id` flag. Either way, +the model is a runtime input, never baked into the harness. + +For everything about providers, model ids, and how `get_model` resolves them, see +[Model providers](./model_providers.md). + +## Configuring a harness for an eval + +Configuration is env-driven. The benchmark reads neutral `AGENT_*` variables and +each harness maps them onto its target. + +**Selecting the harness** + +| Variable | Default | Notes | +| --- | --- | --- | +| `BENCH_AGENT_TYPE` | `gemini-cli` (resolves to `gemini`) | The canonical key or an alias. The `--agent-type` flag overrides it. | + +**Agent config** + +| Variable | Default | Notes | +| --- | --- | --- | +| `AGENT_MODEL` | unset | Model id; flows to the harness's target. | +| `AGENT_PROVIDER` | unset | Provider key (e.g. `gemini`, `anthropic`, `google-vertex`). | +| `AGENT_API_KEY` | unset | Routed onto the provider's key env var(s) via the shared contract; omitted for keyless backends (Vertex/Bedrock ADC). | +| `AGENT_TARGET` | unset | Path to the CLI binary (`gemini` / `oc`). Ignored by `api`. | +| `AGENT_TIMEOUT_SEC` | `600` | Wall-clock budget for each external call. | +| `AGENT_MAX_TURNS` | harness default (50 for `api`) | Caps the `api` tool-use loop. | + +**Capabilities** + +| Variable | Default | Notes | +| --- | --- | --- | +| `BENCH_USE_MCP` | `true` | Master gate. `false` drops the MCP binding entirely. | +| `AGENT_MCP_SERVER` | unset | Shell-quoted argv for the MCP server (e.g. `"uv run gke-mcp"`). | +| `AGENT_ALLOWED_TOOLS` | unset | CSV of pre-approved tool names. | +| `AGENT_SKILLS_PATHS` | unset | CSV of directories to discover `SKILL.md` files under. | +| `AGENT_RULES_TEXT` | unset | Operator-brief text handed to the agent. | + +### Example: gemini CLI with MCP + skills + +```bash +export BENCH_AGENT_TYPE=gemini +export AGENT_PROVIDER=gemini +export AGENT_MODEL=gemini-2.5-pro +export AGENT_API_KEY="$GEMINI_API_KEY" +export AGENT_TARGET=gemini + +export BENCH_USE_MCP=true +export AGENT_MCP_SERVER="uv run gke-mcp" +export AGENT_ALLOWED_TOOLS="list_clusters,get_pods" +export AGENT_SKILLS_PATHS="/opt/skills/gke,/opt/skills/k8s" +``` + +### Example: api harness on Claude with MCP off + +```bash +export BENCH_AGENT_TYPE=api +export AGENT_PROVIDER=anthropic +export AGENT_MODEL=claude-sonnet-4-5 +export AGENT_API_KEY="$ANTHROPIC_API_KEY" + +export BENCH_USE_MCP=false # no MCP server is spawned; tools are dropped +``` + +## Capabilities + +MCP tools, skills, and rules are the three augmentation axes, and they are +independent — an agent may run with any combination, or none. Each is expressed +as a structural Protocol (`SupportsMcp`, `SupportsSkills`, `SupportsRules` in +`devops_bench/agents/capabilities/`): a harness satisfies a Protocol simply by +assigning the matching binding attribute. **MCP** wires the agent to a tool +server, **skills** drop `SKILL.md` files the agent can discover, and **rules** +supply an operator brief. Setting `BENCH_USE_MCP=false` drops the MCP binding +entirely, so the agent sees no tools and the scorer agrees that none ran — skills +and rules are unaffected. + +## Adding your own harness + +Want to wrap a different agent? See +[Add an agent harness](../how-to/add-an-agent-harness.md). diff --git a/docs/how-to/add-an-agent-harness.md b/docs/how-to/add-an-agent-harness.md new file mode 100644 index 000000000..08ffdff5d --- /dev/null +++ b/docs/how-to/add-an-agent-harness.md @@ -0,0 +1,157 @@ +# Add an agent harness + +This guide walks through wrapping a new agent so the benchmark can drive it. The +contract is small: subclass `AgentHarness`, implement `_execute`, register the +class with `@AGENTS.register`, and add your module to the builtin import list. +That's it — no `cli.py` or `run.py` edits. + +For the concepts (harness vs model, capabilities, configuration), read +[Agents](../components/agents.md) first. + +## The contract + +| You do | Where | +| --- | --- | +| Subclass `AgentHarness` | `devops_bench/agents/base.py` | +| Implement `_execute(self, prompt) -> AgentResult` | your new module | +| Register with `@AGENTS.register("")` | your new module | +| Add the module to `_BUILTIN_AGENT_MODULES` | `devops_bench/evalharness/default.py` | + +## Steps + +### 1. Create the module + +Mirror an existing harness. For a CLI-backed agent, follow `gemini_cli` / +`openclaw`: + +```text +devops_bench/agents/cli//agent.py +``` + +For an in-process agent, follow `api`: + +```text +devops_bench/agents//agent.py +``` + +### 2. Subclass `AgentHarness` and assign capability bindings + +Call the base `__init__` with your config, then assign `self.mcp_servers`, +`self.skills`, and `self.rules` from `self.config.capabilities`. Those three +assignments are what make your harness structurally satisfy the capability +Protocols (`SupportsMcp` / `SupportsSkills` / `SupportsRules`) — no mixin needed. + +### 3. Implement only `_execute` + +`_execute(self, prompt: str) -> AgentResult` is the single extension point. +Inside it: + +- Build the invocation for your agent (argv, an API call, whatever it takes). +- Parse the agent's output into canonical `ToolCall` entries + (`devops_bench/agents/result.py`) for the trajectory. +- On a *known* failure (subprocess error, parse miss, timeout), record a message + on `AgentResult.errors` rather than dropping it silently. For a hard failure + with no usable output, return `AgentResult.errored(msg)`. +- Return an `AgentResult`. Leave `latency` at zero — the base `run()` fills it in. + +> [!NOTE] +> Only handle your *known* errors. The base class already catches unexpected +> exceptions and converts them to an errored result, so you don't need a +> catch-all. + +### 4. Register the class + +Decorate it with its canonical key: + +```python +@AGENTS.register("") +class MyAgent(AgentHarness): + ... +``` + +### 5. Wire it for import side-effects + +Registration only fires when the module is imported, so add its path to +`_BUILTIN_AGENT_MODULES` in `devops_bench/evalharness/default.py`: + +```python +_BUILTIN_AGENT_MODULES: tuple[str, ...] = ( + "devops_bench.agents.cli.gemini_cli", + "devops_bench.agents.cli.openclaw", + "devops_bench.agents.api.agent", + "devops_bench.agents..agent", # <- your module +) +``` + +The import loop tolerates `ImportError` / `MissingDependencyError`, so a harness +that needs an optional SDK won't break the host that lacks it. If you want a +friendlier selector name, add an entry to `_AGENT_TYPE_ALIASES` in the same file — +for example, mapping `gemini-cli` to `gemini`. + +### 6. Reuse the shared CLI helpers + +For a CLI agent, don't re-implement capability plumbing. Reuse the helpers in +`devops_bench/agents/shared/cli_capabilities.py`: + +- `build_mcp_servers(...)` — turns granted MCP bindings into a `{name: {command, args}}` launch map. +- `materialize_skills(...)` — copies discovered `SKILL.md` files into a skills directory and returns its path. + +> [!IMPORTANT] +> These helpers stage the files, but they don't tell your agent where to find +> them. Your `_execute` is responsible for pointing the underlying tool at the +> staged locations — whether that's a CLI flag, a config file, or an environment +> variable (e.g. the Gemini CLI agent writes the MCP launch map into its settings +> and the openclaw agent exports its skills dir). Wire the path/env through in +> your harness, or the staged MCP servers and skills won't be picked up. + +### 7. Select it + +Pick your harness with `BENCH_AGENT_TYPE=` (or `--agent-type `). No +other code changes are required — the registry resolves it at run time. + +## Skeleton + +```python +from devops_bench.agents.base import AGENTS, AgentHarness +from devops_bench.agents.config import AgentConfig +from devops_bench.agents.result import AgentResult, ToolCall + + +@AGENTS.register("myagent") +class MyAgent(AgentHarness): + """Harness driving .""" + + def __init__(self, config: AgentConfig | None = None) -> None: + AgentHarness.__init__(self, config) + caps = self.config.capabilities + self.mcp_servers = caps.mcp_servers + self.skills = caps.skills + self.rules = caps.rules + + def _execute(self, prompt: str) -> AgentResult: + # 1. Build and run the invocation for `prompt`. + # 2. Parse output into canonical ToolCall entries. + trajectory: list[dict] = [ + ToolCall(name="example_tool", args={}).to_dict(), + ] + # 3. On a known failure, return AgentResult.errored("..."). + # 4. Return the result (leave latency at zero; the base stamps it). + return AgentResult(output="...", trajectory=trajectory) +``` + +## Test it + +Run a no-infra task with your harness selected. The `noop` deployer skips cluster +provisioning so you can confirm the harness drives the agent and returns a +trajectory end-to-end without standing up infrastructure: + +```bash +export BENCH_AGENT_TYPE=myagent +export BENCH_NO_INFRA=true +export AGENT_PROVIDER=... +export AGENT_MODEL=... +# run a single generation-only task and inspect results.json +``` + +Check the run's `results.json`: a clean run shows your parsed `trajectory`, a +populated `output`, and an empty `errors` list. From 27beb6ab076f88cd9b325a40bdc71947460e6dc7 Mon Sep 17 00:00:00 2001 From: Jessie Liu Date: Tue, 18 Aug 2026 13:32:31 -0700 Subject: [PATCH 2/6] Address review on the agent harness docs - The smoke-test block mixed a concrete BENCH_AGENT_TYPE with elided provider and model values; use concrete stand-ins throughout so the example reads as one coherent thing. - Mark the Codex column unverified. The Antigravity row was confirmed against a live instance and the Claude Code row came from a real portability table, but the Codex cells were inferred from documented behaviour and never checked. A capability map is only useful if its confidence is visible, and every row has a generic fallback anyway. Signed-off-by: Jessie Liu --- .agents/references/harness-capabilities.md | 5 ++++- docs/how-to/add-an-agent-harness.md | 4 ++-- 2 files changed, 6 insertions(+), 3 deletions(-) diff --git a/.agents/references/harness-capabilities.md b/.agents/references/harness-capabilities.md index e42e8ea10..9ab437583 100644 --- a/.agents/references/harness-capabilities.md +++ b/.agents/references/harness-capabilities.md @@ -28,7 +28,10 @@ Notes: harness chains the supporting calls (args, follow-ups, file reads) from it. - **Codex**: Codex CLI / `codex exec`; background via a shell job; state in files; isolation via `git worktree`; prompt the user to clarify; keepalive via periodic - re-check. No native scheduler — re-poll on a `sleep`/cron. + re-check. No native scheduler — re-poll on a `sleep`/cron. **Unverified**: these + cells are inferred from Codex CLI's documented behaviour, not confirmed against a + live instance the way the Antigravity row was. Treat them as a starting point and + fall back to the generic column if a primitive does not behave as described. - **Antigravity tool set** (confirmed from a live instance): files `view_file` · `list_dir` · `grep_search` · `write_to_file` · `replace_file_content` · `multi_replace_file_content`; exec `run_command` · `ask_permission` · diff --git a/docs/how-to/add-an-agent-harness.md b/docs/how-to/add-an-agent-harness.md index 08ffdff5d..8bef1d22b 100644 --- a/docs/how-to/add-an-agent-harness.md +++ b/docs/how-to/add-an-agent-harness.md @@ -148,8 +148,8 @@ trajectory end-to-end without standing up infrastructure: ```bash export BENCH_AGENT_TYPE=myagent export BENCH_NO_INFRA=true -export AGENT_PROVIDER=... -export AGENT_MODEL=... +export AGENT_PROVIDER=myprovider +export AGENT_MODEL=mymodel # run a single generation-only task and inspect results.json ``` From 2a29ff47419f349e293eec47733c69a09f8bb671 Mon Sep 17 00:00:00 2001 From: Jessie Liu Date: Wed, 19 Aug 2026 10:48:40 -0700 Subject: [PATCH 3/6] Drop the Codex column and the notes from the capability map Two reviewers agreed the Codex column should go rather than ship unverified. The word appeared in exactly one file in the repo and the cells were inferred from documented behaviour rather than checked against a live instance, so there was nothing to lose by removing it. The notes section goes with it. The Antigravity tool inventory would stale the moment that tool changes, and the provenance note only existed to caveat Codex. One point is kept, folded into the body: the runner host holds durable run state under RESUME_STAMP, so a bare harness with nothing but a shell can drive and re-attach to a run. That is what makes the generic fallback column meaningful rather than aspirational. Also corrects the contract row for the harness interface. The abstract method is _execute(self, prompt, workspace_path=None) and run() calls it positionally, so a harness written to the two argument signature the doc showed would raise TypeError on first use, which the base class converts into an errored result rather than surfacing as a signature mismatch. Signed-off-by: Jessie Liu --- .agents/references/harness-capabilities.md | 43 +++++++--------------- docs/how-to/add-an-agent-harness.md | 2 +- 2 files changed, 14 insertions(+), 31 deletions(-) diff --git a/.agents/references/harness-capabilities.md b/.agents/references/harness-capabilities.md index 9ab437583..e5f919493 100644 --- a/.agents/references/harness-capabilities.md +++ b/.agents/references/harness-capabilities.md @@ -10,34 +10,17 @@ Skills should express needs generically and consult this map; **degrade gracefully when a capability is absent** — every row has a generic fallback that works on a bare harness with nothing but a shell. -| Capability | Claude Code | Antigravity | Codex | Generic fallback | -|---|---|---|---|---| -| **Spawn a sub-agent** | `Agent` (`subagent_type`) | `invoke_subagent` / `define_subagent` | sub-agent via `codex exec` | run the work inline yourself in one shell | -| **Cheap vs strong model tier** | Haiku / Sonnet / Opus | `/models` (Flash / Pro) | Codex mini / standard | one model for everything; just spend it sparingly | -| **Background / detached run** | `run_in_background` | `manage_task` / `manage_subagents` | shell job (`codex exec` async) | `nohup … &` and poll a file/marker | -| **Scheduled wakeup / timer** | `ScheduleWakeup` | `schedule` | shell cron / `at` | `sleep` between checks, or re-poll each turn | -| **Durable state** | task list | Artifacts / `write_to_file` | a file in the repo | a plain notes file on disk | -| **Isolated worktree** | `EnterWorktree` | `run_command` + `git worktree` | `git worktree` | `git worktree add` + a branch | -| **Ask the operator** | `AskUserQuestion` | `ask_question` | prompt the user | ask in chat | -| **Heartbeat / keepalive** | progress line, no early "done" | progress line, no early "done" | periodic re-check + status line | print a `still working: …` line each tick | +| Capability | Claude Code | Antigravity | Generic fallback | +|---|---|---|---| +| **Spawn a sub-agent** | `Agent` (`subagent_type`) | `invoke_subagent` / `define_subagent` | run the work inline yourself in one shell | +| **Cheap vs strong model tier** | Haiku / Sonnet / Opus | `/models` (Flash / Pro) | one model for everything; just spend it sparingly | +| **Background / detached run** | `run_in_background` | `manage_task` / `manage_subagents` | `nohup … &` and poll a file/marker | +| **Scheduled wakeup / timer** | `ScheduleWakeup` | `schedule` | `sleep` between checks, or re-poll each turn | +| **Durable state** | task list | Artifacts / `write_to_file` | a plain notes file on disk | +| **Isolated worktree** | `EnterWorktree` | `run_command` + `git worktree` | `git worktree add` + a branch | +| **Ask the operator** | `AskUserQuestion` | `ask_question` | ask in chat | +| **Heartbeat / keepalive** | progress line, no early "done" | progress line, no early "done" | print a `still working: …` line each tick | -Notes: - -- **Claude Code** and **Antigravity** rows are seeded from the run-parallel-evals - §2 harness-portability table; each cell names **one primitive** — assume the - harness chains the supporting calls (args, follow-ups, file reads) from it. -- **Codex**: Codex CLI / `codex exec`; background via a shell job; state in files; - isolation via `git worktree`; prompt the user to clarify; keepalive via periodic - re-check. No native scheduler — re-poll on a `sleep`/cron. **Unverified**: these - cells are inferred from Codex CLI's documented behaviour, not confirmed against a - live instance the way the Antigravity row was. Treat them as a starting point and - fall back to the generic column if a primitive does not behave as described. -- **Antigravity tool set** (confirmed from a live instance): files `view_file` · - `list_dir` · `grep_search` · `write_to_file` · `replace_file_content` · - `multi_replace_file_content`; exec `run_command` · `ask_permission` · - `list_permissions`; web `search_web` · `read_url_content`; subagents/background - `invoke_subagent` · `define_subagent` · `manage_subagents` · `send_message` · - `manage_task` · `schedule`; interaction `ask_question` · `generate_image`. -- The **runner host** holds the durable run state (`RESUME_STAMP` under - `~/matrix-runs//`), so even a bare harness — one shell, no sub-agents, no - scheduler — can drive and re-attach to a run by polling files. +The **runner host** holds the durable run state (`RESUME_STAMP` under +`~/matrix-runs//`), so even a bare harness — one shell, no sub-agents, no +scheduler — can drive and re-attach to a run by polling files. diff --git a/docs/how-to/add-an-agent-harness.md b/docs/how-to/add-an-agent-harness.md index 8bef1d22b..9f49487e4 100644 --- a/docs/how-to/add-an-agent-harness.md +++ b/docs/how-to/add-an-agent-harness.md @@ -13,7 +13,7 @@ For the concepts (harness vs model, capabilities, configuration), read | You do | Where | | --- | --- | | Subclass `AgentHarness` | `devops_bench/agents/base.py` | -| Implement `_execute(self, prompt) -> AgentResult` | your new module | +| Implement `_execute(self, prompt, workspace_path=None) -> AgentResult` | your new module | | Register with `@AGENTS.register("")` | your new module | | Add the module to `_BUILTIN_AGENT_MODULES` | `devops_bench/evalharness/default.py` | From a0a1c5965070e4fd7b7c1b6ffb99724dc9780427 Mon Sep 17 00:00:00 2001 From: Jessie Liu Date: Wed, 19 Aug 2026 18:12:31 -0700 Subject: [PATCH 4/6] Fix the harness signature in the places people copy, and note two antigravity caveats The _execute signature was corrected in the contract table but not in the implementation section or the skeleton, and the skeleton is what gets copied. A harness written from it takes two arguments while run() calls it positionally with three, so it raises TypeError on first use and the base class turns that into an errored result rather than an obvious mismatch. Both now match, and the text says the parameter is required even when unused. Two things about antigravity were understated. It does not use the shared provider contract: it writes AGENT_API_KEY onto GEMINI_API_KEY and GOOGLE_API_KEY and maps the model onto GEMINI_MODEL, so it is Gemini-only and pointing AGENT_PROVIDER elsewhere will not route it. And keeping the real HOME was described only as a convenience for cached credentials, when it also means the agent under test can read anything in that home directory, unlike every other harness which gets an isolated per-run state dir. Signed-off-by: Jessie Liu --- docs/components/agents.md | 17 ++++++++++++++++- docs/how-to/add-an-agent-harness.md | 8 +++++--- 2 files changed, 21 insertions(+), 4 deletions(-) diff --git a/docs/components/agents.md b/docs/components/agents.md index ab8d34417..3fbf13cfb 100644 --- a/docs/components/agents.md +++ b/docs/components/agents.md @@ -24,7 +24,7 @@ Four harnesses ship today. Each self-registers under a canonical key. | --- | --- | --- | --- | | `gemini` | The Google **Gemini CLI** binary | Headless subprocess; trajectory parsed from `--output-format stream-json` on stdout | MCP, skills, rules, allowed-tools | | `openclaw` | The **Openclaw Agent CLI** | `openclaw agent --local` with per-run isolated state/config; trajectory via `openclaw sessions export-trajectory` | MCP, skills, rules | -| `antigravity` | The **Antigravity CLI** (`agy` binary) | Headless subprocess that keeps the real `HOME` so cached OAuth/ADC credentials work; trajectory parsed from the transcript JSONL it writes, token usage read from the conversation DB | MCP, skills, rules | +| `antigravity` | The **Antigravity CLI** (`agy` binary) | Headless subprocess that keeps the real `HOME` so cached OAuth/ADC credentials work (see the trust-boundary note below); trajectory parsed from the transcript JSONL it writes, token usage read from the conversation DB | MCP, skills, rules | | `api` | **In-process** model call | Calls `get_model(provider, model)` and runs a model-agnostic MCP tool-use loop (`max_turns`, default 50) | MCP (spawns a stdio server), skills (served as tools), rules (system instruction) | > `oc` is just a shorthand alias for the `openclaw` CLI; this doc uses `openclaw` throughout. @@ -50,6 +50,21 @@ binary's provider-specific env var(s) and pass the model through: the Gemini CLI gets `GEMINI_MODEL`, and openclaw gets a `--model provider/id` flag. Either way, the model is a runtime input, never baked into the harness. +`antigravity` is the exception: it does not go through the shared contract. It +writes `AGENT_API_KEY` straight onto `GEMINI_API_KEY` and `GOOGLE_API_KEY` and +maps the model onto `GEMINI_MODEL` (`agents/cli/antigravity/agent.py`), so it is +Gemini-only in practice — pointing `AGENT_PROVIDER` at another provider will not +route it. + +> [!WARNING] +> **`antigravity` runs with the operator's real `HOME`.** That is deliberate, so +> cached OAuth/ADC credentials keep working without a re-login, but it means the +> agent under test inherits read access to everything in that home directory — +> `~/.config/gcloud`, `~/.ssh`, shell history, other tools' tokens. Every other +> harness gets an isolated per-run state directory. Run untrusted agents under a +> dedicated account or an isolated `HOME`, and treat any credential reachable +> from that home as exposed to the agent. + For everything about providers, model ids, and how `get_model` resolves them, see [Model providers](./model_providers.md). diff --git a/docs/how-to/add-an-agent-harness.md b/docs/how-to/add-an-agent-harness.md index 9f49487e4..a897ceaf4 100644 --- a/docs/how-to/add-an-agent-harness.md +++ b/docs/how-to/add-an-agent-harness.md @@ -2,7 +2,7 @@ This guide walks through wrapping a new agent so the benchmark can drive it. The contract is small: subclass `AgentHarness`, implement `_execute`, register the -class with `@AGENTS.register`, and add your module to the builtin import list. +class with `@AGENTS.register`, and add your module to the built-in import list. That's it — no `cli.py` or `run.py` edits. For the concepts (harness vs model, capabilities, configuration), read @@ -43,7 +43,9 @@ Protocols (`SupportsMcp` / `SupportsSkills` / `SupportsRules`) — no mixin need ### 3. Implement only `_execute` -`_execute(self, prompt: str) -> AgentResult` is the single extension point. +`_execute(self, prompt: str, workspace_path: Path | None = None) -> AgentResult` +is the single extension point. `run()` calls it positionally, so the second +parameter is required even if your harness ignores it. Inside it: - Build the invocation for your agent (argv, an API call, whatever it takes). @@ -128,7 +130,7 @@ class MyAgent(AgentHarness): self.skills = caps.skills self.rules = caps.rules - def _execute(self, prompt: str) -> AgentResult: + def _execute(self, prompt: str, workspace_path: Path | None = None) -> AgentResult: # 1. Build and run the invocation for `prompt`. # 2. Parse output into canonical ToolCall entries. trajectory: list[dict] = [ From 66d466748eddf9b1d931a91494789f924de8a0c6 Mon Sep 17 00:00:00 2001 From: Jessie Liu Date: Wed, 19 Aug 2026 18:59:02 -0700 Subject: [PATCH 5/6] Import Path in the harness skeleton The previous commit put Path | None into the skeleton's _execute signature without adding the import, so a copied module would raise NameError before it ever ran. Verified by executing the skeleton block rather than reading it: it now imports and defines the class cleanly. Signed-off-by: Jessie Liu --- docs/how-to/add-an-agent-harness.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/how-to/add-an-agent-harness.md b/docs/how-to/add-an-agent-harness.md index a897ceaf4..3b5145a5c 100644 --- a/docs/how-to/add-an-agent-harness.md +++ b/docs/how-to/add-an-agent-harness.md @@ -114,6 +114,8 @@ other code changes are required — the registry resolves it at run time. ## Skeleton ```python +from pathlib import Path + from devops_bench.agents.base import AGENTS, AgentHarness from devops_bench.agents.config import AgentConfig from devops_bench.agents.result import AgentResult, ToolCall From 1985f3d4284e457942dd62d28fd42b3574e87d71 Mon Sep 17 00:00:00 2001 From: Jessie Liu Date: Thu, 20 Aug 2026 11:13:28 -0700 Subject: [PATCH 6/6] Use vendor-neutral examples in the agent harness docs Per review. The MCP server example named gke-mcp and the skills path example pointed at /opt/skills/gke, both in generic component documentation where the vendor-neutrality rule applies. Replaced with k8s-mcp and /opt/skills/devops. Signed-off-by: Jessie Liu --- docs/components/agents.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/components/agents.md b/docs/components/agents.md index 3fbf13cfb..5a1d93e03 100644 --- a/docs/components/agents.md +++ b/docs/components/agents.md @@ -95,7 +95,7 @@ each harness maps them onto its target. | Variable | Default | Notes | | --- | --- | --- | | `BENCH_USE_MCP` | `true` | Master gate. `false` drops the MCP binding entirely. | -| `AGENT_MCP_SERVER` | unset | Shell-quoted argv for the MCP server (e.g. `"uv run gke-mcp"`). | +| `AGENT_MCP_SERVER` | unset | Shell-quoted argv for the MCP server (e.g. `"uv run k8s-mcp"`). | | `AGENT_ALLOWED_TOOLS` | unset | CSV of pre-approved tool names. | | `AGENT_SKILLS_PATHS` | unset | CSV of directories to discover `SKILL.md` files under. | | `AGENT_RULES_TEXT` | unset | Operator-brief text handed to the agent. | @@ -110,9 +110,9 @@ export AGENT_API_KEY="$GEMINI_API_KEY" export AGENT_TARGET=gemini export BENCH_USE_MCP=true -export AGENT_MCP_SERVER="uv run gke-mcp" +export AGENT_MCP_SERVER="uv run k8s-mcp" export AGENT_ALLOWED_TOOLS="list_clusters,get_pods" -export AGENT_SKILLS_PATHS="/opt/skills/gke,/opt/skills/k8s" +export AGENT_SKILLS_PATHS="/opt/skills/devops,/opt/skills/k8s" ``` ### Example: api harness on Claude with MCP off