feat(cli): add the CLI entrypoint and benchmark runner (#41) by devops-bench-sync-bot · Pull Request #232 · gke-labs/devops-bench

devops-bench-sync-bot · 2026-07-29T05:23:56Z

feat(cli): add the CLI entrypoint and benchmark runner (#41)

* Add the harness base and scenario manager Signed-off-by: Jessie Liu <jssl@google.com> * Add the default eval harness and package surface Add devops_bench/evalharness/default.py (DefaultEvalHarness: the task loop that provisions, runs the agent, optionally injects chaos and verifies, scores, and tears down) and the evalharness package __init__ exposing the public surface. Includes unit tests for the harness, registry resolution, single env reads, and package-import laziness. Signed-off-by: Jessie Liu <jssl@google.com> --------- Signed-off-by: Jessie Liu <jssl@google.com> GitOrigin-RevId: a945f6e

``DefaultEvalHarness._RECORD_KEYS`` was never read at runtime. Both record builders construct their key set from the ``_empty_record`` literal, so the frozenset was a second, hand-maintained copy of the schema that could silently drift from the records it claimed to describe. Drop the constant and the two docstring references to it. The symmetric-shape invariant stays pinned by the golden tests in ``test_default_harness.py``, which assert both builders emit exactly the expected key set. Signed-off-by: Jessie Liu <jssl@google.com> GitOrigin-RevId: 9cff722

GitOrigin-RevId: bef4468

* feat(agents): add canonical token buckets shared across harnesses Add one canonical six-bucket token schema — input (non-cached), cached, cache_write, reasoning, output (excludes reasoning), total — with None for unreported buckets rather than a fabricated 0, that harnesses map onto, and carry the buckets through to the result row. - agents/result.py: shared TOKEN_BUCKETS + empty_tokens(). - agents/cli/gemini_cli: the terminal result.stats block maps to the canonical dict (input = full input − cached; reasoning derived from the total gap). - results/normalize.py + row.py: rows carry cachedTokens, reasoningTokens, cacheWriteTokens, and totalTokens as additive nullable fields (no SCHEMA_VERSION bump; historical rows stay valid); normalize_tokens reads the canonical keys first with legacy aliases kept for older records. Signed-off-by: Eugene Ng <ngeugene@google.com> * chore(agents): add type hints flagged in review Annotate the exported __all__/TOKEN_BUCKETS constants, fully parameterize _canonical_tokens, and add -> None to the new normalize/build_rows tests. * chore(agents): address review feedback on token buckets - empty_tokens() now returns dict[str, int | None] rather than dict[str, Any]. - normalize_tokens() returns a NormalizedTokens NamedTuple; build_rows reads its named fields (still unpacks/compares as a 6-tuple, so callers are unaffected). - Clamp canonical gemini "input" at 0 so an over-reported "cached" can't push it negative, with coverage. --------- Signed-off-by: Eugene Ng <ngeugene@google.com> GitOrigin-RevId: 6d78692

* feat(agents): add entry-point discovery to the AGENTS registry Give AGENTS the same entry_point_group treatment as the FAULTS, TRIGGERS, METRICS, PROVIDERS, and VERIFIERS registries so an external package can ship an AgentHarness and have it resolve by key with no edit to this tree and no eager import of its module: [project.entry-points."devops_bench.agents"] myagent = "my_pkg.harness:MyHarness" This unblocks downstream repos that pip-install devops-bench as a library and implement their agent harness in their own tree. Resolution stays lazy end-to-end (resolve_agent -> AGENTS.get -> one-time scan), so the group is only consulted on a registry miss. * fix(registry): enforce the lowercase key contract for agents The AGENTS entry-point group documented a lowercase-only key contract but nothing enforced it. `Registry._ensure_entry_points_loaded()` stored `entry_point.name` verbatim, so a downstream package declaring `Dummy-External = "..."` registered a key the harness could never reach — it lowercases the configured agent type before lookup — and the dead key then showed up verbatim in the `available:` list of the NotRegisteredError the user got back. Give `Registry` an optional `key_validator` policy: it returns None to accept a key or a reason to reject it. An explicit `register()` with a rejected key raises the new `InvalidKeyError` (an in-tree mistake should fail loudly); a discovered entry point with a rejected key is skipped with a warning, matching how a failing `entry_point.load()` is already handled so one mispackaged plugin never aborts a run. AGENTS supplies the lowercase policy; every other axis keeps today's accept-anything behavior. * test(core): annotate the new registry key-policy tests Per the tests/**/*.py review guideline, the tests added in this PR now carry type annotations: the sample key policy takes str and returns str | None, and the four key-validator tests declare -> None with typed mocker/caplog fixtures. Scoped to this PR's new code; pre-existing tests in these files are left alone. GitOrigin-RevId: 4d31ee9

…#49) The tofu deployer resolved relative stack names against Path(__file__).parents[2]/"tf", which only exists in a source checkout: under a pip install that lands in site-packages and tf/ is not packaged, so every relative stack: in task.yaml fails. Absolute paths work but skip per-run isolation, so concurrent runs sharing one stack dir contend on .terraform.lock.hcl. Resolve the stack root lazily instead: $BENCH_TF_ROOT when set (blank treated as unset, via the core get_env helper), else the checkout's <repo_root>/tf as before. The resolved root is also threaded into _isolated_work_dir, so stacks under an overridden root keep per-run isolation — installed-library users get the same parallel-run safety as an in-checkout run. The override root is copied whole per isolated run and must not be an ancestor of the run scratch dir; both constraints are documented on _resolve_tf_root. GitOrigin-RevId: cdf2c20

…judge (#47) * feat(verification): add the scored entry schema and combinators VerificationEntry carries the scoring vocabulary alongside the check: role, severity, weight, and a mode that defaults to converge for objectives and assert for safeguards. parse_entries collects authoring failures instead of raising, so one bad entry does not sink a task. Adds the all, any, and none combinators beside the existing sequence and parallel. any is what makes a disjunction expressible at all: correctness is a weighted fraction of passing objectives, and a weighted sum cannot represent "either of these satisfies the requirement". mode: hold parses and is then explicitly rejected, so a task that asks for it gets a clear error rather than silent reinterpretation. Review hardening: the verifier base and every compound spec now reject unknown keys (extra=forbid), and combinator groups require at least one child. * feat(verification): add the score rollup Turns a raw verification report into up to three signals: correctness as the weighted fraction of passing objectives, recoverable safety from recoverable safeguards, and a catastrophic boolean. A signal with no declared entries returns None rather than 0.0. The distinction is load-bearing: zero means the task was measured and failed, None means it was never asked, and collapsing the two would make every task that declares no safeguards look maximally unsafe. Review updates: error-status entries are excluded from numerators and denominators and tallied separately, parse errors fail closed into the objective denominator, and the catastrophic signal is the 0 or 1 gate value consistent with the outcome scoring convention. * feat(verification): add the resource_property verifier Asserts JSONPath properties over kubectl output. Parsing goes through jsonpath_ng.ext, not the plain module: the plain parser accepts filter syntax like [?(@.name=="web")] and then silently matches nothing, which would turn a real assertion into a vacuous pass. Selection and path matching flatten into one set of (object, value) pairs reduced by a single `across_matches: every | none`, which has no default. When that set holds more than one member and no reduction was declared, the check fails and names what it found rather than guessing. The flat model is deliberate. Reducing across objects and across values separately makes a multi-value path collapse before the outer reduction sees it, so three containers with one violator under `none` would report "no object satisfied it" and pass while the violation stood. `across_matches` is named off the combinator vocabulary on purpose: it ranges over matches discovered at runtime, where the all/any/none combinators range over checks declared in the task. An empty object set fails closed, which keeps "nothing matched" distinct from "matched, but the path resolved to nothing". The latter satisfies `none` and fails `every`, so an unobservable predicate never reads as a satisfied one. Carries the review-driven tri-state result status (pass, fail, error) on the shared verifier base and error classification in the pod-health and scaling leaves, plus narrowed eq and ne coercion so plain version strings are never compared numerically. * feat(verification): add run_entry with converge and assert modes converge polls a check to its deadline, for objectives that describe a state the cluster should reach. assert evaluates once, for safeguards that describe a state that must never hold, where polling would only delay the same answer. An assert-mode entry runs with a zero deadline, so combinator children wait without a timeout rather than being marked deadline-reached before they have run. Reworked in review: converge-mode any and none combinators poll in rounds of single-shot child evaluations so no child can starve or poison its siblings' shared deadline; single-shot leaf I/O is floored and the parallel wait is bounded. * feat(evalharness): evaluate verification entries after every run Replaces the chaos-only mapping builder with parse_entries and adds an unconditional post-agent pass that writes raw per-entry results to verification_report on the result record. Verification previously ran only when a chaos_spec referenced it, so no task could score on it. One entry that raises is recorded as a failure and the rest still run, matching how the metrics pipeline isolates a failing evaluator. Parse failures are warn-logged. A dropped entry shrinks the objective denominator and inflates every surviving objective's share, and a key in results.json is not enough evidence for a scoring change that silent. Review updates: a total verification budget bounds the pass, verification also runs on the exception path when infrastructure came up, no_infra skips cluster calls entirely, every record carries verification_status, and the chaos-path verification routes through run_entry so entry modes govern mid-run checks too. * feat(metrics): emit deterministic verification scores Reads verification_report, rolls it up, and emits VerificationCorrectness, VerificationRecoverableSafety, and VerificationCatastrophic beside the judge's scores. outcome_score and scoring.py are untouched. A registered metric rather than a direct write because pipeline.py assigns res['scores'] wholesale after every evaluator has run, and would clobber anything the harness put there itself. VerificationCatastrophic is a boolean, not a float. It names the violation rather than the gate, and feeds the catastrophic parameter on compute_outcome_score_v1 directly. Review updates: parse errors make the metric applicable and score fail-closed, coverage is emitted alongside the three signals, and the catastrophic score is the 0 or 1 gate float. * feat(tasks): add the deploy-hello-app GCP task Ports the task's checklist onto the deterministic verification schema. Every gradable outcome becomes a verification_spec entry, and expected_output keeps only the subjective residue for the judge. The nine hardening bullets split into eight weighted entries so a solution that misses one field loses that field's weight instead of all of them, and so the failure report names the field. Two safeguards are new: a recoverable check that nothing was dumped into the default namespace, and a catastrophic blast-radius trip-wire on kube-system. serving-http is declared but not yet runnable. external_http_probe is not a verifier type in this MVP, so the entry lands in verification_parse_errors. The comment on it says so, and says what it costs the objective total. * feat(tasks): port the opa-remediation task and its kind stack Brings the task and tf/prebuilt/opa-remediation onto the deterministic verification schema. The stack provisions a local kind cluster with Kyverno and three team namespaces, so the task runs without a cloud project. Three schema fixes came with the port. The top-level key was verification_entries, which the Task model drops because its field is verification_spec and extra is ignored, so every check was being discarded silently. Eleven resource_property checks selected their object with `name`, which is BaseVerifier's own label field, so they matched every object of that kind in the namespace rather than the one named; they now use resource_name. Three checks used the removed quantifier field and now use across_matches. repo-remediated is commented out. Its checks need git_repo_sync, which is not a registered verifier on this branch. The comment says so, and says what re-enabling it costs. GitOrigin-RevId: 0022cd6

* Add the safety checklists metric Scores per-task "must-not-do" constraints alongside correctness. Recoverable constraints are judged individually and aggregated into rec_v, rescaled onto [0.1, 1.0]; catastrophic tripwires form a binary cat_v gate that zeroes the outcome when one fires. A tripwire fires only below a dedicated lower threshold than the general pass bar, and a judge error is treated as "not fired" and dropped from the recoverable denominator, so an uncertain or failing judge cannot false-positive a run to zero. Signed-off-by: Jessie Liu <jssl@google.com> * Register the safety metric and add its task fields Adds the recoverable_safety and catastrophic task fields, threads them through the harness record, and registers the safety evaluator in the metrics pipeline so the sub-scores are emitted alongside the existing correctness metrics. Signed-off-by: Jessie Liu <jssl@google.com> * Resolve task placeholders in the safety checklists The judge reads the checklist strings verbatim, so an unresolved {{TARGET_DEPLOYMENT_NAME}} or {{NAMESPACE}} would be graded as literal text and never match what the agent did. Substitute them alongside expected_output, and keep the raw task values when a caller supplies nothing. Signed-off-by: Jessie Liu <jssl@google.com> * Carry resolved safety checklists into failed records The checklists were substituted after the agent ran, and _build_failed_record had no way to receive them, so a run that died during execution persisted raw {{...}} placeholders where a successful run recorded resolved strings. Resolve them right after the prompt, before any failure-prone work, and thread them through _build_failed_record with the same fallback-to-raw behavior the prompt and expected_output already use. Adds failure-path tests for both the resolved and the fallback case. Signed-off-by: Jessie Liu <jssl@google.com> * Disclose unjudged catastrophic tripwires and cover safety in the pipeline A tripwire whose judge call raised was silently treated as "not fired", and the reason string counted it against the total, so a run with an errored tripwire read as fully clean. Track how many were actually judged and mirror the "(N unevaluated)" disclosure the recoverable branch already emits. Adds two batch-level tests for the safety metric: both sub-scores land on the record through evaluate_metrics_batch, and a task declaring neither checklist scores nothing and calls no judge. Signed-off-by: Jessie Liu <jssl@google.com> * Type the GEval stub in the safety pipeline test Signed-off-by: Jessie Liu <jssl@google.com> * Scope the safety metric to judged recoverable safeguards A safeguard's severity decides how it may be evaluated. Catastrophic ones hard gate the outcome to zero, so they must be a fully deterministic check tree and have no judged form; they are declared in a task's verification_spec instead. Recoverable ones may be judged or deterministic. Drop the judged catastrophic path accordingly: the catastrophic task field, _score_catastrophic, its fire threshold, and the harness plumbing that carried the checklist onto the record. No task declared the field, so nothing migrates. Rename the remaining key to JudgedRecoverable, naming its source and severity to sit alongside the deterministic VerificationRecoverable, and emit the raw pass fraction rather than the rescaled value. The [0.1, 1.0] rescale moves to the scoring layer so the judged and deterministic signals share one scale and the floor lives in one place. Signed-off-by: Jessie Liu <jssl@google.com> --------- Signed-off-by: Jessie Liu <jssl@google.com> GitOrigin-RevId: c33b4fa

* Carry the composite outcome score on the result row Widens the flat dashboard row with the correctness and safety sub-scores and the scoring-framework version behind the composite outcome score, so a leaderboard consumer can show the headline number and the components it was built from. Signed-off-by: Jessie Liu <jssl@google.com> * Assemble the composite outcome score after scoring Rolls the per-metric sub-scores into the versioned composite once every metric has run, so it can read the checklist and safety outputs. Correctness falls back to OutcomeValidity when a task defines no checklist, and a record with no correctness signal gets no composite, leaving the row's outcomeScore null. Signed-off-by: Jessie Liu <jssl@google.com> * Guard composite assembly so one record cannot abort the batch The scoring formula raises on an out-of-range sub-score, and assembly runs inside the per-record loop but outside the per-metric guard, so a single malformed score would abandon every remaining record. Wrap it like the metric loop: the offending record loses only its composite. Signed-off-by: Jessie Liu <jssl@google.com> * Share the score-key names and cover the real composite failure path The four score keys the composite reads were declared twice, once privately in metrics/pipeline.py and once publicly in results/normalize.py, with the same literals in both. Move the values to core/score_keys.py and have both layers read from there. core is the shared foundation, so this removes the duplication without creating a metrics to results import edge, and normalize keeps its historical public names as aliases. Also adds two tests for the composite's real failure path. The existing guard test stubs the raise, so it never showed that compute_outcome_score_v1 is what rejects a malformed sub-score; these drive an out-of-range correctness value through the actual formula, once directly and once through the batch. Signed-off-by: Jessie Liu <jssl@google.com> * Read the deterministic verification signals into the composite Adopts the score key convention settled in the deterministic verification PR: VerificationCorrectness, VerificationRecoverable, VerificationCatastrophic, severity as the name. Those four join the judged keys in core/score_keys.py so every layer reads one definition. Each signal is now taken from the first key present in a preference chain, with the deterministic score winning over the judged equivalent for the same quantity: a task that expresses a check in its verification_spec has said what it means exactly, so a judge reading prose should not override it. No task declares both today. Both recoverable producers emit a raw pass fraction, so the [0.1, 1.0] rescale is applied here rather than in either emitter. That keeps the floor in one place and stops a raw 0.0 from zeroing the outcome on a violation that is only recoverable. The row keeps the raw fraction, since results/normalize maps and never scores. Signed-off-by: Jessie Liu <jssl@google.com> * Annotate the changed normalize test functions Signed-off-by: Jessie Liu <jssl@google.com> * Fold the verification score keys into the shared module Now that the deterministic verification metric has landed, its four key names were declared both there and in core/score_keys.py. Point the metric at the shared module and keep its own names as re-exports, so every score key has one definition across the emitters, the composite assembly, and results.normalize. Signed-off-by: Jessie Liu <jssl@google.com> * Read the catastrophic gate before rescaling recoverable safety compute_outcome_score_v1 short-circuits a catastrophic run before validating its other inputs, so a malformed sub-score still yields 0.0. Rescaling the recoverable fraction first defeated that: the rescale validates, so a record with both a catastrophic violation and an out-of-range recoverable value raised and lost its whole composite, including the catastrophic signal. Read the gate first and skip the rescale once it has fired. A malformed value on a non-catastrophic record still raises, which the batch guard handles. Also refreshes the row docstrings, which still described correctness as the checklist score and the recoverable field as rescaled, and annotates the remaining changed test. Signed-off-by: Jessie Liu <jssl@google.com> * Fold the judged recoverable key into the shared module The safety metric landed with its own literal for JudgedRecoverable, which is now also declared in core/score_keys.py. Point the metric at the shared module and keep its name as a re-export, so every score key the composite reads has a single definition. Signed-off-by: Jessie Liu <jssl@google.com> --------- Signed-off-by: Jessie Liu <jssl@google.com> GitOrigin-RevId: 1f62475

devops-bench-sync-bot force-pushed the backsync/from-upstream branch 7 times, most recently from f3f6c47 to 4caaa5c Compare August 5, 2026 05:22

devops-bench-sync-bot force-pushed the backsync/from-upstream branch 8 times, most recently from 51cfb6f to be200ca Compare August 13, 2026 04:53

devops-bench-sync-bot force-pushed the backsync/from-upstream branch from 3253cdc to 7c8b73d Compare August 14, 2026 04:53

jessie1111101 and others added 8 commits August 14, 2026 04:53

feat(cli): add the CLI entrypoint and benchmark runner (#41)

8b150e3

GitOrigin-RevId: bef4468

Navigation Menu

Uh oh!

feat(cli): add the CLI entrypoint and benchmark runner (#41) - #232

feat(cli): add the CLI entrypoint and benchmark runner (#41)#232
devops-bench-sync-bot wants to merge 9 commits into
mainfrom
backsync/from-upstream

devops-bench-sync-bot commented Jul 29, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Uh oh!

Conversation

devops-bench-sync-bot commented Jul 29, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants