Feat/mcp eval phases

abevalflow/mcp/README.md

@@ -0,0 +1,102 @@
+# `abevalflow/mcp/` — MCP evaluation pipeline logic
+
+All evaluation logic for the MCP-server evaluation pipeline (ADR Approach 4,
+Phases 1 and 2). This package **reads** result JSON, applies policy, and
+**decides** pass/fail. No shell-outs to external tools, no network calls.
+Testable with JSON fixtures.
+
+## Structure (organized by role, phase-prefixed)
+
+| Directory | Role | What it does |
+|-----------|------|-------------|
+| `gates/` | **Decide** pass/fail | Read result JSON, count findings, apply policy, produce `GateResult`. One gate per check type. |
+| `checks/` | **Define** check logic | The conformance check functions (what to probe, how to evaluate) and Compass fact/result mapping. Called by the probe runners in `scripts/mcp/probes/`. |
+
+### `gates/` — all pass/fail gates
+
+| File | Phase | What it does |
+|------|-------|-------------|
+| `phase1_secrets_gate.py` | 1 | `SecretsGate` — reads `secrets-scan.json`, counts HIGH findings |
+| `phase1_no_user_code_gate.py` | 1 | `NoUserCodeGate` — reads `no-user-code-scan.json`, counts HIGH findings |
+| `phase1_license_gate.py` | 1 | `LicenseGate` — reads `license-scan.json`, checks against allow-list |
+| `phase2_conformance_gate.py` | 2 | `Phase2Gate` — reads any `<check>-check.json`, maps three-state to `GateResult` |
+| `__init__.py` | 1+2 | Orchestrators: `run_phase1()`, `run_phase2()`, gate lists |
+
+### `checks/` — conformance probes and Compass-consumed checks
+
+| File | Phase | What it does |
+|------|-------|-------------|
+| `phase2_conformance_checks.py` | 2 | 8 `check_*` functions + `probe_all()` — conformance checks against a running MCP server |
+| `phase2_compass_checks.py` | 2 | `map_facts()` + `map_results()` — maps Compass facts/results to `CheckOutcome` |
+
+### Root modules
+
+| File | What it does |
+|------|-------------|
+| `types.py` | Core types: `CheckOutcome`, `STATUS_PASS/FAIL/NOT_EVALUATED` |
+| `helpers.py` | `make_finding()`, `normalize_severity()`, `write_check_result()` |
+| `mcp_client.py` | `MCPClient` — JSON-RPC 2.0 over Streamable HTTP, purpose-built for conformance probing |
+
+## Data flow
+
+```
+Phase 1 (static — no running server)
+┌─────────────────────────────────────────────────────────────────┐
+│ scripts/mcp/scanners/                                           │
+│   phase1_secrets_scan       ──→ secrets-scan.json                │
+│   phase1_no_user_code_scan  ──→ no-user-code-scan.json           │
+│   phase1_license_scan       ──→ license-scan.json                │
+└──────────────────────────────────┬──────────────────────────────┘
+                                   │ JSON on disk
+┌──────────────────────────────────▼──────────────────────────────┐
+│ abevalflow/mcp/gates/                                           │
+│   Phase1SecretsGate     ──→ GateResult (pass/fail)               │
+│   Phase1NoUserCodeGate  ──→ GateResult (pass/fail)               │
+│   Phase1LicenseGate     ──→ GateResult (pass/fail)               │
+└─────────────────────────────────────────────────────────────────┘
+
+Phase 2 (live — against the running MCP server)
+┌─────────────────────────────────────────────────────────────────┐
+│ scripts/mcp/probes/                                             │
+│   phase2_conformance_probe   ──→ <check>-check.json (per check)       │
+│   phase2_compass_probe ──→ <check>-check.json (consumed facts)  │
+└──────────────────────────────────┬──────────────────────────────┘
+                                   │ JSON on disk
+┌──────────────────────────────────▼──────────────────────────────┐
+│ abevalflow/mcp/gates/                                           │
+│   Phase2ConformanceGate (×N)  ──→ GateResult (pass/fail/not_evaluated) │
+└─────────────────────────────────────────────────────────────────┘
+```
+
+## Relationship to `scripts/mcp/`
+
+This package contains the **what** (evaluation logic — reads JSON, decides).
+`scripts/mcp/` contains the **how** (CLI entry points — runs tools, writes JSON).
+The contract between them is JSON files on disk.
+
+## Three-state model (Phase 2)
+
+Every Phase 2 check produces one of three states:
+
+| State | Meaning | Effect on verdict | Effect on grade |
+|---|---|---|---|
+| `pass` | Server is conformant | Does not block | Score 1.0 |
+| `fail` | Server violates requirement | **Blocks** (in block mode) | Score weighted by severity |
+| `not_evaluated` | Check could not produce a signal | Does not block | **Excluded** from grade |
+
+A server is never penalized for something it was never tested for.
+
+## Adding a new check
+
+1. **Add the check function** in `checks/phase2_conformance_checks.py` — takes the
+   client and/or a response, returns a `CheckOutcome`.
+2. **Wire it** in `probe_all()` (same file).
+3. **Add the check name** to `PHASE2_CHECKS` in `gates/__init__.py`.
+4. The gate picks it up automatically — it reads any `<check>-check.json` file.
+   No gate code changes needed.
+
+## Testing
+
+All code in this package is testable with JSON fixtures — no running server,
+no external tools. Tests are in `tests/test_mcp_phase1.py` and
+`tests/test_mcp_phase2.py`.

abevalflow/mcp/__init__.py

@@ -0,0 +1,21 @@
+"""MCP server evaluation pipeline — evaluation logic.
+
+All evaluation logic for the MCP-server evaluation pipeline lives here: gates,
+check functions, data types, the MCP protocol client, and Compass fact mapping.
+Isolated from the skill evaluation pipeline (see ADR: Evaluation Strategy for
+MCP Servers as Standalone Software Components, Approach 4).
+
+Layout (organized by role, phase-prefixed):
+
+- ``types``       - core types: ``CheckOutcome``, ``STATUS_*``
+- ``helpers``     - finding builders, severity normalization, check-result I/O
+- ``mcp_client``  - MCP protocol client (JSON-RPC over Streamable HTTP)
+- ``gates/``      - all pass/fail gates (Phase 1 + Phase 2)
+- ``checks/``     - conformance check functions + Compass mapping (Phase 2)
+
+The CLI entry points that invoke this logic and handle I/O (shell out to
+scanners, create an MCP client, read/write files) live in ``scripts/mcp/``.
+
+Phases 1 and 2 are implemented here; Phase 3 (behavioral, mcpchecker) is out of
+scope for now.
+"""

abevalflow/mcp/checks/__init__.py

@@ -0,0 +1 @@
+"""MCP evaluation checks — conformance probes and Compass-consumed checks."""

abevalflow/mcp/checks/phase2_compass_checks.py

@@ -0,0 +1,224 @@
+"""Phase 2 Compass-consumed check mapping.
+
+A few Phase 2 checks are already collected by Compass as real automated facts, so
+the pipeline consumes them rather than re-probing (avoids duplicating what
+Compass does well). This module maps those Compass SoundCheck facts into the same
+three-state ``CheckOutcome`` the Phase 2 gates read.
+
+Consumed checks and their source facts::
+
+    tool-name-rules     <- mcp:default/tools $.allToolNamesValid
+    oauth-catalog-match <- mcp:default/security $.oauth.enforced
+                                                $.entityAuth.authServerMatch
+                                                $.scopeMatch.allScopesMatch
+
+Only these two are consumed: the Compass tier-1 *automated* facts worth surfacing
+in our aggregate. Metadata Compass owns directly (schema-present, documentation,
+data-source approval) is not re-gated here, per the ADR ("consumed as existing
+Compass Check Results").
+
+Two consumption modes:
+
+- **File mode** (``map_facts``): reads raw Compass facts from a local JSON file
+  and maps subfields to check outcomes. Used for offline/demo runs.
+- **Live mode** (``map_results``): maps SoundCheck API check-result states
+  (passed/failed) directly to check outcomes. Used when the pipeline fetches
+  results from ``GET /api/soundcheck/results``.
+
+The CLI entry point that reads facts from a file or fetches live results from
+the SoundCheck API lives in ``scripts/mcp/probes/phase2_compass_probe.py``.
+"""
+
+from __future__ import annotations
+
+from typing import Any
+
+from abevalflow.mcp.helpers import make_finding
+from abevalflow.mcp.types import (
+    STATUS_FAIL,
+    STATUS_NOT_EVALUATED,
+    STATUS_PASS,
+    CheckOutcome,
+)
+
+FACT_TOOLS = "mcp:default/tools"
+FACT_SECURITY = "mcp:default/security"
+
+# Compass SoundCheck check IDs on the mcp-scorecard track (tier-1 auto-collected).
+CHECK_TOOL_NAME = "mcp-tool-schema-check"
+CHECK_OAUTH = "mcp-oauth-scope-match"
+
+# Maps a SoundCheck check ID to the Phase 2 check name our gates read.
+_CHECK_ID_TO_PHASE2 = {
+    CHECK_TOOL_NAME: "tool-name-rules",
+    CHECK_OAUTH: "oauth-catalog-match",
+}
+
+
+def _get(facts: dict[str, Any], fact_ref: str, *path: str) -> Any:
+    """Walk ``facts[fact_ref]`` down ``path``; return None if any hop is missing."""
+    node: Any = facts.get(fact_ref)
+    for key in path:
+        if not isinstance(node, dict):
+            return None
+        node = node.get(key)
+    return node
+
+
+def _bool_outcome(check: str, value: Any, ok_reason: str, bad_reason: str) -> CheckOutcome:
+    """pass/fail from a boolean fact; not_evaluated when the fact is absent."""
+    if value is None:
+        return CheckOutcome(check, STATUS_NOT_EVALUATED, "Source Compass fact not present.", source="compass")
+    if value is True:
+        return CheckOutcome(check, STATUS_PASS, ok_reason, source="compass")
+    finding = make_finding(severity="high", message=bad_reason, rule_id=check)
+    return CheckOutcome(check, STATUS_FAIL, bad_reason, source="compass", findings=[finding])
+
+
+def map_facts(facts: dict[str, Any]) -> list[CheckOutcome]:
+    """Map Compass facts to the consumed Phase 2 check outcomes."""
+    outcomes: list[CheckOutcome] = []
+
+    outcomes.append(
+        _bool_outcome(
+            "tool-name-rules",
+            _get(facts, FACT_TOOLS, "allToolNamesValid"),
+            "Compass reports all tool names valid.",
+            "Compass reports one or more tool names invalid.",
+        )
+    )
+
+    # OAuth matches catalog = enforced AND auth-server matches AND scopes match.
+    # Distinguish an explicit False (a real violation) from an absent fact (None):
+    # a missing subfield must NOT be read as a violation, or a partially-reported
+    # entity would falsely FAIL and block the phase. Only when every subfield is
+    # present and True can we assert the conjunction; any absent subfield with no
+    # explicit False leaves us unable to evaluate.
+    oauth_fields = [
+        ("oauth-enforced", _get(facts, FACT_SECURITY, "oauth", "enforced"), "OAuth not enforced per Compass"),
+        (
+            "oauth-server-match",
+            _get(facts, FACT_SECURITY, "entityAuth", "authServerMatch"),
+            "OAuth auth server does not match catalog per Compass",
+        ),
+        (
+            "oauth-scope-match",
+            _get(facts, FACT_SECURITY, "scopeMatch", "allScopesMatch"),
+            "OAuth scopes do not match catalog per Compass",
+        ),
+    ]
+    violations = [
+        make_finding(severity="high", message=msg, rule_id=rule) for rule, value, msg in oauth_fields if value is False
+    ]
+    missing = [rule for rule, value, _ in oauth_fields if value is None]
+    if violations:
+        outcomes.append(
+            CheckOutcome(
+                "oauth-catalog-match",
+                STATUS_FAIL,
+                "OAuth configuration does not match the catalog.",
+                source="compass",
+                findings=violations,
+            )
+        )
+    elif not missing:
+        outcomes.append(
+            CheckOutcome(
+                "oauth-catalog-match", STATUS_PASS, "OAuth enforced and matches the catalog.", source="compass"
+            )
+        )
+    elif len(missing) == len(oauth_fields):
+        outcomes.append(
+            CheckOutcome(
+                "oauth-catalog-match", STATUS_NOT_EVALUATED, "No Compass OAuth facts present.", source="compass"
+            )
+        )
+    else:
+        outcomes.append(
+            CheckOutcome(
+                "oauth-catalog-match",
+                STATUS_NOT_EVALUATED,
+                f"Incomplete Compass OAuth facts (missing: {', '.join(missing)}); cannot assert OAuth-matches-catalog.",
+                source="compass",
+            )
+        )
+
+    return outcomes
+
+
+# ---------------------------------------------------------------------------
+# Live mode: map SoundCheck API check results
+# ---------------------------------------------------------------------------
+
+# TODO: fallback opportunity — for tool-name-rules, we already have the tool
+# list from tools/list and the regex is ^[A-Za-z0-9_\-.]{1,128}$. If Compass
+# returns no result (new server not yet onboarded), we could validate names
+# directly instead of reporting not_evaluated. OAuth cannot fall back (needs
+# catalog + PRM data we don't have). Not implemented: ADR says "reuse Compass
+# result", so consume-only for now.
+
+
+def map_results(results: list[dict]) -> list[CheckOutcome]:
+    """Map SoundCheck API check results to Phase 2 check outcomes.
+
+    ``results`` is the ``results`` array from
+    ``GET /api/soundcheck/results?entityRef=...&checks=...``.  Each element has
+    at least ``checkId`` and ``state`` (passed/failed/error/not-applicable).
+
+    Any consumed check not present in the results list is reported
+    ``not_evaluated`` (the entity may not be registered in Compass yet).
+    """
+    by_check_id = {r["checkId"]: r for r in results if isinstance(r, dict) and "checkId" in r}
+    outcomes: list[CheckOutcome] = []
+
+    for check_id, phase2_name in _CHECK_ID_TO_PHASE2.items():
+        result = by_check_id.get(check_id)
+        if result is None:
+            outcomes.append(
+                CheckOutcome(
+                    phase2_name,
+                    STATUS_NOT_EVALUATED,
+                    f"No Compass result for {check_id}; entity may not be registered yet.",
+                    source="compass",
+                )
+            )
+            continue
+
+        state = (result.get("state") or "").lower()
+        notes = ""
+        details = result.get("details")
+        if isinstance(details, dict):
+            notes_obj = details.get("notes")
+            if isinstance(notes_obj, dict):
+                notes = notes_obj.get("data", "")
+            elif isinstance(notes_obj, str):
+                notes = notes_obj
+
+        if state == "passed":
+            outcomes.append(
+                CheckOutcome(phase2_name, STATUS_PASS, notes or f"Compass check {check_id} passed.", source="compass")
+            )
+        elif state == "failed":
+            outcomes.append(
+                CheckOutcome(
+                    phase2_name,
+                    STATUS_FAIL,
+                    notes or f"Compass check {check_id} failed.",
+                    source="compass",
+                    findings=[
+                        make_finding(severity="high", message=notes or f"{check_id} failed", rule_id=phase2_name)
+                    ],
+                )
+            )
+        else:
+            # "error", "not-applicable", or unknown state
+            outcomes.append(
+                CheckOutcome(
+                    phase2_name,
+                    STATUS_NOT_EVALUATED,
+                    f"Compass check {check_id} returned state '{state}': {notes}".strip(),
+                    source="compass",
+                )
+            )
+
+    return outcomes