feat(tier3): support Claude Code on Agent Platform and add active preflight probe

[kweinmeister]

Summary

Adds support for routing Claude Code through Google Cloud Agent Platform (Vertex AI) endpoints, hardens Harbor GKE runtime execution boundaries, and introduces an active 1-token preflight probe for Agent Platform OpenAPI endpoints.

Why this change is needed:

• When evaluating skills with Claude Code via Harbor, users need to route requests to Google Cloud Vertex AI Anthropic endpoints (CLAUDE_CODE_USE_VERTEX=1) and authenticate via Application Default Credentials (ADC).
• Harbor GKE execution mode (--env-mode gke) requires enforcing security boundaries to reject cluster infrastructure settings in skill configs in favor of host CLI flags/environment variables, while ensuring credentials passed in --ek keys/values are redacted from process listings without dropping rate limits or token configs.
• In unprivileged GKE mode (SKILLEVALUATOR_GKE_ALLOW_WORKLOAD_IDENTITY=0), enforces defense-in-depth metadata isolation via a pod-scoped egress NetworkPolicy (preserving UDP/TCP port 53 DNS), a pre-start skillevaluator-metadata-probe-init init container, and a runtime skillevaluator-metadata-probe companion container, with automatic GKE Autopilot 10Gi ephemeral-storage budgeting and fast failure on pod preemption.
• Endpoint validation previously lacked active connectivity checks for Vertex AI OpenAPI endpoints, leading to delayed failures during evaluation runs. The new active 1-token chat/completions probe verifies credentials, permissions, and model availability upfront and fails fast on 401/403/404.

Verification

• I am familiar with the Contributing Guidelines
• Added or updated focused tests
• Updated documentation for user-visible changes
• Ran make lint
• Ran make test
• Ran make build
• Did not add credentials, private datasets, or proprietary benchmark content

Release Impact

• No user-visible release note needed
• Updated CHANGELOG.md

[rng1995]

@kweinmeister - Please address the final 2 review comments: #140 (comment) and #140 (comment)

[kweinmeister]

@rng1995 Both comments have been addressed and replied to, and I ran a live GKE evaluation against the updated branch to verify everything end-to-end. Ready for another look whenever you have a chance!

[rng1995]

@kweinmeister Thank you for your patience and your continued work on this PR, and for the live GKE validation.

I re-reviewed 2436c36:

• MCP placeholder stripping: fixed. With SKILLEVALUATOR_ALLOWED_MCP_HOSTS and SKILLEVALUATOR_ALLOWED_MCP_SECRETS set, Bearer ${DEDICATED_MCP_TOKEN} and Bearer $DEDICATED_MCP_TOKEN are accepted through source validation, _load_mcp_servers, validate_skillevaluators and Claude registration. Operator-owned references, literal tokens and JWTs, unclosed ${VAR, ${VAR:-literal}, $${...} and unapproved variables are rejected. I've resolved that thread.
• GKE metadata isolation: one gap remains. The isolation check runs inside the skill-built image, so a skill can spoof it on clusters that don't enforce NetworkPolicy. Details are in the existing thread.
• New question: whether the metadata-blocking policy also blocks DNS on Dataplane V2 / Autopilot. Details are inline.

I also pushed two maintainer commits to keep the branch mergeable:

• 1a138ba adds an oauthlib>=4.0.0 floor. The Dependency review check was failing because harbor[gke] pulls in kubernetes → requests-oauthlib → oauthlib 3.3.1 (GHSA-hj66-6f7g-4r5v and GHSA-xpv3-w29h-x7cv, both fixed only in 4.0.0).
• c9ab383 resolves the conflicts with #145. The ADC 401 token refresh in LLMClient and the verifier template now wraps #145's retry and schema-fallback calls, and your ADC refresh tests pass through the new path.

Validation: Ruff is clean, the full suite passes locally (9,972 passed, 18 skipped), and all 17 CI checks passed on c9ab383. CI on the latest update from main is running. No live GKE evaluation was performed on my side.

Requesting changes for the isolation gap only.

[kweinmeister]

@rng1995 @chrisknvidia Both comments have been addressed and replied to:

1. The metadata reachability probe now runs from a dedicated companion container (skillevaluator-metadata-probe) instead of the evaluated task container, and the probe script is hardened against dead proxies and /bin/true shims.
2. Added port 53 DNS egress rules for Cloud DNS and kube-dns pods, and removed ::/0 which was causing Cilium Dataplane V2 to bypass IPv4 metadata exclusions.

Synced the branch with main and verified end-to-end locally and on live GKE Autopilot, including a full Tier 3 eval run with claude-code and google-cloud-developer. Ready for another look when you have a chance!

[rng1995]

@kweinmeister Thank you for your patience and the quick, thorough follow-up, and for validating it on a live Autopilot cluster.

I re-reviewed 7601351:

• DNS egress: fixed. I've resolved that thread.
• Direct-mode isolation: the companion probe works as intended: SkillEvaluator-owned image, shared network namespace, hardened script, correct exec routing, fail-closed handling. Details are in the existing thread.

Requesting changes for:

1. [P1] The DinD probe always passes (inline on the wget line). BusyBox wget in the docker:dind image rejects --no-proxy, so compose-mode pods are always reported as isolated.
2. [P2] The ephemeral-storage cap applies outside Autopilot (inline). It silently lowers storage_mb / --override-storage-mb above 10176Mi on every unprivileged direct-mode pod.
3. The window before the probe runs (existing thread). A skill-replaced sleep can reach the metadata server before the probe runs; an init container would close this.
4. Merge blockers:
   • The DCO check fails because 7601351 has no Signed-off-by line. All earlier commits on the branch are signed.
   • The branch now conflicts with main (#161 landed) in tests/test_tier3_public_runtime.py.

Non-blocking, worth a look:

• SKILLEVALUATOR_GKE_METADATA_PROBE_IMAGE isn't in the runner's Harbor subprocess allowlist, so operators can't override the image through skill-eval. The default python:3.12-slim is a mutable Docker Hub tag. Forwarding the variable and pinning a digest would help clusters without Docker Hub egress (today they fail closed). The variable and the companion container also aren't documented yet.
• The probe counts a connection that is accepted but answers after the 2s timeout as "isolated", on both the Python and curl paths. Treating a successful TCP connect as reachable would be stricter.
• The wget branch has no regression test. A BusyBox-style case would have caught item 1.

Validation: Ruff passes, focused GKE/runner/adapter/preflight tests pass (788), and the full local suite passes (10,020 passed, 18 skipped). All CI checks pass except DCO. No live GKE run on my side.

[rng1995]

[P1] BusyBox wget has no --no-proxy option. Its long options are continue, quiet, server-response, output-document, output-file, directory-prefix, proxy (-Y on/off), user-agent, timeout, header, post-data, spider, no-check-certificate and post-file.

In compose mode the probe runs in Harbor's docker:28.3.3-dind container. That image is Alpine with no python3 or curl, so the script reaches this branch. BusyBox treats --no-proxy as an unknown option: it prints its usage text and exits 1. That output isn't empty and has no HTTP/, so the loop finishes with exit 0 ("isolated") whether or not 169.254.169.254 is reachable. Before this commit, the same branch (without --no-proxy) returned 42 for a reachable endpoint.

The script already unsets every proxy variable, so dropping --no-proxy is enough. BusyBox also accepts -Y off if you want the explicit form; GNU wget doesn't. It would also help to make any wget failure other than "connection refused" or "timed out" exit 43 instead of falling through to exit 0, and to add a BusyBox-style case to the probe test.

A quick live check: docker run --rm docker:28.3.3-dind wget --no-proxy -S -O /dev/null http://example.com; echo $?

[kweinmeister]

@rng1995 Dropped --no-proxy from the wget branch (since proxy env vars are already unset at the start of the script) and updated the fallback to exit 43 unless wget fails with an expected connection-refused/timeout error. Also added a BusyBox wget test case.

[rng1995]

[P2] Despite the name, this cap has no Autopilot check. It runs for every unprivileged direct-mode pod on any GKE cluster (call site at line 662) and lowers both the requests and limits ephemeral-storage of main to 10176Mi. A skill with harbor.resources.storage_mb: 20480, or an operator passing --override-storage-mb 20480 on a GKE Standard cluster, silently gets a 10176Mi limit and can be evicted for exceeding it.

Could you apply this only when the cluster is Autopilot (for example by detecting it during preflight, or with an explicit operator flag), or fail with a clear message instead of lowering an explicit request?

[kweinmeister]

@rng1995 Added Autopilot cluster detection (_is_autopilot_cluster, with an optional SKILLEVALUATOR_GKE_AUTOPILOT / --ek autopilot override) and scoped the 10176Mi adjustment so it only applies to non-GPU/TPU pods on Autopilot when main requests the default 10Gi. Explicit storage overrides above 10Gi and GKE Standard clusters are left untouched.

[chrisknvidia]

@kweinmeister I rechecked the current PR head (7601351) against main (6bab56e), and there is still a merge conflict in tests/test_tier3_public_runtime.py. GitHub also reports the PR as conflicting. Could you please resolve this conflict against the latest main and push the updated branch so we can finish the review?

[kweinmeister]

@chrisknvidia @rng1995 Resolved the merge conflict in tests/test_tier3_public_runtime.py against main, signed off the commits for DCO, and addressed the latest review comments on the metadata init probe, BusyBox wget, and Autopilot storage scoping. Ready for another look when you have a chance!