Hosted conformance: deploy + scheduled client runs (CI) and conformance remote

[claude]

Requested by Paul Carleton · Slack thread

Before: the hosted client-conformance server (#501) is deployed to val.town by hand with examples/hosted/deploy-valtown.ts, nothing exercises the live deployment after a deploy, and the only way to drive a client against it is to mint a run with curl and wire the URL/env into the client yourself.

After: pushes to main that touch src/**, examples/hosted/** or package*.json redeploy the rs and relay vals in place and smoke-test them; a scheduled matrix of eight scenarios (core + auth/*) runs the vendored TypeScript everything-client against the live deployment four times a day, after every deploy, on demand, and on PRs that touch the hosted code, with a scenario/result table linking every HTML report in the step summary. A new conformance remote subcommand is the reusable piece both workflows (and humans) use.

This PR adds the CI for the hosted deployment plus the conformance remote CLI command that drives a client against it, and fixes three live-deployment bugs the first matrix runs surfaced.

How: conformance remote --url <RS> --scenario <name> --command "<client>" [--run-id <id>] [--timeout <ms>] [-o result.json] does GET <RS>/s/<scenario> to mint a run, then spawns the client through the very same executeClient() path conformance client --command uses (MCP URL appended as the last argument, MCP_CONFORMANCE_SCENARIO / MCP_CONFORMANCE_PROTOCOL_VERSION in env, MCP_CONFORMANCE_CONTEXT set to the minted context), reads <RS>/results/<runId> back, prints it with the existing printClientResults reporter plus the /results/<runId>.html link, and exits 1 when summary.failed > 0 (or the client timed out). The hosted mint route gained ?runId=<id> so --run-id can pick a predictable id. deploy-hosted.yml (concurrency hosted-deploy, environment hosted-conformance) runs npm ci && npm test, then deploy-valtown.ts --push rs relay, then curls /scenarios and runs one remote --scenario initialize. hosted-conformance.yml has a gate job that probes /scenarios and turns an outage into a skipped (never red) run, a fail-fast: false matrix over initialize, tools_call, json-schema-ref-no-deref, auth/metadata-default, auth/metadata-var1, auth/metadata-var3, auth/basic-cimd, auth/pre-registration (each leg retries once with a fresh run id and uploads its JSON result), and a summary job that aggregates the artifacts into a table. The valtown-manifest.json now carries only the two mcp_conformance org vals (rs, relay) with their live ids so deploys update in place. Unit tests cover remote against a mocked hosted server (mint, URL-as-last-arg, env, ?runId=, results origin, failure exit, unreachable/404) and end to end against the real createHostedApp().

Found by the PR-triggered matrix runs against the live deployment and fixed here:

• The AS relay (examples/hosted/valtown-relay.ts) forwarded the upstream content-length after fetch() had transparently decompressed the body, so every AS-metadata fetch reached the client as full JSON with the compressed length and was truncated (Unterminated string in JSON at position N). The relay now buffers the body, drops the framing/encoding headers, and asks upstream for identity encoding; regression test with a gzipping mock RS.
• The merge of main into the hosted branch pulled src/spec-types/draft.ts (98K chars) and the four spec JSON schemas (90–180K) into the RS closure, and val.town rejects files over 80,000 characters, so a deploy aborted halfway through the upload. deploy-valtown.ts now strips comments from oversized TS modules, stages .json imports as generated .json.ts modules split into string chunks when needed, and refuses to push if anything is still over the cap; stage-only regression test. The live rs and relay vals were redeployed from this branch, which also brought the issuer field (added on main in fix(auth): add issuer to pre-registration context #423) into the hosted auth/pre-registration context that the vendored client requires.
• On val.town the isolate serving GET /results is usually not the one that saw the OAuth flow; SessionManager.results() re-judges the persisted raw log in a fresh scenario instance. The metadata-discovery and token-endpoint-auth scenarios kept their RFC 8707 resource observations in private fields, so the fresh instance reported resource-parameter-in-authorization / resource-parameter-in-token as "not provided" even though the persisted incoming-auth-request checks show the client sent it in both places. observeResourceParameters() now reads those facts back from the log (the PRM route also records the identifier it served), and getChecks() falls back to it; single-process judging is unchanged. The hosted relay test snapshots the raw log and re-judges it through finalizeChecks(), failing without the fallback.

Setup needed from a maintainer:

• GitHub environment hosted-conformance with secret VAL_TOWN_TOKEN (a val.town API token for the mcp_conformance org)
• Repository variables HOSTED_RS_URL (RS origin) and HOSTED_AS_URL (AS relay origin); both workflows fall back to the current RS val URL when HOSTED_RS_URL is unset
• workflow_run and schedule triggers only fire once the workflow files are on main; the pull_request trigger on this PR runs the matrix against the live deployment now

Stacked on #501 (claude/hosted-client-conformance); merge that first.

🤖 Generated with Claude Code

https://claude.ai/code/session_01FXWixCiyW8eEfwFeZcADEK

[pkg-pr-new]

Open in StackBlitz

npx https://pkg.pr.new/@modelcontextprotocol/conformance@502

commit: 617cce8