Hosted client-conformance server: public MCP endpoints per scenario, auth relay, val.town deploy, declarative client steps - #501
Draft
claude[bot] wants to merge 10 commits into
Draft
Conversation
Mounts every (non-auth) client-testing scenario at /s/<name> on a single long-lived HTTP server, so clients-under-test can point at a public URL instead of being spawned by the runner. - src/hosted/: session manager + loopback proxy + express app - /results/<id>[.html]: JSON or pretty-printed checks per session - /mcp: the hosted server is itself an MCP server with list_scenarios, start_session, get_results tools - examples/hosted/valtown.ts: self-contained Request->Response variant for serverless hosts that can't bind loopback ports - conformance hosted --port <n> [--public-origin <url>] [--ttl <ms>] Auth scenarios are excluded - they need a second public origin for the authorization server, which a single-host proxy can't expose. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ateless Replaces the loopback-proxy approach with direct mounting: - Scenario gains optional handler(getBaseUrl) -> RequestListener and mcpPath. New HandlerScenario base class implements start()/stop() as a thin wrapper around handler(), so the CLI runner and hosted runner share identical scenario code with no port binding required for hosted mode. - Refactored every non-auth scenario (initialize, tools_call, elicitation-defaults, sse-retry, request-metadata, mrtr-client, json-schema-ref-deref, plus all BaseHttpScenario subclasses) onto HandlerScenario. json-schema-ref-deref now derives its canary URL from getBaseUrl so it points at the public mounted path. - URL scheme is now /s/<scenario>/<run-id> with the run-id in the path, not the mcp-session-id header — works for stateless-transport clients (every draft scenario using sessionIdGenerator: undefined). - Dropped src/hosted/proxy.ts. - examples/hosted/valtown.ts is now a Request->Response bridge around the real createHostedApp(), not a reimplementation, so the same scenarios run on val.town/Deno/Bun. sse-retry is 501'd through the buffered bridge but works under 'conformance hosted'. 236/236 tests pass (13 new). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Auth scenarios need ≥2 public origins because RFC 8414/9728 well-known paths
and issuer validation are origin-rooted — they can't live under the
/s/<scenario>/<id>/ prefix. This adds an AS-relay topology where a stateless
second deployment forwards everything to the RS app's /__aux/<role>/*
backchannel; all scenario state (closures, checks[]) stays in one process.
- types: AuthHandlerScenario base — authHandlers(ctx)→{rs,aux} mirrors
HandlerScenario; start()/stop() bind one localhost port per origin so the
CLI runner path is unchanged.
- scenarios: refactor basic-cimd, discovery-metadata×4, pre-registration to
the new shape (rest are mechanical follow-up; still work via start()).
- hosted/server: --as-origin/--as2-origin/--idp-origin enable auth/* mounts;
/__aux/<role>/* dispatch (extracts /r/<runId>, strips it, hands to the
run's aux handler) guarded by x-relay-secret + timingSafeEqual; root-level
/.well-known/oauth-protected-resource/s/* dispatch for RFC 9728 discovery.
- examples/hosted/valtown-relay.ts: ~40 LOC stateless relay (curated header
forward, redirect:manual, shared secret). One val per role.
- hosted-auth.test.ts: spins RS+relay on ephemeral ports and walks discovery
→ DCR → authorize → token → MCP → results end-to-end.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…hecker Hosted MCP checker for the stateless draft protocol (2026-07-28): - src/scenarios/client/stateless-gauntlet.ts — single stateless server whose tools each validate one aspect of the request that carried it. Transport obligations (Accept, Content-Type, MCP-Protocol-Version, Mcp-Method/Mcp-Name, io.modelcontextprotocol/* _meta) checked on every POST. Results carry the required resultType, and discover/list carry ttlMs/cacheScope. - src/scenarios/client/auth-checker.ts — auth-chain checker scenario. - examples/hosted/valtown-checker.ts + valtown-auth-checker.ts — val.town entrypoints; each val IS the checker (origin-rooted, no /x/<scenario> path). - examples/hosted/deploy-valtown.ts — stages the import closure with Deno-style specifiers and pushes via the val.town v2 API. - examples/hosted/fetch-bridge.ts — adapts express RequestListener → fetch handler for serverless runtimes. - src/scenarios/client/auth/helpers/createAuthServer.ts — encode PKCE challenge + scopes into the auth code itself so the mock AS is stateless across serverless isolates. Builds on paulc/hosted-auth (second-origin relay) and paulc/hosted-server.
… SQLite store val.town load-balances one run's requests across short-lived isolates, so GET /results on the in-memory server flapped between "unknown run", an empty check list and the real one depending on which isolate answered. - src/hosted/store.ts: RunStore interface (+ MemoryRunStore). Each process writes its raw check log through after every request, keyed by (run, writer) and replaced wholesale so concurrent writers never clobber; run metadata is persisted so a cold process can rebuild handlers. - SessionManager: optional store, ensure() (used by the /__aux relay backchannel), persist()/flush(), and results() that merges every writer's log and re-judges it once with a fresh scenario instance (finalizeChecks) instead of trusting one process's getChecks(). - server.ts: results routes and meta get_results go through the merged view; RS-side PRM well-known uses getOrCreate; dispatch writes through on res.end when a store is configured; minted context carries the scenario name so it can be passed verbatim as MCP_CONFORMANCE_CONTEXT. - examples/hosted/valtown-store.ts: SqliteRunStore on the account SQLite API (6h retention, swept on run creation). valtown.ts wires it up and awaits sessions.flush() before returning each response so the last request's write isn't abandoned when the isolate idles. - json-schema-ref-no-deref: keep observed state in the raw log as _state/* INFO events (+ rawChecks()) so it is multi-process safe. - everything-client: tools_call alias and actually call the tool. - valtown-manifest: client-rs / client-relay vals.
First slice of "generic steering" for client conformance: a scenario may declare the client-side choreography it needs as data, the runner ships it in MCP_CONFORMANCE_CONTEXT as `steps`, and a client with no bespoke handler for the scenario name interprets it. Checks stay in the scenario; only the instructions to the client under test become data. - src/steps: closed op set (tools/list, tools/call, wait, disconnect) as a zod schema, one `$from` capture form, resolveFrom/resolveArguments. - Scenario.steps (types.ts); declared on initialize, tools_call, json-schema-ref-no-deref and elicitation-sep1034-client-defaults. - runner/client.ts merges steps into the context blob; the hosted server does the same for minted runs and lists steps on / and /scenarios. - everything-client: fallback interpreter (standing defaults: connect first, accept elicitation with schema defaults, disconnect last). Named handlers still win; MCP_CONFORMANCE_FORCE_STEPS=1 forces the generic path so it can be exercised against scenarios that also have handlers.
🏠 Remote-Dev: homespace
Bring the hosted runner (src/hosted, examples/hosted), declarative client steps (src/steps) and the HandlerScenario/AuthHandlerScenario refactor onto current main. Merge resolution notes: - Scenario.handler() and authHandlers() now receive main's ScenarioContext (resolved spec version + mock factories). HandlerScenario and AuthHandlerScenario.start(ctx) forward it; the hosted runner builds one per run (hostedScenarioContext) mirroring the CLI runner's default version resolution. - mock-server: split createServerStateful/Stateless into createHandler* (unbound RequestListener + recorded log) plus a shared listenMockHandler, exposed as ScenarioContext.createHandler(). tools_call keeps main's version-aware MockServer rewrite and stays hostable through it. - auth scenarios keep main's createAuthServer(ctx, ...) / createServer(ctx, ...) signatures; discovery-metadata carries main's RFC 8707 resource tracking; pre-registration's context includes the AS issuer as on main. - json-schema-ref-deref: main's server/discover shim and MCP-Protocol-Version rewrite live inside handler(). - hosted tests updated for SEP-2575's -32022 version-unsupported code and the pre-registration issuer field. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FXWixCiyW8eEfwFeZcADEK
commit: |
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
…ux-path regex Fixes the CodeQL alerts raised on the hosted runner: - src/hosted/html.ts: escape the run id in the results report's <title> (it was interpolated raw); export the escape helper as escapeHtml and also escape single quotes. - src/scenarios/client/stateless-gauntlet.ts: HTML-escape the consent page's continue link (it carried the re-encoded query and the Host-derived issuer with only '&' escaped) and the mounted base URL embedded in the landing page. - src/hosted/server.ts: locate the /r/<run-id> pair in /__aux paths by splitting segments instead of a lazy `^(.*?)/r/…` regex, which could backtrack polynomially on adversarial paths. Same prefix/run-id/suffix result for every well-formed path. Tests: renderResults escaping, the consent page with a hostile X-Forwarded-Host, and aux-path parsing (invalid id, unknown run, adversarial repeated /r/- prefix). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FXWixCiyW8eEfwFeZcADEK
CodeQL (js/missing-rate-limiting) flagged the /authorize and /token route handlers in createAuthServer.ts as "performing authorization without rate limiting". The only change this branch made to those handlers is calling encodeAuthCode()/decodeAuthCode(), and CodeQL's heuristic treats any callee whose name looks authorization-related as a credential check. These helpers only pack/unpack a base64url JSON envelope carrying per-flow PKCE state — no secret, no signature, no verification — so rename them to packFlowCode()/unpackFlowCode() (and AuthCodeState to FlowCodeState) and document that in the doc comment. No behaviour change; the mock AS is a test fixture, not a production authorization server. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FXWixCiyW8eEfwFeZcADEK
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Requested by Paul Carleton · Slack thread
Before: Client-conformance scenarios only run under the CLI runner, which spawns the client-under-test and binds a fresh loopback port per scenario. A client that cannot be spawned locally (a hosted agent, a browser, another team's service) has no public URL to point at, and the
auth/*scenarios need two local origins so they cannot be exercised that way at all.After:
npx @modelcontextprotocol/conformance hostedserves every hostable client scenario from one long-lived origin (/s/<scenario>/<run-id>/mcp, results at/results/<run-id>, mint-a-run atGET /s/<scenario>), with theauth/*scenarios behind a stateless second-origin relay and a val.town deployment underexamples/hostedwhose run state survives serverless isolates. Scenarios exposehandler()/authHandlers()instead of binding ports, and plumbing-only scenarios publish declarativestepsin the client context so a generic client can be steered without bespoke per-scenario code.This lands the
paulc/hosted-client-sketchstack (experimental-stateless-checker → hosted-auth → hosted-server → sketch) onto current main as a single merge; the PR diff is the new work only.How:
src/hostedmounts each scenario'shandler()under a path prefix and rewritesreq.url, so the CLI runner and the hosted runner exercise identical scenario code (HandlerScenario/AuthHandlerScenarioinsrc/types.tssupplystart()/stop()for free). Auth scenarios return{ rs, aux }handlers; the relay (examples/hosted/valtown-relay.ts) forwards every AS-origin request to<rs>/__aux/<role>/...with a shared secret, and the per-run issuer is<as-origin>/r/<run-id>so the run is recoverable from any path the client derives from it.src/stepsis the data format for client steering;examples/clients/typescript/everything-client.tslearns to follow it.Merge notes (where main and the branch had diverged):
Scenario.handler()andauthHandlers()now receive main'sScenarioContext(resolved spec version + mock factories);HandlerScenario/AuthHandlerScenario.start(ctx)forward it, and the hosted runner builds one per run (hostedScenarioContext) mirroring the CLI runner's default version resolution.src/mock-server:createServerStateful/Statelessare split intocreateHandler*(an unbound listener plus therecordedlog) and a sharedlistenMockHandler, exposed asScenarioContext.createHandler().tools_callkeeps main's version-awareMockServerrewrite and stays hostable through it.createAuthServer(ctx, ...)/createServer(ctx, ...)signatures;discovery-metadatacarries main's RFC 8707 resource tracking;pre-registration's context includes the ASissueras on main.json-schema-ref-deref: main'sserver/discovershim andMCP-Protocol-Versionrewrite now live insidehandler().-32022version-unsupported code and theissuerfield.Verification:
npm run typecheckandnpm run lintare clean;npm testis 607/608. The one failure (src/scenarios/server/negative.test.ts› sep-2164-resource-not-found, a 2026-07-28ReadResourceResultwire-schema violation) fails identically on pristineorigin/mainand is unrelated to this change. The val.town deployment has been pushed to the shared org account but not yet smoke-tested end to end from the requesting environment (its egress policy blocks*.val.run); the vendoredeverything-clientruns against the hosted endpoints are still to be confirmed.🤖 Generated with Claude Code
https://claude.ai/code/session_01FXWixCiyW8eEfwFeZcADEK
Generated by Claude Code