Skip to content

test(e2e): cover gateway_unavailable and pin the non-JSON body 400 - #263

Merged
rambohe-ch merged 3 commits into
kaito-project:mainfrom
rambohe-ch:e2e-dataplane-error-coverage
Nov 3, 2026
Merged

rambohe-ch merged 3 commits into
kaito-project:mainfrom
rambohe-ch:e2e-dataplane-error-coverage

Conversation

@rambohe-ch

@rambohe-ch rambohe-ch commented Nov 2, 2026 •

Copy link
Copy Markdown
Collaborator

Reason for Change:

Closes two data-plane E2E gaps against the data-plane error-handling proposal:

  • 502 gateway_unavailable (outage local_reply mapper feat: add production stack arch contents #3) had no E2E coverage.
  • The non-JSON body case accepted any status from 400 to 500, so it could not catch a regression to a misleading 404 or a 5xx.
 test/e2e/
+├── gateway_unavailable_spec.go   # new: mapper #3
 ├── cases.go                      # new case: gateway-unavailable
 ├── model_routing_spec.go         # non-JSON body: 400..500 -> exactly 400
 └── utils/ginkgo.go               # Outage label doc lists the new spec

gateway_unavailable needs a failure before model routing (no X-Gateway-Model-Name) with a router flag. Scaling the namespace Gateway itself to zero cannot produce that, because no Envoy is left to answer. Model discovery is the request path that reaches an upstream without model routing:

GET /v1/models                      # bodyless: BBR injects no model header
  Gateway route models-api-list
    upstream: productionstack-status-reporter   # scaled to 0
    -> local 503, flag UH
  outage local_reply mapper #3      # header absent + router flag
    -> 502 gateway_unavailable, x-kaito-error-source: gateway

The spec:

  • is a regular case (gateway-unavailable) with one mocked model. The model is never requested, but on an App Routing cluster the Gateway hostname is published to DNS from the namespace's HTTPRoute hostnames, and only a modeldeployment renders an HTTPRoute;
  • finds the reporter Deployment by its app.kubernetes.io/name label rather than a fixed namespace and name;
  • restores the original replica count on cleanup;
  • is Serial, because the reporter is a cluster-wide singleton.

Per #262 it does not carry StandardK8sOnly: scaling a shared Deployment works on a managed control plane.

For a non-JSON body, BBR answers with its own plain-text 400 (an ext_proc immediate response), which the gateway passes through without the error envelope. The test now asserts exactly 400 and records the raw response (status, content-type, x-kaito-error-source, body) with AddReportEntry, so the body is visible in the report without asserting its shape.

Requirements

  • added unit tests and e2e tests (if applicable).

Issue Fixed:

Notes for Reviewers:

Evidence

The new specs have not been run on a live cluster yet; they run in the PR e2e workflow (!Nightly && !Perf && !Scaling). Local checks:

  • go vet ./test/e2e/... (with -mod=readonly): passed.
  • gofmt -l test/e2e: no output.
  • make verify-boilerplate: passed.
  • codespell on the changed files: no findings.
  • ginkgo --dry-run --label-filter='Outage && ModelDiscovery' ./test/e2e: selects exactly the 2 new specs.

Merge Danger

Door: two-way. Test-only change, no chart or code change.

Blast Radius: e2e-runtime. The new spec briefly scales the status reporter to zero, which pauses model discovery and control-plane Events cluster-wide; Serial keeps every other spec from overlapping it.

The companion proposal update is in a separate PR.

- Add gateway_unavailable_spec.go: scale the cluster-wide
  productionstack-status-reporter to zero and assert GET /v1/models
  returns 502 gateway_unavailable with x-kaito-error-source: gateway
  (outage local_reply mapper kaito-project#3: no model header + UH). The harness is
  installed without a model, so the spec needs no GPU node; it is
  Serial because the reporter is a cluster-wide singleton.
- Tighten the non-JSON body case in model_routing_spec.go to BBR's
  native 400 and record the raw response in the report. The body is
  passed through unchanged, so its shape is not asserted.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: rambohe-ch <rambohe.ch@gmail.com>
@rambohe-ch

Copy link
Copy Markdown
Collaborator Author

Companion proposal update: #264 (its Test Plan references the specs added here).

… wait

The baseline GET /v1/models failed in CI with "no such host": the spec
installed its harness in a random namespace, so on an App Routing
cluster the Gateway's per-namespace DNS record was new and not yet
resolvable within the 2-minute baseline wait.

Use a fixed namespace like every case namespace, and wait for the
baseline as long as the model cases wait for gateway routing
(InferenceSetReadyTimeout, polled every 10s).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: rambohe-ch <rambohe.ch@gmail.com>
The baseline GET /v1/models still failed with "no such host" for the
full 20 minutes with a fixed namespace, so the DNS record was never
published rather than slow. On an App Routing cluster external-dns
publishes the Gateway hostname from the namespace's HTTPRoute
hostnames, and only a modeldeployment renders an HTTPRoute; a harness
without a model has no resolvable hostname.

Turn the spec into a regular case (CaseGatewayUnavailable) with one
mocked model installed through InstallCase, like epp-outage and
model-unavailable. The model is never requested. InstallCase already
waits for gateway routing, so the baseline wait goes back to 2 minutes,
and the NoGPU label is dropped.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: rambohe-ch <rambohe.ch@gmail.com>
@rambohe-ch
rambohe-ch merged commit fcb37f4 into kaito-project:main Nov 3, 2026
13 checks passed

This branch was previously deployed

1 inactive deployment
e2e-test — b972319e Deployed Nov 3, 2026 by rambohe-ch via e2e-tests #690
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant