Repository navigation
test(e2e): cover gateway_unavailable and pin the non-JSON body 400 - #263
Merged
rambohe-ch merged 3 commits intoNov 3, 2026
Merged
Conversation
- Add gateway_unavailable_spec.go: scale the cluster-wide productionstack-status-reporter to zero and assert GET /v1/models returns 502 gateway_unavailable with x-kaito-error-source: gateway (outage local_reply mapper kaito-project#3: no model header + UH). The harness is installed without a model, so the spec needs no GPU node; it is Serial because the reporter is a cluster-wide singleton. - Tighten the non-JSON body case in model_routing_spec.go to BBR's native 400 and record the raw response in the report. The body is passed through unchanged, so its shape is not asserted. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: rambohe-ch <rambohe.ch@gmail.com>
rambohe-ch
requested review from
Fei-Guo,
techworldhello and
zhuangqh
as code owners
November 2, 2026 23:58
1 task
Collaborator
Author
|
Companion proposal update: #264 (its Test Plan references the specs added here). |
… wait The baseline GET /v1/models failed in CI with "no such host": the spec installed its harness in a random namespace, so on an App Routing cluster the Gateway's per-namespace DNS record was new and not yet resolvable within the 2-minute baseline wait. Use a fixed namespace like every case namespace, and wait for the baseline as long as the model cases wait for gateway routing (InferenceSetReadyTimeout, polled every 10s). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: rambohe-ch <rambohe.ch@gmail.com>
The baseline GET /v1/models still failed with "no such host" for the full 20 minutes with a fixed namespace, so the DNS record was never published rather than slow. On an App Routing cluster external-dns publishes the Gateway hostname from the namespace's HTTPRoute hostnames, and only a modeldeployment renders an HTTPRoute; a harness without a model has no resolvable hostname. Turn the spec into a regular case (CaseGatewayUnavailable) with one mocked model installed through InstallCase, like epp-outage and model-unavailable. The model is never requested. InstallCase already waits for gateway routing, so the baseline wait goes back to 2 minutes, and the NoGPU label is dropped. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: rambohe-ch <rambohe.ch@gmail.com>
This branch was previously deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Reason for Change:
Closes two data-plane E2E gaps against the data-plane error-handling proposal:
502 gateway_unavailable(outagelocal_replymapper feat: add production stack arch contents #3) had no E2E coverage.test/e2e/ +├── gateway_unavailable_spec.go # new: mapper #3 ├── cases.go # new case: gateway-unavailable ├── model_routing_spec.go # non-JSON body: 400..500 -> exactly 400 └── utils/ginkgo.go # Outage label doc lists the new specgateway_unavailableneeds a failure before model routing (noX-Gateway-Model-Name) with a router flag. Scaling the namespace Gateway itself to zero cannot produce that, because no Envoy is left to answer. Model discovery is the request path that reaches an upstream without model routing:The spec:
gateway-unavailable) with one mocked model. The model is never requested, but on an App Routing cluster the Gateway hostname is published to DNS from the namespace's HTTPRoute hostnames, and only a modeldeployment renders an HTTPRoute;app.kubernetes.io/namelabel rather than a fixed namespace and name;Serial, because the reporter is a cluster-wide singleton.Per #262 it does not carry
StandardK8sOnly: scaling a shared Deployment works on a managed control plane.For a non-JSON body, BBR answers with its own plain-text
400(an ext_proc immediate response), which the gateway passes through without the error envelope. The test now asserts exactly400and records the raw response (status,content-type,x-kaito-error-source, body) withAddReportEntry, so the body is visible in the report without asserting its shape.Requirements
Issue Fixed:
Notes for Reviewers:
Evidence
The new specs have not been run on a live cluster yet; they run in the PR
e2eworkflow (!Nightly && !Perf && !Scaling). Local checks:go vet ./test/e2e/...(with-mod=readonly): passed.gofmt -l test/e2e: no output.make verify-boilerplate: passed.codespellon the changed files: no findings.ginkgo --dry-run --label-filter='Outage && ModelDiscovery' ./test/e2e: selects exactly the 2 new specs.Merge Danger
Door: two-way. Test-only change, no chart or code change.
Blast Radius: e2e-runtime. The new spec briefly scales the status reporter to zero, which pauses model discovery and control-plane Events cluster-wide;
Serialkeeps every other spec from overlapping it.The companion proposal update is in a separate PR.