Skip to content

Commit 433be0f

Browse files
rambohe-chCopilot
andauthored
test(e2e): narrow StandardK8sOnly to specs a managed control plane cannot run (#262)
* test(e2e): narrow StandardK8sOnly to specs a managed control plane cannot run With cluster-admin access on an AKS AI Manager cluster, 57 of the 82 StandardK8sOnly specs pass: scaling shared Deployments, kubectl exec into the Gateway and scraping pod metrics through pods/proxy all work. Drop the label from those specs and keep it only where a managed control plane still cannot run them: - ModelDeploymentValues a managed backend cannot express: scaling (ScalingMetrics, EPPScorerWeights), model_routing load distribution (EPPScorerWeights), modeldeployment_chart autoUpgrade (AutoUpgrade). - State reconciled away by a managed controller: the sustained-slow ModelMirror case of model_download_slow. - Not yet verified: the cloud provisioning-failure cases of control_plane_error (skipped under gpu-node-mocker) and the EPP metrics spec (skipped everywhere). Update the label documentation to describe these reasons. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: rambohe-ch <rambohe.ch@gmail.com> * test(e2e): unskip the namespace discovery-label check on AI Manager AI Manager now stamps productionstack.kaito.sh/managed-by: modelharness on the namespaces it provisions, and the spec passes on a live AI Manager cluster. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: rambohe-ch <rambohe.ch@gmail.com> * test(e2e): enable the EPP metrics spec and drop the dead debug-filter spec EPP routing success (metrics): the Skip blamed an Istio sidecar for a 401 on the EPP metrics port. The modeldeployment chart has passed --metrics-endpoint-auth=false since it was introduced, and EPP pods run without a sidecar. The spec passes on an AKS AI Manager cluster, so remove the Skip and its StandardK8sOnly label. Debug EnvoyFilter log chain: the inference-debug-filter manifest it relies on was removed in #46 and nothing installs it, and its Gateway pod selector predates per-case gateways. It could not pass on any backend, so delete it together with findDebugLogTriple and its scenario entry. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: rambohe-ch <rambohe.ch@gmail.com> * test(e2e): run load distribution and autoUpgrade on AI Manager Both specs pass on an AKS AI Manager cluster, which now accepts EPPScorerWeights and AutoUpgrade through modeldeployment overrides, so drop their StandardK8sOnly labels. Restore the load-distribution scorer weights in an AfterEach instead of a DeferCleanup registered by the BeforeEach. Ginkgo runs AfterAll before cleanups registered by the last spec of an Ordered container, so a focused or filtered run in which load distribution was the last spec tried to restore weights on deployments AfterAll had already removed. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: rambohe-ch <rambohe.ch@gmail.com> * test(e2e): run the scaling specs on AI Manager scaling (0 -> 1 -> 2 -> 1 -> 0 and anti-flapping, 17 specs) and scaling_transitions (manual <-> autoscaling, min 0 parking, 9 specs) pass on an AKS AI Manager cluster with cluster-admin access, where the backend maps the chart's scaling values onto the AI Manager scale profile and scaling overrides. Patching an InferenceSet's replicas works there too. Drop their StandardK8sOnly labels and the caller requirement on SetInferenceSetReplicas. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: rambohe-ch <rambohe.ch@gmail.com> * test(e2e): label model_unavailable StandardK8sOnly It empties the inference pool with a manual replica count of 0. AKS AI Manager rejects that (scale.manual.replicas must be >= 1), and its only zero-replica mode, autoscaling with minReplicas 0, wakes on the requests the spec sends instead of staying empty. Update the label docs, which cited this spec as one any backend can honour. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: rambohe-ch <rambohe.ch@gmail.com> * test(e2e): run the ModelMirror download specs where KAITO reconciles mirrors The mock ModelMirror was Static, which KAITO's ModelMirror controller marks Ready immediately, so on clusters with the ModelMirror feature gate on (AKS AI Manager) the sustained-slow spec never saw a slow download and the dip and partial-window specs passed without testing anything. Create the mock as a Managed mirror on a StorageClass with no provisioner instead. KAITO creates the PVC and download Job, but the PVC never binds and the Job pod is never scheduled, so the controller has no progress to report and leaves the status the spec patches untouched. Nothing is downloaded and no node capacity is reserved. With the gate off nothing reconciles the mirror, as before. Read the reporter's --event-refresh-seconds instead of assuming the E2E install's 120s: a managed install may keep the chart default of 300s, and the refresh and quiet windows must outlast it. At 120s the windows are unchanged. All four model_download_slow specs pass on an AI Manager cluster; drop the StandardK8sOnly label from the sustained-slow spec. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: rambohe-ch <rambohe.ch@gmail.com> * test(e2e): label the exact-origin CORS context StandardK8sOnly It configures an exact-origin CORS allowlist on the modelharness. AKS AI Manager installs the harness with the chart's wildcard default and has no setting for it, so the context fails there and, being in the same Ordered container, skips the wildcard context. The wildcard context matches that default and passes 12/12 on an AI Manager cluster. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: rambohe-ch <rambohe.ch@gmail.com> --------- Signed-off-by: rambohe-ch <rambohe.ch@gmail.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
1 parent fd8d6d2 commit 433be0f

26 files changed

Lines changed: 192 additions & 254 deletions

‎test/e2e/README.md‎

Lines changed: 16 additions & 21 deletions
Original file line numberDiff line numberDiff line change
@@ -92,33 +92,28 @@ specs. They remain enabled by default upstream. Consumers whose deployed stack
9292
does not yet support these endpoints can exclude them with `!ModelDiscovery`.
9393

9494
`StandardK8sOnly` cuts across the other two. It marks specs that require a
95-
standard Kubernetes cluster and cannot run on an opinionated managed one such as
96-
AKS Automatic: they reshape running workloads
97-
directly rather than going through the `Deployer` — `ScaleDeployment` on a
98-
Deployment's scale subresource ([`utils/cluster.go`](utils/cluster.go)) or
99-
`SetInferenceSetReplicas` patching an InferenceSet
100-
([`utils/scaling.go`](utils/scaling.go)) — or their assertion only holds under a
101-
`ModelDeploymentValues` field a managed backend cannot express, or they scrape
102-
metrics through `pods/proxy`, directly or via the helpers in
103-
[`utils/metrics.go`](utils/metrics.go). When running against a managed cluster:
95+
standard Kubernetes cluster and cannot run on a managed control plane such as
96+
AKS AI Manager, even with cluster-admin access: they need a setting a managed
97+
backend cannot express (a manual replica count of 0, an exact-origin CORS
98+
allowlist), or they have not been verified on a managed control plane yet.
99+
Scaling shared Deployments through the scale subresource
100+
([`utils/cluster.go`](utils/cluster.go)), patching an InferenceSet's replicas
101+
([`utils/scaling.go`](utils/scaling.go)), `kubectl exec` into the Gateway, and
102+
scraping metrics through `pods/proxy` ([`utils/metrics.go`](utils/metrics.go))
103+
do not need it. When running against a managed cluster:
104104

105105
```bash
106106
E2E_LABEL='!StandardK8sOnly' make test-e2e
107107
```
108108

109-
It currently covers `bbr_outage`, `ext_authz_outage`, `epp_outage`,
110-
`cluster_filter_ha`, `cluster_status`, `control_plane_error`,
111-
`model_download_slow`, `scaling`, `scale_to_zero`, and the
112-
`Load distribution` Context of `model_routing` (which turns prefix-cache scoring
113-
off through `EPPScorerWeights` so identical prompts spread across replicas).
114-
It also covers `prefix_cache_routing`, `prefix_cache_perf`, the cross-model
115-
isolation and EPP metrics Contexts and backend 4xx counter assertion in
116-
`model_routing`, and the full-chain counter assertions in `filter_order`.
109+
It currently covers `model_unavailable`, the exact-origin `Browser CORS` Context
110+
of `apikey_auth`, and the two cloud provisioning-failure cases of
111+
`control_plane_error`.
117112
Label the smallest container that needs it, not the whole file: routing and
118113
filter-order specs that do not require these mechanisms stay runnable everywhere.
119-
Note it describes the MECHANISM, not the blast radius: `model_unavailable`
120-
empties an inference pool too, but does so through `UpgradeModelDeployment`, so
121-
any backend can honour it and it stays unlabelled.
114+
Note it describes the MECHANISM, not the blast radius: a disruptive spec stays
115+
unlabelled when a managed backend can honour it, such as the scaling specs that
116+
park a deployment at zero through autoscaling.
122117

123118

124119
### Bring up a cluster from scratch
@@ -271,7 +266,7 @@ Conventions worth knowing when writing a spec:
271266
errors. `WaitForDeploymentReplicas` accepts at least the requested ready count,
272267
except zero, which waits for scale-down convergence. `DeploymentReplicaGuard`
273268
captures the original count and provides exact `ScaleAndWait` and repeatable
274-
`Restore` operations for `StandardK8sOnly` fault-injection specs.
269+
`Restore` operations for fault-injection specs.
275270
Register case teardown and restoration independently in `BeforeAll`, with
276271
restoration registered last so it runs first after the Ordered container.
277272
Keep outage attribution, labels, and recovery assertions visible in each spec.

‎test/e2e/apikey_auth_spec.go‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -175,7 +175,11 @@ var _ = Describe("API Key Authentication", Ordered, utils.GinkgoLabelAuth, utils
175175
})
176176
})
177177

178-
Context("Browser CORS", utils.GinkgoLabelCORS, func() {
178+
// Exact-origin CORS needs the modelharness allowlist configured per
179+
// namespace. AKS AI Manager installs the harness with the chart's wildcard
180+
// default and has no setting for it, hence StandardK8sOnly; the wildcard
181+
// Context below matches that default and runs everywhere.
182+
Context("Browser CORS", utils.GinkgoLabelCORS, utils.GinkgoLabelStandardK8sOnly, func() {
179183
const disallowedOrigin = "https://disallowed.e2e.test"
180184

181185
sendPreflight := func(origin, method, headers string) (*http.Response, error) {

‎test/e2e/bbr_outage_spec.go‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -55,7 +55,7 @@ import (
5555
// the mesh. Scaling it to zero breaks every other in-flight inference
5656
// request, so this suite must not run concurrently with any other spec.
5757
var _ = Describe("BBR outage (fail-closed cluster filter)",
58-
Ordered, Serial, utils.GinkgoLabelOutage, utils.GinkgoLabelStandardK8sOnly, func() {
58+
Ordered, Serial, utils.GinkgoLabelOutage, func() {
5959

6060
const bbrDeploymentName = "body-based-router"
6161

‎test/e2e/cluster_filter_ha_spec.go‎

Lines changed: 2 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -49,7 +49,7 @@ import (
4949
// so that the loss of ONE replica is a transparent failover and the
5050
// fail-closed `502` path only fires when ALL replicas are down.
5151
//
52-
// This suite perturbs the shared kaito-system BBR Deployment, so it is
52+
// This suite perturbs the shared BBR Deployment, so it is
5353
// decorated Serial — no other spec may run while BBR is degraded.
5454
//
5555
// Scope note: the structured `502 bbr_unavailable` envelope (+
@@ -59,8 +59,7 @@ import (
5959
// It is intentionally NOT asserted here; the all-replicas-down case below
6060
// asserts only that the request fails closed (5xx, never a silent 404).
6161
var _ = Describe("BBR cluster-filter HA",
62-
Ordered, Serial, utils.GinkgoLabelOutage, utils.GinkgoLabelSmoke,
63-
utils.GinkgoLabelStandardK8sOnly, func() {
62+
Ordered, Serial, utils.GinkgoLabelOutage, utils.GinkgoLabelSmoke, func() {
6463

6564
const (
6665
bbrDeployment = "body-based-router"

‎test/e2e/cluster_status_spec.go‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -36,7 +36,7 @@ import (
3636
// These perturb cluster-wide singletons every other namespace shares, so the
3737
// suite is decorated Serial.
3838
var _ = Describe("Cluster Status Reporter",
39-
Serial, utils.GinkgoLabelStatusReporter, utils.GinkgoLabelNoGPU, utils.GinkgoLabelStandardK8sOnly, func() {
39+
Serial, utils.GinkgoLabelStatusReporter, utils.GinkgoLabelNoGPU, func() {
4040

4141
const (
4242
// Allow a couple of resyncs plus startup grace and rollout slack.

‎test/e2e/control_plane_error_spec.go‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -51,7 +51,7 @@ import (
5151
// gpu-node-mocker fakes provisioning — and the az CLI to confirm the
5252
// precondition, and skip otherwise.
5353
var _ = Describe("Control-plane Error Reporter",
54-
utils.GinkgoLabelStatusReporter, utils.GinkgoLabelNoGPU, utils.GinkgoLabelStandardK8sOnly, func() {
54+
utils.GinkgoLabelStatusReporter, utils.GinkgoLabelNoGPU, func() {
5555

5656
const (
5757
setupTimeout = 3 * time.Minute
@@ -176,7 +176,7 @@ var _ = Describe("Control-plane Error Reporter",
176176
Expect(found).To(BeFalse(), "a missing EPP Deployment must not be reported")
177177
})
178178

179-
It("passes through the terminal provisioning failure of an instance type without quota", func() {
179+
It("passes through the terminal provisioning failure of an instance type without quota", utils.GinkgoLabelStandardK8sOnly, func() {
180180
requireRealProvisioner()
181181
instanceType := envOr(envZeroQuotaInstanceType, "Standard_NV36ads_A10_v5")
182182
family := envOr(envZeroQuotaFamily, "StandardNVADSA10v5Family")
@@ -216,7 +216,7 @@ var _ = Describe("Control-plane Error Reporter",
216216
expectPassThrough(kaitoReason, isName, emitTimeout)
217217
})
218218

219-
It("passes through an AsyncProvisioningError for an instance type the subscription cannot use", func() {
219+
It("passes through an AsyncProvisioningError for an instance type the subscription cannot use", utils.GinkgoLabelStandardK8sOnly, func() {
220220
requireRealProvisioner()
221221
region, err := utils.ClusterRegion(ctx)
222222
Expect(err).NotTo(HaveOccurred())

‎test/e2e/epp_outage_spec.go‎

Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -57,8 +57,7 @@ import (
5757
// namespace's request path, not any cluster-wide singleton, so the suite
5858
// is safe to run alongside other (non-overlapping) specs.
5959
var _ = Describe("EPP outage (fail-closed InferencePool ext_proc)",
60-
Ordered, utils.GinkgoLabelOutage, utils.GinkgoLabelNightly,
61-
utils.GinkgoLabelStandardK8sOnly, func() {
60+
Ordered, utils.GinkgoLabelOutage, utils.GinkgoLabelNightly, func() {
6261

6362
var (
6463
ctx context.Context

‎test/e2e/ext_authz_outage_spec.go‎

Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -60,8 +60,7 @@ import (
6060
// authenticated request, so this suite must not run concurrently with any
6161
// other spec.
6262
var _ = Describe("ext_authz outage (fail-closed cluster filter)",
63-
Ordered, Serial, utils.GinkgoLabelOutage, utils.GinkgoLabelAuth,
64-
utils.GinkgoLabelStandardK8sOnly, func() {
63+
Ordered, Serial, utils.GinkgoLabelOutage, utils.GinkgoLabelAuth, func() {
6564

6665
const authDeploymentName = "apikey-authz"
6766

‎test/e2e/filter_order_spec.go‎

Lines changed: 2 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -197,7 +197,7 @@ var _ = Describe("Filter execution order",
197197
// actually deliver requests to the workload pod (vLLM).
198198
// ─────────────────────────────────────────────────────────────────
199199

200-
Context("P0: bbr precedes EPP, full chain delivers traffic", utils.GinkgoLabelStandardK8sOnly, func() {
200+
Context("P0: bbr precedes EPP, full chain delivers traffic", func() {
201201

202202
// B2 — Send N valid authenticated requests, then assert that
203203
// vLLM's `vllm:request_success_total{model_name=<modelName>}`
@@ -281,13 +281,8 @@ var _ = Describe("Filter execution order",
281281
// of the five filters we care about. This is the strongest
282282
// possible assertion because it does not depend on any business
283283
// request behaviour: it checks the rendered xDS config itself.
284-
//
285-
// Reaching the admin port means `kubectl exec` into the gateway
286-
// pod, which a managed control plane does not allow — hence
287-
// StandardK8sOnly. Every other spec here asserts the same
288-
// ordering through request behaviour and stays runnable.
289284
It("F1: HCM filter order is CORS → ext_authz → bbr → ext_proc → router",
290-
utils.GinkgoLabelStandardK8sOnly, func() {
285+
func() {
291286
gwPod, err := firstRunningPod(ctx, caseNS, gatewayLabel)
292287
Expect(err).NotTo(HaveOccurred(),
293288
"per-namespace Gateway pod should be Running")

‎test/e2e/gpu_mocker_spec.go‎

Lines changed: 7 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -87,7 +87,7 @@ var _ = Describe("GPU Mocker E2E", Ordered, func() {
8787

8888
Context("InferenceSet and InferencePool lifecycle", utils.GinkgoLabelInfra, func() {
8989

90-
Context("InferenceSet lifecycle", utils.GinkgoLabelStandardK8sOnly, func() {
90+
Context("InferenceSet lifecycle", func() {
9191
It("should have EPP pods running for each InferencePool", func() {
9292
clientset, err := utils.GetK8sClientset()
9393
Expect(err).NotTo(HaveOccurred())
@@ -158,7 +158,7 @@ var _ = Describe("GPU Mocker E2E", Ordered, func() {
158158
})
159159
})
160160

161-
Context("HTTPRoute status", utils.GinkgoLabelStandardK8sOnly, func() {
161+
Context("HTTPRoute status", func() {
162162
It("should have HTTPRoutes with Accepted=True condition for each deployment", func() {
163163
dynClient, err := utils.GetDynamicClient()
164164
Expect(err).NotTo(HaveOccurred())
@@ -198,7 +198,7 @@ var _ = Describe("GPU Mocker E2E", Ordered, func() {
198198

199199
Context("Fake node and shadow pod lifecycle", utils.GinkgoLabelInfra, func() {
200200

201-
Context("Fake nodes", utils.GinkgoLabelStandardK8sOnly, func() {
201+
Context("Fake nodes", func() {
202202
It("should have fake nodes with correct labels", func() {
203203
clientset, err := utils.GetK8sClientset()
204204
Expect(err).NotTo(HaveOccurred())
@@ -240,7 +240,7 @@ var _ = Describe("GPU Mocker E2E", Ordered, func() {
240240
})
241241
})
242242

243-
Context("Shadow pods", utils.GinkgoLabelStandardK8sOnly, func() {
243+
Context("Shadow pods", func() {
244244
It("should have shadow pods running in the shadow namespace", func() {
245245
clientset, err := utils.GetK8sClientset()
246246
Expect(err).NotTo(HaveOccurred())
@@ -541,7 +541,7 @@ var _ = Describe("GPU Mocker E2E", Ordered, func() {
541541
})
542542
})
543543

544-
Context("Terminating pod reaping", utils.GinkgoLabelStandardK8sOnly, func() {
544+
Context("Terminating pod reaping", func() {
545545
It("should force-delete a KAITO pod stuck Terminating on a fake node", func() {
546546
clientset, err := utils.GetK8sClientset()
547547
Expect(err).NotTo(HaveOccurred())
@@ -664,7 +664,7 @@ var _ = Describe("GPU Mocker E2E", Ordered, func() {
664664

665665
Context("Garbage collection", utils.GinkgoLabelInfra, func() {
666666

667-
Context("Fake node GC", utils.GinkgoLabelStandardK8sOnly, func() {
667+
Context("Fake node GC", func() {
668668
It("should delete orphaned fake nodes and leases when the NodeClaim is removed", func() {
669669
clientset, err := utils.GetK8sClientset()
670670
Expect(err).NotTo(HaveOccurred())
@@ -749,7 +749,7 @@ var _ = Describe("GPU Mocker E2E", Ordered, func() {
749749
})
750750
})
751751

752-
Context("Shadow pod GC", utils.GinkgoLabelStandardK8sOnly, func() {
752+
Context("Shadow pod GC", func() {
753753
It("should delete shadow pods via native Kubernetes GC when the original pod is removed", func() {
754754
clientset, err := utils.GetK8sClientset()
755755
Expect(err).NotTo(HaveOccurred())

0 commit comments

Comments
 (0)