Repository navigation
Commit 433be0f
test(e2e): narrow StandardK8sOnly to specs a managed control plane cannot run (#262)
* test(e2e): narrow StandardK8sOnly to specs a managed control plane cannot run
With cluster-admin access on an AKS AI Manager cluster, 57 of the 82
StandardK8sOnly specs pass: scaling shared Deployments, kubectl exec into
the Gateway and scraping pod metrics through pods/proxy all work. Drop the
label from those specs and keep it only where a managed control plane
still cannot run them:
- ModelDeploymentValues a managed backend cannot express: scaling
(ScalingMetrics, EPPScorerWeights), model_routing load distribution
(EPPScorerWeights), modeldeployment_chart autoUpgrade (AutoUpgrade).
- State reconciled away by a managed controller: the sustained-slow
ModelMirror case of model_download_slow.
- Not yet verified: the cloud provisioning-failure cases of
control_plane_error (skipped under gpu-node-mocker) and the EPP metrics
spec (skipped everywhere).
Update the label documentation to describe these reasons.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: rambohe-ch <rambohe.ch@gmail.com>
* test(e2e): unskip the namespace discovery-label check on AI Manager
AI Manager now stamps productionstack.kaito.sh/managed-by: modelharness on
the namespaces it provisions, and the spec passes on a live AI Manager
cluster.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: rambohe-ch <rambohe.ch@gmail.com>
* test(e2e): enable the EPP metrics spec and drop the dead debug-filter spec
EPP routing success (metrics): the Skip blamed an Istio sidecar for a 401
on the EPP metrics port. The modeldeployment chart has passed
--metrics-endpoint-auth=false since it was introduced, and EPP pods run
without a sidecar. The spec passes on an AKS AI Manager cluster, so remove
the Skip and its StandardK8sOnly label.
Debug EnvoyFilter log chain: the inference-debug-filter manifest it relies
on was removed in #46 and nothing installs it, and its Gateway pod
selector predates per-case gateways. It could not pass on any backend, so
delete it together with findDebugLogTriple and its scenario entry.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: rambohe-ch <rambohe.ch@gmail.com>
* test(e2e): run load distribution and autoUpgrade on AI Manager
Both specs pass on an AKS AI Manager cluster, which now accepts
EPPScorerWeights and AutoUpgrade through modeldeployment overrides, so
drop their StandardK8sOnly labels.
Restore the load-distribution scorer weights in an AfterEach instead of a
DeferCleanup registered by the BeforeEach. Ginkgo runs AfterAll before
cleanups registered by the last spec of an Ordered container, so a focused
or filtered run in which load distribution was the last spec tried to
restore weights on deployments AfterAll had already removed.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: rambohe-ch <rambohe.ch@gmail.com>
* test(e2e): run the scaling specs on AI Manager
scaling (0 -> 1 -> 2 -> 1 -> 0 and anti-flapping, 17 specs) and
scaling_transitions (manual <-> autoscaling, min 0 parking, 9 specs) pass
on an AKS AI Manager cluster with cluster-admin access, where the backend
maps the chart's scaling values onto the AI Manager scale profile and
scaling overrides. Patching an InferenceSet's replicas works there too.
Drop their StandardK8sOnly labels and the caller requirement on
SetInferenceSetReplicas.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: rambohe-ch <rambohe.ch@gmail.com>
* test(e2e): label model_unavailable StandardK8sOnly
It empties the inference pool with a manual replica count of 0. AKS AI
Manager rejects that (scale.manual.replicas must be >= 1), and its only
zero-replica mode, autoscaling with minReplicas 0, wakes on the requests
the spec sends instead of staying empty. Update the label docs, which
cited this spec as one any backend can honour.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: rambohe-ch <rambohe.ch@gmail.com>
* test(e2e): run the ModelMirror download specs where KAITO reconciles mirrors
The mock ModelMirror was Static, which KAITO's ModelMirror controller marks
Ready immediately, so on clusters with the ModelMirror feature gate on (AKS
AI Manager) the sustained-slow spec never saw a slow download and the
dip and partial-window specs passed without testing anything.
Create the mock as a Managed mirror on a StorageClass with no provisioner
instead. KAITO creates the PVC and download Job, but the PVC never binds
and the Job pod is never scheduled, so the controller has no progress to
report and leaves the status the spec patches untouched. Nothing is
downloaded and no node capacity is reserved. With the gate off nothing
reconciles the mirror, as before.
Read the reporter's --event-refresh-seconds instead of assuming the E2E
install's 120s: a managed install may keep the chart default of 300s, and
the refresh and quiet windows must outlast it. At 120s the windows are
unchanged.
All four model_download_slow specs pass on an AI Manager cluster; drop the
StandardK8sOnly label from the sustained-slow spec.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: rambohe-ch <rambohe.ch@gmail.com>
* test(e2e): label the exact-origin CORS context StandardK8sOnly
It configures an exact-origin CORS allowlist on the modelharness. AKS AI
Manager installs the harness with the chart's wildcard default and has no
setting for it, so the context fails there and, being in the same Ordered
container, skips the wildcard context. The wildcard context matches that
default and passes 12/12 on an AI Manager cluster.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: rambohe-ch <rambohe.ch@gmail.com>
---------
Signed-off-by: rambohe-ch <rambohe.ch@gmail.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>1 parent fd8d6d2 commit 433be0f
26 files changed
Lines changed: 192 additions & 254 deletions
File tree
- test/e2e
- utils
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
92 | 92 | | |
93 | 93 | | |
94 | 94 | | |
95 | | - | |
96 | | - | |
97 | | - | |
98 | | - | |
99 | | - | |
100 | | - | |
101 | | - | |
102 | | - | |
103 | | - | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
104 | 104 | | |
105 | 105 | | |
106 | 106 | | |
107 | 107 | | |
108 | 108 | | |
109 | | - | |
110 | | - | |
111 | | - | |
112 | | - | |
113 | | - | |
114 | | - | |
115 | | - | |
116 | | - | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
117 | 112 | | |
118 | 113 | | |
119 | | - | |
120 | | - | |
121 | | - | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
122 | 117 | | |
123 | 118 | | |
124 | 119 | | |
| |||
271 | 266 | | |
272 | 267 | | |
273 | 268 | | |
274 | | - | |
| 269 | + | |
275 | 270 | | |
276 | 271 | | |
277 | 272 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
175 | 175 | | |
176 | 176 | | |
177 | 177 | | |
178 | | - | |
| 178 | + | |
| 179 | + | |
| 180 | + | |
| 181 | + | |
| 182 | + | |
179 | 183 | | |
180 | 184 | | |
181 | 185 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
55 | 55 | | |
56 | 56 | | |
57 | 57 | | |
58 | | - | |
| 58 | + | |
59 | 59 | | |
60 | 60 | | |
61 | 61 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
49 | 49 | | |
50 | 50 | | |
51 | 51 | | |
52 | | - | |
| 52 | + | |
53 | 53 | | |
54 | 54 | | |
55 | 55 | | |
| |||
59 | 59 | | |
60 | 60 | | |
61 | 61 | | |
62 | | - | |
63 | | - | |
| 62 | + | |
64 | 63 | | |
65 | 64 | | |
66 | 65 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
36 | 36 | | |
37 | 37 | | |
38 | 38 | | |
39 | | - | |
| 39 | + | |
40 | 40 | | |
41 | 41 | | |
42 | 42 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
51 | 51 | | |
52 | 52 | | |
53 | 53 | | |
54 | | - | |
| 54 | + | |
55 | 55 | | |
56 | 56 | | |
57 | 57 | | |
| |||
176 | 176 | | |
177 | 177 | | |
178 | 178 | | |
179 | | - | |
| 179 | + | |
180 | 180 | | |
181 | 181 | | |
182 | 182 | | |
| |||
216 | 216 | | |
217 | 217 | | |
218 | 218 | | |
219 | | - | |
| 219 | + | |
220 | 220 | | |
221 | 221 | | |
222 | 222 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
57 | 57 | | |
58 | 58 | | |
59 | 59 | | |
60 | | - | |
61 | | - | |
| 60 | + | |
62 | 61 | | |
63 | 62 | | |
64 | 63 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
60 | 60 | | |
61 | 61 | | |
62 | 62 | | |
63 | | - | |
64 | | - | |
| 63 | + | |
65 | 64 | | |
66 | 65 | | |
67 | 66 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
197 | 197 | | |
198 | 198 | | |
199 | 199 | | |
200 | | - | |
| 200 | + | |
201 | 201 | | |
202 | 202 | | |
203 | 203 | | |
| |||
281 | 281 | | |
282 | 282 | | |
283 | 283 | | |
284 | | - | |
285 | | - | |
286 | | - | |
287 | | - | |
288 | | - | |
289 | 284 | | |
290 | | - | |
| 285 | + | |
291 | 286 | | |
292 | 287 | | |
293 | 288 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
87 | 87 | | |
88 | 88 | | |
89 | 89 | | |
90 | | - | |
| 90 | + | |
91 | 91 | | |
92 | 92 | | |
93 | 93 | | |
| |||
158 | 158 | | |
159 | 159 | | |
160 | 160 | | |
161 | | - | |
| 161 | + | |
162 | 162 | | |
163 | 163 | | |
164 | 164 | | |
| |||
198 | 198 | | |
199 | 199 | | |
200 | 200 | | |
201 | | - | |
| 201 | + | |
202 | 202 | | |
203 | 203 | | |
204 | 204 | | |
| |||
240 | 240 | | |
241 | 241 | | |
242 | 242 | | |
243 | | - | |
| 243 | + | |
244 | 244 | | |
245 | 245 | | |
246 | 246 | | |
| |||
541 | 541 | | |
542 | 542 | | |
543 | 543 | | |
544 | | - | |
| 544 | + | |
545 | 545 | | |
546 | 546 | | |
547 | 547 | | |
| |||
664 | 664 | | |
665 | 665 | | |
666 | 666 | | |
667 | | - | |
| 667 | + | |
668 | 668 | | |
669 | 669 | | |
670 | 670 | | |
| |||
749 | 749 | | |
750 | 750 | | |
751 | 751 | | |
752 | | - | |
| 752 | + | |
753 | 753 | | |
754 | 754 | | |
755 | 755 | | |
| |||
0 commit comments