fix(recipes): roll nodewright-operator back to v0.17.1; qualify hook image for OKE - #2633
Merged
Merged
Conversation
…ook image for OKE v0.18.0 renames Skyhook to NodeWright and writes status only on the new kind, so the deployment readiness gate times out on every recipe carrying nodewright-customizations, and upstream documents the upgrade as a manual migration AICR does not yet guide. Pin back to v0.17.1 until that guidance lands. OKE's cri-o rejects the v0.17.1 chart's short-name alpine/kubectl hook image (NVIDIA/nodewright#481); webhook.removalImage is set to docker.io/alpine/kubectl with the chart's tag and digest pins unchanged. Fixes #2595 Signed-off-by: Atif Mahmood <atif1996@users.noreply.github.com>
atif1996
added
the
theme/recipes
Contributor
|
🌿 Preview your docs: https://nvidia-preview-fix-nodewright-pin-rollback.docs.buildwithfern.com/aicr |
Contributor
Recipe evidence check
Protected recipesRecipes with committed evidence (
Other affected recipes without evidence yet: 71These recipes are affected by this PR but carry no committed evidence pointer, so there is
How to refresh evidenceRun on a cluster matching the recipe's aicr snapshot -o snapshot.yaml
# Profiled families (AKS/GKE gpuStack): hydrate the recipe with the
# pointer's recorded 'profile:' selection first — validating the raw
# overlay resolves only the declaration default, and 'aicr validate'
# has no --profile flag. AKS additionally needs the pool projection
# (GKE uses the plain snapshot above):
# az aks nodepool list -g <rg> --cluster-name <cluster> -o json > pools.json
# aicr snapshot --aks-gpu-pools pools.json -o snapshot.yaml
# aicr recipe -s snapshot.yaml --intent <intent> [--platform <platform>] \
# --profile <name>=<value> -o recipe.yaml
# State the target leaf's intent/platform explicitly (the snapshot
# fingerprint supplies service/accelerator/OS but intent and platform
# default to 'any') and pass -r recipe.yaml below instead of the raw
# overlay.
aicr validate \
-r recipes/overlays/<slug>.yaml \
-s snapshot.yaml \
--emit-attestation ./out \
--push ghcr.io/<your-fork>/aicr-evidence
# Copy to the per-source path printed in the emit 'copyTo' hint:
# recipes/evidence/<slug>/<source>/<bundle-digest>.yamlThis gate is warning-only and never blocks merge. See ADR-007 for the trust model. |
ayuskauskas
approved these changes
Sep 8, 2026
Contributor
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: ASSERTIVE Plan: Enterprise Run ID: 📒 Files selected for processing (5)
Included review availability: Your plan provides up to 12 included reviews per hour; 11 remain after this review. 📝 WalkthroughWalkthroughThe NodeWright operator default chart version changes from Priority: ➖ Normal — Schedule the chart rollback because v0.18.0 times out deployment readiness gates and the OKE image fix affects recipes using nodewright-customizations. Estimated code review effort: 2 (Simple) | ~10 minutes Severity of issue fixed: Medium Merge Risk: ⚪ Minimal · up to This restores the NodeWright v0.17.1 compatibility contract and uses a fully qualified webhook-removal image for OKE. The generated recipe data and image inventory are updated consistently, with no current merge-blocking risk identified. Suggested reviewers: 🚥 Pre-merge checks | ✅ 3 | ❌ 1❌ Failed checks (1 inconclusive)
✅ Passed checks (3 passed)
Full details: Linked Issues checkExplanation The changes satisfy the pinning, OKE image, and regenerated output requirements from issue ✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
Contributor
Coverage Report ✅
Coverage BadgeNo Go source files changed in this PR. |
mchmarny
requested changes
Sep 8, 2026
mchmarny
approved these changes
Sep 8, 2026
This was referenced Sep 17, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Roll the
nodewright-operatorpin back fromv0.18.0tov0.17.1and keep OKE working by fully qualifying the maintenance-hook image (webhook.removalImage: docker.io/alpine/kubectl) in the component values.Motivation / Context
The
v0.17.1→v0.18.0bump landed as part of the OKE gpuStack PR (#2355). v0.18.0 renamesSkyhooktoNodeWrightand writesstatus.statusonly on the new kind, so every recipe carryingnodewright-customizationstimes out the deployment readiness gate on a healthy cluster; this took down all AKS/EKS/GKE UAT lanes since 9/4 (RCCA in the v0.21 release thread). Upstream also documents the v0.18.0 upgrade as a multi-step manual migration that AICR does not yet guide (design 021), so the team decided not to release on v0.18.0 until that guidance exists.The bump was not cosmetic for OKE: cri-o rejects v0.17.1's short-name
alpine/kubectlhook image (NVIDIA/nodewright#481). The v0.17.1 chart exposes the repository aswebhook.removalImage, so setting it todocker.io/alpine/kubectl(tag and digest unchanged at the chart's pins) resolves that without the bump.Fixes: #2595
Related: #2355, #2593, #2594, #2597
Type of Change
Component(s) Affected
pkg/recipe)recipes/registry.yaml,recipes/components/nodewright-operator/values.yamlImplementation Notes
v0.18.0→v0.17.1, with a comment pointing at recipes: downgrade nodewright-operator pin to v0.17.x to restore the Skyhook status contract #2595 so the next bump waits for the upgrade guidance.webhook.removalImage: docker.io/alpine/kubectlin the shared component values (applies to every recipe; the fully qualified name is correct everywhere, not just OKE). The BOM confirms the rendered hook image isdocker.io/alpine/kubectl:1.36.2@sha256:01d138ce….docs/user/container-images.mdregenerated (make bom-docs); the BOM version column is CI-gated.NodeWrightwith aSkyhookfallback, taint gate from the operator's configured value) remains open as forward-compatibility work for when the pin does move; it is not required for this rollback.Testing
No Go source changes, so no coverage delta.
Risk Assessment
Rollout notes: No AICR release has shipped v0.18.0; the bump only ever existed on
main(9/3 → now), so this is a mitigation, not a downgrade for released users. Without it, v0.21 would have been the first release to carry v0.18.0 and every user upgrading from v0.20 would have hit the multi-step upstream migration (design 021) with no AICR guidance. The only clusters that see a chart downgrade aremain-tracking test and UAT clusters deployed in that window; upstream keeps the legacyskyhook.nvidia.comstate for a rollback window (LEGACY_CLEANUP_DELAY, default 24h). Fresh installs are unaffected.Checklist
make testwith-race)make lint)git commit -S) — GPG signing info