Repository navigation
recipes: downgrade nodewright-operator pin to v0.17.x to restore the Skyhook status contract #2595
Description
Summary
Pin nodewright-operator back to v0.17.x in recipes/registry.yaml, reverting the v0.17.1 → v0.18.0 bump made in #2355.
v0.18.0 migrates skyhook.nvidia.com/v1alpha1 Skyhook CRs to nodewright.nvidia.com/v1alpha1 NodeWright and writes completion status only on the new kind; its legacy-status mirror fails in a reconcile conflict loop, so Skyhook.status never populates. AICR's deployment readiness gate polls the legacy CR, so every recipe carrying nodewright-customizations fails the deployment phase on a v0.18.0 cluster even when node tuning has fully completed (observed live on bare-metal GB300: NodeWright reports complete 2/2, Skyhook.status is {}).
Downgrading restores a working status contract immediately, without waiting on an upstream operator fix or a validator change.
Constraint to resolve before merging
The v0.18.0 bump was not cosmetic: OKE's cri-o rejects v0.17.1's short-name alpine/kubectl maintenance-hook image (NVIDIA/nodewright#481). A straight revert to v0.17.1 reintroduces that failure on OKE.
Options:
- Pin a
v0.17.xpatch that carries the fully-qualified maintenance-hook image, if one exists. - Pin
v0.17.xglobally and override the image for the OKE family. - Hold the downgrade if neither is available, and track the fix under the alternatives below.
Alternatives (tracked separately)
- nodewright v0.18.0 upgrade changed the CR status contract (Skyhook -> NodeWright); legacy status never populates #2594 — keep v0.18.0 and fix the broken legacy-status mirror upstream in NVIDIA/nodewright.
- deployment validator polls deprecated Skyhook CR; nodewright v0.18.0 writes status only on NodeWright #2593 — validator-side fix: read
NodeWrightwith aSkyhookfallback. Needed regardless of the pinned version, sinceSkyhookis deprecated and slated for removal.
Acceptance criteria
-
nodewright-operatordefaultVersionpinned to av0.17.xthat works on OKE cri-o - Goldens/BOM regenerated;
make qualifygreen - Deployment-phase validation passes on a live cluster with
nodewright-customizationsenabled
Activity
atif1996 commentedon Sep 5, 2026
Upstream version research (2026-09-05) — this narrows the options materially:
No
v0.17.xcarries the OKE fix. The short-name fix is NVIDIA/nodewright#490 (fix(chart): fully qualify alpine/kubectl image for short-name-enforcing runtimes), merged 2026-08-17 ontomainand first shipped inchart/v0.18.0(released the same day). The newestv0.17.xischart/v0.17.1(2026-06-26), two months earlier. So option 1 from the description — "pin a v0.17.x patch that carries the qualified image" — does not exist; a downgrade must pair with an OKE image override, or accept the OKE cri-o breakage.v0.19.0shipped 2026-09-04 (chart and operator), one day after we picked up v0.18.0. Its published notes cover a drain-completion fix andruntimeRequiredCordonAfter; they do not mention the Skyhook→NodeWright legacy-status mirror, so whether it fixes this regression is unverified. Worth testing before committing to a downgrade — if v0.19.0 mirrors status correctly, a forward bump is cleaner than a downgrade plus an image override.Suggested order of investigation:
v0.19.0on a live cluster — doesSkyhook.statuspopulate?v0.17.1with an OKE-family image override for the migration hook, and keep nodewright v0.18.0 upgrade changed the CR status contract (Skyhook -> NodeWright); legacy status never populates #2594 open upstream.mchmarny commentedon Sep 8, 2026
Triaged: confirmed at unset / Backlog. Scope remains valid but is not ready for immediate execution; current placement is intentional.