Skip to content

deployment validator polls deprecated Skyhook CR; nodewright v0.18.0 writes status only on NodeWright #2593

Closed
#2829
Closed
deployment validator polls deprecated Skyhook CR; nodewright v0.18.0 writes status only on NodeWright#2593
#2829
Assignees
ayuskauskas
Labels
area/validatortheme/validationConstraint evaluation, health checks, and conformance evidence
Milestone
v1

Description

@atif1996
atif1996
opened on Sep 5, 2026

Summary

The deployment-phase readiness gate polls the deprecated Skyhook CR for tuning completion, but nodewright-operator v0.18.0 (the version pinned in recipes/registry.yaml) migrates every Skyhook to a NodeWright (nodewright.nvidia.com/v1alpha1) and writes completion status only on the NodeWright. The legacy Skyhook.status stays empty. So on a healthy, fully-tuned cluster the gate polls a status that never populates and times out after its 8m deadline, failing the deployment phase while tuning has genuinely completed.

Evidence (live GB300 bare-metal cluster, nodewright-operator v0.18.0)

$ kubectl get skyhook no-op -o jsonpath='{.status.status}'
                 # empty
$ kubectl get nodewright no-op -o json | jq .status
  "completeNodes": "2/2", "status": "complete",
  conditions: [NodesConverged, complete]

The API server itself warns on apply:
skyhook.nvidia.com/v1alpha1 Skyhook is deprecated; migrate to nodewright.nvidia.com/v1alpha1 NodeWright ... The Skyhook kind will be removed in a future release.

The operator also logs a reconcile conflict loop while trying (and failing) to mirror status back to the legacy object:
error updating during migration nodewright status [no-op]: the object has been modified

Where

  • validators/deployment/expected_resources.go: nodewrightGVR = {Group: "skyhook.nvidia.com", Version: "v1alpha1", Resource: "skyhooks"} and the verifyNodewrightReady poll.
  • recipes/checks/nodewright-customizations/health-check.yaml: chainsaw assert on Skyhook.status.status == "complete".

Fix options

  1. Read NodeWright (nodewright.nvidia.com/v1alpha1) — the authoritative, non-deprecated kind — with a fallback to Skyhook for older operator versions.
  2. Or gate on the operator version and select the CR accordingly.

Blocks live-hardware validation evidence for any recipe using nodewright-customizations on a v0.18.0 cluster (found while collecting GB300 evidence for #2568).

Activity

  1. atif1996
    added
    theme/validationConstraint evaluation, health checks, and conformance evidence
    on Sep 5, 2026
  2. github-actions
    added
    needs-triage
    area/validator
    and removed
    needs-triage
    on Sep 5, 2026

  3. atif1996 commented on Sep 5, 2026

    @atif1996
    atif1996
    on Sep 5, 2026
    ContributorAuthor

    Regression provenance: the validator's read of the deprecated Skyhook kind is long-standing, but it became load-bearing when the nodewright-operator pin moved v0.17.1 → v0.18.0 in #2355 (merged 0cf961fb, 2026-09-03). v0.17.1 still populated the legacy Skyhook.status; v0.18.0 migrates to NodeWright and writes status only there, with a legacy mirror that fails in a reconcile conflict loop (error updating during migration nodewright status [no-op]: the object has been modified).

    The bump itself should not be reverted — it is required on OKE, where cri-o rejects v0.17.1's short-name alpine/kubectl maintenance-hook image (NVIDIA/nodewright#481). The fix belongs on the validator side: read nodewright.nvidia.com/v1alpha1 NodeWright (with a Skyhook fallback for older operators).

    Impact today: this blocks live-hardware deployment-phase validation for any recipe carrying nodewright-customizations on a v0.18.0 cluster — found while collecting GB300 evidence for #2568, but not specific to it.

  4. atif1996 commented on Sep 5, 2026

    @atif1996
    atif1996
    on Sep 5, 2026
    ContributorAuthor

    Related: #2594 covers the upgrade-side contract change (v0.18.0 migration + broken legacy mirror). This issue remains the validator-side fix (read NodeWright, fall back to Skyhook).

  5. atif1996 commented on Sep 5, 2026

    @atif1996
    atif1996
    on Sep 5, 2026
    ContributorAuthor

    Version-disposition alternatives now tracked separately: #2595 (downgrade to v0.17.x) and #2594 (stay on v0.18.0, fix the mirror upstream). This validator-side fix is needed either way, since Skyhook is deprecated and slated for removal.

  6. atif1996
    added a commit that references this issue on Sep 7, 2026

    fix(validator): read NodeWright CR and configured runtime-required taint

    5b85420

  7. mchmarny commented on Sep 8, 2026

    @mchmarny
    mchmarny
    on Sep 8, 2026
    Member

    Triaged: P2→P1. The current nodewright v0.18.0 status contract makes healthy deployments time out in validation; PR #2597 is actively addressing it.

  8. atif1996
    added a commit that references this issue on Sep 8, 2026

    fix(validator): read NodeWright CR and configured runtime-required taint

    3b0d4ac
  9. mchmarny
    assigned
    ayuskauskas
    on Sep 8, 2026
  10. mchmarny
    added this to the v1 milestone on Sep 17, 2026

  11. mchmarny commented on Sep 17, 2026

    @mchmarny
    mchmarny
    on Sep 17, 2026
    Member

    Added to the v1 milestone.

    This gates the remaining v1 coverage work rather than sitting beside it: the deployment phase times out on any cluster running the pinned nodewright-operator v0.18.0, so neither the VR200/RKE2 evidence refresh nor a GB300 run on a tuned cluster (PR #2772) can produce deployment-phase evidence until it lands.

    ROADMAP §2 requires published deployment, conformance, and performance evidence for every Supported coordinate.

  12. ayuskauskas
    added 2 commits that reference this issue on Sep 18, 2026

    fix(validator): name an empty runtime-required taint env for what it is

    c8dc2b2

    fix(bundler): see a pre-rename operator before stripping its taints

    f0a2f6f

  13. 3 remaining items

  14. ayuskauskas
    added 14 commits that reference this issue on Sep 18, 2026

    feat(recipes): pin nodewright v0.19.0; safe boundaries stop blocking …

    96c062d

    fix(validator): name an empty runtime-required taint env for what it is

    91a9900

    fix(bundler): see a pre-rename operator before stripping its taints

    01ca42a

    fix(validator): retarget the RDMA readiness assert and probe both ope…

    cdf897c

    fix(upgrade): an all-safe crossing is safe, not blocked

    dc3a4ff

    fix(upgrade): an unassessed origin is not rescued by crossing more bo…

    5ee73cc

    fix(validator): resolve the nodewright API group by discovery; pin v0…

    c5a0e33

    feat(recipes): pin nodewright v0.19.0; safe boundaries stop blocking …

    1868a40

    fix(validator): name an empty runtime-required taint env for what it is

    94ddadf

    fix(bundler): see a pre-rename operator before stripping its taints

    e36b0b4

    fix(validator): retarget the RDMA readiness assert and probe both ope…

    3cda5f5

    fix(upgrade): an all-safe crossing is safe, not blocked

    3e5ac62

    fix(upgrade): an unassessed origin is not rescued by crossing more bo…

    7f27eb5

    fix(validator): bind nodewright discovery to the validator context

    49b379c
  15. yuanchen8911
    closed this as completedin #2829on Sep 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

area/validatortheme/validationConstraint evaluation, health checks, and conformance evidence

Type

No type

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions