Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
7c8892e
fix(bundler): recognize OCP component aliases in DRA rollout wait, ch…
Aug 22, 2026
4034f18
test: cover OCP alias fallback in DRA annotation and driver-absent re…
Aug 22, 2026
e378fe8
fix: address review — OCP DaemonSet name-prefix match, per-component …
Aug 22, 2026
a29142c
fix: gate DRA kubelet plugin restart on migration-wait fail-closed state
Aug 22, 2026
1af93d2
fix(deploy.sh): block DRA restart when driver-managed and migration g…
Aug 24, 2026
1bd133e
test: cover DRA restart gating on DriverOperatorManaged
Aug 25, 2026
dffca83
fix: address njhensley review — needs-retry exit signal, OCP render c…
mohityadav8 Aug 27, 2026
7437732
fix(deploy.sh): block DRA restart when migration wait times out
mohityadav8 Sep 3, 2026
95879b3
fix: define missing OCP consts in bundler.go, regenerate deploy.sh go…
mohityadav8 Sep 3, 2026
ddd53a0
test(bundler): regenerate deploy.sh goldens after rebase onto upstrea…
mohityadav8 Sep 20, 2026
4ccf203
test(bundler): regenerate deploy.sh goldens
mohityadav8 Sep 20, 2026
dd414fa
fix fix fix review
mohityadav8 Oct 2, 2026
0085b5a
Merge branch 'main' into fix/2135-ocp-alias-coverage
mohityadav8 Oct 2, 2026
f37c671
test: regenerate readiness-gate and stock-render goldens
mohityadav8 Oct 2, 2026
0db0d30
fix(deploy.sh.tmpl): restore missing fi closing
mohityadav8 Oct 2, 2026
254eb20
Merge remote-tracking branch 'origin/main' into fix/2135-ocp-alias-co…
mohityadav8 Oct 2, 2026
61589f6
test(deploy.sh.tmpl): add bash -n syntax check for canonical and OCP …
mohityadav8 Oct 2, 2026
1df25b7
Merge branch 'main' into fix/2135-ocp-alias-coverage
mchmarny Oct 2, 2026
78abe52
DRA restart
mohityadav8 Oct 3, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/user/component-catalog.md
Original file line number Diff line number Diff line change
Expand Up @@ -59,7 +59,7 @@ The source of truth is [`recipes/registry.yaml`](https://github.com/NVIDIA/aicr/
| **cert-manager-ocp-olm** | OLM installer for cert-manager on OpenShift. Creates the OperatorGroup and Subscription resources that install the certified cert-manager Operator via the Operator Lifecycle Manager. Paired with `cert-manager-ocp`. OCP-specific. | [cert-manager (Certified)](https://catalog.redhat.com/software/container-stacks/detail/5ec3f5a5eebc3d6acb0ee71c) |
| **cert-manager-ocp** | cert-manager CertManager CR for OpenShift. The operand Deployments (controller, cainjector, webhook) land in a hardcoded `cert-manager` namespace regardless of the operator's own namespace. Deployed after `cert-manager-ocp-olm`. OCP-specific. | [cert-manager](https://github.com/cert-manager/cert-manager) |
| **prometheus-adapter-ocp** | Prometheus Adapter for OpenShift. Reuses the same upstream chart as `prometheus-adapter`, pointed at OCP's built-in Thanos Querier instead of kube-prometheus-stack (which stays disabled on OCP). No certified OCP operator exists for this component. OCP-specific. | [prometheus-adapter](https://github.com/kubernetes-sigs/prometheus-adapter) |
| **nvidia-dra-driver-gpu-ocp** | NVIDIA DRA GPU driver for OpenShift. Reuses the same upstream chart as `nvidia-dra-driver-gpu`, with an added SCC RoleBinding granting the kubelet-plugin DaemonSet the host device access OCP's default restricted-v2 SCC forbids. No certified OCP operator exists for this component. OCP-specific. Known limitation: some GPU-driver rollout protections and remedy hints do not yet cover the OCP aliases (`gpu-operator-ocp`, `nvidia-dra-driver-gpu-ocp`) — the deployer's stale-NVML migration wait/restart, driver-version annotation injection, and the driver-absent remedy's `gpuoperator:`/`dradriver:` override keys; tracked in [#2136](https://github.com/NVIDIA/aicr/issues/2136). | [NVIDIA DRA Driver](https://github.com/kubernetes-sigs/dra-driver-nvidia-gpu) |
| **nvidia-dra-driver-gpu-ocp** | NVIDIA DRA GPU driver for OpenShift. Reuses the same upstream chart as `nvidia-dra-driver-gpu`, with an added SCC RoleBinding granting the kubelet-plugin DaemonSet the host device access OCP's default restricted-v2 SCC forbids. No certified OCP operator exists for this component. OCP-specific. Known limitation: the driver-version annotation injected onto the DRA pod templates falls back to the `gpu-operator-ocp-olm` Subscription channel, which changes on a channel re-pin but not on every in-channel OLM auto-upgrade — so the stale-NVML rollout gate (#973) can still miss an in-channel driver bump on OCP; tracked in [#2135](https://github.com/NVIDIA/aicr/issues/2135). | [NVIDIA DRA Driver](https://github.com/kubernetes-sigs/dra-driver-nvidia-gpu) |
Comment thread
mohityadav8 marked this conversation as resolved.
Comment thread
yuanchen8911 marked this conversation as resolved.
| **k8s-nim-operator-ocp** | NVIDIA NIM Operator for OpenShift. Reuses the same upstream chart as `k8s-nim-operator`, with OCP-specific RBAC. Requires `cert-manager-ocp` for admission-webhook TLS. OCP-specific. | [K8s NIM Operator](https://github.com/NVIDIA/k8s-nim-operator) |

## VR200 Preview coverage
Expand Down
39 changes: 33 additions & 6 deletions pkg/bundler/bundler.go
Original file line number Diff line number Diff line change
Expand Up @@ -3418,11 +3418,13 @@ const draChartVersionAnnotation = header.Domain + "/gpu-operator-chart-version"
// filtered resolved recipe before derived values are written; recipes that
// disable either remain untouched.
const (
gpuOperatorComponentName = "gpu-operator"
draComponentName = "nvidia-dra-driver-gpu"
draEvictionEnvName = "NODE_LABEL_FOR_GPU_POD_EVICTION"
draEvictionNodeSelectorPath = "kubeletPlugin.nodeSelector"
gpuOperatorDRAEvictionEnvPath = "driver.manager.env"
gpuOperatorComponentName = "gpu-operator"
gpuOperatorOCPComponentName = "gpu-operator-ocp"
gpuOperatorOCPOLMComponentName = "gpu-operator-ocp-olm"
draComponentName = "nvidia-dra-driver-gpu"
draEvictionEnvName = "NODE_LABEL_FOR_GPU_POD_EVICTION"
draEvictionNodeSelectorPath = "kubeletPlugin.nodeSelector"
gpuOperatorDRAEvictionEnvPath = "driver.manager.env"

// draNodeLabelerComponentName is the manifest-only component that mirrors
// GFD's nvidia.com/gpu.present onto the eviction label, so the label is
Expand All @@ -3442,7 +3444,7 @@ const (
)

var (
gpuOperatorComponentNames = []string{gpuOperatorComponentName, "gpu-operator-ocp"}
gpuOperatorComponentNames = []string{gpuOperatorComponentName, gpuOperatorOCPComponentName}
draComponentNames = []string{draComponentName, "nvidia-dra-driver-gpu-ocp"}
)

Expand Down Expand Up @@ -3972,6 +3974,31 @@ func (b *DefaultBundler) injectDRAChartVersionAnnotation(
// is exercised by the disabled-component unit tests.
return
}
if gpuOperatorComponentName == gpuOperatorOCPComponentName && gpuOperatorVersion == "" {
Comment thread
yuanchen8911 marked this conversation as resolved.
// gpu-operator-ocp is a ClusterPolicy CR, not a Helm chart, so
// ComponentRef.Version is never populated for it — the empty
// check below would always skip injection on OCP. Fall back to
// the OLM Subscription channel (gpu-operator-ocp-olm) as the
// rollout-trigger value instead.
//
// KNOWN LIMITATION: the channel pin (e.g. "v25.10") only
// changes on a channel re-pin, not on every operator update.
// With installPlanApproval: Automatic (the default —
// components/gpu-operator-ocp-olm/values.yaml), OLM can
// upgrade to newer CSVs inside the same channel — reloading
// the driver — without the channel string changing, so this
// annotation catches bundle-driven operator bumps (a recipe
// regenerated against a different channel) but NOT in-channel
// auto-upgrades. The stale-NVML gap this annotation exists to
// close (#973) remains open for that case on OCP. See #2135.
if olmValues, ok := componentValues[gpuOperatorOCPOLMComponentName]; ok {
if sub, ok := olmValues["subscription"].(map[string]any); ok {
if channel, ok := sub["channel"].(string); ok {
gpuOperatorVersion = channel
}
}
}
}
Comment thread
mohityadav8 marked this conversation as resolved.
if gpuOperatorVersion == "" {
// gpu-operator is enabled but the resolver produced an empty
// Version string. This shouldn't happen in normal recipe
Expand Down
41 changes: 41 additions & 0 deletions pkg/bundler/bundler_dra_annotation_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -193,6 +193,47 @@ func TestInjectDRAChartVersionAnnotation_PreservesExistingValues(t *testing.T) {
}
}

// TestInjectDRAChartVersionAnnotation_OCPFallbackToOLMChannel pins the
// OCP fallback added for #2135: gpu-operator-ocp is a ClusterPolicy
// CR, not a Helm chart, so ComponentRef.Version is always empty for
// it. Instead of skipping injection (the pre-fix behavior), the
// helper reads the OLM Subscription channel from the
// gpu-operator-ocp-olm component's values and mirrors that onto both
// nvidia-dra-driver-gpu-ocp pod templates.
func TestInjectDRAChartVersionAnnotation_OCPFallbackToOLMChannel(t *testing.T) {
b, err := New()
if err != nil {
t.Fatalf("New() error = %v", err)
}

const draOCPComponentName = "nvidia-dra-driver-gpu-ocp"
componentValues := map[string]map[string]any{
gpuOperatorOCPComponentName: {},
draOCPComponentName: {},
gpuOperatorOCPOLMComponentName: {
"subscription": map[string]any{
"channel": "v25.10",
},
},
}
rr := &recipe.RecipeResult{
ComponentRefs: []recipe.ComponentRef{
{Name: gpuOperatorOCPComponentName, Version: ""},
{Name: draOCPComponentName, Version: "0.4.1"},
},
}

b.injectDRAChartVersionAnnotation(componentValues, rr)

for _, podPath := range []string{"controller", "kubeletPlugin"} {
got := dig(componentValues[draOCPComponentName], podPath, "podAnnotations", draChartVersionAnnotation)
if got != "v25.10" {
t.Errorf("podAnnotations[%s][%s] = %v, want v25.10 (OLM channel fallback)",
podPath, draChartVersionAnnotation, got)
}
}
}

// TestInjectDRAChartVersionAnnotation_OverridesUserSet pins the
// "internal annotation always reflects the actual chart version"
// invariant. A user --set that wrote a stale value into the
Expand Down
64 changes: 56 additions & 8 deletions pkg/bundler/deployer/helm/helm.go
Original file line number Diff line number Diff line change
Expand Up @@ -62,6 +62,15 @@ type ComponentData struct {
IsOCI bool
Tag string // Git ref for Kustomize-typed components (tag/branch/commit)
Path string // Path within the repository to the kustomization

// DriverOperatorManaged is true when the bundle's effective values
// select an operator-managed NVIDIA driver — gpu-operator's or
// gpu-operator-ocp's driver.enabled is true. deploy.sh's DRA
// migration-wait block (see #2135, #973) uses this to tell "driver
// is host-managed" apart from "driver is operator-managed but the
// DaemonSet/node-label migration signal isn't observable yet",
// which live cluster state alone cannot distinguish.
DriverOperatorManaged bool
}

// compile-time interface check
Expand Down Expand Up @@ -287,6 +296,42 @@ func (g *Generator) Generate(ctx context.Context, outputDir string) (*deployer.O

// buildComponentDataList builds a sorted list of ComponentData from the recipe.
// It validates that all component names are safe for use as directory names.
// driverOperatorManaged reports whether this bundle's effective values
// select an operator-managed NVIDIA driver: gpu-operator's or
// gpu-operator-ocp's driver.enabled is true. Checks both component names
// since only one is ever enabled in a given recipe (see
// pkg/bundler/bundler.go's gpuOperatorComponentNames for the canonical
// list this mirrors).
// gpuOperatorComponentName and gpuOperatorOCPComponentName are this
// package's copy of the canonical/OCP gpu-operator component names (a
// 4th duplicate alongside pkg/bundler/bundler.go, pkg/bundler/validations
// /checks.go, and their override-key constants — this package cannot
// import pkg/bundler due to the dependency cycle noted at
// componentOverrideKeys' godoc equivalent). Named here, rather than an
// inline literal, so a `grep gpuOperatorOCPComponentName` across the repo
// surfaces every copy that needs updating together.
const (
gpuOperatorComponentName = "gpu-operator"
gpuOperatorOCPComponentName = "gpu-operator-ocp"
)

func (g *Generator) driverOperatorManaged() bool {
for _, name := range []string{gpuOperatorComponentName, gpuOperatorOCPComponentName} {
values, ok := g.ComponentValues[name]
if !ok {
Comment thread
yuanchen8911 marked this conversation as resolved.
continue
}
driver, ok := values["driver"].(map[string]any)
if !ok {
continue
}
if enabled, ok := driver["enabled"].(bool); ok && enabled {
return true
}
}
return false
}

// Only the fields consumed by the orchestration templates are populated.
func (g *Generator) buildComponentDataList() ([]ComponentData, error) {
// Sort by deployment order
Expand All @@ -295,6 +340,8 @@ func (g *Generator) buildComponentDataList() ([]ComponentData, error) {
g.RecipeResult.DeploymentOrder,
)

driverOperatorManaged := g.driverOperatorManaged()

components := make([]ComponentData, 0, len(sorted))
for _, ref := range sorted {
if !deployer.IsSafePathComponent(ref.Name) {
Expand All @@ -305,14 +352,15 @@ func (g *Generator) buildComponentDataList() ([]ComponentData, error) {
chartName := ref.EffectiveChart()

components = append(components, ComponentData{
Name: ref.Name,
Namespace: ref.Namespace,
Repository: ref.Source,
ChartName: chartName,
Version: ref.Version,
IsOCI: strings.HasPrefix(ref.Source, "oci://"),
Tag: ref.Tag,
Path: ref.Path,
Name: ref.Name,
Namespace: ref.Namespace,
Repository: ref.Source,
ChartName: chartName,
Version: ref.Version,
IsOCI: strings.HasPrefix(ref.Source, "oci://"),
Tag: ref.Tag,
Path: ref.Path,
DriverOperatorManaged: driverOperatorManaged,
})
}

Expand Down
Loading
Loading