Repository navigation
recipes: design grounded qualification for the OKE gpuStack profile #2363
Description
Activity
- added 5 commits that reference this issue
on Aug 29, 2026 Decision
Settled with @yuanchen8911 (Slack, 2026-08-31) and implemented in #2355 (
681a5362):Selected: Option A, narrowed to two values.
oci-managed(Oracle GPU image driver + theNvidiaGpuPluginadd-on advertising) andoperator-managed(driverless image, add-on removed, operator owns driver/toolkit/plugin). The hybrid image-driver + operator-plugin value is withdrawn — NKX confirmed no consumer — which collapses both ownership axes onto one distinguishing signal and dissolves this issue's two-axis problem the same way #2360 dissolved GKE's.Canonical signal: the add-on's control-plane state, observed — not declared:
aicr snapshot|validate --oke-addons <oci ce cluster list-addons --all --output json dump>projectsNvidiaGpuPluginintoK8s.oke-addons.nvidia-gpu-plugin(installedwhen present+ACTIVE,absentwhen removed), mirroring the AKS--aks-gpu-poolsprojection (#1967) end to end.Answers to the completion criteria:
- Option: A (two grounded facts degenerate to one under the two-value model); durable profile constraints only — no
readinessConstraints, no ClusterPolicy readback (Option C stays rejected), noGPU.hardware.driver-loaded. - Mixed/unavailable: fail closed. Any non-ACTIVE lifecycle state projects a marker no constraint accepts (
addon-<state>, named in the resolution error); a snapshot captured without the flag fails as reading-unavailable with recapture guidance; a malformed dump fails the snapshot before any cluster work. - Both phases: the same
--oke-addonsflag onaicr snapshot(generation) andaicr validate(pre-flight re-capture; ignored with a warning when--snapshotsupplies a pre-captured artifact, which must already carry the reading). - Label route / add-on interplay: per-node
oci.oraclecloud.com/disable-gpu-device-plugindisablement is out of contract — it leaves the add-on installed, which reads asoci-managed. Fleets using it (NVCF today) migrate to add-on removal, the shapeterraform-oci-okeclusteralready produces (NvidiaGpuPlugin = { remove = true }, verified live: the add-on is absent fromlist-addonson a DGXC cluster and present on NVCF's).
Deferred follow-up (the one remaining item from this issue's gating question): a supported-image consistency check (driver-bearing vs driverless image classification) as a secondary signal — not required for qualification under the two-value model, tracked here.
Leaving this open only for that follow-up; the qualification design itself is resolved.
- Option: A (two grounded facts degenerate to one under the two-value model); durable profile constraints only — no
Owner confirmation
I agree with the narrowed two-mode v1 model:
Profile NVIDIA GPU driver/toolkit owner Device-plugin owner oci-managedOKE node image OKE NvidiaGpuPluginadd-onoperator-managedGPU Operator GPU Operator The
operator-pluginhybrid remains unsupported. The current NKX/DGXC configuration maps tooperator-managed; its preinstalled MOFED driver belongs to the network stack and does not change GPU-stack ownership.For device-plugin qualification, the OKE control-plane add-on state is preferable to making
oci.oraclecloud.com/disable-gpu-device-plugin=truecanonical:- Node label: simpler and Kubernetes-visible, but it declares intent rather than proving that the OKE add-on was removed.
- Add-on projection: observes the actual control-plane state, but requires OCI API access or an exported
oci ce cluster list-addonsJSON input.
Because the supported
operator-managedcontract requires cluster-wide removal ofNvidiaGpuPlugin, the add-on projection is the stronger fail-closed qualifier. Per-pool label-based disablement remains outside the v1 contract.This resolves device-plugin ownership qualification. The supported-image consistency check for NVIDIA GPU driver ownership remains the follow-up.
- added 9 commits that reference this issue
on Aug 31, 2026 Triaged: confirmed at P2 / Backlog. Scope remains valid but is not ready for immediate execution; current placement is intentional.
- added a parent issue
on Oct 9, 2026
The problem
OKE is the only family whose
gpuStackvaries two dimensions independently:oci-defaultoperator-pluginoperator-managedGKE's coupled modes can be separated by one grounded fact, and AKS's single varying axis by one provider projection. #2355 uses two ClusterPolicy readings — but both read back fields rendered by the selected value; neither observes external advertiser or driver ownership. So each value satisfies its own readiness constraints by construction, and a wrong external mode passes pre-flight.
Gating question
OCI exposes
NodePool.nodeSourceDetails.imageId, but OKE custom images can carry arbitrary modifications, so an OCID alone does not establish driver ownership. Resolving this likely needs an allowlist of known image OCIDs, an image tag or naming contract, provisioning metadata, or an explicit supported-image contract.Second assumption, not yet resolved
The advertiser axis is not settled either. The
oci.oraclecloud.com/disable-gpu-device-plugin=truelabel covers the node-label disablement route, but DGXC disables the plugin by removing the add-on, which leaves no node marker. Either NKX-9804 makes the label the canonical contract even under add-on removal, or AICR needs a normalized projection combining cluster add-on state with per-pool labels — an add-on-state projection alone does not cover per-pool label opt-outs while the add-on remains installed. The choice below is not decided by the driver signal alone.Options
--aks-gpu-poolspattern. All three values qualify on ordinary durable profile constraints — evaluated at generation when a snapshot is supplied and re-evaluated at validation. NoreadinessConstraints, no new phase. Needs both assumptions resolved.oci-defaultandoperator-plugin, both restricted to driver-bearing images and requiringGPU.hardware.driver-loaded=true; distinguish them by the advertiser fact. Deferoperator-managed. Needs one distinguishing fact, but still needs that common driver-present invariant.driver-loadedusable on fresh pools, but mis-selects when regenerating from an already-deployed cluster. Not viable as the primary mechanism.What is needed to decide
Representative evidence for every intended class — not another PR revision:
Completion criteria
This issue is done when it:
Candidate ADR-015 amendment
To land separately, in its own PR, once this resolves:
Related
#2355, #2356, #2347, #2359, NKX-9804, and ADR-015 (
docs/design/015-recipe-configuration-profiles.md)