-
Notifications
You must be signed in to change notification settings - Fork 110
feat(recipes): OKE RDMA fabric wiring (L40S RoCE + GB200 IB) #2356
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
3 commits
Select commit
Hold shift + click to select a range
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
62 changes: 62 additions & 0 deletions
62
recipes/components/network-operator/manifests/nic-cluster-policy-oke-gb200.yaml
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,62 @@ | ||
| # Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. | ||
| # | ||
| # Licensed under the Apache License, Version 2.0 (the "License"); | ||
| # you may not use this file except in compliance with the License. | ||
| # You may obtain a copy of the License at | ||
| # | ||
| # http://www.apache.org/licenses/LICENSE-2.0 | ||
| # | ||
| # Unless required by applicable law or agreed to in writing, software | ||
| # distributed under the License is distributed on an "AS IS" BASIS, | ||
| # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| # See the License for the specific language governing permissions and | ||
| # limitations under the License. | ||
|
|
||
| # NicClusterPolicy for GB200 OKE (OCI) — rdmaSharedDevicePlugin over InfiniBand. | ||
| # | ||
| # Mirrors the AOR OCI GB200 config validated on gb200-ew. No ofedDriver (host | ||
| # MOFED), no SR-IOV: the NVL72 east-west fabric is IB on the rdma0-3 netdevs | ||
| # (oci_hpc.rdma_device_names_mode=2 kernel cmdline names them deterministically). | ||
| # | ||
| # The IB devices are advertised as nvidia.com/mlnxnics — the same resource | ||
| # name the L40S SR-IOV path uses, so workloads request RDMA uniformly | ||
| # across OKE fabrics. | ||
| apiVersion: mellanox.com/v1alpha1 | ||
| kind: NicClusterPolicy | ||
| metadata: | ||
| name: nic-cluster-policy | ||
| annotations: | ||
| helm.sh/hook: post-install,post-upgrade | ||
| helm.sh/hook-weight: "5" | ||
| helm.sh/hook-delete-policy: before-hook-creation | ||
| labels: | ||
| app.kubernetes.io/managed-by: {{ .Release.Service }} | ||
| helm.sh/chart: {{ printf "%s-%s" .Chart.Name .Chart.Version | replace "+" "_" | trunc 63 | trimSuffix "-" }} | ||
| spec: | ||
| rdmaSharedDevicePlugin: | ||
| image: k8s-rdma-shared-dev-plugin | ||
| repository: nvcr.io/nvidia/mellanox | ||
| version: network-operator-v26.4.1 | ||
| config: | | ||
| { | ||
| "configList": [ | ||
| { | ||
| "resourcePrefix": "nvidia.com", | ||
| "resourceName": "mlnxnics", | ||
| "rdmaHcaMax": 63, | ||
| "selectors": { | ||
| "linkTypes": ["infiniband"], | ||
| "ifNames": ["rdma0", "rdma1", "rdma2", "rdma3"] | ||
| } | ||
| } | ||
| ] | ||
| } | ||
| deploymentTolerations: | ||
| - key: CriticalAddonsOnly | ||
| operator: Exists | ||
| tolerations: | ||
| # RDMA DaemonSets must land on tainted GPU nodes. | ||
| - key: nvidia.com/gpu | ||
| operator: Exists | ||
| - key: CriticalAddonsOnly | ||
| operator: Exists | ||
87 changes: 87 additions & 0 deletions
87
recipes/components/network-operator/manifests/nic-cluster-policy-oke-l40s.yaml
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,87 @@ | ||
| # Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. | ||
| # | ||
| # Licensed under the Apache License, Version 2.0 (the "License"); | ||
| # you may not use this file except in compliance with the License. | ||
| # You may obtain a copy of the License at | ||
| # | ||
| # http://www.apache.org/licenses/LICENSE-2.0 | ||
| # | ||
| # Unless required by applicable law or agreed to in writing, software | ||
| # distributed under the License is distributed on an "AS IS" BASIS, | ||
| # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| # See the License for the specific language governing permissions and | ||
| # limitations under the License. | ||
|
|
||
| # NicClusterPolicy for L40S OKE (OCI) SR-IOV RoCE. | ||
|
yuanchen8911 marked this conversation as resolved.
|
||
| # | ||
| # The network-operator Helm chart installs the operator + CRD but does not template | ||
| # a NicClusterPolicy CR (values-oke-l40s.yaml sets deployCR: false). This manifest | ||
| # creates it so the operator reconciles the RoCE fabric stack. Hand-rendered from | ||
| # AOR's network-operator/nicclusterpolicy.yaml.tmpl (provider: oci branch, with | ||
| # network.type == roce → nvIpam + secondaryNetwork included). | ||
| # | ||
| # OCI specifics (vs Forge IB): NO ofedDriver — OCI nodes carry host MOFED, consumed | ||
| # by the GPU Operator driver via driver.rdma.useHostMofed (l40s-oke-ubuntu leaf). One | ||
| # sriovDevicePlugin resource, nvidia.com/mlnxnics, selecting the OCI ConnectX VF | ||
| # device IDs (101a = ConnectX-5 Ex VF, 101e = mlx5Gen VF). RoCE also needs nv-ipam | ||
| # (VF IP allocation) + secondaryNetwork/multus (attach the VF into workload pods). | ||
| # vendor 15b3 = Mellanox. | ||
| apiVersion: mellanox.com/v1alpha1 | ||
| kind: NicClusterPolicy | ||
| metadata: | ||
| name: nic-cluster-policy | ||
| annotations: | ||
| helm.sh/hook: post-install,post-upgrade | ||
| helm.sh/hook-weight: "5" | ||
| helm.sh/hook-delete-policy: before-hook-creation | ||
| labels: | ||
| app.kubernetes.io/managed-by: {{ .Release.Service }} | ||
| helm.sh/chart: {{ printf "%s-%s" .Chart.Name .Chart.Version | replace "+" "_" | trunc 63 | trimSuffix "-" }} | ||
| spec: | ||
| # RoCE: allocate IPs for the RDMA VFs and wire them into pods via multus. | ||
| nvIpam: | ||
| image: nvidia-k8s-ipam | ||
| repository: ghcr.io/mellanox | ||
| version: v0.2.0 | ||
| enableWebhook: false | ||
| containerResources: | ||
| - name: nv-ipam-node | ||
| requests: | ||
| cpu: 500m | ||
| memory: 1Gi | ||
| limits: | ||
| cpu: "1" | ||
| memory: 2Gi | ||
| secondaryNetwork: | ||
| cniPlugins: | ||
| image: plugins | ||
| repository: ghcr.io/k8snetworkplumbingwg | ||
| version: v1.6.2-update.1 | ||
| multus: | ||
| image: multus-cni | ||
| repository: ghcr.io/k8snetworkplumbingwg | ||
| version: v4.2.1 | ||
| sriovDevicePlugin: | ||
| image: sriov-network-device-plugin | ||
| repository: ghcr.io/k8snetworkplumbingwg | ||
| version: v3.9.0 | ||
| config: | | ||
| { | ||
| "resourceList": [ | ||
| { | ||
| "resourcePrefix": "nvidia.com", | ||
| "resourceName": "mlnxnics", | ||
| "selectors": {"isRdma":true,"vendors":["15b3"],"devices":["101a","101e"]} | ||
| } | ||
| ] | ||
| } | ||
| # Operator DaemonSet placement: system/monitoring nodes only (matches AOR). | ||
| deploymentTolerations: | ||
| - key: CriticalAddonsOnly | ||
| operator: Exists | ||
| tolerations: | ||
| # RDMA DaemonSets must land on tainted GPU nodes. | ||
| - key: nvidia.com/gpu | ||
| operator: Exists | ||
| - key: CriticalAddonsOnly | ||
| operator: Exists | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,46 @@ | ||
| # Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. | ||
| # | ||
| # Licensed under the Apache License, Version 2.0 (the "License"); | ||
| # you may not use this file except in compliance with the License. | ||
| # You may obtain a copy of the License at | ||
| # | ||
| # http://www.apache.org/licenses/LICENSE-2.0 | ||
| # | ||
| # Unless required by applicable law or agreed to in writing, software | ||
| # distributed under the License is distributed on an "AS IS" BASIS, | ||
| # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| # See the License for the specific language governing permissions and | ||
| # limitations under the License. | ||
|
|
||
| # network-operator Helm values for GB200 OKE (OCI) InfiniBand. | ||
|
yuanchen8911 marked this conversation as resolved.
|
||
| # | ||
| # OCI GB200 NVL72 model (vs L40S RoCE / Forge IB): NO ofedDriver — nodes carry host | ||
| # MOFED — and no SR-IOV/nv-ipam/multus either. East-west is InfiniBand (rdma0-3), | ||
| # served by rdmaSharedDevicePlugin from the post-install NicClusterPolicy manifest, | ||
| # NOT the chart. deployCR off so that manifest CR is authoritative. | ||
| # nfd.enabled: false — GPU Operator's NFD is used; no second NFD. | ||
| # deployNodeFeatureRules stays at the chart default (true): the chart's | ||
| # nvidia-nics-rules NodeFeatureRule labels vendor-15b3 class-0200/0207 nodes | ||
| # feature.node.kubernetes.io/pci-15b3.present=true, which the deployment | ||
| # validator's RDMA readiness gate uses as its node cohort. Do NOT copy AKS's | ||
| # deployNodeFeatureRules: false here — AKS disables it only because it ships | ||
| # its own targeted rule manifest; without either source the gate's cohort is | ||
| # empty and it fails closed on every deploy. | ||
| deployCR: false | ||
| nvIpam: | ||
| enabled: false | ||
| secondaryNetwork: | ||
| deploy: false | ||
| nfd: | ||
| enabled: false | ||
| # Operator placement comes from the bundler's system-node scheduling | ||
| # injection (registry nodeScheduling: operator.nodeSelector / | ||
| # operator.tolerations) — no hardcoded affinity here. | ||
| operator: | ||
| resources: | ||
| limits: | ||
| cpu: "1" | ||
| memory: 2Gi | ||
| requests: | ||
| cpu: 500m | ||
| memory: 2Gi | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,55 @@ | ||
| # Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. | ||
| # | ||
| # Licensed under the Apache License, Version 2.0 (the "License"); | ||
| # you may not use this file except in compliance with the License. | ||
| # You may obtain a copy of the License at | ||
| # | ||
| # http://www.apache.org/licenses/LICENSE-2.0 | ||
| # | ||
| # Unless required by applicable law or agreed to in writing, software | ||
| # distributed under the License is distributed on an "AS IS" BASIS, | ||
| # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| # See the License for the specific language governing permissions and | ||
| # limitations under the License. | ||
|
|
||
| # network-operator Helm values for L40S OKE (OCI) SR-IOV RoCE. | ||
|
yuanchen8911 marked this conversation as resolved.
|
||
| # Hand-rendered from AOR's network-operator/values.yaml.tmpl (provider: oci) + | ||
| # nicclusterpolicy.yaml.tmpl (oci branch, network.type == roce). | ||
| # | ||
| # OCI model (vs Forge IB / Mistral DOCA): NO ofedDriver — OCI bare-metal nodes carry | ||
| # host MOFED, so the GPU Operator uses it via driver.rdma.useHostMofed (set on the | ||
| # l40s-oke-ubuntu leaf). network-operator's job here is the SR-IOV VF device plugin | ||
| # (advertises nvidia.com/mlnxnics RDMA VFs) plus nv-ipam + secondaryNetwork (multus) | ||
| # for RoCE — all supplied by the post-install NicClusterPolicy manifest, NOT the chart. | ||
| # | ||
| # deployCR/nvIpam/secondaryNetwork: AICR's wrapper defaults are on (deployCR: true, | ||
| # nvIpam.enabled: true, secondaryNetwork.deploy: true) — they template the wrapper's | ||
| # own NicClusterPolicy. Turn deployCR off so our manifest CR is authoritative (it is | ||
| # the only place the OCI VF selectors 101a/101e can be expressed); the operator | ||
| # reconciles nv-ipam + secondaryNetwork + sriovDevicePlugin from that CR regardless. | ||
| # nfd.enabled: false — GPU Operator's NFD is used; no second NFD. | ||
| # deployNodeFeatureRules stays at the chart default (true): the chart's | ||
| # nvidia-nics-rules NodeFeatureRule labels vendor-15b3 class-0200/0207 nodes | ||
| # feature.node.kubernetes.io/pci-15b3.present=true, which the deployment | ||
| # validator's RDMA readiness gate uses as its node cohort. Do NOT copy AKS's | ||
| # deployNodeFeatureRules: false here — AKS disables it only because it ships | ||
| # its own targeted rule manifest; without either source the gate's cohort is | ||
| # empty and it fails closed on every deploy. | ||
| deployCR: false | ||
| nvIpam: | ||
| enabled: false | ||
| secondaryNetwork: | ||
| deploy: false | ||
| nfd: | ||
| enabled: false | ||
| # Operator placement comes from the bundler's system-node scheduling | ||
| # injection (registry nodeScheduling: operator.nodeSelector / | ||
| # operator.tolerations) — no hardcoded affinity here. | ||
| operator: | ||
| resources: | ||
| limits: | ||
| cpu: "1" | ||
| memory: 2Gi | ||
| requests: | ||
| cpu: 500m | ||
| memory: 2Gi | ||
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.