Repository navigation
Allow the device plugin to register multiple identical devices sharing the same IOMMU group #139
Description
As it is today, the device plugin only advertises one device per IOMMU group.
Some hardware boards by default can have more than 1 GPU or NVSwitch per IOMMU group.
In this case the number of allocatable devices on the node will be equal to the number of IOMMU groups and will be less than the actual number of devices.
We have a real customer use case to create a single Kubevirt VM on a node and passthrough all Nvidia GPUs and NVSwitches, but currently this is not possible.
The proposal is to allow the device plugin to conditionally register all the devices when explicitly asked by the deployer(for example by setting a CLI flag).
For more details and implementation reference, see: kubevirt/kubevirt#13347
Activity
andreyod commentedon Jul 1, 2025
Please let me know if this is acceptable and we will implement and open a PR for this feature.
ezrasilvera commentedon Jul 1, 2025
Just to add to that - we also implemented and tested such a change on this code and it indeed fixed the issue and the changes seems relatively simple.
As @andreyod mention we understand the reasoning behind the design decision of trying to block the user from using devices from the same IOMMU group in two different VMs but this also blocks legitimate use cases that work perfectly when using qemu/kvm directly (not through KubeVirt).
We can leave the default behavior as is and enable the "full discovery" only if explicitly requested by the user.
andreyod commentedon Jul 7, 2025
Hi @rthallisey . WDYT?
zvonkok commentedon Jul 10, 2025
Independent of the shortcomings of the device-plugin how it handles DGX, HGX systems your IOMMU groups look off. All of those components should be each in its own IOMMU Group.
Having all those components in one IOMMU Group means only one VM can allocate them. Have you enabled ACS? What are you running as the host OS and kernel?
Depending on the number of GPUs you pass-through into a single VM it will depend how many NVSwitch'es you also need to pass-through.
What is the NVSwitch virtualization model you're using? Looking through the output from the PR, you have A100 is it a DGX or HGX?
Have you reviewed the documentation at https://docs.nvidia.com/datacenter/tesla/fabric-manager-user-guide/index.html?
This is related to the DRA topic @alaypatel07 @rmohr @rthallisey
ezrasilvera commentedon Jul 10, 2025
@zvonkok Thanks for the input - Few comments:
I want to emphasis that we are now looking to support a very specific use case where we use single VM per node and assign all GPUs and NVSwitches into that VM. This is fully supported if we use KVM/QEMU directly without the need to enable ACS (as we use single VM) .
Is there any special reason why not enable an optional feature/flag that will allow the device plugin to discover all devices?
BTW, please note that the current design doesn't really protect users as it's working according to the device type - which means if we have NVSwitch and GPU in the same group a user can still try to assign GPU to one VM and NVSWitch to anothe VM and get a failure .
zvonkok commentedon Jul 11, 2025
I understand that you have only one use case and pass all GPUs and NVSwitches through into the VM. The question here is how the device plugin should handle that.
Usually, you would have something like this:
Are we saying that for every DGX/HGX system, we always want to pass through all the GPUs and NVSwitches?
ezrasilvera commentedon Jul 11, 2025
We are mainly saying that because we don't know what the user wants to do we shouldn't filter devices to "protect the user" or at least allow a mode that discovers snd exposes all devices if the user request it explicitly.
BTW, what about the GPUDirect RDMA issue with ACS?
Also as I pointed out in the shared MVSwitches mode that NVIdia recommends we assign only the nvswitches to the special service VM so we basically don't need to enable ACS and still partition the nodd into multiple VMs
ezrasilvera commentedon Jul 15, 2025
Hi @zvonkok . I enabled the ACS on our SupeMicro but unfortunetly the NVSwitches are still in one group (the GPUs are in separate groups)
IOMMU Group 130 00:10.0 PCI bridge [0604]: Intel Corporation Device [8086:1bb0] (rev 11)
Kernel driver in use: pcieport
IOMMU Group 130 03:00.0 PCI bridge [0604]: PMC-Sierra Inc. Device [11f8:4128]
Kernel driver in use: pcieport
IOMMU Group 130 03:00.1 Memory controller [0580]: PMC-Sierra Inc. Device [11f8:4128]
Subsystem: NVIDIA Corporation Device [10de:1643]
IOMMU Group 130 04:00.0 PCI bridge [0604]: PMC-Sierra Inc. Device [11f8:4128]
Kernel driver in use: pcieport
IOMMU Group 130 04:01.0 PCI bridge [0604]: PMC-Sierra Inc. Device [11f8:4128]
Kernel driver in use: pcieport
IOMMU Group 130 04:02.0 PCI bridge [0604]: PMC-Sierra Inc. Device [11f8:4128]
Kernel driver in use: pcieport
IOMMU Group 130 04:03.0 PCI bridge [0604]: PMC-Sierra Inc. Device [11f8:4128]
Kernel driver in use: pcieport
IOMMU Group 130 05:00.0 Bridge [0680]: NVIDIA Corporation GH100 [H100 NVSwitch] [10de:22a3] (rev a1)
Subsystem: NVIDIA Corporation Device [10de:1796]
Kernel driver in use: vfio-pci
IOMMU Group 130 06:00.0 Bridge [0680]: NVIDIA Corporation GH100 [H100 NVSwitch] [10de:22a3] (rev a1)
Subsystem: NVIDIA Corporation Device [10de:1796]
Kernel driver in use: vfio-pci
IOMMU Group 130 07:00.0 Bridge [0680]: NVIDIA Corporation GH100 [H100 NVSwitch] [10de:22a3] (rev a1)
Subsystem: NVIDIA Corporation Device [10de:1796]
Kernel driver in use: vfio-pci
IOMMU Group 130 08:00.0 Bridge [0680]: NVIDIA Corporation GH100 [H100 NVSwitch] [10de:22a3] (rev a1)
Subsystem: NVIDIA Corporation Device [10de:1796]
Kernel driver in use: vfio-pci
zvonkok commentedon Jul 15, 2025
There is no issue with ACS and GPUDirect RDMA, you need to configure ACS, ATS properly depending on who talks to whom.
zvonkok commentedon Jul 16, 2025
We have several overlapping projects that require this functionality, and there is no consensus on whether we should invest effort in extending the device plugin, which is currently insufficient to cover all the use cases that we might encounter on DGX/HGX-like systems.
Another idea that is being considered is to invest effort in a capable DRA driver/plugin, which is the path forward for all GPU-related allocations in a Kubernetes cluster.
A straightforward and effective way to expose those devices is to ignore all IOMMU groups and expose each vendor:10de + device-id (GPU + NvSwitch) as an extended resource. This will not give us any logical grouping, just all the VFIO devices as a single extended resource.
ezrasilvera commentedon Jul 22, 2025
Thanks @zvonkok . We have looked into DRA and indeed it's a good direction we are investigation as well.
However we need something relatively urgent for some customer use-case and a simple change to the device-plugin (we can derive the PR) can help us (and also DRA is still Beta in k8s )
In general I think the current design of exposing limited resources (one per IOMMU group) is not a good approach especially that this scenario on this board configuration can be easily achieved with libvirt/KVM (without KubeVirt) .
Do you see any special reason why not to add an optional flag that allow discovery of all devices ?
zvonkok commentedon Jul 24, 2025
@ezrasilvera As mentioned before, your system does not properly set up ACS, and there are different reasons. Having said that, I am in favour of having the capability in the device plugin to use on HGX/DGX-like systems.
Please do not rely on your IOMMU Group setup, since this will break other systems.
What combination of GPUs and NVSwitches do we want to support?
If we want all GPUs and all NVSwitches, rather than parsing or relying on the IOMMU Group, expose every GPU and NvSwitches that are bound to vfio-pci.
Can you please post a PR for review? I will tag the necessary people to move this forward. I will also need to test this on my side to ensure it doesn't break other use cases.
andreyod commentedon Jul 29, 2025
@ezrasilvera @zvonkok
PR #141 adds support to register all devices.