expected minimum latency with GB10

[mionica-e1]

Hi,

as previously described here, out-of-box achievable latency for the RoCE loopback test using HSB 2.7.0 with a GB10-powered machine was 5.57μs.

This is pretty close to the 4μs latency required for QEC; close enough that it is worth exploring whether optimizations to the data flow might be worth pursuing.

First question would be, is it reasonable to expect that such a latency (4μs max) could be achieved, in the considered use case, assuming the minimal possible (software) overhead?

Second question would be, would you have any suggestions on what to pursue, and, more importantly, in which order?

The current thinking on our side is that, since GB10 cannot do GPUdirect, the lowest hanging fruit would be to move all things related to buffers management into the kernel (for proof-of-concept, this'd mean, direct changes to the drivers; if it works, and if it would be feasible to do so, extract the code into a standalone kernel module).
Assuming that made a difference, another venue we'd consider is, running a "minimal-noise" environment - potentially reducing the in-use ARM core count, while using the smallest possible userland - the general idea being, to minimize the pressure put on RAM by the cores, by keeping as much as possible in the cache hierarchy.

Any comments, suggestions or advice would be appreciated.

Best regards,
MI

[mkhazraee]

Hi @mionica-e1,

First let me make a minor clarification about GPUDirect part, the part that we care about GPUDirect is for the data to be directly written into the GPU memory, instead of for example going to host memory and doing an extra copy which was the case back in the days. In case of GB10, there is no extra copy. The same memory is shared by the CPU and GPU, so normal RDMA would write to that memory which GPU can anyways directly read from. So GPUDirect RDMA missing is more of there is no separate GPU memory to need it, basic RDMA achieves what we need.

That being said, there is a catch for GB10, there are some limitations about how NIC is exposed to the GPU, which causes some limitations on how well GPUNetIO can work, and we have to use a CPU proxy. That's the first issue. On a similar note, those ARM cores being less capable can add to this latency of CPU proxy, so isolating them might reduce the latency to some extent. And finally, the GPU cores are not as capable of the RTX PRO 6000 that we got the below 4us numbers on, so they would add some latency.

So your idea of minimal noise and assigning some CPU cores for that CPU proxy might be lowest hanging fruit. If that doesn't help much, then some driver changes would be required, which hard to say how much effort would be required.

Are you specifically interested in using GB10 for a workload? GB10 support was mostly to enable more users to try things out, and we didn't expect similar 4us from it. Since the GPU is not that large and capable, it might not be a practical use case. It would be great if you can provide some insights into your application, and we can ask more from the driver and hardware folks even if that's possible.

Moein