Repository navigation
Replies: 1 comment
|
Multi-node statevector simulation in CUDA-Q is possible, but it’s also the most sensitive and least “plug-and-play” execution mode in the entire stack. The confusion you’re running into is very common because:
You need distributed runtime coordination + launch orchestration + backend configuration. 1. Key idea: what “multi-node statevector” actually meansIn CUDA-Q SVS (statevector simulator), multi-node execution means:
So you are effectively running: This requires MPI-level coordination, not just CUDA_VISIBLE_DEVICES. 2. Why multi-GPU worked but multi-node didn’tMulti-GPU single node works because:
Multi-node requires: ✔ process launch per node 3. Minimum required architectureTo run multi-node SVS you need: ✔ (A) Identical environment on all nodesEach node must have:
✔ (B) Shared networking setupNodes must be able to communicate via:
✔ (C) CUDA-Q SVS multi-node backend enabledYou must configure backend to use distributed SVS, typically via:
4. Typical way to run (MPI-based)Step 1: compile/run as MPI programFor C++: mpicxx my_program.cpp -o simStep 2: run across nodesExample: mpirun -np 8 \
-host node1,node2,node3,node4 \
-x CUDA_VISIBLE_DEVICES=0 \
./sim
``` id="mpi_run1"or Slurm: srun -N 4 -n 8 ./sim5. CUDA-Q specific requirementYou must ensure backend is configured for distributed SVS. Typical environment variables: export CUDAQ_BACKEND=svs
export CUDAQ_SVS_ENABLE_MPI=1(Some builds may auto-detect MPI; others require explicit enablement.) 6. Python version (if using CUDA-Q Python)You typically still run via MPI launcher: mpirun -np 8 python my_kernel.pyKey point:
7. Common missing steps (why it fails for most users)❌ 1. Running without MPI launcherIf you do: python script.py👉 only single node executes ❌ 2. Mismatched CUDA-Q builds across nodesEven small differences cause:
❌ 3. No network backend configuredSVS requires:
❌ 4. Firewall / port blockingMulti-node SVS needs:
8. How to verify multi-node is actually workingAdd debug env: export CUDAQ_LOG_LEVEL=debugLook for logs like:
If you don’t see ranks → you are NOT running multi-node. 9. Recommended minimal working setupNode 1 (head):mpirun -np 4 -host node1,node2,node3,node4 ./your_appEach node:
10. Reality check (important)Multi-node SVS is:
11. Direct answer
If you wantI can help you with:
If this answer helped or pointed you in the right direction, I'd appreciate it if you could mark it as the accepted answer so it's easier for others with the same issue to find. Also, if you found my contribution useful, I'd appreciate it if you could check out my GitHub profile, follow me, and star any repositories you find interesting. GitHub: https://github.com/Advait251206 |
Uh oh!
There was an error while loading. Please reload this page.
Hi
I want to simulate the statevector on multi nodes.
I checked this
https://nvidia.github.io/cuda-quantum/latest/using/backends/sims/svsims.html#multi-gpu-multi-node
and successfully implemented on multi gpu but not in multi nodes.
i am not sure how i set up env in each machines and how to run the application?
would you please help me out for solving this issue?
Thanks in advance.
All reactions