Enforce decoder GPU ownership in the realtime server - #687
Closed
kvmto wants to merge 9 commits into
Closed
Conversation
Decoders can now be pinned to a specific CUDA device at construction via a cuda_device_id parameter, settable through C++/Python kwargs and the YAML realtime config (top-level decoder field, next to type/ transport). Model: one thread owns one decoder. decoder::get() validates the id (negative or >= device count throws), persistently pins the constructing thread with cudaSetDevice (no restore), strips the key before the plugin constructor, and stores it (get_cuda_device_id()). Plugin constructors therefore allocate on the right device with zero plugin changes -- this covers trt_decoder and the closed-source nv-qldpc-decoder transparently. decode_async() is the one exception: its fresh std::async worker pins itself for the call's duration via a lib-private RAII CudaDeviceGuard (libs/qec/lib/hardware_guards.h). The realtime host dispatcher (one thread serving all decoders) applies each decoder's device with set-if-different before enqueue and before DEVICE-mode graph capture; no-op for unpinned decoders. Tests: 7 C++ unit tests incl. 2-GPU placement and async-worker pinning, YAML round-trip + prepare_decoder_params coverage, Python kwargs tests. NUMA/mempolicy/cpu_affinity and thread-binding APIs are deferred to a follow-up PR per review feedback on NVIDIA#634. Signed-off-by: kvmto <kmato@nvidia.com>
cuda_device_id pinned only the constructing thread, so two dispatch paths decoded on whatever device was current: - decoding-server DecodingSession workers start unpinned; every session decoded on the default device regardless of its pin. The worker now pins itself once at worker_loop entry. - the direct (no realtime session) path decodes on the caller thread, which configure_decoders leaves on the LAST decoder's device. It now selects the decoder's device before each decode. Both paths share a new fail-fast helper (hardware_guards.h, set-if-different, throws). apply_decoder_cuda_device also uses it now: a cudaSetDevice failure previously warned and continued on the wrong device; it now surfaces as a dispatch error response, and graph capture aborts during initialization. Signed-off-by: Melody Ren <melodyr@nvidia.com>
Two paths could previously run with a decoder's cuda_device_id silently violated -- the worst failure mode, because decoding can still succeed on the wrong GPU with nothing but an unread log line as evidence: - DecodingSession workers logged a cudaSetDevice failure and kept serving. The pin now runs on the worker thread behind a promise/future handshake in start_worker(): the worker either starts pinned or the exception is rethrown on the registry thread and server startup aborts, the same channel as a decoder that fails to construct. - gpu_roce chose its GPU from HOLOLINK_GPU_ID (default 0) with no knowledge of the decoder's pin, splitting graph capture (pinned device) from ring buffers and device-side graph launch (env device). Both knobs name the same topology fact -- the FPGA-affine GPU -- so they are now reconciled at transport creation: agreement or a single set knob resolves the device, a conflict throws, and an unset environment defers to the decoder's pin. The reconciliation is a plain function compiled in every configuration and unit-tested; the gpu_roce call site remains behind CUDAQ_GPU_ROCE_AVAILABLE and needs hardware validation. Signed-off-by: Melody Ren <melodyr@nvidia.com>
- set_cuda_device_for_decode: -1 is a no-op and an impossible device id throws -- both runnable on GPU-less CI (cudaSetDevice past the device count fails there too), covering the failure transport the worker handshake rides on, which cannot be induced end-to-end after construction-time validation. - DecodingSession handshake smoke: a decoder pinned to device 0 starts its worker through the promise/future handshake and serves a queued item (skips below 1 GPU). Signed-off-by: Melody Ren <melodyr@nvidia.com>
Signed-off-by: kvmto <kmato@nvidia.com>
Propagate cuda_device_id through realtime decoder configuration, including YAML serialization, Python bindings, and decoder factory parameters. Transfer CUDA device ownership from the configuration thread to the runtime components that actually use each decoder. Dedicated decoding-server workers now select their configured device before accepting work, while shared host dispatchers select the appropriate device for each request. Ensure decoder construction, CUDA graph capture, scheduler initialization, decode execution, graph release, and decoder teardown all occur on the decoder's owning device. Restore the caller's original device when ownership is transferred to a runtime worker. Align GPU RoCE transport placement with decoder placement. Inherit HOLOLINK_GPU_ID when cuda_device_id is omitted, reject conflicting settings, and reject ambiguous device placement across graph-dispatch decoders. Add YAML, Python, server-worker, realtime-dispatch, teardown, and multi-GPU coverage for the completed decoder-server placement lifecycle. Signed-off-by: kvmto <kmato@nvidia.com>
bmhowe23
reviewed
Jul 13, 2026
Merge Melody Ren's decoder pinning follow-up so its production-tested fixes remain in history with their original authorship and this work becomes a true descendant of PR NVIDIA#678. Adopt the shared fail-fast CUDA device-selection helper, the decoding-session worker startup handshake, explicit HOLOLINK_GPU_ID tracking, and the tested GPU RoCE device-reconciliation contract. Retain the additional decoder-server ownership guarantees from this branch: select the owning device across enqueue, correction, and reset operations; restore configuration threads after decoder construction; and perform CUDA graph capture, scheduler initialization, graph release, decoder destruction, and transport teardown on the owning device. Resolve GPU RoCE placement before decoder construction so graph capture and transport allocation cannot land on different devices. Reject conflicting environment and decoder placement, explicitly negative device IDs, and ambiguous graph-dispatch configurations. Combine Melody's CI-runnable reconciliation and worker-handshake coverage with the stronger multi-GPU execution, realtime dispatch, and device-correct teardown tests from this branch. Signed-off-by: kvmto <kmato@nvidia.com>
melody-ren
added a commit
that referenced
this pull request
Jul 13, 2026
Summary: - Worker threads in the decoding server now pin themselves to the decoder's cuda_device_id before serving. Construction only pinned the registry thread, so every worker was decoding on the default device regardless of the pin prior to this patch. - The direct call path (no realtime session) had the same problem: configure_decoders leaves the thread on the last decoder's device, so decoder 0 would decode on decoder N's GPU. Each decode now selects its own decoder's device first. - If a pin can't be honored we fail instead of decoding on the wrong device: a worker that can't pin aborts server startup, and a cudaSetDevice failure during dispatch returns an error response instead of logging a warning and carrying on. - Hololink: HOLOLINK_GPU_ID and cuda_device_id both name the GPU the FPGA is attached to, so they must agree. Either one alone selects the device; setting both to different values throws at transport creation. Single decoder only, same as the rest of the gpu_roce path. - Nothing changes if cuda_device_id is not set. - Decoder construction is transactional: if a plugin constructor throws after its device was selected, the calling thread's CUDA device is restored to its previous value instead of being left on the failed decoder's device. Originally first introduced in #687. Adopting the change here. Outdated: This is a follow-up PR to #664, which should be merged before this PR This PR depends on #690 --------- Signed-off-by: kvmto <kmato@nvidia.com> Signed-off-by: Melody Ren <melodyr@nvidia.com> Co-authored-by: kvmto <kmato@nvidia.com>
Collaborator
Signed-off-by: kvmto <kmato@nvidia.com>
kvmto
force-pushed
the
decoder-server-gpu-pinning
branch
from
July 15, 2026 14:03
312d264 to
4593528
Compare
Collaborator
Collaborator
Author
|
@melody-ren is this PR needed? |
Collaborator
I picked the critical bug fix from this PR and included it in #678. But a lot has changed since then and I'm not sure how much in this PR is still update to date |
Collaborator
Author
|
If this is obsolete we can close it. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Depends on #664.
This follow-up completes the realtime decoder-server integration for the
cuda_device_idplacement contract introduced by #664. It transfers device ownership from decoder construction to the runtime threads and resources that actually use each decoder.Because the base PR comes from a fork, this draft temporarily targets
mainand includes the #664 changes until that PR merges. The follow-up-only delta is available at kvmto/cudaqx@decoder-gpu-pinning...decoder-server-gpu-pinning.The change:
cuda_device_idthrough YAML, Python bindings, and realtime decoder factory parameters;HOLOLINK_GPU_IDwhen needed, and rejects conflicts;Runtime / performance impact
Dedicated decoder workers select their CUDA device once during startup. Shared host-dispatch and direct realtime paths query the current device per request and call
cudaSetDeviceonly when a switch is required. No new third-party dependencies are introduced.Testing
allbuild: 493/493 steps passed.clang-format --dry-run --Werror: passed.git diff --check: passed.CudaDeviceIdsuite: 15/15 passed on two visible GPUs.One unrelated Python app test fails locally because the installed CUDA-Q Python module does not expose
cudaq.measure_handle; this PR does not modify that application. GPU RoCE-enabled compilation also remains for CI because local HSB and DOCA dependencies are unavailable.Self-review checklist
Before requesting review
Scope and size
Tests
Documentation
Code style
Dependencies