Add end-to-end GPU labs (Kueue + cluster autoscaler) to the Ray on AKS example

[Li Chen (AaronLiChen)]

Summary

Adds a gpu-lab module to examples/kueue-and-ray-on-aks/ — a three-part lab sequence that takes a customer from GPU pool provisioning through hardware verification, multi-node distributed training, and queue contention, plus the metrics to check afterwards.

This extends #5867 (CPU-only ProvisioningRequest example) to GPUs, where atomic provisioning stops being a convenience and becomes what keeps an expensive cluster from deadlocking.

Stacked on #5867. Until that merges, the diff here includes its commit.

New/changed:

• 2-kueue-queues/manifests/50-gpu-autoscale-queue.yaml — GPU ResourceFlavor (with the nvidia.com/gpu=present:NoSchedule toleration AKS puts on GPU pools), ProvisioningRequestConfig with managedResources: [nvidia.com/gpu], AdmissionCheck, ClusterQueue, LocalQueue
• 3-workloads/gpu-lab/ — README + three lab manifests (hardware verify, multi-node torchrun DDP, queue contention)
• Parent README updates in the example root, module 2, and module 3

Validated on real hardware

Run against A100 80GB pools; measured output is in the README rather than illustrative numbers.

• Lab 1 (8×A100, one node): nvidia-smi, per-device torch matmul, and NCCL all_reduce_perf all pass — Avg bus bandwidth : 224.991 GB/s (NVLink-class)
• Lab 2 single-node (8 ranks): 50 steps in 0.7s, 17568.3 samples/s
• Lab 2 cross-node (2 nodes × 4 GPUs): completes, but 19.8 samples/s per GPU — ~110× slower. Cause: NCCL INFO NET/IB : No device found → Using network Socket, i.e. TCP over eth0 instead of RDMA. The README makes this the central teaching point: reading the NCCL transport line is how you tell a working multi-node job from a working-but-pointless one.
• Lab 3 (Kueue v0.18.2): A admitted, B admitted 4m later, C stayed suspended with zero pods — insufficient unused quota for nvidia.com/gpu in flavor gpu-a100, 2 more needed

Caveats, documented in the README

• The cross-node run used 4 GPUs/node rather than 8 because another tenant held a GPU on one node.
• Lab 3 was validated on quota gating only — the ProvisioningRequest CRD is absent from both test clusters, so the CAS scale-up path there is described, not measured. The CPU equivalent is exercised in Add Kueue and Cluster Autoscaler Gang Scheduling on AKS example #5867.
• An A100 scale-up in centraluseuap hit AllocationFailed (pinned availability set out of capacity). Kept in the README as a real illustration of why GPU capacity is worth gating.

Test plan

• Lab 1 run on an 8×A100 node
• Lab 2 run single-node and cross-node
• Lab 3 run against a real Kueue ClusterQueue
• All test resources cleaned up; autoscaling pool reverted

[Copilot]

🟡 Changes recommended

There are documentation inconsistencies (e.g., incorrect configuration count and unclear “choose one” guidance, plus a manifest-vs-output mismatch) that should be corrected before merging.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Adds an end-to-end “gpu-lab” walkthrough to the existing Kueue + Ray on AKS example, extending the autoscaler-driven ProvisioningRequest pattern from CPU to GPU workloads.

Changes:

• Adds CPU and GPU autoscale queue configurations (ProvisioningRequest AdmissionCheck + ClusterQueue/LocalQueue wiring).
• Adds two plain Kubernetes Job demos in Module 3: a CPU cas-batch-job and a three-part gpu-lab (verify hardware, multi-node training, queue contention).
• Updates the module READMEs to document the new queues and workloads.

File summaries

File	Description
examples/kueue-and-ray-on-aks/README.md	Updates the top-level module summary to include autoscale queues and the new workloads.
examples/kueue-and-ray-on-aks/2-kueue-queues/README.md	Documents the new autoscale (CPU/GPU) queue options and how they relate to the existing demos.
examples/kueue-and-ray-on-aks/2-kueue-queues/manifests/40-autoscale-queue.yaml	Introduces a CPU autoscale queue using ProvisioningRequest to drive cluster autoscaler scale-up before admission.
examples/kueue-and-ray-on-aks/2-kueue-queues/manifests/50-gpu-autoscale-queue.yaml	Adds the GPU variant (GPU-managedResources + GPU pool taint toleration) for atomic GPU provisioning.
examples/kueue-and-ray-on-aks/3-workloads/README.md	Adds documentation and comparison table entries for the new cas-batch-job and gpu-lab workloads.
examples/kueue-and-ray-on-aks/3-workloads/cas-batch-job/README.md	Adds an end-to-end CPU-only ProvisioningRequest→CAS demo description and runbook.
examples/kueue-and-ray-on-aks/3-workloads/cas-batch-job/manifests/job.yaml	Adds the suspended Job manifest that triggers CAS scale-up via Kueue admission.
examples/kueue-and-ray-on-aks/3-workloads/gpu-lab/README.md	Adds the GPU lab guide (provisioning, verification, distributed training, contention, and metrics).
examples/kueue-and-ray-on-aks/3-workloads/gpu-lab/manifests/10-gpu-verify.yaml	Lab 1 Job manifest to validate GPU visibility, CUDA compute, and NCCL all-reduce performance.
examples/kueue-and-ray-on-aks/3-workloads/gpu-lab/manifests/20-multinode-training.yaml	Lab 2 indexed Job + headless Service manifest to demonstrate multi-node torchrun DDP with gang semantics.
examples/kueue-and-ray-on-aks/3-workloads/gpu-lab/manifests/30-queue-contention.yaml	Lab 3 manifests to demonstrate quota-based queuing/serialization under GPU contention.

Review details

Suppressed comments (1)

examples/kueue-and-ray-on-aks/2-kueue-queues/README.md:89

• The "Choose one" note only mentions 20-/30- manifests, but the Apply section now also includes the 40- and 50- autoscale demos. Update this note so it matches the updated guidance about applying just one configuration (or clarify which combinations are supported).

> **⚠️ Choose one.** `20-single-queue.yaml` and `30-team-queues.yaml` are
> independent configurations. To switch between them, delete the active one
> first:
> ```bash
> kubectl delete -f manifests/20-single-queue.yaml   # then apply 30-team-queues.yaml

• Files reviewed: 11/11 changed files
• Comments generated: 2
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

🔵 Needs a closer look

There are a few concrete correctness/docs issues in the new lab materials (notably a hard-coded NCCL GPU count and queue option count/mutual-exclusion wording) that should be fixed before merging.

Review details

Suppressed comments (3)

Previously missed (2) — in code that hasn't changed since the last review.

examples/kueue-and-ray-on-aks/2-kueue-queues/README.md:89

• The "Choose one" warning only mentions 20-single-queue.yaml and 30-team-queues.yaml, but this PR adds two more mutually exclusive queue configs (40-autoscale-queue.yaml and 50-gpu-autoscale-queue.yaml). The warning should list all options so readers don't accidentally apply multiple demos.

> **⚠️ Choose one.** `20-single-queue.yaml` and `30-team-queues.yaml` are
> independent configurations. To switch between them, delete the active one
> first:
> ```bash
> kubectl delete -f manifests/20-single-queue.yaml   # then apply 30-team-queues.yaml

examples/kueue-and-ray-on-aks/3-workloads/gpu-lab/manifests/10-gpu-verify.yaml:89

• all_reduce_perf is hard-coded to -g 8, but the README explicitly suggests adapting these labs to other GPU SKUs (for example, 2-GPU H100 nodes). If someone updates the Job's nvidia.com/gpu limit but forgets to also change -g, the NCCL test will run with the wrong GPU count and can fail or hang.

examples/kueue-and-ray-on-aks/2-kueue-queues/README.md:11

• The intro says "Three queue configurations" but the table lists four (Single, Team, Autoscale, GPU autoscale). This mismatch is confusing for readers trying to choose the right manifest.

Three queue configurations are provided as **independent demos** — apply one,
not several:

• Files reviewed: 11/11 changed files
• Comments generated: 0 new
• Review effort level: Lite