fix(validator): skip secure-accelerator-access for Slinky Slurm leaves - #2724
Conversation
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: ASSERTIVE Plan: Enterprise Run ID: 📒 Files selected for processing (2)
Included review availability: Your plan provides up to 12 included reviews per hour; 11 remain after this review. 📝 WalkthroughWalkthrough
Priority: ⬇️ Low Estimated code review effort: 2 (Simple) | ~10 minutes Change: Bug fix Suggested reviewers: Merge Risk: ⚪ Minimal · up to The validator now avoids the incompatible Kubernetes GPU probe only for enabled Slinky Slurm recipes while preserving validation for other configurations. No actionable merge risk remains. 🚥 Pre-merge checks | ✅ 4✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
Slinky NodeSet pods reserve a node's GPUs through their own Kubernetes pod spec, and Slurm allocates per job internally via GRES and cgroups. CheckSecureAcceleratorAccess schedules a Kubernetes-native GPU test pod, which either hangs Pending with no spare capacity or passes using capacity elsewhere without exercising Slurm's own isolation. Skip the check when slinky-slurm is in the recipe, mirroring CheckRobustController's existing component-gated skip pattern. slinky-slurm-health and slinky-slurm-imex-channel already validate Slurm's own GPU access path. Signed-off-by: Mike Cook <micook@nvidia.com>
fa8ce58 to
2572106
Compare
Summary
Skip
secure-accelerator-accessfor Slinky Slurm leaves instead of running it against a GPU allocation path Slurm never uses.Motivation / Context
Slinky NodeSet pods reserve a node's GPUs through their own Kubernetes pod spec, and Slurm allocates per job internally via GRES and cgroups.
CheckSecureAcceleratorAccessschedules a Kubernetes-native GPU test pod, which either hangs Pending with no spare capacity or passes using capacity elsewhere without exercising Slurm's own isolation.slinky-slurm-healthandslinky-slurm-imex-channelalready validate Slurm's own GPU access path, so this check adds nothing but a false result on Slurm leaves.Fixes: N/A
Related: N/A
Type of Change
Component(s) Affected
cmd/aicr,pkg/cli)cmd/aicrd,pkg/server)pkg/recipe)pkg/bundler,pkg/component/*)pkg/collector,pkg/snapshotter)pkg/validator)pkg/errors,pkg/k8s)docs/,examples/)Implementation Notes
Mirrors
CheckRobustController's existing component-gated skip pattern.recipeHasComponent(ctx, "slinky-slurm")returnsvalidators.Skip(...)before any Kubernetes resource is touched, so the skip applies to every Slurm leaf regardless of which overlay declared the check. This is a fix at the check level rather than a per-overlay exclusion, since no exclusion mechanism exists in the recipe engine and four pre-existing h100 Slurm leaves (AKS, EKS, GKE, Kind) carry the same latent bug today.Testing
Passed. Added
TestCheckSecureAcceleratorAccess_SkipsForSlinkySlurm, a table-driven test covering slinky-slurm present, slinky-slurm disabled, slinky-slurm absent, and a nil recipe.Risk Assessment
Rollout notes: N/A
Checklist
make testwith-race)make lint)git commit -S)