Skip to content

docs(evidence): attest gb300-generic-ubuntu-training recipe-evidence v3 - #2851

Merged
atif1996 merged 1 commit into
mainfrom
docs/evidence-gb300-generic-ubuntu-training
Sep 19, 2026
Merged

atif1996 merged 1 commit into
mainfrom
docs/evidence-gb300-generic-ubuntu-training

Conversation

@atif1996

@atif1996 atif1996 commented Sep 19, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Add the signed recipe-evidence v3 attestation for gb300-generic-ubuntu-training, collected on a bare-metal GB300 NVL72 cluster, and allowlist the signer source it was published under.

Motivation / Context

gb300-generic-ubuntu-training (#2568) shipped without committed evidence: the run that qualified it predated the leaf rename, so its bundle was never pointed to. This is the first pointer for the leaf, produced against current main so the content-only recipe digest (173b1baf…, main at ebddbb2ad with nodewright-operator v0.19.0 from #2829) matches what the evidence gate computes from the overlay.

Fixes: N/A
Related: #2568, #2813, #2829, #2846, #2847, #2848

Type of Change

  • Documentation update

Component(s) Affected

  • Other: recipes/evidence/ (pointer + allowlist.yaml)

Implementation Notes

  • Pointer at the canonical per-source path recipes/evidence/gb300-generic-ubuntu-training/728b071aed7124418b1fa9c7f0521e24/sha256-74c42dc3….yaml, signed keyless (Fulcio/Rekor public-good, Rekor index 116117498) via aicr evidence publish; bundle at ghcr.io/atif1996/aicr-evidence:gb300-generic-ubuntu-training-6d21f4209def (public package). A first run against pre-fix(validator): read NodeWright by discovery; pin nodewright v0.19.0 #2829 main attested 1f87a241… and went stale on rebase; the cluster was re-bundled (nodewright-operator v0.17.1 → v0.19.0 via deploy.sh), re-validated and re-published against the current pin.
  • 728b071aed7124418b1fa9c7f0521e24 is a new signer source (GitHub OAuth, noreply identity), added under community with label: atif1996 — the same class as the existing OAuth entries.
  • Validators ran from the :edge image (AICR_VALIDATOR_IMAGE_TAG=edge, built from c35f47577) because :latest predates fix(validator): suppress dra-node-labeler health check when eviction not opted in #2847 and fails expected-resources on the not-opted-in dra-node-labeler (expected-resources fails NOT_FOUND on dra-node-labeler when DRA eviction not opted in (#2813) #2846). The evidence attests recipe content and validator results; the image tag is not part of the digest.
  • The NVLS NCCL check was scoped with --node-selector nvidia.com/gpu.clique=<id> to the two GPU nodes sharing one NVLink clique. The cluster's third GPU node sits in a different rack/clique; without the selector NCCL cannot form the NVLS tree across cliques and falls back to sockets (0.1–0.4 GB/s, then the 30m budget). The check itself could group by gpu.clique or fail fast — noted for a follow-up issue.

Testing

aicr snapshot -o snapshot.yaml
aicr recipe -s snapshot.yaml --service generic --intent training
# -> components=14 overlays=4 (base, gb300-any, os-ubuntu mixin, gb300-generic-ubuntu-training)

aicr bundle -r recipe.yaml --accelerated-node-selector nodeGroup=customer-gpu \
  --accelerated-node-toleration nvidia.com/gpu=present:NoSchedule \
  --system-node-selector nodeGroup=system-cpu --system-node-toleration CriticalAddonsOnly=true:NoSchedule \
  --workload-gate skyhook.nvidia.com=runtime-required:NoSchedule --workload-selector workload-type=training --nodes 3
./deploy.sh   # 14/14 installed; ClusterPolicy + NicClusterPolicy ready; every GPU node nvidia.com/gpu: 4, rdma/ib: 64
# re-bundled from main @ ebddbb2ad and replayed: nodewright-operator upgraded to v0.19.0, tuning complete 3/3 on both Skyhook and NodeWright kinds

AICR_VALIDATOR_IMAGE_TAG=edge aicr validate -r recipes/overlays/gb300-generic-ubuntu-training.yaml -s snapshot.yaml --phase all \
  --node-selector nodeGroup=customer-gpu --node-selector nvidia.com/gpu.clique=<clique> \
  --toleration nvidia.com/gpu=present:NoSchedule --emit-attestation out
# CTRF summary: tests=10 passed=10 failed=0 skipped=0
#   deployment  4/4 (1m36s)   conformance 5/5 (48s)   performance 1/1 (2m53s)
#   nccl-all-reduce-bw-nvls: 840.57 GB/s (floor >= 823; 842.02 on the pre-#2829 run)

aicr evidence publish out --push ghcr.io/atif1996/aicr-evidence
aicr evidence verify recipes/evidence/gb300-generic-ubuntu-training/728b…/sha256-74c42dc3….yaml   # bundle valid, 4/4 steps
go run ./tools/evidence-pointercheck -root recipes/evidence                                       # OK
yamllint recipes/evidence/                                                                          # clean

Cluster: bare-metal GB300 NVL72, Kubernetes v1.35.3, Ubuntu 24.04 on a 64k-page nvidia kernel, 2 system + 3 GPU nodes (4 GPUs each), skyhook tuning 3/3 complete before the driver rolled out.

Risk Assessment

  • Low — Additive: one pointer file and one allowlist entry; no code or recipe content changes.

Rollout notes: N/A.

Checklist

  • Tests pass locally (make test with -race) — N/A for a pointer; recipes/... package tests pass
  • Linter passes (make lint) — yamllint clean on the changed files
  • I did not skip/disable tests to make CI green
  • I added/updated tests for new functionality — N/A (pointer file only)
  • I updated docs if user-facing behavior changed — N/A
  • Changes follow existing patterns in the codebase
  • Commits are cryptographically signed (git commit -S -s)

@atif1996 atif1996 added the theme/supply-chain SLSA, SBOM, Sigstore, and provenance verification label Sep 19, 2026
@coderabbitai

coderabbitai Bot commented Sep 19, 2026 •

Copy link
Copy Markdown
Contributor

Review Change StackReview Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Repository: NVIDIA/aicr/.coderabbit.yaml

Review profile: ASSERTIVE

Plan: Enterprise

Run ID: 3925d443-5be8-45ac-9e57-afa323b0579c

📥 Commits

Reviewing files that changed from the base of the PR and between 4b2ca21 and 498dde6.

📒 Files selected for processing (1)
  • recipes/evidence/gb300-generic-ubuntu-training/728b071aed7124418b1fa9c7f0521e24/sha256-74c42dc3d793da22081a2e9507ca78842fe3795059f245d9b03c2edf4eccd8ab.yaml

Included review availability: Your plan provides up to 12 included reviews per hour; 11 remain after this review.


📝 Walkthrough

Walkthrough

The change adds a community signer entry to the evidence allowlist. It also adds attestation metadata for the gb300-generic-ubuntu-training recipe, including the evidence digest, OCI reference, signer, issuer, Rekor index, and schema version.

Priority: ⬇️ Low

Estimated code review effort: 1 (Trivial) | ~5 minutes

Change: Other

Suggested reviewers: yuanchen8911

🚥 Pre-merge checks | ✅ 4
✅ Passed checks (4 passed)
Check name Status Explanation
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Title check ✅ Passed The title clearly identifies the documentation change: adding an attestation for the gb300-generic-ubuntu-training recipe evidence.
Description check ✅ Passed The description directly explains the signed evidence attestation, signer allowlist entry, validation results, and changed files. It is relevant to the pull request.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Commit to this branch
  • Create a new PR

Comment @coderabbitai help to get the list of available commands.

@github-actions

github-actions Bot commented Sep 19, 2026 •

Copy link
Copy Markdown
Contributor

Recipe evidence check

Protected recipes

Recipes with committed evidence (recipes/evidence/<slug>/<source>/<digest>.yaml) that this PR affects: 1

Recipe Source Pointer Verify Digest match
gb300-generic-ubuntu-training 728b071aed7124418b1fa9c7f0521e24 sha256-74c42dc3d793da22081a2e9507ca78842fe3795059f245d9b03c2edf4eccd8ab ✅ passed ✅ matches

This gate is warning-only and never blocks merge. See ADR-007 for the trust model.

@github-actions

github-actions Bot commented Sep 19, 2026 •

Copy link
Copy Markdown
Contributor

Coverage Report ✅

Metric Value
Coverage 84.9%
Threshold 83%
Status Pass
Coverage Badge
![Coverage](https://img.shields.io/badge/coverage-84.9%25-brightgreen)

No Go source files changed in this PR.

@atif1996
atif1996 marked this pull request as ready for review September 19, 2026 02:13
@atif1996
atif1996 requested a review from a team as a code owner September 19, 2026 02:13
@atif1996
atif1996 force-pushed the docs/evidence-gb300-generic-ubuntu-training branch from 58b182f to 4b2ca21 Compare September 19, 2026 02:14
lockwobr
lockwobr previously approved these changes Sep 19, 2026
Signed-off-by: Atif Mahmood <atif1996@users.noreply.github.com>
@atif1996
atif1996 force-pushed the docs/evidence-gb300-generic-ubuntu-training branch from 4b2ca21 to 498dde6 Compare September 19, 2026 03:04
@atif1996
atif1996 requested a review from lockwobr September 19, 2026 03:07
@atif1996
atif1996 enabled auto-merge (squash) September 19, 2026 03:13
@atif1996
atif1996 merged commit 8be4c82 into main Sep 19, 2026
76 checks passed
@atif1996
atif1996 deleted the docs/evidence-gb300-generic-ubuntu-training branch September 19, 2026 03:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/recipes size/S theme/supply-chain SLSA, SBOM, Sigstore, and provenance verification

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants