VR200 RKE2: metadata-collector crash-loops on host glibc mount (rc1 regression) #3103

Summary

On the v1.0.0-rc1 VR200 RKE2 recipes, nvsentinel's metadata-collector crash-loops on startup, so nvsentinel never becomes ready and deploy.sh stalls (the last component, prometheus-adapter, never installs).

Cause

#2711 re-enabled metadata-collector on the VR200 RKE2 overlays (vr200-rke2-ubuntu-training.yaml, vr200-rke2-ubuntu-inference.yaml) by:

• setting runtimeClassName: "" (the intended NRI-mode design),
• mounting the host's whole /usr/lib/aarch64-linux-gnu at /usr/local/nvidia/lib,
• setting LD_LIBRARY_PATH=/usr/local/nvidia/lib.

That directory contains the host glibc. The VR200 reference image is Ubuntu 26.04 (glibc 2.43) and the metadata-collector image is Ubuntu 22.04 (glibc 2.35), so the binary loads the host libc.so.6 with the container's older loader:

/app/metadata-collector: symbol lookup error: /usr/local/nvidia/lib/libc.so.6: undefined symbol: __tunable_is_initialized, version GLIBC_PRIVATE

Exit 127 on every restart, on both workers.

Scope

• All four VR200 RKE2 leaves (training, training-kubeflow, inference, inference-dynamo). Only these two overlays use the host mount; other platforms use a NVIDIA RuntimeClass.
• New in v1.0.0-rc1: v0.22.0 had metadata-collector disabled on VR200. The v1.25.0 image bump (fix(recipes): bump nvsentinel to v1.25.0 #2997) is not involved; v1.22.0 and v1.25.0 images both ship glibc 2.35.
• fix(recipes): nvsentinel v1.22.0 + re-enable VR200 metadata-collector #2711 was validated with lint and unit/golden tests only; this is its first run on VR200 hardware.

Environment

RKE2 v1.35.6+rke2r1, Ubuntu 26.04, kernel 7.0.0-2008-nvidia-bos-64k, host-managed driver 615.23, nvsentinel v1.25.0.

Fix

Mount only libnvidia-ml.so.1 (type: File) into /usr/local/nvidia/lib. Verified on the same cluster: metadata-collector runs, enumerates 4 GPUs and the NVSwitch, records NVLink topology, and writes gpu_metadata.json.