[Bug]: nvidia-dcgm hostengine can start without NVML during a driver restart and never recover

[gengwg]

Describe the bug

With the standalone DCGM hostengine enabled (dcgm.enabled: true), the nvidia-dcgm pod can start while the nvidia kernel module is not loaded and then run indefinitely with no NVML. Both it and nvidia-dcgm-exporter stay 1/1 Running and Ready while exporting zero DCGM_FI_* series for that node, so GPU health is silently invisible until someone notices.

The mechanism:

1. The nvidia-dcgm DaemonSet's only init container is toolkit-validation, which checks /run/nvidia/validations/toolkit-ready. That file survives an in-place driver container restart, so the gate passes while the driver is mid-reload. Only nvidia-container-toolkit-daemonset and nvidia-operator-validator carry driver-validation.
2. The toolkit-installed host wrapper (/usr/local/nvidia/toolkit/nvidia-container-runtime, from nvidia-ctk-installer with CheckModules) does grep "^nvidia " /proc/modules || exec runc "$@". When the module is unloaded at container create time the container gets plain runc with no library or device injection. The hostengine pod logs:

   nvidia driver modules are not yet loaded, invoking /usr/local/bin/runc directly
   Started host engine version 4.4.2 using port number: 5555
   

3. nv-hostengine has no libnvidia-ml.so in its filesystem, starts anyway, and never re-initialises. The container has no liveness probe, so nothing restarts it.
4. nvidia-dcgm-exporter (started a little later, after the module loaded; it even hit failed to stat CDI host device /run/nvidia/driver/dev/nvidia-modeset a few times until the driver finished) connects via DCGM_REMOTE_HOSTENGINE_INFO=nvidia-dcgm:5555 and logs:

   Not collecting GPU metrics; error getting devices count: Cannot perform the requested operation because NVML doesn't exist on this system.
   

   It then serves an empty metrics page while Ready. Restarting only the exporter does not help; it caches the empty device list from the still-deaf hostengine.

Triggers observed: node boot racing an operator rollout, an in-place nvidia-driver-ctr restart with no reboot, and an operator rollout that recreates the dcgm DaemonSet pods while the driver container restarts (node had 70+ days uptime). We have hit this on multiple nodes across several clusters over the last month; one node stayed dark for five days because the only alert covering it was suppressed.

The exporter-side half of this is tracked in NVIDIA/dcgm-exporter#523 (and #717 proposes an NVML init retry), but those cannot fix the standalone hostengine, and the operator renders these DaemonSets itself, so the dcgm-exporter chart's initContainer support (#525) does not apply here.

To Reproduce

On a node running the standalone hostengine:

1. Restart the driver container in place (kubectl exec <driver-pod> -c nvidia-driver-ctr -- kill 1, or trigger an operator rollout that does it).
2. Within the window where grep "^nvidia " /proc/modules on the host fails, delete the nvidia-dcgm pod on that node.
3. The replacement logs the invoking runc directly line. Once the driver is back, kubectl exec <dcgm-pod> -c nvidia-dcgm-ctr -- dcgmi discovery -l reports 0 GPUs found and keeps doing so indefinitely. nvidia-smi in the driver container shows all GPUs healthy.
4. Delete nvidia-dcgm-exporter on the node: it comes back, connects, and still exports no DCGM_FI_* series.

Recovery is to delete the nvidia-dcgm pod (after the driver is up), verify dcgmi discovery -l sees GPUs, then delete nvidia-dcgm-exporter.

Expected behavior

Any of these would prevent the silent state:

• The nvidia-dcgm and nvidia-dcgm-exporter DaemonSets carry the driver-validation init container like the toolkit DaemonSet does, or toolkit-validation re-checks the driver rather than trusting a possibly stale toolkit-ready.
• The standalone nvidia-dcgm container exits non-zero when NVML cannot be initialised, so kubelet restarts it once the driver is ready. A liveness probe keyed on dcgmi discovery -l seeing at least one GPU would do the same.
• ClusterPolicy exposes livenessProbe / initContainers for dcgm and dcgmExporter, so operators can add a gate without a chart change. Today neither section has probe or init fields, so we could not fix this from the ClusterPolicy side and ended up running an out-of-band CronJob that does the two-step restart.

Environment (please provide the following information):

• GPU Operator Version: v25.10.1 (chart v25.10.1); DCGM 4.4.2-1-ubuntu22.04; dcgm-exporter 4.4.2-4.7.0-distroless; container toolkit v1.18.1; driver 580.105.08; H100 80GB HBM3 nodes
• OS: Ubuntu 24.04.4 LTS
• Kernel Version: 6.8.0-124-generic
• Container Runtime Version: containerd 2.2.3
• Kubernetes Distro and Version: kubeadm/kubespray, v1.34.3

Information to attach

Relevant pod state during the incident, other pods omitted:

NAME                                   READY   STATUS    RESTARTS
nvidia-dcgm-<hash>                     1/1     Running   0          # started ~1 min before driver-ctr restart
nvidia-dcgm-exporter-<hash>            1/1     Running   4          # 4x StartError exitCode 128 (CDI stat), then ran blind
nvidia-driver-daemonset-<hash>         1/1     Running   2          # nvidia-driver-ctr restarted in place during rollout

Exporter lastState:

reason: StartError, exitCode: 128
failed to create containerd task: failed to create shim task: OCI runtime create failed: could not apply required modification to OCI specification: error modifying OCI spec: failed to inject CDI devices: failed to inject devices: failed to stat CDI host device "/run/nvidia/driver/dev/nvidia-modeset": no such file or directory

nvidia-smi in the driver container: all 8 GPUs present, normal temperatures, customer workloads running throughout. dcgmi discovery -l in the deaf hostengine pod: 0 GPUs found.

Happy to provide a must-gather bundle privately if useful.

[rajathagasthya]

Thanks for reporting this @gengwg! We've discovered this is a generic issue across all operands, where the operands can start without driver libraries (NVML) loaded. We're tracking it in #2874. Any fix for that issue should also take care of this.