Reload vGPU licensing config without restarting the driver

[abrarshivani]

Problem

When the Secret referenced by driver.licensingConfig is updated — an NLS token rotated by ExternalSecrets/Vault, say — running driver pods never see the new files. Recovering the new license today requires a kubectl rollout restart of nvidia-driver-daemonset, which unloads the kernel module and evicts every GPU workload on the node.

Reported as NVIDIA/gpu-operator#2279.

There are three independent freeze points. The operator's subPath mounts are one (addressed in the companion PR below). This PR addresses the other: the entrypoint copies /drivers/* into /etc/nvidia once at startup, and nvidia-gridd reads those files only at process start. nvidia-gridd has no reload signal — the process must be restarted.

Approach

Watch the licensing configuration and bounce only nvidia-gridd, leaving the kernel module and running workloads untouched.

This is safe because a vGPU license is a lease, not a presence check. There is a 20-minute grace period before any performance degradation begins, and compute is throttled rather than disabled after that. A restart taking a couple of seconds is invisible to running workloads. Full capability is restored immediately on re-acquisition.

SIGTERM, never SIGKILL. A clean SIGTERM returns the lease to the license server. An abrupt kill matches NVIDIA's documented "abrupt shutdown may not release the license" failure mode, which would leak the seat while the new nvidia-gridd acquires another — cluster-wide, that double-counts a seat on every node at once.

What this does

Adds a background watcher, started from the vGPU branch of _start_daemons (_load_driver in the precompiled variants):

• Source resolution — prefers the new /drivers/licensing-config/ directory mount, falls back per-file to the existing subPath paths. With an older operator the directory is simply absent, the watcher observes no change, and this is completely inert.
• Content comparison, not mtime — kubelet swaps a projected volume by atomically relinking ..data, which makes timestamps an unreliable signal. Comparing content also makes a broken configuration non-looping: a config that does not change again is not applied again.
• Jitter before acting — kubelet propagates to every node in the same 1–2 minute window, so without it every driver pod in the cluster would hit the license server as one synchronized burst.
• Last-known-good rollback — if nvidia-gridd does not come back (a malformed gridd.conf, for example), the previous config is restored and nvidia-gridd restarted with it. Failures are logged loudly and never propagated: exiting would fail the pod, unload the module and evict every GPU workload — a wildly disproportionate response to a typo in a secret.
• Liveness check only — license acquisition is deliberately not verified; the 20-minute grace period makes a failure to acquire non-urgent.

Applied to the 8 entrypoints that carry the vGPU gridd block. The helper block is byte-identical across all 8 (verified by hash); the only per-file difference is the VGPU_GRIDD_LD_LIBRARY_PATH line, which already differed. OpenShift needs no extra file — Makefile:261-262 maps build-vgpuguest-rhcos% to SUBDIR = rhel9.

Configuration

Both reach the container through the operator's existing driver.env passthrough, so no operator change is needed for them:

Variable	Default	Effect
VGPU_LICENSE_AUTO_RELOAD	true	Set to anything else to not start the watcher
VGPU_LICENSE_RELOAD_INTERVAL	30	Poll interval in seconds

Notes for reviewers

No new runtime dependencies. Content is read with the bash $(< file) builtin that these entrypoints already use for pid files, rather than sha256sum/md5sum (coreutils) or cmp (diffutils), none of which is guaranteed across ubuntu, UBI, and the CUDA UBI images the precompiled variants build from.

Startup digest ordering. The digest is recorded before the initial copy, not after. Recording it after would make the watcher treat an update landing between the copy and the snapshot as already-applied, silently dropping a rotation. Failing this way round costs at most one spurious restart.

Shutdown ordering. _stop_vgpu_license_watcher runs first in _shutdown and blocks until the watcher has exited, so the watcher can never be relaunching nvidia-gridd while _unload_driver is stopping it. The watcher's trap handler only sets a flag and never exits, and bash runs traps between commands, so an in-flight restart always completes first.

Testing

• bash -n clean on all 8.
• shellcheck — no new findings on any of the 8 versus baseline.
• A throwaway harness (not committed) that stubs nvidia-gridd and /drivers exercises 15 scenarios against all 8 entrypoints: legacy fallback, directory precedence, partial directory, content-vs-mtime, rotation, no-restart-when-unchanged, broken-config rollback, no retry loop, recovery, watcher reaping, both env knobs, non-NLS, set -e safety, invalid interval, and the shutdown race.

Not tested: anything requiring a real vGPU driver, a real nvidia-gridd, a real kubelet ..data relink, or the actual base images. This needs verification on a cluster with vGPU hardware and a live NLS server before merge — hence draft.

Unrelated pre-existing bug found

All three */precompiled/nvidia-driver files call _start_vgpu_topology_daemon but never define it (0 definitions, versus 1 in the non-precompiled files). Under set -eu that would abort the vGPU path in precompiled images today. Untouched here as out of scope, but it suggests the vGPU path in those three is not exercised upstream. Worth a separate issue.

Companion PR

This is one half of a two-repo change and does nothing on its own. The operator half adds the /drivers/licensing-config/ directory mount: NVIDIA/gpu-operator#2677

Neither ordering breaks anything — this PR is inert without the operator change, and the operator change is ignored by images without this one.