Skip to content

Mount vGPU licensing config as a directory for live updates#2677

Draft
abrarshivani wants to merge 1 commit into
NVIDIA:mainfrom
abrarshivani:feat/licensing-config-directory-mount
Draft

Mount vGPU licensing config as a directory for live updates#2677
abrarshivani wants to merge 1 commit into
NVIDIA:mainfrom
abrarshivani:feat/licensing-config-directory-mount

Conversation

@abrarshivani

Copy link
Copy Markdown
Contributor

Problem

Closes half of #2279.

When the Secret/ConfigMap referenced by driver.licensingConfig is updated — an NLS token rotated by ExternalSecrets/Vault, say — running nvidia-driver-daemonset pods never see the new files. Recovering the new license today requires a kubectl rollout restart, which unloads the kernel module and evicts every GPU workload on the node.

The licensing volume is mounted only via subPath:

/drivers/gridd.conf                                      (subPath: gridd.conf)
/drivers/ClientConfigToken/client_configuration_token.tok (subPath: client_configuration_token.tok)

subPath mounts are resolved once when the container starts, and kubelet explicitly excludes them from its atomic-symlink refresh. An in-place update of the backing object is therefore never visible to a running pod.

Change

Add a third mount of the same licensing volume as a plain directory at /drivers/licensing-config, with no subPath, so kubelet keeps its contents in sync while the pod runs.

- mountPath: /drivers/licensing-config
  name: licensing-config
  readOnly: true

The two existing subPath mounts are retained unchanged. Every published driver image entrypoint copies from those exact paths; removing them would break all of them. Keeping both means no driver-version detection is needed and no ordering constraint exists between this PR and its companion.

Applied to both the NVIDIADriver CR path (internal/state) and the ClusterPolicy path (controllers) — the latter is still the default whenever driver.useNvidiaDriverCRD is false.

Why a new path rather than fixing the existing ones

/drivers/gridd.conf is a single file inside the driver payload directory; you cannot mount a directory over a file path, nor over /drivers itself. Making it live requires a new path regardless. Since mount liveness does nothing without the entrypoint watcher anyway, there is no value in preserving backward-compatible paths, and one uniform directory keeps the entrypoint to a single code branch.

The new subdirectory collides with nothing under /drivers: every consumer uses an exact filename (/drivers/nvidia.conf, /drivers/vgpuDriverCatalog.yaml, NVIDIA-Linux-*.run), and vgpu-util filters its ReadDir on ^NVIDIA-Linux-x86_64-(.*)-grid.run.

Why the volume is reused rather than duplicated

The Items projection would be byte-identical, so a second volume would only add a redundant copy of the same projected keys and a second source of truth to keep in sync — and that list already varies conditionally on NLS. One volume, three mounts, one place to edit.

Keys are still projected through Items, so extra keys carried by a user's Secret — common when one is synced from Vault — do not land in the mounted directory.

Heads-up for upgraders

DRIVER_CONFIG_DIGEST changes, so upgrading rolls the driver DaemonSet once for clusters using vGPU licensing. This is unavoidable for any pod-spec change. It is a one-time cost on upgrade, not on subsequent license rotations — which is precisely the rollout this feature exists to eliminate.

Testing

  • go test ./internal/state/... ./controllers/... passes.
  • Adds TestDriverAdditionalConfigsVGPULicensing, covering both the Secret/NLS-enabled and ConfigMap/NLS-disabled cases: the directory mount carries no subPath, the legacy subPath mounts survive, and the Items projection stays restricted to the known keys. Verified that it fails without the production change.
    • This closes a real gap: getDriverAdditionalConfigs was previously reached only by the RHEL subscription-mounts test, so the licensing branch of the NVIDIADriver CR path had no direct coverage.
  • Golden files regenerated.

Pre-existing gap noted, not addressed here

TestDriverVGPULicensing/...Secret hand-construct their additionalConfigs literal rather than calling getDriverAdditionalConfigs, so they are template-rendering tests and do not exercise this code. One visible symptom: the golden YAML shows no readOnly: true on any licensing mount, including the two pre-existing ones, even though the production code sets it. Worth a follow-up.

Companion PR

This is one half of a two-repo change and does nothing on its ownnvidia-gridd reads its config only at process start and has no reload signal, so a live mount alone changes nothing. The driver-container half adds a watcher that bounces nvidia-gridd when the config actually changes: NVIDIA/gpu-driver-container#881

Neither ordering breaks anything: this PR is ignored by images without the watcher, and the watcher falls back to the legacy paths without this PR.

…ates

The licensing Secret/ConfigMap is mounted into the driver container only via
subPath mounts at /drivers/gridd.conf and
/drivers/ClientConfigToken/client_configuration_token.tok. subPath mounts are
resolved once when the container starts and kubelet explicitly excludes them
from its atomic-symlink refresh, so an in-place update of the backing object
(for example an NLS token rotated by ExternalSecrets/Vault) is never visible to
a running nvidia-driver-daemonset pod. Recovering the new license today
requires a `kubectl rollout restart`, which unloads the kernel module and
evicts every GPU workload on the node.

Add a third mount of the same licensing volume as a plain directory at
/drivers/licensing-config with no subPath, so kubelet keeps its contents in
sync while the pod runs. This gives the driver container entrypoint a live view
of gridd.conf (and client_configuration_token.tok when NLS is enabled) that it
can watch in order to restart nvidia-gridd in place instead of rolling the
daemonset.

The volume is reused rather than duplicated: the Items projection is identical,
so a second volume would only add a redundant copy of the same projected keys
and a second source of truth to keep in sync. Keys are still projected through
Items so that extra keys carried by a user's Secret do not land in the mount.

The two existing subPath mounts are retained unchanged. Every published driver
image entrypoint copies from those exact paths, and removing them would break
all of them.

Applied to both the NVIDIADriver CR path (internal/state) and the ClusterPolicy
path (controllers), which is still live whenever driver.useNvidiaDriverCRD is
false (the default).

Refs NVIDIA#2279
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant