fix(vgpu-device-manager): wait for host-installed vGPU Manager readiness

[lexfrei]

Description

The vgpu-manager-validation init container of the nvidia-vgpu-device-manager DaemonSet waited only for the vgpu-manager-ready status file, which the validator writes when the vGPU Manager is deployed as a container. When the vGPU Manager driver is pre-installed on the host (driver.enabled=false), the validator instead writes host-vgpu-manager-ready (and deletes both files at the start of each run), so the init container blocked indefinitely on "waiting for NVIDIA vGPU Manager to be setup" and the vGPU Device Manager never started.

host-vgpu-manager-ready is written by the validator but read nowhere else in the tree, so the host-installed-driver path never had a working readiness gate. There was already a TODO next to the gate acknowledging this.

This makes the init container wait for either status file, so the operand starts in both the container-managed and host-installed driver modes, and resolves that TODO.

Checklist

• No secrets, sensitive information, or unrelated changes
• Lint checks passing (make lint)
• Generated assets in-sync (make validate-generated-assets)
• Go mod artifacts in-sync (make validate-modules)
• Test cases are added for new code paths

Testing

Added TestVGPUDeviceManagerReadinessGate (controllers package) which decodes the daemonset asset and asserts the init container's readiness gate waits for both vgpu-manager-ready and host-vgpu-manager-ready, combined with || (OR) so it passes when either file is present. Verified red/green: the test fails on the previous single-file gate and on an accidental &&, and passes with the fix. go test ./controllers/, go vet ./controllers/, and golangci-lint run on the changed package all pass. This changes only a static asset and a unit test, so api/, config/, bundle/, deployments/, and the module files are untouched.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[rajathagasthya]

/ok-to-test c03bb17

[lexfrei]

The failing e2e-tests / e2e-tests-nvidiadriver job did not execute any gpu-operator code. It failed while Holodeck was provisioning the EC2 GPU runner, before the cluster came up:

InsufficientInstanceCapacity: We currently do not have sufficient g4dn.xlarge capacity in the Availability Zone you requested (us-west-1a).

The test VM was never created, so the operator was never installed and the vGPU validator this PR touches never ran. The e2e-tests-containerd job on the same commit passed. Could a maintainer re-run the nvidiadriver job? Thanks.

[lexfrei]

Update, superseding my earlier note: the re-run got past Holodeck provisioning and stood up the cluster and operator, but e2e-tests-nvidiadriver still fails — and it does not look related to this PR.

What the log shows: the operator Helm-installs cleanly (STATUS: deployed), then the driver pod is never created. The test polls the test-operator namespace for ~50 min ("No resources found in test-operator namespace", 542×) and times out. The operator itself is healthy — e2e-tests-containerd passes on the same operator image, and no reconcile/parse error (or any reference to this PR's asset) appears in the log.

The same signature shows up on unrelated changes:

• feat(validator): re-enable SR-IOV vGPU VFs before waiting for them #2601 (validator SR-IOV VFs, a different diff): nvidiadriver fails identically, containerd passes.
• Add GPUCluster CRD and controller for DRA-based stack #2571 (GPUCluster golden render tests, not mine): same — nvidiadriver fails, containerd passes.
• main ran nvidiadriver green earlier the same day (run 29751380127, ~15:33 UTC); every PR run after ~16:46 UTC fails the same way.

So this looks like an environmental break in the nvidiadriver e2e — the driver DaemonSet is never created — that appeared after main was last green, not a regression from this change. The operator-side logs weren't archived in the run ("No files were found ... ./logs/"), so I can't yet pin the exact upstream cause; glad to dig further if the captured cluster logs are available somewhere.

[tariq1890]

@lexfrei Can you rebase your PR? I think the e2e test failure may be related to a bug introduced in our main branch which we have since fixed.

[lexfrei]

Rebased onto current main (also rebased #2601, which hit the same nvidiadriver e2e failure). Thanks for the pointer!

[tariq1890]

/ok to test 2121660

[tariq1890]

Thanks @lexfrei, this looks good to me. I'd like to get an approve from @cdesiniotis before merging this.

[cdesiniotis]

Thanks for the contribution @lexfrei