Support NVIDIADriver reconciliation and driver upgrades without a ClusterPolicy

[karthikvetrivel]

Stacked on #2571 (kv-gpuclusterconfig-crd). This PR's diff is just the two commits below. GitHub will retarget it onto main once #2571 lands.

Summary

This PR lets the NVIDIADriver controller reconcile without a ClusterPolicy, so a GPUCluster (the DRA stack) can use operator-managed drivers standalone.

• The driver host root is now sourced via the info catalog instead of the ClusterPolicy CR. The driver renderer consumes a new InfoTypeHostRoot catalog entry rather than reading the ClusterPolicy directly, which decouples it from the gpuv1 types. When both CRs exist, the ClusterPolicy's hostPaths.rootFS takes precedence. Otherwise the GPUCluster's value is used.
• The consumer gate now accounts for both configuration CRs. If the ClusterPolicy delegates its driver (useNvidiaDriverCRD) or a GPUCluster exists, the NVIDIADriver reconciles normally. If a ClusterPolicy is present but not delegating and no GPUCluster exists, the NVIDIADriver goes disabled ("useNvidiaDriverCRD is not enabled in ClusterPolicy and no GPUCluster exists"). If neither CR exists, it goes notReady ("no ClusterPolicy or GPUCluster object found in the cluster").
• The controller now watches GPUCluster. Generation changes fan out to all NVIDIADriver instances, so creating or deleting a GPUCluster transitions NVIDIADrivers automatically without a manual poke. Operator RBAC gains get, list, and watch on gpuclusters.

Testing

There is unit coverage for the host-root sourcing and the gate transitions, plus two live validations on a single-node Tesla T4 cluster (Kubernetes 1.34, containerd 2.2).

• Standalone end-to-end: with no ClusterPolicy, a GPUCluster plus a default NVIDIADriver rolled out the driver DaemonSet. The DRA kubelet-plugin came up against the operator-installed driver root (devRoot=/run/nvidia/driver) and published its DeviceClasses and ResourceSlice. A pod with a DRA claim ran nvidia-smi against the operator-installed driver.
• Gate matrix, observed via .status.state transitions on a cluster converted from a released operator install. A ClusterPolicy that is not delegating with no GPUCluster produced disabled. A GPUCluster alone produced ready. With neither CR, the NVIDIADriver went notReady with the expected message. The ClusterPolicy plus GPUCluster coexistence path is covered by the unit tests.
• All transitions were watch-driven. Deleting or creating a GPUCluster flipped the NVIDIADriver state without manual intervention, and a single GPUCluster creation transitioned two NVIDIADriver instances at once.

[coveralls]

coverage: 34.822% (+1.3%) from 33.481% — kv-nvidiadriver-standalone into main

[tariq1890]

Suggested change

	logger.Error(err, "failed to get a cluster-wide configuration object")
	logger.Error(err, "failed to retrieve hostPaths information. No ClusterPolicy or GPUCluster object found in the cluster")

[tariq1890]

We can merge two if blocks into one with this change

Suggested change

	if clusterPolicy == nil {
	if clusterPolicy == nil || clusterPolicy.Spec.Driver.UseNvidiaDriverCRDType() {

[cdesiniotis]

+1

[cdesiniotis]

Actually, isn't it possible now to have both ClusterPolicy-backed drivers and NVIDIADriver-backed drivers (we can continue to debate whether this is actually desired or not in a separate thread)? IIUC the upgrade-controller, as currently implemented, does not account for that use case -- it only calls reconcileNVIDIADriverUpgrades() OR reconcileClusterPolicyDriverUpgrades.

[cdesiniotis]

The problematic scenario -- ClusterPolicy and GPUCluster exist; clusterpolicy.spec.driver.useNvidiaDriverCRD=false; NVIDIADriver CRs exist which target DRA nodes. In this case, the upgrade-controller will not be able to manage the upgrades for both the ClusterPolicy-owned driver daemonsets and the NVIDIADriver-owned driver daemonsets.

[tariq1890]

Correct me if I am wrong, but in the scenario where both ClusterPolicy and GPUCluster exist, will the .spec.hostPaths.rootFS from GPUCluster have the higher order of precedence?

[tariq1890]

Thanks @karthikvetrivel! Can you also squash your commit history here?