Skip to content

Support NVIDIADriver reconciliation without a ClusterPolicy#2572

Open
karthikvetrivel wants to merge 2 commits into
kv-gpuclusterconfig-crdfrom
kv-nvidiadriver-standalone
Open

Support NVIDIADriver reconciliation without a ClusterPolicy#2572
karthikvetrivel wants to merge 2 commits into
kv-gpuclusterconfig-crdfrom
kv-nvidiadriver-standalone

Conversation

@karthikvetrivel

@karthikvetrivel karthikvetrivel commented Jun 23, 2026

Copy link
Copy Markdown
Member

Stacked on #2571 (kv-gpuclusterconfig-crd). This PR's diff is just the two commits below. GitHub will retarget it onto main once #2571 lands.

Summary

This PR lets the NVIDIADriver controller reconcile without a ClusterPolicy, so a GPUCluster (the DRA stack) can use operator-managed drivers standalone.

  • The driver host root is now sourced via the info catalog instead of the ClusterPolicy CR. The driver renderer consumes a new InfoTypeHostRoot catalog entry rather than reading the ClusterPolicy directly, which decouples it from the gpuv1 types. When both CRs exist, the ClusterPolicy's hostPaths.rootFS takes precedence. Otherwise the GPUCluster's value is used.
  • The consumer gate now accounts for both configuration CRs. If the ClusterPolicy delegates its driver (useNvidiaDriverCRD) or a GPUCluster exists, the NVIDIADriver reconciles normally. If a ClusterPolicy is present but not delegating and no GPUCluster exists, the NVIDIADriver goes disabled ("useNvidiaDriverCRD is not enabled in ClusterPolicy and no GPUCluster exists"). If neither CR exists, it goes notReady ("no ClusterPolicy or GPUCluster object found in the cluster").
  • The controller now watches GPUCluster. Generation changes fan out to all NVIDIADriver instances, so creating or deleting a GPUCluster transitions NVIDIADrivers automatically without a manual poke. Operator RBAC gains get, list, and watch on gpuclusters.

Testing

There is unit coverage for the host-root sourcing and the gate transitions, plus two live validations on a single-node Tesla T4 cluster (Kubernetes 1.34, containerd 2.2).

  • Standalone end-to-end: with no ClusterPolicy, a GPUCluster plus a default NVIDIADriver rolled out the driver DaemonSet. The DRA kubelet-plugin came up against the operator-installed driver root (devRoot=/run/nvidia/driver) and published its DeviceClasses and ResourceSlice. A pod with a DRA claim ran nvidia-smi against the operator-installed driver.
  • Gate matrix, observed via .status.state transitions on a cluster converted from a released operator install. A ClusterPolicy that is not delegating with no GPUCluster produced disabled. A GPUCluster alone produced ready. With neither CR, the NVIDIADriver went notReady with the expected message. The ClusterPolicy plus GPUCluster coexistence path is covered by the unit tests.
  • All transitions were watch-driven. Deleting or creating a GPUCluster flipped the NVIDIADriver state without manual intervention, and a single GPUCluster creation transitioned two NVIDIADriver instances at once.

@karthikvetrivel
karthikvetrivel marked this pull request as draft June 23, 2026 15:54
@karthikvetrivel
karthikvetrivel force-pushed the kv-nvidiadriver-standalone branch 3 times, most recently from 8ffcfc3 to 479562c Compare June 24, 2026 19:05
@karthikvetrivel
karthikvetrivel force-pushed the kv-gpuclusterconfig-crd branch from 5691bbc to e5dcecd Compare July 1, 2026 15:24
@karthikvetrivel
karthikvetrivel force-pushed the kv-nvidiadriver-standalone branch from 479562c to 0e75d4c Compare July 1, 2026 15:24
@coveralls

coveralls commented Jul 1, 2026

Copy link
Copy Markdown

Coverage Status

Coverage is 32.929%kv-nvidiadriver-standalone into kv-gpuclusterconfig-crd. No base build found for kv-gpuclusterconfig-crd.

@karthikvetrivel
karthikvetrivel force-pushed the kv-gpuclusterconfig-crd branch 6 times, most recently from 3b7cc55 to 4ad2c9d Compare July 8, 2026 18:48
@karthikvetrivel
karthikvetrivel force-pushed the kv-gpuclusterconfig-crd branch 8 times, most recently from 1a06b01 to b101993 Compare July 16, 2026 16:01
Signed-off-by: Karthik Vetrivel <kvetrivel@nvidia.com>
Signed-off-by: Karthik Vetrivel <kvetrivel@nvidia.com>
@karthikvetrivel
karthikvetrivel force-pushed the kv-nvidiadriver-standalone branch from 0e75d4c to 7f38f41 Compare July 16, 2026 16:11
@karthikvetrivel
karthikvetrivel marked this pull request as ready for review July 16, 2026 16:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants