Handle gdrdrv kernel module in _unload_driver()

[shaq918]

Summary

The _unload_driver() function in the driver container entrypoint script accounts for nvidia_modeset, nvidia_uvm, and nvidia_peermem when computing nvidia_deps, but does not account for gdrdrv (loaded by the nvidia-gdrcopy-ctr sidecar container). This causes the driver container to enter CrashLoopBackOff when it restarts on nodes where GDRCopy is enabled.

Problem

When RDMA is enabled in the GPU Operator ClusterPolicy (driver.rdma.enabled: true), the driver pod runs three containers:

Container	Kernel module
nvidia-driver-ctr	nvidia, nvidia_modeset, nvidia_uvm
nvidia-peermem-ctr	nvidia_peermem
nvidia-gdrcopy-ctr	gdrdrv

All four dependent modules (nvidia_modeset, nvidia_uvm, nvidia_peermem, gdrdrv) hold references on the nvidia module. However, _unload_driver() only counts three of them in nvidia_deps. When the driver container restarts (e.g. due to upgrade, pod eviction, or node reboot), it calls _unload_driver() while gdrdrv is still loaded by the co-located gdrcopy sidecar. The nvidia module refcount (4) exceeds the expected nvidia_deps count (3), and the function bails out with:

Could not unload NVIDIA driver kernel modules, driver is in use

The driver container then enters CrashLoopBackOff. The only recovery is to delete the entire driver pod (which terminates the gdrcopy sidecar and unloads gdrdrv), but the issue recurs on any subsequent driver container restart.

Observed in production across multiple clusters running H100/H200 GPU nodes with InfiniBand and GDRCopy enabled.

Fix

Add gdrdrv to the set of known nvidia-dependent modules in _unload_driver(), matching the existing pattern used for nvidia_modeset, nvidia_uvm, and nvidia_peermem:

if [ -f /sys/module/gdrdrv/refcnt ]; then
    rmmod_args+=("gdrdrv")
    ((++nvidia_deps))
fi

When gdrdrv is present, it is added to rmmod_args (so it gets unloaded) and nvidia_deps is incremented (so the refcount check passes). When gdrdrv is not present (no GDRCopy), the block is a no-op.

Applied to all 11 nvidia-driver scripts across ubuntu22.04, ubuntu24.04, ubuntu26.04, rhel8, rhel9, and rhel10 (both compiled and precompiled variants).

Reproduction

1. Deploy GPU Operator with driver.rdma.enabled: true (enables peermem + gdrcopy sidecars)
2. Verify gdrdrv is loaded: lsmod | grep gdrdrv → nvidia refcnt=4
3. Delete the driver container (not the pod) or trigger a driver restart
4. Driver container enters Init:CrashLoopBackOff with "Could not unload NVIDIA driver kernel modules"

Testing

• With fix: driver container restart successfully unloads gdrdrv along with the other modules, nvidia refcount check passes, clean reinstall
• Without GDRCopy: gdrdrv sysfs entry doesn't exist, the new block is skipped entirely — no behavior change

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.