fix: Create NVIDIA control device nodes in nvidia-cdi-refresh.service

[ehfd]

Reviewers: @elezar @myeolenv

Associated with NVIDIA/libnvidia-container#388.

Closes #1982

Summary

Re-enables Wayland and associated Vulkan and OpenGL capabilities on the container toolkit without workarounds.

CDI spec generation only discovered NVIDIA control device nodes that already existed on the host; on systems where no host component triggers their creation, /dev/nvidia-modeset (and, after a reboot, the UVM nodes) were silently omitted from the generated specifications. This change creates missing control device nodes ahead of discovery — the programmatic equivalent of running nvidia-ctk system create-device-nodes --control-devices --load-kernel-modules at the points where specifications are produced. The legacy-mode counterpart (mounting the node for the graphics capability) is NVIDIA/libnvidia-container#388, keeping each half in the component that owns the behavior.

Why This Exists

The NVIDIA driver does not register its device nodes with devtmpfs; nodes such as /dev/nvidia-modeset are created on demand by userspace driver components. On headless Docker hosts (typical Debian, Ubuntu, and Arch installations) nothing ever triggers modeset creation, so:

• nvidia-ctk cdi generate, the nvidia-cdi-refresh service, and jit-cdi mode (the default resolution of mode = "auto") generated specs without the node, and containers requesting any capability set never received it. Wayland/KMS clients in containers fail as a result.
• In legacy mode, nvidia-container-cli creates the node under --load-kmods but mounts it only for the display capability; fix: Discover /dev/nvidia-modeset instead of assuming it exists libnvidia-container#388 extends that mount to graphics.
• The Kubernetes toolkit installer already pre-creates control device nodes, which explains why GPU Operator nodes behaved differently from plain Docker hosts for identical images.

This closes the device half of the container Wayland/X11 enablement tracked in #1477 and #563.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[Copilot]

Pull request overview

This PR ensures NVIDIA control device nodes (notably /dev/nvidia-modeset and UVM nodes) are proactively created when missing, so CDI spec generation and legacy graphics injection reliably include the nodes needed for Wayland/KMS/X11 workflows on hosts that don’t auto-create them.

Changes:

• Add an internal/system/devicenodes package to best-effort load kernel modules and mknod missing control device nodes (with per-node error collection and unit tests).
• Extend pkg/nvcdi with a DeviceNodeCreator interface + option, and invoke node creation during NVML-mode common discovery (guarded by a new disable-device-node-creation feature flag).
• Wire node creation into toolkit entrypoints: nvidia-ctk cdi generate, jit-CDI modifier, and legacy graphics modifier (adds /dev/nvidia-modeset injection).

Reviewed changes

Copilot reviewed 9 out of 9 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
pkg/nvcdi/options.go	Adds WithDeviceNodeCreator option and stores creator in library options.
pkg/nvcdi/lib.go	Plumbs deviceNodeCreator into the constructed NVCDI library instance.
pkg/nvcdi/common-nvml.go	Calls creator during NVML common discovery, gated by feature flag.
pkg/nvcdi/api.go	Introduces DeviceNodeCreator interface and disable-device-node-creation feature flag constant.
internal/system/devicenodes/devicenodes.go	New implementation for best-effort module load + device-node creation with error aggregation.
internal/system/devicenodes/devicenodes_test.go	Unit tests for existence short-circuiting, module mapping, and error collection.
internal/modifier/graphics.go	Legacy graphics modifier now attempts to create/discover /dev/nvidia-modeset for Wayland/KMS clients.
internal/modifier/cdi.go	Injects a device node creator into NVCDI for automatic (jit) CDI spec modification.
cmd/nvidia-ctk/cdi/generate/generate.go	Wires a device node creator into nvidia-ctk cdi generate during run().

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[ehfd]

@cdesiniotis For after v1.20.0.

[henry118]

@ehfd Thanks for putting this PR together. I definitely see why this change is needed.

As an alternative, do you think it would be simpler to just add the following directly in nvidia-cdi-refresh.service?

nvidia-ctk system create-device-nodes --control-devices --load-kernel-modules

[ehfd]

@henry118 It could address it partially, but has some rough spots.

Most representatively, the first container injection won't work with nvidia-ctk system create-device-nodes --control-devices --load-kernel-modules; it will work from the second time.

[ehfd]

@henry118 The footprint is now reduced greatly. The logic is baked into ExecStart=/usr/bin/nvidia-ctk cdi generate instead of nvidia-ctk system create-device-nodes --control-devices --load-kernel-modules.

[ehfd]

Now, this PR also depends on NVIDIA/libnvidia-container#388 merged to be fully fixed in both CDI and legacy pathways.

[henry118]

@henry118 It could address it partially, but has some rough spots.

Most representatively, the first container injection won't work with nvidia-ctk system create-device-nodes --control-devices --load-kernel-modules; it will work from the second time.

Could you please explain this a bit more? Is it because of startup races between docker and nvidia-cdi-refresh?

[ehfd]

@henry118 I will answer in-depth tomorrow.

[ehfd]

At boot, it is exactly a docker to nvidia-cdi-refresh race, but the race is structural, not an ordering oversight. Node absence is not a boot-only condition so that no systemd unit ordering can close it.

The below is an AI analysis.

The mechanism

1. The driver never registers control nodes with devtmpfs. internal/system/devicenodes/devicenodes.go:57-60 : /dev/nvidia-modeset, /dev/nvidia-uvm* exist only if some userspace agent created them since the last boot or module unload. /dev resets every boot; nodes also vanish on any modprobe -r / driver reload.
2. The refresh service is unordered against every container engine. deployments/systemd/nvidia-cdi-refresh.service: Type=oneshot, WantedBy=multi-user.target, no After=/Requires= on docker/containerd/crio. A GPU container started early at boot (e.g. restart=always, or an immediate docker run) reaches the runtime before the oneshot has run any ExecStart — henry118's extra ExecStart included. First container loses; second (started after the oneshot) works. That is the boot instance of the race.
3. Default containers don't even use the on-disk spec. internal/info/auto.go:89 — mode = "auto" resolves to jit-cdi on NVML platforms. The spec is generated in-memory per container at container-start (internal/modifier/cdi.go:44-87 → newAutomaticCDISpecModifier), so /var/run/cdi/nvidia.yaml — the only artifact henry118's edit fixes — is irrelevant to the default path. Only node existence on the host matters to jit-cdi, and that's guaranteed only if the service's ExecStart already ran.
4. The path unit doesn't fire on the events that actually remove the nodes. deployments/systemd/nvidia-cdi-refresh.path watches only /lib/modules/%v/modules.dep{,.bin} and /usr/bin/nvidia-ctk — install/upgrade/kernel-change events. A driver reload (modprobe -r nvidia_uvm nvidia_modeset, driver update that reloads modules) deletes the nodes but touches none of those paths, so the service never re-runs. With henry118's approach, every container started from that moment until reboot/upgrade fails — not just the first.
5. Discovery is fail-open, so the failure is silent. internal/discover/mounts.go:66-68 — a missing node produces Could not locate /dev/nvidia-modeset at warning level and the spec is emitted without it. That's precisely pauljeary's [Bug]: nvidia-cdi-refresh service at boot causing container Vulkan errors #1982 log: boot run at 09:58:45 misses it (nothing had created it yet → broken Wayland/Vulkan in containers), manual restart at 10:00:42 finds it (the COSMIC session had triggered creation in between). Also note [Bug]: nvidia-cdi-refresh service at boot causing container Vulkan errors #1982's own workaround was systemctl disable nvidia-cdi-refresh.service — any fix living solely in that unit dies with it.
6. Why it must be in-process, not a prior command. devicenodes.go:46-49 — the UVM major is read from /proc/devices at constructor time and "only becomes available once the associated kernel modules have been loaded". Creation must be ordered after module load and immediately before discovery reads /dev. Baking it into nvidia-ctk cdi generate (generate.go:320) and the jit/graphics modifiers (cdi.go:187, graphics.go:37) makes that ordering guaranteed by construction; an external actor can only make the race window smaller, never zero.

Timeline under henry118's original proposal:

boot ─┬─ multi-user.target ─┬─ nvidia-cdi-refresh.service (oneshot):
     │                     │     create-device-nodes → cdi generate  ← spec+nodes OK *after this point*
     │                     └─ docker.service (unordered, parallel)
     │                            └─ GPU container #1 (restart=always)  ← jit-cdi spec built NOW: no
modeset → FAIL
     │                                  GPU container #2 (later)         ← nodes exist → OK
     └─ driver reload mid-lifetime ──→ nodes gone, path unit NOT fired   ← containers #3..N all FAIL

Not only docker↔refresh-service — that's the boot-time instance of it. The general condition is: any spec generation or container start that happens before some agent has created the control nodes since the driver was last (re)loaded gets a silently degraded spec.

Concretely, with the unit-only approach:

1. nvidia-cdi-refresh.service is Type=oneshot, WantedBy=multi-user.target, with no ordering against docker/containerd/crio. A GPU container with restart=always (or an early docker run) can start before the oneshot's extra ExecStart runs. Since mode = "auto" resolves to jit-cdi on NVML platforms, that container builds its spec in-memory at start time and misses /dev/nvidia-modeset — first injection fails, the second one (after the oneshot) works. There's no systemd edge that fixes this: the service must also re-run on install/upgrade events, so engines can't simply be ordered After= it, and that wouldn't cover crio/podman/manual runs anyway.
2. Node absence isn't boot-only. The driver doesn't register control nodes with devtmpfs, so they also disappear on every driver reload/modprobe -r. The path trigger only watches modules.dep and the nvidia-ctk binary — reloads don't fire it — so from that moment all containers lose the node until reboot or upgrade, not just the first.
3. The unit is optional: tarball/non-systemd installs, and users who disable it (the exact workaround in [Bug]: nvidia-cdi-refresh service at boot causing container Vulkan errors #1982), get no coverage at all. Moving creation into cdi generate and the jit-cdi/graphics modifiers makes it happen in the same process, right before discovery reads /dev — and after the module load, which is what makes the dynamic UVM major readable from /proc/devices. It's idempotent (skips existing nodes) and best-effort (warn, never abort), so ordering no longer depends on any external actor having run first.

One caveat worth adding in review: the boot race for static-spec consumers (docker --device nvidia.com/gpu=…, mode = "cdi") additionally depends on henry118's ExecStart being ordered before the cdi generate line — silent breakage if appended after. In-process creation can't be misordered.

[ehfd]

@henry118 Redesigned the PR based on your suggestions. However, the dependency on NVIDIA/libnvidia-container#388 is now stronger.

[henry118]

@ehfd This approach LGTM overall.

@cdesiniotis @tariq1890 PTAL.

[henry118]

/ok to test c3f82ff

[ehfd]

Applied, thank you. @henry118

I used /usr/bin/grep to match the ExecCondition= line above; otherwise it's your suggestion as written. Verified on a live host: with nvgpu absent the grep exits 1 and the chain proceeds normally, and with an nvgpu module loaded the step short-circuits before nvidia-ctk and the unit still exits 0.

On the fallback: --load-kernel-modules calls nvmodules.LoadAll(), which modprobes nvidia, nvidia-uvm and nvidia-modeset and returns on the first failure, and run() propagates that error before CreateNVIDIAControlDevices() is reached. So a single failing modprobe means no control device nodes are created at all, including for the modules that are loaded and working. That is reachable in practice because modprobe returns 1 for a module that is loaded but no longer resolvable on disk (FATAL: Module nvidia not found in directory /lib/modules/<kver> — I reproduced this against a loaded module missing from modules.dep). The second invocation then creates whichever nodes have a major in /proc/devices.

However, if you think this should still be removed after my explanation and the case I am stating is niche, I can remove it.

[ehfd]

Additional considerations across all three PRs: NVIDIA/libnvidia-container#388 (comment)

[henry118]

Just call out that we've historically had issues with adding dependencies, see #1735 .

So for this case, since it introduced a Before= rather than After=, in order to form a cycle among, say docker, us, other, either

1. docker has a Before=us; or
2. other has After=docker and Before=us;

The question is how likely are these two cases in practice. They seem unlikely to me.

[cdesiniotis]

I think these two scenarios are unlikely, so I would be open to adding this ordering dependency. However, can we propose this change in a separate PR? I would prefer if this PR was scoped to just addressing the race condition where CDI spec generation occurs before all NVIDIA device nodes exist and / or all NVIDIA driver kernel modules are loaded.

[ehfd]

Removed as well.

[ehfd]

Following up on #2019.

[cdesiniotis]

/ok to test 8269969

[cdesiniotis]

/cherry-pick release-1.20

[github-actions]

🤖 Backport PR created for release-1.20: #2017 ✅