fix: Discover /dev/nvidia-modeset instead of assuming it exists

[ehfd]

Reviewers: @elezar @henry118 @cdesiniotis

Summary

lookup_devices() reports /dev/nvidia-modeset as present unconditionally, filling the entry from a hardcoded major/minor rather than from the host. When the node does not exist, requesting the display capability fails container creation instead of skipping the mount.

Why This Exists

The NVIDIA driver does not register its control device nodes with devtmpfs; they are created on demand by userspace. The driver's own udev rule runs bare nvidia-modprobe when the nvidia module is loaded:

# /lib/udev/rules.d/60-nvidia.rules
ACTION=="add|bind", KERNEL=="nvidia", RUN+="/usr/bin/nvidia-modprobe"

That creates /dev/nvidia{N,ctl} but not /dev/nvidia-modeset, which requires nvidia-modprobe -m. A host can therefore legitimately have the driver loaded and no modeset node.

Every other node discovered in this function already tolerates that. /dev/nvidia-uvm and /dev/nvidia-uvm-tools, immediately above, go through find_device_node(), which logs missing device … and returns false. /dev/nvidia-modeset is the only node asserted into existence, and so the only one that can turn a missing device into a container-creation failure.

Callers that pass --load-kmods are insulated, because the library creates the node itself before discovery — the NVIDIA Container Toolkit sets load-kmods = true by default. Callers that do not pass it are not insulated: enroot invokes nvidia-container-cli configure directly without --load-kmods (conf/hooks/98-nvidia.sh), and a host using it has neither nvidia-cdi-refresh.service nor the toolkit's udev rule to create the node.

Resolution

Discover the modeset node with find_device_node(), exactly as the UVM nodes above it are discovered.

Behavior Changes

• display on a host where /dev/nvidia-modeset does not exist: the container is created, missing device /dev/nvidia-modeset is logged at warning level, and the node is not mounted. Previously container creation failed.
• No change when the node exists.
• No change to which capabilities receive the node.
• The WSL path is unaffected; the change sits inside the !dxcore.initialized branch.

Implementation Summary

• Replace the hardcoded assignment in lookup_devices() with a find_device_node() call, matching the surrounding UVM device discovery.

Verification

2x Tesla P100-SXM2-16GB, driver 580.178.04, Ubuntu 26.04. shared and tools built with WITH_LIBELF=yes WITH_TIRPC=yes (LIB_VERSION and REVISION supplied for a tag-less tree), compared against the packaged 1.20.0 build.

Host node removed, display requested:

$ sudo rm -f /dev/nvidia-modeset

# packaged 1.20.0
$ nvidia-container-cli --debug=/dev/stderr configure --no-cgroups \
      --ldconfig=@/sbin/ldconfig --device=0 --display --utility /tmp/rootfs
I nvc_info.c:572] listing device /dev/nvidia-modeset
nvidia-container-cli: mount error: stat failed: /dev/nvidia-modeset: no such file or directory
exit code 1

# with this patch
W nvc_info.c:327] missing device /dev/nvidia-modeset
exit code 0

The container is configured without the node rather than failing to be created.

With the node present there is no change, and the capability gate is untouched:

patched build, host node present	/dev/nvidia-modeset in container
--graphics --utility	absent
--display --utility	present

The packaged 1.20.0 build produces the same two rows.

This PR has been rescoped. It originally also widened the capability gate in nvc_mount.c so that graphics received /dev/nvidia-modeset. Per the discussion below that change has been dropped, and the remaining capability question is a documentation matter rather than a code one. The toolkit-side halves of this work merged as NVIDIA/nvidia-container-toolkit#1979 and NVIDIA/nvidia-container-toolkit#1980.

[Copilot]

Pull request overview

This PR aligns nvidia-container-cli’s device injection behavior with the NVIDIA Container Toolkit’s OCI-level semantics by ensuring /dev/nvidia-modeset is mounted for containers that request either the display or graphics capability (previously only display).

Changes:

• Extend the modeset-device gating check in nvc_driver_mount() to allow provisioning when OPT_GRAPHICS_LIBS is set, not only OPT_DISPLAY.
• Update the in-code comment to reflect the expanded gating condition.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[henry118]

/ok to test 43ec8ca

[henry118]

what's the rational for this change? I'm not sure if it's necessary for the purpose of this PR.

[ehfd]

The old code hard-coded the modeset entry: modeset.id = makedev(NV_DEVICE_MAJOR, NV_MODESET_DEVICE_MINOR); has_modeset = 1; so /dev/nvidia-modeset was listed unconditionally, even when the node doesn't exist on the host (i.e. the nvidia-modeset kernel module isn't loaded). Using find_device_node makes the entry conditional on the device actually being present: it stats the node and warn-and-skips on ENOENT instead of registering a device that would later fail to mount, and it records the real st_rdev rather than assuming the fixed major/minor. It also matches how nvidia-uvm and nvidia-uvm-tools are discovered in the same function, just above.

So, it's a sanity fix that closes a real source of failure.

/dev/nvidia-modeset is a userspace-created device, like /dev/nvidia-uvm or /dev/nvidia-uvm-tools. It should work identically.

[ehfd]

Full code around the edit location:

        else {
                if (!(flags & OPT_NO_UVM)) {
                        if ((has_uvm = find_device_node(err, root, NV_UVM_DEVICE_PATH, &uvm)) < 0)
                                return (-1);
                        if ((has_uvm_tools = find_device_node(err, root, NV_UVM_TOOLS_DEVICE_PATH, &uvm_tools)) < 0)
                                return (-1);
                }
                if (!(flags & OPT_NO_MODESET)) {
                        if ((has_modeset = find_device_node(err, root, NV_MODESET_DEVICE_PATH, &modeset)) < 0)
                                return (-1);
                }
                nvidiactl.path = (char *)NV_CTL_DEVICE_PATH;
                nvidiactl.id = makedev(NV_DEVICE_MAJOR, NV_CTL_DEVICE_MINOR);
                has_nvidiactl = 1;
        }

[henry118]

Per the original logic the existence of a modeset device is a prerequisite. But the after this change it is no longer true. Have you seen a real failure case? If not I'd rather keep the original logic intact.

[ehfd]

Explanation:

The original logic doesn't check for the device — that's the part I'd like to correct.

Before this change, lookup_devices() registered the node unconditionally: a fixed makedev(NV_DEVICE_MAJOR, NV_MODESET_DEVICE_MINOR) and has_modeset = 1, with no stat anywhere. The existence check happens later, at mount time — mount_device() stats the source (xstat(), src/nvc_mount.c:222) and returns NULL if it isn't there, which sends nvc_driver_mount() to goto fail and aborts the container start.

So the prerequisite isn't "a modeset device exists". It's "a modeset device exists, or the container fails to start."

The failure case: a host where /dev/nvidia-modeset was never created. The driver doesn't register control nodes with devtmpfs, so on a headless host that's simply the state until something creates them. A container requesting display there, with nvidia-container-cli invoked without --load-kmods, fails to start — on a node that isn't required for anything else it asked for.

That's rare today only because display is rare. It stops being rare with the other half of this PR: once the gate includes graphics, every graphics container on such a host takes the same hard failure. That's why the change is in this PR rather than standing alone — it's the guard that makes widening the gate safe, not a cleanup.

After the change the node behaves exactly like nvidia-uvm and nvidia-uvm-tools immediately above it: find_device_node() stats it, warns missing device %s and skips on ENOENT, and records the real st_rdev instead of assuming 195:254 — which is also what mount_device() validates against (s.st_rdev != dev->id).

Ordering is unchanged. nvc_init() runs load_kernel_modules() under OPT_LOAD_KMODS (src/nvc.c:423), and the CLI calls libnvc.init() at src/cli/configure.c:304, before driver_info_new() at :323. So --load-kmods still creates the node before discovery looks for it. The only behavior that changes is the genuinely-absent case, which goes from "abort the container start" to "warn and skip".

[ehfd]

$ sudo rm -f /dev/nvidia-modeset
$ nvidia-container-cli --debug=/dev/stderr configure … --display --utility /tmp/rootfs
I nvc_info.c:572] listing device /dev/nvidia-modeset
nvidia-container-cli: mount error: stat failed: /dev/nvidia-modeset: no such file or directory
exit code 1

# with the patch
W nvc_info.c:327] missing device /dev/nvidia-modeset
exit code 0

[henry118]

@ehfd, do you have any material or a concrete example that supports the following statement?

/dev/nvidia-modeset backs the display-facing paths of the graphics APIs themselves — Vulkan direct-to-display, the EGLDevice/EGLOutput platform, and the presentation machinery used by Wayland compositor stacks — not only the X.Org driver.

I’m not fully convinced that those paths require userspace access to /dev/nvidia-modeset. For example, NVIDIA’s current Linux driver documentation describes Wayland/non-X11 presentation through DRM KMS, exposed through the DRM device: (link).

More generally, even for the Wayland case, if a workload needs DRM/KMS to drive an actual connector, that seems semantically closer to the display capability than to graphics alone.

A couple of things that would help clarify this:

• Is there a concrete workload or reproducer that requests graphics only and fails specifically because /dev/nvidia-modeset is unavailable?
• For such workloads, why is requesting graphics,display not the appropriate capability combination?

[ehfd]

@henry118

1. The docs contradict the behavior. NVIDIA's own table (https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/docker-specialized.html#driver-capabilities) documents graphics as "Required for running OpenGL, EGL, and Vulkan applications" and display as "required for leveraging X11 display." A user requesting graphics for a Vulkan app is following the documented contract and gets a driver that can't produce a swapchain.

2. display doesn't describe the workload. This isn't about driving a connector — it's a client requesting a swapchain against a surface something else owns. Using display as the switch for client-side WSI gives it a second, unrelated meaning.

3. It's only needed on the legacy path. ResolveRuntimeMode returns JitCDIRuntimeMode for PlatformNVML, and jit-CDI injects the node via controlDeviceNodeDiscoverer() with no capability gate. So graphics alone already works on a default install and fails only under legacy. "Add display" asks users to compensate for a divergence between NVIDIA's own modes that they can't see from inside the container.

In short, it's a fix that brings legacy behavior with CDI on parity.

The below has happened frequently in non-CDI legacy in real life; it just wasn't fixed until now.

The reproducer

Host. Tesla V100-SXM2-32GB, driver 580.173.02, nvidia_drm not loaded, no /dev/dri present — the headless server configuration this PR targets, where NVKMS is reachable only through /dev/nvidia-modeset.

Method. An LD_PRELOAD shim that returns ENOENT for open("/dev/nvidia-modeset") and passes every other path through — what a process sees inside a container that was not given the node. Nothing else differs; three runs each way, deterministic.

Result: Vulkan WSI breaks.

With the node:

Presentable Surfaces:
  GPU id : 0 (Tesla V100-SXM2-32GB):
    Surface types: VK_KHR_xcb_surface, VK_KHR_xlib_surface
    Present Modes: count = 4
  GPU id : 1 (llvmpipe (LLVM 20.1.2, 256 bits)): ...

Without it:

ERROR while creating surface for extension VK_KHR_xcb_surface : vkGetPhysicalDeviceSurfacePresentModesKHR failed with ERROR_UNKNOWN
ERROR while creating surface for extension VK_KHR_xlib_surface : vkGetPhysicalDeviceSurfacePresentModesKHR failed with ERROR_UNKNOWN

Presentable Surfaces:
  GPU id : 1 (llvmpipe (LLVM 20.1.2, 256 bits)): ...

The NVIDIA GPU drops out of Presentable Surfaces entirely — no formats, no present modes, no VkSurfaceCapabilitiesKHR. A Vulkan application cannot create a swapchain on the GPU; it fails outright or silently falls back to llvmpipe.

Who opens the node. Backtrace captured at the open():

/lib/x86_64-linux-gnu/libnvidia-glcore.so.580.173.02
/lib/x86_64-linux-gnu/libGLX_nvidia.so.0
/lib/x86_64-linux-gnu/libvulkan.so.1

libGLX_nvidia.so.0 is the NVIDIA Vulkan ICD (/etc/vulkan/icd.d/nvidia_icd.json → "library_path": "libGLX_nvidia.so.0"). The open comes from the graphics driver itself, client-side, with no display-server component anywhere in the stack.

[henry118]

Thanks for the reproducer, that does establish that the Vulkan X11 WSI path requires userspace access to /dev/nvidia-modeset.

What I’m still not convinced of is the conclusion that graphics should therefore always receive the modeset device.

The reproducer exercises VK_KHR_xcb_surface and VK_KHR_xlib_surface, i.e. Vulkan presentation through X11. That seems consistent with a workload requiring both graphics and display, rather than demonstrating that a graphics-only workload is incomplete.

There also seems to be a capability-model question here. As I understand the current implementation:

• graphics enables the graphics library set.
• display implies the graphics + gates /dev/nvidia-modeset.

If /dev/nvidia-modeset is granted to graphics as well, then graphics and display become effectively equivalent. What distinct semantic purpose would display retain after this change?

I do agree that the documentation is ambiguous today. Saying that graphics is required for OpenGL/EGL/Vulkan applications while describing display only as being required for X11 does not explain clearly that a Vulkan application using X11 presentation may need graphics,display.

So I’m wondering whether the better fix is to clarify the capability contract, e.g. that display covers display/presentation/modesetting functionality, including X11 and other direct-display paths, while graphics covers rendering functionality.

[ehfd]

@henry118 Thank you. The capability-contract question is the right one to ask, and "it matches CDI" is an argument about consistency, not about meaning. So let me answer it properly: here is a boundary that leaves both capabilities with distinct, checkable content, and it turns out not to be my taxonomy; NVIDIA already ships it.

I'll be as detailed as possible due to this being a rare opportunity to make things work out for everyone (please excuse me for the length).

1. The line is client vs server, not X11 vs not-X11

graphics — render and present as a client. A process submits work to the GPU and hands finished buffers to a surface owned by something else (an X server, a Wayland compositor), or to no surface at all. This is the whole GL/GLES/EGL/GLX/Vulkan client stack, including WSI for X11, Wayland and GBM, and including the device nodes those libraries open during initialisation.

display — be the display server. Load the NVIDIA X.Org modules, take DRM master, drive the display engine. display implies graphics, exactly as it does today.

I dropped my earlier "anything touched by a graphics API" formulation, because you would have been right to reject it: libglxserver_nvidia.so is literally the GLX implementation loaded by the X server, so under that rule it would land in graphics, which is not what I want either.

2. This is NVIDIA's own line

The driver ships /usr/share/nvidia/files.d/sandboxutils-filelist.json, described in the README as:

A JSON file (/usr/share/nvidia/files.d/sandboxutils-filelist.json), which lists all the driver files (library, firmware, binary, configuration, ...) used by container runtime environments such as nvidia-container-toolkit and enroot.

Every entry carries a category. On 580.178.04 the X-server-side category xdriver is exactly three files:

{ "name": "nvidia_drv.so",                     "type": "XMODULE_SHARED_LIB",   "category": ["xdriver"] }
{ "name": "libglxserver_nvidia.so.580.178.04", "type": "GLX_MODULE_SHARED_LIB","category": ["xdriver"] }
{ "name": "nvidia-drm-outputclass.conf",       "type": "CONFIG",               "category": ["xdriver"] }

And the entire X11 client platform is categorised as EGL, not as X:

{ "name": "libnvidia-egl-xcb.so.1.0.5",  "category": ["egl", "egl_x11"] }
{ "name": "libnvidia-egl-xlib.so.1.0.5", "category": ["egl", "egl_x11"] }
{ "name": "20_nvidia_xcb.json",          "category": ["egl", "egl_x11"] }
{ "name": "20_nvidia_xlib.json",         "category": ["egl", "egl_x11"] }

So "X11 appears in the path" is not NVIDIA's criterion for the X category. The criterion is client vs server, which is the definition above. (pkg/nvcdi/driver-nvml.go already carries a TODO(ArangoGutierrez) to drive its library list from this same file, so this isn't a foreign concept to the toolkit either.)

nvSandboxUtilsGetGpuResource() says the same thing on the device side: it enumerates a GPU's nodes as NV_DEV_NVIDIA, NV_DEV_NVIDIA_CTL, NV_DEV_NVIDIA_UVM, NV_DEV_NVIDIA_MODESET, NV_DEV_DRI_CARD, NV_DEV_DRI_RENDERD — with no capability dimension at all.

3. The material you asked for

You asked whether anything supports the claim that the graphics APIs themselves reach /dev/nvidia-modeset. NVIDIA's own README does, and the text is identical in 580.178.04 and 610.57.04:

A kernel module (nvidia-modeset.ko); this kernel module is responsible for programming the display engine of the GPU. User-mode NVIDIA driver components such as the NVIDIA X driver, OpenGL driver, and VDPAU driver communicate with nvidia-modeset.ko through the /dev/nvidia-modeset device file.

Three userspace consumers are named. The X driver is one of them. A model that hands the node only to display serves one of the three.

4. It is not X11-specific — reproducer on Wayland

You were right that my previous reproducer only exercised X11. So I removed X11 from it entirely.

Host. 2x Tesla P100-SXM2-16GB, Ubuntu 26.04, driver 580.178.04, Docker 29.7.2, container toolkit 1.19.1.
Method. Same as before: an LD_PRELOAD shim returning ENOENT for open("/dev/nvidia-modeset") and passing every other path through — what a process sees inside a container that was not given the node. Run inside --gpus all with NVIDIA_DRIVER_CAPABILITIES=all, so the only variable is the node.

Wayland, VK_KHR_wayland_surface, surface provided by weston --backend=headless. No X server running, no X11 library in the process:

-------- WITH /dev/nvidia-modeset --------
vkCreateWaylandSurfaceKHR -> VkResult=0
  vkGetPhysicalDeviceSurfacePresentModesKHR  VkResult=0    count=4

-------- WITHOUT /dev/nvidia-modeset --------
vkCreateWaylandSurfaceKHR -> VkResult=0
  vkGetPhysicalDeviceSurfacePresentModesKHR  VkResult=-13  count=0   <-- VK_ERROR_UNKNOWN

Same call, same VK_ERROR_UNKNOWN as the X11 case. The compositor is headless, so this is the driver's WSI query path rather than a full swapchain — but the point stands: the failure is in the NVIDIA client driver, and it is not about X11.

X11, all the way through to a real swapchain. Same shim, VK_KHR_xcb_surface against Xvfb — a pure software X server whose framebuffer is ordinary memory — with nvidia_drm unloaded on the host, so no NVIDIA DRM device exists at all:

-------- WITH /dev/nvidia-modeset --------
  presentation supported on some queue family: YES
  vkGetPhysicalDeviceSurfaceFormatsKHR       VkResult=0  count=2
  vkGetPhysicalDeviceSurfacePresentModesKHR  VkResult=0  count=3
  vkCreateSwapchainKHR                       VkResult=0   <-- swapchain created

-------- WITHOUT /dev/nvidia-modeset --------
  presentation supported on some queue family: YES
  vkGetPhysicalDeviceSurfaceFormatsKHR       VkResult=0  count=2
  vkGetPhysicalDeviceSurfacePresentModesKHR  VkResult=-13  count=0   <-- FAILURE

Worth stating plainly, because it speaks to the kms.html point: there is no NVIDIA display engine anywhere in this configuration. No connector, no KMS, no NVIDIA DRM device, and an X server that is a memory buffer. The path being exercised is NVKMS through /dev/nvidia-modeset, not DRM/KMS through /dev/dri. Requiring a capability named display for it means requiring a display capability for a workload that never touches a display.

And the honest control — offscreen is unaffected:

-------- offscreen: no surface, no WSI, no display server --------
WITH modeset:     vkCreateDevice ok / vkCreateImage ok / vkAllocateMemory ok / vkBindImageMemory ok
WITHOUT modeset:  vkCreateDevice ok / vkCreateImage ok / vkAllocateMemory ok / vkBindImageMemory ok

The scope of the claim is exactly presentation. Pure offscreen and compute rendering do not need the node — this PR is not asking for it to reach compute.

One more detail from strace on that offscreen run:

openat(AT_FDCWD, "/dev/nvidia-modeset", O_RDWR|O_CLOEXEC) = 16

The client driver opens the node during initialisation whether or not a surface is ever created. It is part of the GL/Vulkan driver's normal startup, not something a display server asks for on its behalf.

5. On "why is graphics,display not the appropriate combination?"

Because on a default install it is not a combination at all — it changes nothing.

ResolveRuntimeMode resolves mode = "auto" to jit-cdi on NVML platforms; internal/modifier/mode.go runs no graphics modifier for CDI/jit-CDI; and pkg/nvcdi contains no reference to DriverCapabilities anywhere. Measured on the host above, docker run --gpus all:

mode	caps	/dev/nvidia-modeset	nvidia_drv.so + libglxserver_nvidia.so	/dev/dri/*	egl_vendor.d
default (auto→jit-cdi)	compute,utility	yes	yes	—	yes
default	graphics / display / all	yes	yes	—	yes
legacy	compute,utility	no	no	no	no
legacy	graphics	no	yes	yes	yes
legacy	display / graphics,display / all	yes	yes	yes	yes

(/dev/dri empty in the default rows only because nvidia_drm was unloaded for that run.)

Two things fall out of this table:

1. In the mode almost everyone runs, NVIDIA_DRIVER_CAPABILITIES has no effect at all on graphics files or control devices — compute,utility receives /dev/nvidia-modeset, libGLX_nvidia, 10_nvidia.json and nvidia_drv.so. So telling a user to add display is asking them to set a variable that does nothing on their machine, and that only matters if they happen to be on legacy — a mode they cannot observe from inside the container.
2. The legacy + graphics row is the entire bug. It is the only configuration in the product that receives the X.Org driver module, the DRM nodes and the EGL vendor config, but not the device node the GL/Vulkan driver opens. That is not a capability design; it is one line that was never revisited when everything around it moved.

6. What display retains — and how to make that real in this repo

You are right that display must not be left meaningless, and I want to be precise about where it currently stands: OPT_DISPLAY occurs exactly once outside options.h (nvc_mount.c, this mount), and libnvidia-container deliberately ships none of the xdriver files —

/*
 * Display libraries are not needed.
 *
 * "libnvidia-gtk2.so" // GTK2 (used by nvidia-settings)
 * ...
 * "nvidia_drv.so"     // Driver module for X server
 * "libglx.so"         // GLX extension module for X server
 */

So after this PR alone, display would gate nothing here, which is a fair objection.

I am happy to fix that in the same breath. Add the three xdriver-category files under OPT_DISPLAY:

• nvidia_drv.so
• libglxserver_nvidia.so.<RM_VERSION>
• nvidia-drm-outputclass.conf

Then the change reads "move the device to where the driver actually opens it, and give display its own non-overlapping content" rather than "empty out display". It also brings the legacy path to parity with what the toolkit already injects, and it is the natural companion to #1980. The known hard part is locating the X module path on hosts that install outside the default X.Org module path — that is what #563 is about, and buildXOrgSearchPaths() already solves it on the Go side, so the search-path logic can be mirrored rather than invented. Tell me whether you want that in this PR or as an immediate follow-up and I will do it either way.

One implementation detail to flag before either of us commits to that. In legacy mode both this library and the toolkit's Go graphics modifier act on the same container (internal/modifier/mode.go returns {"feature-gated", "graphics", "mode"}), and the toolkit already mounts nvidia_drv.so and libglxserver_nvidia.so.<RM_VERSION> at their host paths. If OPT_DISPLAY also mounted them, legacy + display would mount the same host paths twice, which is the case internal/discover/graphics.go already works around for libnvidia-allocator.so:

The library libnvidia-allocator.so is already handled by either the *.RM_VERSION injection or by libnvidia-container. We therefore filter it out here as a workaround for the case where libnvidia-container will re-mount this in the container, which causes issues with shared mount propagation.

That filter covers only libnvidia-allocator.so today, so the OPT_DISPLAY addition has to be paired with extending it to the two X.Org modules in #1980. nvidia-drm-outputclass.conf is better left to the toolkit alone, since #1980 already mounts it at the canonical container path and a host-path mount from here would place it there twice.

Two things to state:

• I checked before writing this: graphics already receives nvidia_drv.so and libglxserver_nvidia.so today in both modes (table above). However, these shared object files are only ever used for running an X.Org X11 server, so it could be adequately located in display. To be explicit about the cost: an image that runs an X server inside the container while requesting only graphics would then have to add display, and that is the one behaviour change in this proposal. The xorg.conf.d snippets are a separate matter and should keep being delivered on the current shared gate either way — /usr/share/X11/xorg.conf.d/nvidia-drm-outputclass.conf and the distribution-provided 10-nvidia.conf are inert without an X server, and they are what lets a container-hosted X server find the modules at all, so withholding them from graphics would be a second behaviour change for no benefit.

• Nothing here requires the CDI path to become capability-aware, which it currently is not. The graphics/display distinction stays a legacy-mode concept plus documentation; expressing it in CDI would need a separate device class and is out of scope.

7. The component lists, as concretely as I can make them

Derived from sandboxutils-filelist.json on 580.178.04, using <RM_VERSION> for version-suffixed files. The 32-bit variants of anything marked below are the compat32 set.

display — X.Org server-side only (NVIDIA category xdriver)

/usr/lib/xorg/modules/drivers/nvidia_drv.so
/usr/lib/xorg/modules/extensions/libglxserver_nvidia.so.<RM_VERSION>
/usr/share/X11/xorg.conf.d/nvidia-drm-outputclass.conf
/usr/share/X11/xorg.conf.d/10-nvidia.conf          (distribution-provided, not shipped by the NVIDIA installer)

The two xorg.conf.d files are listed here because that is the category NVIDIA gives them, not because they should be withheld from graphics — see the note in §6.

Arguably also, though NVIDIA categorises these as utils rather than xdriver:

/usr/bin/nvidia-xconfig
/usr/bin/nvidia-settings   + libnvidia-gtk2.so.<RM_VERSION> / libnvidia-gtk3.so.<RM_VERSION>
libnvidia-wfb.so.<RM_VERSION>

#1980 injects nvidia-xconfig on the shared graphics/display gate rather than display-only, and I am not proposing to move it.

graphics — client-side rendering and presentation (categories glx, egl, egl_x11, egl_wayland, egl_gbm, egl_headless, gbm, vulkan, optix, nvpresent)

Libraries:

libGLX_nvidia.so.<RM_VERSION>
libEGL_nvidia.so.<RM_VERSION>
libGLESv1_CM_nvidia.so.<RM_VERSION>
libGLESv2_nvidia.so.<RM_VERSION>
libnvidia-eglcore.so.<RM_VERSION>
libnvidia-glcore.so.<RM_VERSION>
libnvidia-glsi.so.<RM_VERSION>
libnvidia-glvkspirv.so.<RM_VERSION>
libnvidia-tls.so.<RM_VERSION>
libnvidia-fbc.so.<RM_VERSION>
libnvidia-allocator.so.<RM_VERSION>
libnvidia-rtcore.so.<RM_VERSION>
libnvoptix.so.<RM_VERSION>
libnvidia-present.so.<RM_VERSION>          (Smooth Motion Vulkan layer)
libnvidia-vksc-core.so.<RM_VERSION>        (VulkanSC)
libnvidia-egl-gbm.so.*.*
libnvidia-egl-wayland.so.*.*
libnvidia-egl-wayland2.so.*.*              (newer drivers; absent on 580/610)
libnvidia-egl-xcb.so.*.*
libnvidia-egl-xlib.so.*.*
gbm/nvidia-drm_gbm.so                      (symlink -> ../libnvidia-allocator.so.1)
libGLX_indirect.so.0                       (symlink -> libGLX_nvidia.so.<RM_VERSION>)

Configuration and data:

/usr/share/glvnd/egl_vendor.d/10_nvidia.json
/usr/share/egl/egl_external_platform.d/10_nvidia_wayland.json
/usr/share/egl/egl_external_platform.d/09_nvidia_wayland2.json
/usr/share/egl/egl_external_platform.d/15_nvidia_gbm.json
/usr/share/egl/egl_external_platform.d/20_nvidia_xcb.json
/usr/share/egl/egl_external_platform.d/20_nvidia_xlib.json
/etc/vulkan/icd.d/nvidia_icd.json
/etc/vulkan/implicit_layer.d/nvidia_layers.json
/etc/vulkansc/icd.d/nvidia_icd_vksc.json
/usr/share/nvidia/nvoptix.bin
/usr/share/nvidia/nvidia-application-profiles-<RM_VERSION>-rc

Device nodes:

/dev/nvidia<N>
/dev/nvidiactl
/dev/nvidia-modeset        <-- what this PR is about
/dev/dri/renderD*          (render node)

Shared with compute and already reachable through it, listed only so the set is complete: libnvidia-gpucomp, libcuda, libnvidia-ptxjitcompiler, libnvidia-nvvm70, libnvidia-pkcs11*.

10-nvidia.conf is distribution packaging rather than an NVIDIA-shipped file; libnvidia-vulkan-producer.so does not exist on 580 or 610 (it is in neither installedcomponents.html nor sandboxutils-filelist.json, and nvidia-ctk cdi generate logs Could not locate libnvidia-vulkan-producer.so.580.178.04); and libnvidia-allocator.so currently sits in compute_libs in nvc_info.c rather than graphics_libs, so graphics alone does not get it today and gbm/nvidia-drm_gbm.so dangles.

8. If you still prefer to keep graphics out of it

There is a smaller framing that reaches the same outcome without touching the capability semantics at all:

/dev/nvidia-modeset is a control device, not a capability-scoped resource. It is discovered in lookup_devices() next to /dev/nvidiactl; CDI groups it with /dev/nvidiactl, /dev/nvidia-uvm and /dev/nvidia-uvm-tools in controlDeviceNodeDiscoverer(); and nvSandboxUtilsGetGpuResource() returns it as NV_DEV_NVIDIA_MODESET alongside every other per-GPU node with no capability dimension. Treat it like the other control devices in legacy too, and display keeps whatever meaning you want to give it.

NVIDIA/nvidia-container-toolkit#1979, which you have been reviewing, already takes this position on the creation side: nvidia-cdi-refresh.service runs nvidia-ctk system create-device-nodes --control-devices --load-kernel-modules, and the udev rule fires the unit on nvidia module load, so /dev/nvidia-modeset is created alongside /dev/nvidiactl and the UVM nodes with no capability dimension anywhere. If the toolkit treats it as a control device when creating it, gating it on a capability when mounting it is the part that is out of step.

That version also covers the VDPAU consumer named in the README, which OPT_DISPLAY|OPT_GRAPHICS_LIBS on its own does not — VDPAU is a video library.

I am fine with either resolution. What I would like to avoid is the status quo, where the same image on the same host gets a working Vulkan swapchain or a broken one depending on a runtime mode the user cannot see.

[henry118]

@ehfd Thanks for the detail. Let me be precise about what I'm now convinced of vs what I'm still not.

Conceded

The mechanism is real. Your Wayland/Xvfb reproducers and the offscreen control answered my question. The Vulkan ICD indeed opens /dev/nvidia-modeset even without display server.

Not conceded

This belongs under graphics.

Every failure you've shown is the presentation-path:

• vkGetPhysicalDeviceSurfacePresentModesKHR
• vkCreateSwapchainKHR

Nothing non-presentation has ever failed.

Independent testing on my end confirms it too. ICD negotiation, vkCreateInstance, vkEnumeratePhysicalDevices, vkGetPhysicalDeviceProperties/Features/QueueFamilyProperties, vkEnumerateDeviceExtensionProperties all succeed without the modeset device being present, although the ICD tries to open it and get open("/dev/nvidia-modeset", ...) = -1 ENOENT.

As to the mode boundary argument, the Vulkan spec draws the line the other way. From the WSI chapter:

Since the Vulkan API can be used without displaying results, WSI is provided through the use of optional Vulkan extensions.

VK_KHR_surface/VK_KHR_swapchain are extensions, not core. The Vulkan Tutorial's swapchain chapter is explicit about this and about why:

Swapchains are not in the core Vulkan spec as they are optional, and often unique to the different platforms

and:

since image presentation is heavily tied into the window system and the surfaces associated with windows, it is not actually part of the Vulkan core.

Even inside Vulkan's object model, presentation support isn't implied by VK_QUEUE_GRAPHICS_BIT, it is queried per (queueFamily, VkSurfaceKHR) pair.

Vulkan's own architecture treats "can render" and "can present" as structurally independent, and explicitly attributes that split to presentation's dependence on the window system.

Where I land

Presentation/WSI/swapchain workloads should request graphics,display together. That's the combination the evidence actually supports. I don't think this needs "fixing" by merging the two flags.

And I'd also push back on the cross-mode "consistency" framing too. The capability-aware behavior is a legacy-only concept to begin with, and CDI/jit-CDI doesn't read NVIDIA_DRIVER_CAPABILITIES at all (as you already figured out). Legacy only activates via an explicit mode = "legacy" opt-in. So there's no ambient inconsistency a user silently falls into.

The gap here isn't a bug to reconcile across modes, it is a documentation gap. Today's docs don't say a Vulkan/EGL app that presents needs graphics,display under legacy. We can fix that with something that separates rendering from presentation.

[ehfd]

@henry118 Now that NVIDIA/nvidia-container-toolkit#1979 and NVIDIA/nvidia-container-toolkit#1980 have been merged, considering your conclusion, I think that the following are left:

• For full end-to-end Vulkan to work, compute,graphics,display, or more simply put, all, is required (due to the dependence on libnvidia-gpucomp.so.*); documentation fix. This is now in fix: Clarify the graphics and display driver capabilities cloud-native-docs#477.

• The fix in src/nvc_info.c is still a reachable surface, because the combination of display and the lack of /dev/nvidia-modeset fails to create the container. So this needs to stay. This is remaining in fix: Discover /dev/nvidia-modeset instead of assuming it exists #388.

[ehfd]

@henry118

Only #388 (comment) is what remains now, and it's a sanity fix. The PR summary has been updated.

The standalone case is the one above — a host where /dev/nvidia-modeset was never created, reached by a caller that doesn't pass --load-kmods, so the library doesn't create it either. enroot is the concrete example: it execs nvidia-container-cli configure directly (conf/hooks/98-nvidia.sh) with no --load-kmods, and such a host has neither nvidia-cdi-refresh.service nor the toolkit's udev rule. Requesting display there fails the container start over a node that nothing else it asked for needs.

$ sudo rm -f /dev/nvidia-modeset
$ nvidia-container-cli --debug=/dev/stderr configure … --display --utility /tmp/rootfs
I nvc_info.c:572] listing device /dev/nvidia-modeset
nvidia-container-cli: mount error: stat failed: /dev/nvidia-modeset: no such file or directory
exit code 1

# with the patch
W nvc_info.c:327] missing device /dev/nvidia-modeset
exit code 0

[ehfd]

Examples of problematic scenarios:
dstackai/dstack#4004
dstackai/dstack#4006