fix: NVIDIA X.Org driver modules mount fixes

[ehfd]

Reviewers: @elezar @cdesiniotis @henry118

Fixes #1477
Fixes #563

Summary

Graphics config files for the NVIDIA driver were mounted into the container at their host paths, so on hosts that do not install them under /usr/share (or /etc for the OpenCL ICD) they land where the loaders in the container never look. This mounts them at the canonical container locations, mirroring the existing Vulkan ICD handling, and injects the EGL X11 platform libraries and nvidia-xconfig.

The create-xorg-config CDI hook from the previous revision of this PR is gone. Per @cdesiniotis' review, the host's xorg.conf.d snippet is what carries the ModulePath when the driver installs the X.Org modules outside the default X.Org module path; mounting that snippet where the container's X server reads it is sufficient, and a host without one is treated as a host-side misconfiguration.

Why This Exists

EGL vendor, EGL external platform and GLVND config files were mounted at their host paths. On hosts with non-FHS layouts these land where container glvnd never looks, forcing workarounds such as __EGL_VENDOR_LIBRARY_FILENAMES. The same applies to the xorg.conf.d snippet that makes the injected X.Org modules loadable, and to the OpenCL ICD.

The EGL X11 platform libraries (libnvidia-egl-xcb, libnvidia-egl-xlib) and their external platform config files were not discovered at all, even though the GBM and Wayland platform libraries next to them are.

Resolution

• EGL/GLVND/X11 config files are mounted at /usr/share/... and the OpenCL ICD at /etc/OpenCL/vendors, mirroring the existing Vulkan ICD handling.
• The X.Org driver modules keep being mounted at their host paths, so an absolute ModulePath in the mounted snippet stays valid in the container, and modules installed to the default X.Org module path on the host are found at the same path in the container.
• libnvidia-egl-xcb.so.*.*, libnvidia-egl-xlib.so.*.* and their external platform JSON files are discovered and injected.
• nvidia-xconfig is injected when present on the host.
• All the mount discoverers in graphics.go are wrapped in WithCache (per @henry118's earlier review); the X.Org modules were previously discovered twice because graphicsDriverLibraries reads Mounts() from both Mounts() and Hooks().

Behavior Changes

• Containers requesting graphics or display receive the config files at canonical paths, the EGL X11 platform libraries and their config files, and nvidia-xconfig where installed.
• No new hooks are generated. The generated spec differs from main only by those mounts.
• Hosts without the relevant files see per-file discovery warnings, as with existing optional components.

Implementation Summary

• Split the graphics config discovery out into newGraphicsConfigsDiscoverer, using mountsToContainerPath for /usr/share and /etc.
• Add the EGL X11 platform libraries to the graphics library discoverer and nvidia-xconfig to a binaries discoverer.
• Add TestGraphicsConfigsDiscoverer, covering both standard and non-standard host locations.

Verification

go test ./... failing set is identical to the unmodified main baseline on the validation host (all failures are pre-existing environment failures in nvidia-ctk-installer, internal/modifier CSV mode and pkg/nvcdi); golangci-lint run ./... reports 0 issues; gofmt is clean.

Real-host validation on 2x Tesla P100, driver 580.178.04, Ubuntu 26.04, Docker 29.7.2, container ubuntu:24.04 with only xserver-xorg-core and mesa-utils installed:

1. Host with modules in the default X.Org module path (/usr/lib/xorg/modules/{drivers,extensions}, Ubuntu packaging, snippet without a ModulePath): X.Org loads nvidia_drv.so and libglxserver_nvidia.so from /usr/lib/xorg/modules/..., glxinfo -B reports Tesla P100-SXM2-16GB/PCIe/SSE2. Unchanged from main.

2. Host with modules outside the default X.Org module path (/usr/lib/x86_64-linux-gnu/nvidia/xorg, with the host 10-nvidia.conf carrying the matching ModulePath): the mounted snippet is what makes it work.

   (==) ModulePath set to "/usr/lib/xorg/modules"
   (**) OutputClass "nvidia" ModulePath extended to "/usr/lib/x86_64-linux-gnu/nvidia/xorg,/usr/lib/xorg/modules"
   (II) LoadModule: "nvidia"
   (II) Loading /usr/lib/x86_64-linux-gnu/nvidia/xorg/nvidia_drv.so
   (II) LoadModule: "glxserver_nvidia"
   (II) Loading /usr/lib/x86_64-linux-gnu/nvidia/xorg/libglxserver_nvidia.so
   (II) NVIDIA GLX Module  580.178.04
   

   glxinfo -B and the EGL X11 platform both report Tesla P100-SXM2-16GB/PCIe/SSE2.

   Note: MatchDriver "nvidia-drm" only matches when the DRM device nodes are present in the container, so this path requires nvidia-drm loaded on the host and the display/graphics capability. With nvidia_drm unloaded the OutputClass never matches and no ModulePath is applied.

3. nvidia-xconfig in the container generates a working BusID xorg.conf; libEGL_nvidia dlopens the injected libnvidia-egl-xcb.so.1/libnvidia-egl-xlib.so.1 when their config files are present (verified with LD_DEBUG=libs). On 580 the built-in X11 platform still services EGL_PLATFORM_X11, so injecting them is about keeping the container's EGL platform set matching the host's rather than fixing an observed failure today.

4. Legacy-mode parity (mode = "legacy"): graphics receives the same set as CDI mode; compute,utility receives none of it, unchanged from main.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[ehfd]

@cdesiniotis For after v1.20.0.

[henry118]

Thanks @ehfd. overall lgtm. left a few comments, ptal.

[henry118]

/ok to test 27972c1

[henry118]

This LGTM now. @cdesiniotis can you also ptal?

[cdesiniotis]

@ehfd thanks for your contribution. I am not too keen on introducing a CDI hook for this. Wouldn't the host's Xorg conf file (i.e. /usr/share/X11/xorg.conf.d/10-nvidia.conf) handle the scenario where the NVIDIA-specific libraries are installed at non-standard locations? Would the issue you are encountering go away if the xorg.conf.d file was also mounted into the container?

You mention a use case where the xorg.conf.d file does not exist on the host, but in my opinion, that is a misconfiguration and this type of scenario is out-of-scope for the NVIDIA Container Toolkit to address. A prerequisite is that the host has a functioning (and complete) driver installation. If a specific application does not work on the host, then we cannot expect it to work in a container.

[ehfd]

@cdesiniotis I understand your perspective. I will find a solution which works for all parties.

[ehfd]

@cdesiniotis @henry118

I've reworked the PR around your suggestion. The create-xorg-config hook is gone, and the PR now relies on the host's xorg.conf.d snippet being mounted where the container's X server reads it.

I validated this on a 2x Tesla P100 host (driver 580.178.04, Ubuntu 26.04, Docker 29.7.2) with a stock ubuntu:24.04 container that only has xserver-xorg-core and mesa-utils installed. With the X.Org modules deliberately installed outside the default X.Org module path (/usr/lib/x86_64-linux-gnu/nvidia/xorg) and the host 10-nvidia.conf carrying the matching ModulePath, the mounted snippet is exactly what makes it work:

(==) ModulePath set to "/usr/lib/xorg/modules"
(**) OutputClass "nvidia" ModulePath extended to "/usr/lib/x86_64-linux-gnu/nvidia/xorg,/usr/lib/xorg/modules"
(II) LoadModule: "nvidia"
(II) Loading /usr/lib/x86_64-linux-gnu/nvidia/xorg/nvidia_drv.so
(II) LoadModule: "glxserver_nvidia"
(II) Loading /usr/lib/x86_64-linux-gnu/nvidia/xorg/libglxserver_nvidia.so
(II) NVIDIA GLX Module  580.178.04

glxinfo -B and the EGL X11 platform both report the P100. One prerequisite worth recording for anyone who finds this later: MatchDriver "nvidia-drm" only matches when the DRM device nodes are in the container, so this needs nvidia-drm loaded on the host and the display/graphics capability requested. With nvidia_drm unloaded the OutputClass never matches and no ModulePath is applied — which lines up with your point that the host needs a functioning driver installation.

What's left in the PR:

• Config files (EGL vendor, EGL external platform, GLVND, xorg.conf.d, OpenCL ICD) mounted at the canonical container locations rather than the host paths, mirroring the existing Vulkan ICD handling. This is what makes the xorg.conf.d snippet land where the container's X server reads it on hosts that do not install it under /usr/share, and it removes the need for __EGL_VENDOR_LIBRARY_FILENAMES-style workarounds for the same reason.
• X.Org modules still mounted at their host paths — unchanged from main — so an absolute ModulePath in the mounted snippet stays valid.
• libnvidia-egl-xcb / libnvidia-egl-xlib and their external platform JSON files injected alongside the GBM and Wayland platform libraries. libEGL_nvidia does dlopen them once the JSON files are present (confirmed with LD_DEBUG=libs), so this is about the container's EGL platform set matching the host's.
• nvidia-xconfig injected when present on the host (preferred in Dynamically expose NVIDIA X.Org X11 display server libraries and configure the container correctly #563; the container loses configuration-controlling mechanisms without this).

[henry118]

/ok to test d8b95c8

[henry118]

LGTM

[coveralls]

Coverage Report for CI Build 32267242093

Warning

Build has drifted: This PR's base is out of sync with its target branch, so coverage data may include unrelated changes.
Quick fix: rebase this PR. Learn more →

Coverage increased (+0.1%) to 44.087%

Details

• Coverage increased (+0.1%) from the base build.
• Patch coverage: 25 uncovered changes across 1 file (19 of 44 lines covered, 43.18%).
• No coverage regressions found.

Uncovered Changes

File	Changed	Covered	%
internal/discover/graphics.go	44	19	43.18%

Coverage Regressions

No coverage regressions found.

---

Coverage Stats

Relevant Lines:	15313
Covered Lines:	6751
Line Coverage:	44.09%
Coverage Strength:	0.49 hits per line

---

💛 - Coveralls

[ehfd]

Additional considerations across all three PRs: NVIDIA/libnvidia-container#388 (comment)

[ehfd]

@cdesiniotis Can you review this too?

[cdesiniotis]

LGTM. @ehfd thanks for your contribution and patience during reviews.

[cdesiniotis]

/cherry-pick release-1.20

[github-actions]

🤖 Backport PR created for release-1.20: #2016 ✅

[ehfd]

@cdesiniotis @henry118 Thank you for your time investment for the community!