[Bug]: nvidia-cdi-refresh service at boot causing container Vulkan errors

[pauljeary]

Similar to #1611 the nvidia-cdi-refresh service seems to run too early (?) and creates a cdi file that doesn't quite work - in this particular case it is the vulkan drivers that fail in the container as they can't create a surface (e.g. trying to run vkcube).

I'm running Fedora 44 with cosmic desktop.

For now I disable the service

$ sudo systemctl disable nvidia-cdi-refresh.service
Removed '/etc/systemd/system/multi-user.target.wants/nvidia-cdi-refresh.service'.
Disabling 'nvidia-cdi-refresh.service', but its triggering units are still active:
nvidia-cdi-refresh.path

But have to change the location of the generated cdi (by modifying /etc/systemd/system/nvidia-cdi-refresh.service)

#Environment=NVIDIA_CTK_CDI_OUTPUT_FILE_PATH=/var/run/cdi/nvidia.yaml
Environment=NVIDIA_CTK_CDI_OUTPUT_FILE_PATH=/etc/cdi/nvidia.yaml

The log below is from the service running at boot up and then running again (manual restart). The line level=warning msg="Could not locate /dev/nvidia-modeset: /dev/nvidia-modeset: not found" stands out but I'm not sure what that means?

Is there a better workaround/fix that will survive nvidia-ctk updates?

journalctl -u nvidia-cdi-refresh.service -b                                                                                                                                      [11:03:55]
Aug 12 09:58:44 lnx0021 systemd[1]: Starting nvidia-cdi-refresh.service - Refresh NVIDIA CDI specification file...
Aug 12 09:58:45 lnx0021 sh[1970]: GPU 0: NVIDIA GeForce RTX 3080 Ti Laptop GPU (UUID: GPU-8b41da4e-1fb9-c561-350c-2a6584c3b378)
Aug 12 09:58:45 lnx0021 nvidia-ctk[2159]: time="2026-08-12T09:58:45+01:00" level=info msg="Using /usr/lib64/libnvidia-sandboxutils.so.610.57.04"
Aug 12 09:58:45 lnx0021 nvidia-ctk[2159]: time="2026-08-12T09:58:45+01:00" level=info msg="Using /usr/lib64/libnvidia-ml.so.610.57.04"
Aug 12 09:58:45 lnx0021 nvidia-ctk[2159]: time="2026-08-12T09:58:45+01:00" level=info msg="Auto-detected mode as 'nvml'"
Aug 12 09:58:45 lnx0021 nvidia-ctk[2159]: time="2026-08-12T09:58:45+01:00" level=info msg="Using driver version 610.57.04"
Aug 12 09:58:45 lnx0021 nvidia-ctk[2159]: time="2026-08-12T09:58:45+01:00" level=warning msg="Could not locate /dev/nvidia-modeset: /dev/nvidia-modeset: not found"
Aug 12 09:58:45 lnx0021 nvidia-ctk[2159]: time="2026-08-12T09:58:45+01:00" level=info msg="Selecting /dev/nvidia-uvm-tools as /dev/nvidia-uvm-tools"
Aug 12 09:58:45 lnx0021 nvidia-ctk[2159]: time="2026-08-12T09:58:45+01:00" level=info msg="Selecting /dev/nvidia-uvm as /dev/nvidia-uvm"
Aug 12 09:58:45 lnx0021 nvidia-ctk[2159]: time="2026-08-12T09:58:45+01:00" level=info msg="Selecting /dev/nvidiactl as /dev/nvidiactl"
Aug 12 09:58:45 lnx0021 nvidia-ctk[2159]: time="2026-08-12T09:58:45+01:00" level=info msg="Selecting /usr/lib64/libnvidia-egl-gbm.so.1.1.3 as /usr/lib64/libnvidia-egl-gbm.so.1.1.3"
Aug 12 09:58:45 lnx0021 nvidia-ctk[2159]: time="2026-08-12T09:58:45+01:00" level=info msg="Selecting /usr/lib64/libnvidia-egl-wayland.so.1.1.21 as /usr/lib64/libnvidia-egl-wayland.so.1.1.21"
Aug 12 09:58:45 lnx0021 nvidia-ctk[2159]: time="2026-08-12T09:58:45+01:00" level=info msg="Selecting /usr/lib64/libnvidia-egl-wayland2.so.1.0.1 as /usr/lib64/libnvidia-egl-wayland2.so.1.0.1"
Aug 12 09:58:45 lnx0021 nvidia-ctk[2159]: time="2026-08-12T09:58:45+01:00" level=info msg="Selecting /usr/lib64/libnvidia-allocator.so.610.57.04 as /usr/lib64/libnvidia-allocator.so.610.57.>
Aug 12 09:58:45 lnx0021 nvidia-ctk[2159]: time="2026-08-12T09:58:45+01:00" level=warning msg="Could not locate libnvidia-vulkan-producer.so.610.57.04: libnvidia-vulkan-producer.so.610.57.04>
Aug 12 09:58:45 lnx0021 nvidia-ctk[2159]: time="2026-08-12T09:58:45+01:00" level=info msg="Selecting /usr/lib64/xorg/modules/drivers/nvidia_drv.so as /usr/lib64/xorg/modules/drivers/nvidia_>
Aug 12 09:58:45 lnx0021 nvidia-ctk[2159]: time="2026-08-12T09:58:45+01:00" level=warning msg="Could not locate libglxserver_nvidia.so.610.57.04: libglxserver_nvidia.so.610.57.04: not found"
Aug 12 09:58:45 lnx0021 nvidia-ctk[2159]: time="2026-08-12T09:58:45+01:00" level=info msg="Selecting /usr/share/glvnd/egl_vendor.d/10_nvidia.json as /usr/share/glvnd/egl_vendor.d/10_nvidia.>
Aug 12 09:58:45 lnx0021 nvidia-ctk[2159]: time="2026-08-12T09:58:45+01:00" level=info msg="Selecting /usr/share/egl/egl_external_platform.d/15_nvidia_gbm.json as /usr/share/egl/egl_external>
Aug 12 09:58:45 lnx0021 nvidia-ctk[2159]: time="2026-08-12T09:58:45+01:00" level=info msg="Selecting /usr/share/egl/egl_external_platform.d/10_nvidia_wayland.json as /usr/share/egl/egl_exte>
Aug 12 09:58:45 lnx0021 nvidia-ctk[2159]: time="2026-08-12T09:58:45+01:00" level=info msg="Selecting /usr/share/egl/egl_external_platform.d/09_nvidia_wayland2.json as /usr/share/egl/egl_ext>
Aug 12 09:58:45 lnx0021 nvidia-ctk[2159]: time="2026-08-12T09:58:45+01:00" level=info msg="Selecting /usr/share/nvidia/nvoptix.bin as /usr/share/nvidia/nvoptix.bin"
Aug 12 09:58:45 lnx0021 nvidia-ctk[2159]: time="2026-08-12T09:58:45+01:00" level=warning msg="Could not locate X11/xorg.conf.d/10-nvidia.conf: X11/xorg.conf.d/10-nvidia.conf: not found"
Aug 12 09:58:45 lnx0021 nvidia-ctk[2159]: time="2026-08-12T09:58:45+01:00" level=warning msg="Could not locate X11/xorg.conf.d/nvidia-drm-outputclass.conf: X11/xorg.conf.d/nvidia-drm-output>
Aug 12 09:58:45 lnx0021 nvidia-ctk[2159]: time="2026-08-12T09:58:45+01:00" level=warning msg="Could not locate vulkan/icd.d/nvidia_icd.json: vulkan/icd.d/nvidia_icd.json: not found\nvulkan/>
Aug 12 09:58:45 lnx0021 nvidia-ctk[2159]: time="2026-08-12T09:58:45+01:00" level=warning msg="Could not locate vulkan/icd.d/nvidia_layers.json: vulkan/icd.d/nvidia_layers.json: not found\nv>
Aug 12 09:58:45 lnx0021 nvidia-ctk[2159]: time="2026-08-12T09:58:45+01:00" level=info msg="Selecting /usr/share/vulkan/implicit_layer.d/nvidia_layers.json as /etc/vulkan/implicit_layer.d/nv>
Aug 12 09:58:45 lnx0021 nvidia-ctk[2159]: time="2026-08-12T09:58:45+01:00" level=info msg="Selecting /usr/share/vulkan/icd.d/nvidia_icd.x86_64.json as /etc/vulkan/icd.d/nvidia_icd.x86_64.js>
Aug 12 09:58:45 lnx0021 nvidia-ctk[2159]: time="2026-08-12T09:58:45+01:00" level=info msg="Selecting /usr/lib64/libEGL_nvidia.so.610.57.04 as /usr/lib64/libEGL_nvidia.so.610.57.04"
Aug 12 09:58:45 lnx0021 nvidia-ctk[2159]: time="2026-08-12T09:58:45+01:00" level=info msg="Selecting /usr/lib64/libGLESv1_CM_nvidia.so.610.57.04 as /usr/lib64/libGLESv1_CM_nvidia.so.610.57.>
Aug 12 09:58:45 lnx0021 nvidia-ctk[2159]: time="2026-08-12T09:58:45+01:00" level=info msg="Selecting /usr/lib64/libGLESv2_nvidia.so.610.57.04 as /usr/lib64/libGLESv2_nvidia.so.610.57.04"
Aug 12 09:58:45 lnx0021 nvidia-ctk[2159]: time="2026-08-12T09:58:45+01:00" level=info msg="Selecting /usr/lib64/libGLX_nvidia.so.610.57.04 as /usr/lib64/libGLX_nvidia.so.610.57.04"
Aug 12 09:58:45 lnx0021 nvidia-ctk[2159]: time="2026-08-12T09:58:45+01:00" level=info msg="Selecting /usr/lib64/libcuda.so.610.57.04 as /usr/lib64/libcuda.so.610.57.04"
Aug 12 09:58:45 lnx0021 nvidia-ctk[2159]: time="2026-08-12T09:58:45+01:00" level=info msg="Selecting /usr/lib64/libcudadebugger.so.610.57.04 as /usr/lib64/libcudadebugger.so.610.57.04"
Aug 12 09:58:45 lnx0021 nvidia-ctk[2159]: time="2026-08-12T09:58:45+01:00" level=info msg="Selecting /usr/lib64/libnvcuvid.so.610.57.04 as /usr/lib64/libnvcuvid.so.610.57.04"
Aug 12 09:58:45 lnx0021 nvidia-ctk[2159]: time="2026-08-12T09:58:45+01:00" level=info msg="Selecting /usr/lib64/libnvidia-allocator.so.610.57.04 as /usr/lib64/libnvidia-allocator.so.610.57.>
Aug 12 09:58:45 lnx0021 nvidia-ctk[2159]: time="2026-08-12T09:58:45+01:00" level=info msg="Selecting /usr/lib64/libnvidia-cfg.so.610.57.04 as /usr/lib64/libnvidia-cfg.so.610.57.04"
Aug 12 09:58:45 lnx0021 nvidia-ctk[2159]: time="2026-08-12T09:58:45+01:00" level=info msg="Selecting /usr/lib64/libnvidia-eglcore.so.610.57.04 as /usr/lib64/libnvidia-eglcore.so.610.57.04"
Aug 12 09:58:45 lnx0021 nvidia-ctk[2159]: time="2026-08-12T09:58:45+01:00" level=info msg="Selecting /usr/lib64/libnvidia-encode.so.610.57.04 as /usr/lib64/libnvidia-encode.so.610.57.04"
Aug 12 09:58:45 lnx0021 nvidia-ctk[2159]: time="2026-08-12T09:58:45+01:00" level=info msg="Selecting /usr/lib64/libnvidia-fbc.so.610.57.04 as /usr/lib64/libnvidia-fbc.so.610.57.04"
Aug 12 09:58:45 lnx0021 nvidia-ctk[2159]: time="2026-08-12T09:58:45+01:00" level=info msg="Selecting /usr/lib64/libnvidia-glcore.so.610.57.04 as /usr/lib64/libnvidia-glcore.so.610.57.04"
Aug 12 09:58:45 lnx0021 nvidia-ctk[2159]: time="2026-08-12T09:58:45+01:00" level=info msg="Selecting /usr/lib64/libnvidia-glsi.so.610.57.04 as /usr/lib64/libnvidia-glsi.so.610.57.04"
Aug 12 09:58:45 lnx0021 nvidia-ctk[2159]: time="2026-08-12T09:58:45+01:00" level=info msg="Selecting /usr/lib64/libnvidia-glvkspirv.so.610.57.04 as /usr/lib64/libnvidia-glvkspirv.so.610.57.>
Aug 12 09:58:45 lnx0021 nvidia-ctk[2159]: time="2026-08-12T09:58:45+01:00" level=info msg="Selecting /usr/lib64/libnvidia-gpucomp.so.610.57.04 as /usr/lib64/libnvidia-gpucomp.so.610.57.04"
Aug 12 09:58:45 lnx0021 nvidia-ctk[2159]: time="2026-08-12T09:58:45+01:00" level=info msg="Selecting /usr/lib64/libnvidia-gtk3.so.610.57.04 as /usr/lib64/libnvidia-gtk3.so.610.57.04"
Aug 12 09:58:45 lnx0021 nvidia-ctk[2159]: time="2026-08-12T09:58:45+01:00" level=info msg="Selecting /usr/lib64/libnvidia-ml.so.610.57.04 as /usr/lib64/libnvidia-ml.so.610.57.04"
Aug 12 09:58:45 lnx0021 nvidia-ctk[2159]: time="2026-08-12T09:58:45+01:00" level=info msg="Selecting /usr/lib64/libnvidia-ngx.so.610.57.04 as /usr/lib64/libnvidia-ngx.so.610.57.04"
Aug 12 09:58:45 lnx0021 nvidia-ctk[2159]: time="2026-08-12T09:58:45+01:00" level=info msg="Selecting /usr/lib64/libnvidia-nvvm.so.610.57.04 as /usr/lib64/libnvidia-nvvm.so.610.57.04"
Aug 12 09:58:45 lnx0021 nvidia-ctk[2159]: time="2026-08-12T09:58:45+01:00" level=info msg="Selecting /usr/lib64/libnvidia-opencl.so.610.57.04 as /usr/lib64/libnvidia-opencl.so.610.57.04"
Aug 12 09:58:45 lnx0021 nvidia-ctk[2159]: time="2026-08-12T09:58:45+01:00" level=info msg="Selecting /usr/lib64/libnvidia-opticalflow.so.610.57.04 as /usr/lib64/libnvidia-opticalflow.so.610>
Aug 12 09:58:45 lnx0021 nvidia-ctk[2159]: time="2026-08-12T09:58:45+01:00" level=info msg="Selecting /usr/lib64/libnvidia-pkcs11-openssl3.so.610.57.04 as /usr/lib64/libnvidia-pkcs11-openssl>
Aug 12 09:58:45 lnx0021 nvidia-ctk[2159]: time="2026-08-12T09:58:45+01:00" level=info msg="Selecting /usr/lib64/libnvidia-present.so.610.57.04 as /usr/lib64/libnvidia-present.so.610.57.04"
Aug 12 09:58:45 lnx0021 nvidia-ctk[2159]: time="2026-08-12T09:58:45+01:00" level=info msg="Selecting /usr/lib64/libnvidia-ptxjitcompiler.so.610.57.04 as /usr/lib64/libnvidia-ptxjitcompiler.>
Aug 12 09:58:45 lnx0021 nvidia-ctk[2159]: time="2026-08-12T09:58:45+01:00" level=info msg="Selecting /usr/lib64/libnvidia-rtcore.so.610.57.04 as /usr/lib64/libnvidia-rtcore.so.610.57.04"
Aug 12 09:58:45 lnx0021 nvidia-ctk[2159]: time="2026-08-12T09:58:45+01:00" level=info msg="Selecting /usr/lib64/libnvidia-sandboxutils.so.610.57.04 as /usr/lib64/libnvidia-sandboxutils.so.6>
Aug 12 09:58:45 lnx0021 nvidia-ctk[2159]: time="2026-08-12T09:58:45+01:00" level=info msg="Selecting /usr/lib64/libnvidia-tileiras.so.610.57.04 as /usr/lib64/libnvidia-tileiras.so.610.57.04"
Aug 12 09:58:45 lnx0021 nvidia-ctk[2159]: time="2026-08-12T09:58:45+01:00" level=info msg="Selecting /usr/lib64/libnvidia-tls.so.610.57.04 as /usr/lib64/libnvidia-tls.so.610.57.04"
Aug 12 09:58:45 lnx0021 nvidia-ctk[2159]: time="2026-08-12T09:58:45+01:00" level=info msg="Selecting /usr/lib64/libnvidia-vksc-core.so.610.57.04 as /usr/lib64/libnvidia-vksc-core.so.610.57.>
Aug 12 09:58:45 lnx0021 nvidia-ctk[2159]: time="2026-08-12T09:58:45+01:00" level=info msg="Selecting /usr/lib64/libnvidia-wayland-client.so.610.57.04 as /usr/lib64/libnvidia-wayland-client.>
Aug 12 09:58:45 lnx0021 nvidia-ctk[2159]: time="2026-08-12T09:58:45+01:00" level=info msg="Selecting /usr/lib64/libnvoptix.so.610.57.04 as /usr/lib64/libnvoptix.so.610.57.04"
Aug 12 09:58:45 lnx0021 nvidia-ctk[2159]: time="2026-08-12T09:58:45+01:00" level=info msg="Selecting /usr/lib64/vdpau/libvdpau_nvidia.so.610.57.04 as /usr/lib64/vdpau/libvdpau_nvidia.so.610>
Aug 12 09:58:45 lnx0021 nvidia-ctk[2159]: time="2026-08-12T09:58:45+01:00" level=warning msg="Could not locate /nvidia-persistenced/socket: /nvidia-persistenced/socket: not found"
Aug 12 09:58:45 lnx0021 nvidia-ctk[2159]: time="2026-08-12T09:58:45+01:00" level=warning msg="Could not locate /nvidia-fabricmanager/socket: /nvidia-fabricmanager/socket: not found"
Aug 12 09:58:45 lnx0021 nvidia-ctk[2159]: time="2026-08-12T09:58:45+01:00" level=warning msg="Could not locate /tmp/nvidia-mps: /tmp/nvidia-mps: not found"
Aug 12 09:58:45 lnx0021 nvidia-ctk[2159]: time="2026-08-12T09:58:45+01:00" level=info msg="Selecting /lib/firmware/nvidia/610.57.04/gsp_ga10x.bin as /lib/firmware/nvidia/610.57.04/gsp_ga10x>
Aug 12 09:58:45 lnx0021 nvidia-ctk[2159]: time="2026-08-12T09:58:45+01:00" level=info msg="Selecting /lib/firmware/nvidia/610.57.04/gsp_tu10x.bin as /lib/firmware/nvidia/610.57.04/gsp_tu10x>
Aug 12 09:58:45 lnx0021 nvidia-ctk[2159]: time="2026-08-12T09:58:45+01:00" level=info msg="Selecting /usr/bin/nvidia-smi as /usr/bin/nvidia-smi"
Aug 12 09:58:45 lnx0021 nvidia-ctk[2159]: time="2026-08-12T09:58:45+01:00" level=info msg="Selecting /usr/bin/nvidia-debugdump as /usr/bin/nvidia-debugdump"
Aug 12 09:58:45 lnx0021 nvidia-ctk[2159]: time="2026-08-12T09:58:45+01:00" level=info msg="Selecting /usr/bin/nvidia-persistenced as /usr/bin/nvidia-persistenced"
Aug 12 09:58:45 lnx0021 nvidia-ctk[2159]: time="2026-08-12T09:58:45+01:00" level=info msg="Selecting /usr/bin/nvidia-cuda-mps-control as /usr/bin/nvidia-cuda-mps-control"
Aug 12 09:58:45 lnx0021 nvidia-ctk[2159]: time="2026-08-12T09:58:45+01:00" level=info msg="Selecting /usr/bin/nvidia-cuda-mps-server as /usr/bin/nvidia-cuda-mps-server"
Aug 12 09:58:45 lnx0021 nvidia-ctk[2159]: time="2026-08-12T09:58:45+01:00" level=warning msg="Could not locate nvidia-imex: nvidia-imex: not found"
Aug 12 09:58:45 lnx0021 nvidia-ctk[2159]: time="2026-08-12T09:58:45+01:00" level=warning msg="Could not locate nvidia-imex-ctl: nvidia-imex-ctl: not found"
Aug 12 09:58:45 lnx0021 nvidia-ctk[2159]: time="2026-08-12T09:58:45+01:00" level=info msg="Generated CDI spec with version 0.7.0"
Aug 12 09:58:45 lnx0021 systemd[1]: nvidia-cdi-refresh.service: Deactivated successfully.
Aug 12 09:58:45 lnx0021 systemd[1]: Finished nvidia-cdi-refresh.service - Refresh NVIDIA CDI specification file.
Aug 12 09:58:45 lnx0021 systemd[1]: nvidia-cdi-refresh.service: Consumed 334ms CPU time over 643ms wall clock time, 90.1M memory peak.
Aug 12 10:00:42 lnx0021 systemd[1]: Starting nvidia-cdi-refresh.service - Refresh NVIDIA CDI specification file...
Aug 12 10:00:42 lnx0021 sh[10844]: GPU 0: NVIDIA GeForce RTX 3080 Ti Laptop GPU (UUID: GPU-8b41da4e-1fb9-c561-350c-2a6584c3b378)
Aug 12 10:00:42 lnx0021 nvidia-ctk[10846]: time="2026-08-12T10:00:42+01:00" level=info msg="Using /usr/lib64/libnvidia-sandboxutils.so.610.57.04"
Aug 12 10:00:42 lnx0021 nvidia-ctk[10846]: time="2026-08-12T10:00:42+01:00" level=info msg="Using /usr/lib64/libnvidia-ml.so.610.57.04"
Aug 12 10:00:42 lnx0021 nvidia-ctk[10846]: time="2026-08-12T10:00:42+01:00" level=info msg="Auto-detected mode as 'nvml'"
Aug 12 10:00:42 lnx0021 nvidia-ctk[10846]: time="2026-08-12T10:00:42+01:00" level=info msg="Using driver version 610.57.04"
Aug 12 10:00:42 lnx0021 nvidia-ctk[10846]: time="2026-08-12T10:00:42+01:00" level=info msg="Selecting /dev/nvidia-modeset as /dev/nvidia-modeset"
Aug 12 10:00:42 lnx0021 nvidia-ctk[10846]: time="2026-08-12T10:00:42+01:00" level=info msg="Selecting /dev/nvidia-uvm-tools as /dev/nvidia-uvm-tools"
Aug 12 10:00:42 lnx0021 nvidia-ctk[10846]: time="2026-08-12T10:00:42+01:00" level=info msg="Selecting /dev/nvidia-uvm as /dev/nvidia-uvm"
Aug 12 10:00:42 lnx0021 nvidia-ctk[10846]: time="2026-08-12T10:00:42+01:00" level=info msg="Selecting /dev/nvidiactl as /dev/nvidiactl"
Aug 12 10:00:42 lnx0021 nvidia-ctk[10846]: time="2026-08-12T10:00:42+01:00" level=info msg="Selecting /usr/lib64/libnvidia-egl-gbm.so.1.1.3 as /usr/lib64/libnvidia-egl-gbm.so.1.1.3"
Aug 12 10:00:42 lnx0021 nvidia-ctk[10846]: time="2026-08-12T10:00:42+01:00" level=info msg="Selecting /usr/lib64/libnvidia-egl-wayland.so.1.1.21 as /usr/lib64/libnvidia-egl-wayland.so.1.1.2>
Aug 12 10:00:42 lnx0021 nvidia-ctk[10846]: time="2026-08-12T10:00:42+01:00" level=info msg="Selecting /usr/lib64/libnvidia-egl-wayland2.so.1.0.1 as /usr/lib64/libnvidia-egl-wayland2.so.1.0.>
Aug 12 10:00:42 lnx0021 nvidia-ctk[10846]: time="2026-08-12T10:00:42+01:00" level=info msg="Selecting /usr/lib64/libnvidia-allocator.so.610.57.04 as /usr/lib64/libnvidia-allocator.so.610.57>
Aug 12 10:00:42 lnx0021 nvidia-ctk[10846]: time="2026-08-12T10:00:42+01:00" level=warning msg="Could not locate libnvidia-vulkan-producer.so.610.57.04: libnvidia-vulkan-producer.so.610.57.0>
Aug 12 10:00:42 lnx0021 nvidia-ctk[10846]: time="2026-08-12T10:00:42+01:00" level=info msg="Selecting /usr/lib64/xorg/modules/drivers/nvidia_drv.so as /usr/lib64/xorg/modules/drivers/nvidia>
Aug 12 10:00:42 lnx0021 nvidia-ctk[10846]: time="2026-08-12T10:00:42+01:00" level=warning msg="Could not locate libglxserver_nvidia.so.610.57.04: libglxserver_nvidia.so.610.57.04: not found"
Aug 12 10:00:42 lnx0021 nvidia-ctk[10846]: time="2026-08-12T10:00:42+01:00" level=info msg="Selecting /usr/share/glvnd/egl_vendor.d/10_nvidia.json as /usr/share/glvnd/egl_vendor.d/10_nvidia>
Aug 12 10:00:42 lnx0021 nvidia-ctk[10846]: time="2026-08-12T10:00:42+01:00" level=info msg="Selecting /usr/share/egl/egl_external_platform.d/15_nvidia_gbm.json as /usr/share/egl/egl_externa>
Aug 12 10:00:42 lnx0021 nvidia-ctk[10846]: time="2026-08-12T10:00:42+01:00" level=info msg="Selecting /usr/share/egl/egl_external_platform.d/10_nvidia_wayland.json as /usr/share/egl/egl_ext>
Aug 12 10:00:42 lnx0021 nvidia-ctk[10846]: time="2026-08-12T10:00:42+01:00" level=info msg="Selecting /usr/share/egl/egl_external_platform.d/09_nvidia_wayland2.json as /usr/share/egl/egl_ex>
Aug 12 10:00:42 lnx0021 nvidia-ctk[10846]: time="2026-08-12T10:00:42+01:00" level=info msg="Selecting /usr/share/nvidia/nvoptix.bin as /usr/share/nvidia/nvoptix.bin"
Aug 12 10:00:42 lnx0021 nvidia-ctk[10846]: time="2026-08-12T10:00:42+01:00" level=warning msg="Could not locate X11/xorg.conf.d/10-nvidia.conf: X11/xorg.conf.d/10-nvidia.conf: not found"
Aug 12 10:00:42 lnx0021 nvidia-ctk[10846]: time="2026-08-12T10:00:42+01:00" level=warning msg="Could not locate X11/xorg.conf.d/nvidia-drm-outputclass.conf: X11/xorg.conf.d/nvidia-drm-outpu>
Aug 12 10:00:42 lnx0021 nvidia-ctk[10846]: time="2026-08-12T10:00:42+01:00" level=warning msg="Could not locate vulkan/icd.d/nvidia_icd.json: vulkan/icd.d/nvidia_icd.json: not found\nvulkan>
Aug 12 10:00:42 lnx0021 nvidia-ctk[10846]: time="2026-08-12T10:00:42+01:00" level=warning msg="Could not locate vulkan/icd.d/nvidia_layers.json: vulkan/icd.d/nvidia_layers.json: not found\n>
Aug 12 10:00:42 lnx0021 nvidia-ctk[10846]: time="2026-08-12T10:00:42+01:00" level=info msg="Selecting /usr/share/vulkan/implicit_layer.d/nvidia_layers.json as /etc/vulkan/implicit_layer.d/n>
Aug 12 10:00:42 lnx0021 nvidia-ctk[10846]: time="2026-08-12T10:00:42+01:00" level=info msg="Selecting /usr/share/vulkan/icd.d/nvidia_icd.x86_64.json as /etc/vulkan/icd.d/nvidia_icd.x86_64.j>
Aug 12 10:00:42 lnx0021 nvidia-ctk[10846]: time="2026-08-12T10:00:42+01:00" level=info msg="Selecting /usr/lib64/libEGL_nvidia.so.610.57.04 as /usr/lib64/libEGL_nvidia.so.610.57.04"
Aug 12 10:00:42 lnx0021 nvidia-ctk[10846]: time="2026-08-12T10:00:42+01:00" level=info msg="Selecting /usr/lib64/libGLESv1_CM_nvidia.so.610.57.04 as /usr/lib64/libGLESv1_CM_nvidia.so.610.57>
Aug 12 10:00:42 lnx0021 nvidia-ctk[10846]: time="2026-08-12T10:00:42+01:00" level=info msg="Selecting /usr/lib64/libGLESv2_nvidia.so.610.57.04 as /usr/lib64/libGLESv2_nvidia.so.610.57.04"
Aug 12 10:00:42 lnx0021 nvidia-ctk[10846]: time="2026-08-12T10:00:42+01:00" level=info msg="Selecting /usr/lib64/libGLX_nvidia.so.610.57.04 as /usr/lib64/libGLX_nvidia.so.610.57.04"
Aug 12 10:00:42 lnx0021 nvidia-ctk[10846]: time="2026-08-12T10:00:42+01:00" level=info msg="Selecting /usr/lib64/libcuda.so.610.57.04 as /usr/lib64/libcuda.so.610.57.04"
Aug 12 10:00:42 lnx0021 nvidia-ctk[10846]: time="2026-08-12T10:00:42+01:00" level=info msg="Selecting /usr/lib64/libcudadebugger.so.610.57.04 as /usr/lib64/libcudadebugger.so.610.57.04"
Aug 12 10:00:42 lnx0021 nvidia-ctk[10846]: time="2026-08-12T10:00:42+01:00" level=info msg="Selecting /usr/lib64/libnvcuvid.so.610.57.04 as /usr/lib64/libnvcuvid.so.610.57.04"
Aug 12 10:00:42 lnx0021 nvidia-ctk[10846]: time="2026-08-12T10:00:42+01:00" level=info msg="Selecting /usr/lib64/libnvidia-allocator.so.610.57.04 as /usr/lib64/libnvidia-allocator.so.610.57>
Aug 12 10:00:42 lnx0021 nvidia-ctk[10846]: time="2026-08-12T10:00:42+01:00" level=info msg="Selecting /usr/lib64/libnvidia-cfg.so.610.57.04 as /usr/lib64/libnvidia-cfg.so.610.57.04"
Aug 12 10:00:42 lnx0021 nvidia-ctk[10846]: time="2026-08-12T10:00:42+01:00" level=info msg="Selecting /usr/lib64/libnvidia-eglcore.so.610.57.04 as /usr/lib64/libnvidia-eglcore.so.610.57.04"
Aug 12 10:00:42 lnx0021 nvidia-ctk[10846]: time="2026-08-12T10:00:42+01:00" level=info msg="Selecting /usr/lib64/libnvidia-encode.so.610.57.04 as /usr/lib64/libnvidia-encode.so.610.57.04"
Aug 12 10:00:42 lnx0021 nvidia-ctk[10846]: time="2026-08-12T10:00:42+01:00" level=info msg="Selecting /usr/lib64/libnvidia-fbc.so.610.57.04 as /usr/lib64/libnvidia-fbc.so.610.57.04"
Aug 12 10:00:42 lnx0021 nvidia-ctk[10846]: time="2026-08-12T10:00:42+01:00" level=info msg="Selecting /usr/lib64/libnvidia-glcore.so.610.57.04 as /usr/lib64/libnvidia-glcore.so.610.57.04"
Aug 12 10:00:42 lnx0021 nvidia-ctk[10846]: time="2026-08-12T10:00:42+01:00" level=info msg="Selecting /usr/lib64/libnvidia-glsi.so.610.57.04 as /usr/lib64/libnvidia-glsi.so.610.57.04"
Aug 12 10:00:42 lnx0021 nvidia-ctk[10846]: time="2026-08-12T10:00:42+01:00" level=info msg="Selecting /usr/lib64/libnvidia-glvkspirv.so.610.57.04 as /usr/lib64/libnvidia-glvkspirv.so.610.57>
Aug 12 10:00:42 lnx0021 nvidia-ctk[10846]: time="2026-08-12T10:00:42+01:00" level=info msg="Selecting /usr/lib64/libnvidia-gpucomp.so.610.57.04 as /usr/lib64/libnvidia-gpucomp.so.610.57.04"
Aug 12 10:00:42 lnx0021 nvidia-ctk[10846]: time="2026-08-12T10:00:42+01:00" level=info msg="Selecting /usr/lib64/libnvidia-gtk3.so.610.57.04 as /usr/lib64/libnvidia-gtk3.so.610.57.04"
Aug 12 10:00:42 lnx0021 nvidia-ctk[10846]: time="2026-08-12T10:00:42+01:00" level=info msg="Selecting /usr/lib64/libnvidia-ml.so.610.57.04 as /usr/lib64/libnvidia-ml.so.610.57.04"
Aug 12 10:00:42 lnx0021 nvidia-ctk[10846]: time="2026-08-12T10:00:42+01:00" level=info msg="Selecting /usr/lib64/libnvidia-ngx.so.610.57.04 as /usr/lib64/libnvidia-ngx.so.610.57.04"
Aug 12 10:00:42 lnx0021 nvidia-ctk[10846]: time="2026-08-12T10:00:42+01:00" level=info msg="Selecting /usr/lib64/libnvidia-nvvm.so.610.57.04 as /usr/lib64/libnvidia-nvvm.so.610.57.04"
Aug 12 10:00:42 lnx0021 nvidia-ctk[10846]: time="2026-08-12T10:00:42+01:00" level=info msg="Selecting /usr/lib64/libnvidia-opencl.so.610.57.04 as /usr/lib64/libnvidia-opencl.so.610.57.04"
Aug 12 10:00:42 lnx0021 nvidia-ctk[10846]: time="2026-08-12T10:00:42+01:00" level=info msg="Selecting /usr/lib64/libnvidia-opticalflow.so.610.57.04 as /usr/lib64/libnvidia-opticalflow.so.61>
Aug 12 10:00:42 lnx0021 nvidia-ctk[10846]: time="2026-08-12T10:00:42+01:00" level=info msg="Selecting /usr/lib64/libnvidia-pkcs11-openssl3.so.610.57.04 as /usr/lib64/libnvidia-pkcs11-openss>
Aug 12 10:00:42 lnx0021 nvidia-ctk[10846]: time="2026-08-12T10:00:42+01:00" level=info msg="Selecting /usr/lib64/libnvidia-present.so.610.57.04 as /usr/lib64/libnvidia-present.so.610.57.04"
Aug 12 10:00:42 lnx0021 nvidia-ctk[10846]: time="2026-08-12T10:00:42+01:00" level=info msg="Selecting /usr/lib64/libnvidia-ptxjitcompiler.so.610.57.04 as /usr/lib64/libnvidia-ptxjitcompiler>
Aug 12 10:00:42 lnx0021 nvidia-ctk[10846]: time="2026-08-12T10:00:42+01:00" level=info msg="Selecting /usr/lib64/libnvidia-rtcore.so.610.57.04 as /usr/lib64/libnvidia-rtcore.so.610.57.04"
Aug 12 10:00:42 lnx0021 nvidia-ctk[10846]: time="2026-08-12T10:00:42+01:00" level=info msg="Selecting /usr/lib64/libnvidia-sandboxutils.so.610.57.04 as /usr/lib64/libnvidia-sandboxutils.so.>
Aug 12 10:00:42 lnx0021 nvidia-ctk[10846]: time="2026-08-12T10:00:42+01:00" level=info msg="Selecting /usr/lib64/libnvidia-tileiras.so.610.57.04 as /usr/lib64/libnvidia-tileiras.so.610.57.0>
Aug 12 10:00:42 lnx0021 nvidia-ctk[10846]: time="2026-08-12T10:00:42+01:00" level=info msg="Selecting /usr/lib64/libnvidia-tls.so.610.57.04 as /usr/lib64/libnvidia-tls.so.610.57.04"
Aug 12 10:00:42 lnx0021 nvidia-ctk[10846]: time="2026-08-12T10:00:42+01:00" level=info msg="Selecting /usr/lib64/libnvidia-vksc-core.so.610.57.04 as /usr/lib64/libnvidia-vksc-core.so.610.57>
Aug 12 10:00:42 lnx0021 nvidia-ctk[10846]: time="2026-08-12T10:00:42+01:00" level=info msg="Selecting /usr/lib64/libnvidia-wayland-client.so.610.57.04 as /usr/lib64/libnvidia-wayland-client>
Aug 12 10:00:42 lnx0021 nvidia-ctk[10846]: time="2026-08-12T10:00:42+01:00" level=info msg="Selecting /usr/lib64/libnvoptix.so.610.57.04 as /usr/lib64/libnvoptix.so.610.57.04"
Aug 12 10:00:42 lnx0021 nvidia-ctk[10846]: time="2026-08-12T10:00:42+01:00" level=info msg="Selecting /usr/lib64/vdpau/libvdpau_nvidia.so.610.57.04 as /usr/lib64/vdpau/libvdpau_nvidia.so.61>
Aug 12 10:00:42 lnx0021 nvidia-ctk[10846]: time="2026-08-12T10:00:42+01:00" level=warning msg="Could not locate /nvidia-persistenced/socket: /nvidia-persistenced/socket: not found"
Aug 12 10:00:42 lnx0021 nvidia-ctk[10846]: time="2026-08-12T10:00:42+01:00" level=warning msg="Could not locate /nvidia-fabricmanager/socket: /nvidia-fabricmanager/socket: not found"
Aug 12 10:00:42 lnx0021 nvidia-ctk[10846]: time="2026-08-12T10:00:42+01:00" level=warning msg="Could not locate /tmp/nvidia-mps: /tmp/nvidia-mps: not found"
Aug 12 10:00:42 lnx0021 nvidia-ctk[10846]: time="2026-08-12T10:00:42+01:00" level=info msg="Selecting /lib/firmware/nvidia/610.57.04/gsp_ga10x.bin as /lib/firmware/nvidia/610.57.04/gsp_ga10>
Aug 12 10:00:42 lnx0021 nvidia-ctk[10846]: time="2026-08-12T10:00:42+01:00" level=info msg="Selecting /lib/firmware/nvidia/610.57.04/gsp_tu10x.bin as /lib/firmware/nvidia/610.57.04/gsp_tu10>
Aug 12 10:00:42 lnx0021 nvidia-ctk[10846]: time="2026-08-12T10:00:42+01:00" level=info msg="Selecting /usr/bin/nvidia-smi as /usr/bin/nvidia-smi"
Aug 12 10:00:42 lnx0021 nvidia-ctk[10846]: time="2026-08-12T10:00:42+01:00" level=info msg="Selecting /usr/bin/nvidia-debugdump as /usr/bin/nvidia-debugdump"
Aug 12 10:00:42 lnx0021 nvidia-ctk[10846]: time="2026-08-12T10:00:42+01:00" level=info msg="Selecting /usr/bin/nvidia-persistenced as /usr/bin/nvidia-persistenced"
Aug 12 10:00:42 lnx0021 nvidia-ctk[10846]: time="2026-08-12T10:00:42+01:00" level=info msg="Selecting /usr/bin/nvidia-cuda-mps-control as /usr/bin/nvidia-cuda-mps-control"
Aug 12 10:00:42 lnx0021 nvidia-ctk[10846]: time="2026-08-12T10:00:42+01:00" level=info msg="Selecting /usr/bin/nvidia-cuda-mps-server as /usr/bin/nvidia-cuda-mps-server"
Aug 12 10:00:42 lnx0021 nvidia-ctk[10846]: time="2026-08-12T10:00:42+01:00" level=warning msg="Could not locate nvidia-imex: nvidia-imex: not found"
Aug 12 10:00:42 lnx0021 nvidia-ctk[10846]: time="2026-08-12T10:00:42+01:00" level=warning msg="Could not locate nvidia-imex-ctl: nvidia-imex-ctl: not found"
Aug 12 10:00:42 lnx0021 nvidia-ctk[10846]: time="2026-08-12T10:00:42+01:00" level=info msg="Generated CDI spec with version 0.7.0"
Aug 12 10:00:42 lnx0021 systemd[1]: nvidia-cdi-refresh.service: Deactivated successfully.
Aug 12 10:00:42 lnx0021 systemd[1]: Finished nvidia-cdi-refresh.service - Refresh NVIDIA CDI specification file.

[henry118]

Could be related to #1979

@pauljeary Can you confirm if /dev/nvidia-modeset exists on your system?

If not, does running the following command before running the container resolve the issue?

sudo nvidia-ctk system create-device-nodes --control-devices --load-kernel-modules

[ehfd]

I believe that #1979 is a fix for this issue.

Explained in #1979 (comment).

[pauljeary]

Could be related to #1979

@pauljeary Can you confirm if /dev/nvidia-modeset exists on your system?

If not, does running the following command before running the container resolve the issue?

sudo nvidia-ctk system create-device-nodes --control-devices --load-kernel-modules

Hi thanks for picking this up, yes /dev/nvidia-modeset exists at the point I want to run the containers (interactively).

Don't know if these logs help?

sudo journalctl -b -k | grep -iE 'nvidia|modeset' 
Aug 18 09:17:22 lnx0021 kernel: NVRM: loading NVIDIA UNIX Open Kernel Module for x86_64  610.57.04  Release Build  (dvs-builder@U22-I3-AF05-32-1)  Wed Jul 29 02:45:17 UTC 2026
Aug 18 09:17:22 lnx0021 kernel: nvidia-modeset: Loading NVIDIA UNIX Open Kernel Mode Setting Driver for x86_64  610.57.04  Release Build  (dvs-builder@U22-I3-AF05-32-1)  Wed Jul 29 02:31:30 UTC 2026
Aug 18 09:17:22 lnx0021 kernel: [drm] [nvidia-drm] [GPU ID 0x00000100] Loading driver
Aug 18 09:17:24 lnx0021 kernel: [drm] Initialized nvidia-drm 0.0.0 for 0000:01:00.0 on minor 0
Aug 18 09:17:24 lnx0021 kernel: nvidia 0000:01:00.0: [drm] fb1: nvidia-drmdrmfb frame buffer device
Aug 18 09:19:18 lnx0021 kernel: vkcube[9599]: segfault at 268 ip 00007fb2b054ce20 sp 00007fffbc5218b0 error 4 in libnvidia-glcore.so.610.57.04[d4ce20,7fb2afc00000+c00000] likely on CPU 4 (core 8, socket 0)

systemctl status nvidia-cdi-refresh.service
...
Aug 18 09:17:24 lnx0021 systemd[1]: nvidia-cdi-refresh.service: Deactivated successfully.

[ehfd]

@pauljeary It depends on whether that device is correctly inside the container or not.

[henry118]

@pauljeary Missing nvidia-modeset device within the container is likely causing the segfault.

Please confirm whether that device is found inside the container. It could be the generated CDI spec did not include it due to the issue described in #1979.

If /dev/nvidia-modeset is not found inside the container or in the CDI spec (but exists on host), a manual workaround is regenerate the spec by running:

systemctl restart nvidia-cdi-refresh.service

[pauljeary]

so, no - it's not found in the container or in the CDI spec - and yes this seems to be a duplicate of #1979 so happy to close this bug and await a resolution to that.

The workaround is fine (although updating the location of NVIDIA_CTK_CDI_OUTPUT_FILE_PATH=/etc/cdi/nvidia.yaml in /etc/nvidia-container-toolkit/nvidia-cdi-refresh.env and disabling the service works better for me so I don't have to restart the service after every boot).

[ehfd]

Addressed reviewer comments in #1979 to accommodate the direction of @henry118.