Add Ubuntu 26.04 driver container with package-based driver install#857
Open
rajathagasthya wants to merge 2 commits into
Open
Add Ubuntu 26.04 driver container with package-based driver install#857rajathagasthya wants to merge 2 commits into
rajathagasthya wants to merge 2 commits into
Conversation
rajathagasthya
force-pushed
the
worktree-driver-package-install
branch
3 times, most recently
from
July 8, 2026 15:14
55e4294 to
96850ac
Compare
The workflow pinned tonistiigi/binfmt:qemu-v6.2.0 (December 2021), which cannot run the Ubuntu 26.04 arm64 userland: the ubuntu26.04 image build hangs indefinitely with no output when its first apt layer starts under emulation, while the same layer completes natively and on older Ubuntu releases under the same emulator. qemu-v8.1.5 fixes the hang but segfaults in libc-bin on Ubuntu 22.04 (tonistiigi/binfmt#240): newer QEMU breaks the old jammy glibc, while the old QEMU cannot run the new resolute userland. No single version handles both, so select the emulator by distribution: ubuntu22.04 keeps qemu-v6.2.0, everything else uses qemu-v8.1.5. All other distributions passed CI on qemu-v8.1.5. v9.x is avoided due to a reported apt regression on fresh Ubuntu images (tonistiigi/binfmt#245). Signed-off-by: Rajath Agasthya <ragasthya@nvidia.com>
For the passthrough driver type, install the NVIDIA driver from apt packages in the ubuntu2604 CUDA repository and build the kernel modules with DKMS, replacing the .run installer. The vGPU driver type keeps the .run flow. The image build preinstalls the driver userspace (nvidia-headless-no-dkms-open plus the display/codec libraries it omits, for .run parity), pinned to the exact release by nvidia-driver-pinning-<version>, and downloads the metapackage shims, the DKMS/kernel-source packages for both kernel module flavors, and nvidia-firmware into a local file: apt repository baked into the image. At container start the flavor-appropriate metapackage (nvidia-open for open/auto, cuda-drivers for proprietary) installs offline from the local repo and DKMS builds the modules against the host kernel headers; dkms autoinstall -k reruns as a check since the package postinst exits 0 even when it skips the build. The fast path installs nothing, so container restarts stay offline as with the baked .run file. Runtime network access is needed only for kernel headers. nvidia-firmware is reinstalled at container start: the GPU operator mounts /lib/firmware in the driver container from the host so the kernel can read the GSP firmware, and that mount hides the copy installed at image build. The reinstall writes the files into the mounted host directory, which the .run installer achieves by writing the firmware at install time. The fast path mounts the driver rootfs before starting the daemons so that GPU initialization always finds the firmware. Ubuntu 26.04 kernel debs ship their payload under usr/lib/modules (merged-usr), so the kernel prerequisites step extracts module files from there. zstd is installed at build so DKMS compresses built modules to match the kernel's module compression. KERNEL_MODULE_TYPE=auto resolves to open without runtime detection: every driver branch in the ubuntu2604 repository (>= 595) supports Turing+ GPUs only, where open is the recommended default. CI builds ubuntu26.04 for 595.71.05; branch 580 is excluded since the ubuntu2604 repository does not carry it. MAX_THREADS (init -m) is not honored on the package path: the NVIDIA dkms.conf hardcodes -j$(nproc). The compressed image is 0.84 GB on amd64, vs 0.68 GB for the ubuntu24.04 .run image built from the same commit. Signed-off-by: Rajath Agasthya <ragasthya@nvidia.com>
rajathagasthya
force-pushed
the
worktree-driver-package-install
branch
from
July 8, 2026 22:01
96850ac to
e5b8c1c
Compare
rajathagasthya
marked this pull request as ready for review
July 8, 2026 22:03
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a new
ubuntu26.04/driver container. For the passthrough (baremetal) driver type itinstalls the NVIDIA driver from apt packages in the
ubuntu2604CUDA repository and builds thekernel modules with DKMS, replacing the
.runinstaller. The vGPU driver type keeps the existing.runflow.How it works
Image build installs the driver userspace (
nvidia-headless-no-dkms-openplus thedisplay/codec libraries, matching what the
.runfile ships today) and the extrapackages (fabricmanager, nscq, imex, nvlsm, libnvsdm). It also downloads the driver metapackages,
the DKMS/kernel-source packages for both kernel module flavors, and
nvidia-firmware(~185 MBtotal) into a local
file:apt repository baked into the image, then removes the CUDA networkrepository so NVIDIA packages resolve only from the local one.
Container start installs the flavor-appropriate driver metapackage from the local
repository —
nvidia-openfor open,cuda-driversfor proprietary. At runtime,aptskips everything already installed at build, so only the DKMS and kernel-source packages
are installed, and DKMS builds the modules against the host kernel headers.
nvidia-firmwareis reinstalled at container start: the GPU operator mounts/lib/firmwareinthe driver container from the host so the kernel can read the GSP firmware, and that mount
hides the copy installed at image build. The reinstall writes the files into the mounted host
directory, which the
.runinstaller achieves by writing the firmware at install time. The fast pathmounts the driver rootfs before starting the daemons so that GPU initialization always finds the firmware.
dkms autoinstall -k $(uname -r)runs afterwards as a check: the DKMS package postinstexits 0 even when it skips the build, so this rerun surfaces any build failure and fails init
instead of continuing without modules.
Fast path (driver already loaded, configuration unchanged) installs nothing: no apt, no
network. Runtime network access is needed only for kernel headers, same as the
.runflow.Package landscape
The two NVIDIA-documented install entry points are
nvidia-open(open) andcuda-drivers(proprietary). Both are empty metapackage shims whose dependency trees share the userspace
and differ only in the kernel-module rows at the bottom:
nvidia-kernel-source[-open]is the module source tree under/usr/src;nvidia-dkms[-open]registers it with DKMS. The two flavors are mutually exclusive (
nvidia-kernel-source-opendeclares
Provides/Conflicts/Replaces: nvidia-kernel-source). This makes the runtimekernel module type change just an
apt-get install.The same trees split by install time: the left side carries the
opentree's dependencies,the right side contains the
proprietarytree's. The packages that come vianvidia-headless-no-dkms-openare boxed.All userspace packages are installed at build time. Container start adds only the metapackage
shims and the DKMS packages that build the kernel modules.
Design decisions
Install from packages, not the
.runfile (passthrough only). Removes the.run/nvidia-installerdependency for the passthrough type.Driver metapackage plus a version pinning package, not a curated package list. The build
installs
nvidia-driver-pinning-<version>, which pins the whole driver tree (drivermetapackage, persistenced, fabricmanager, nscq, imex, libnvsdm) to the exact release, so builds
are reproducible without hand-pinning each package. The metapackage is the NVIDIA-documented
install path and needs no changes when package names change.
Full driver, not headless.
libnvidia-gl,libnvidia-decode/encode/fbc1and the EGLlibraries are kept for parity with the
.runimage. Graphics workloads consume them from thedriver rootfs.
KERNEL_MODULE_TYPE=autoresolves to open; no runtime detection. Every driver branch inthe
ubuntu2604repository (>= 595) supports Turing+ GPUs only, where the open modules are therecommended default, so detection via
nvidia-installer --print-recommended-kernel-module-typewould always return "open". Proprietary remains an explicit opt-in (
cuda-drivers).vGPU keeps the
.runflow. vGPU guest drivers need to be installed using.runfiles fromthe licensing portal with no package equivalent. Only the install step differs between the two
types; module load, daemons, and the rootfs mount are shared.
Driver payload baked into the image; no network dependency at container start. Installing
from the network CUDA repository at runtime would have made every container restart depend on
repository access, which is a regression from the
.runimage where the fast path works offline.Cost: 0.84 GB compressed (amd64) vs 0.68 GB for the ubuntu24.04
.runimage built from thesame commit (arm64: 0.83 GB vs 0.60 GB).
CI
ubuntu2604repository carries only branches >= 595).
resolutetags.tonistiigi/binfmt:qemu-v6.2.0pin (December 2021) cannot run the Ubuntu 26.04 arm64userland: the build hangs indefinitely, with no output, at the first apt layer under
emulation.
qemu-v8.1.5fixes that but segfaults inlibc-binon Ubuntu 22.04(tonistiigi/binfmt#240), so no single
version covers both. ubuntu22.04 keeps
qemu-v6.2.0; every other distribution usesqemu-v8.1.5(all passed CI on it).Known limitations
init -m/MAX_THREADSis not honored on the package path: the NVIDIAdkms.confhardcodes-j$(nproc), so DKMS's own-joption has no effect. The vGPU.runpath still honors it via--concurrency-level.Testing
KERNEL_MODULE_TYPE=proprietary).runflow unchanged