Skip to content

Add Ubuntu 26.04 driver container with package-based driver install#857

Open
rajathagasthya wants to merge 2 commits into
mainfrom
worktree-driver-package-install
Open

Add Ubuntu 26.04 driver container with package-based driver install#857
rajathagasthya wants to merge 2 commits into
mainfrom
worktree-driver-package-install

Conversation

@rajathagasthya

@rajathagasthya rajathagasthya commented Jul 7, 2026

Copy link
Copy Markdown
Contributor

Adds a new ubuntu26.04/ driver container. For the passthrough (baremetal) driver type it
installs the NVIDIA driver from apt packages in the ubuntu2604 CUDA repository and builds the
kernel modules with DKMS, replacing the .run installer. The vGPU driver type keeps the existing
.run flow.

How it works

Image build installs the driver userspace (nvidia-headless-no-dkms-open plus the
display/codec libraries, matching what the .run file ships today) and the extra
packages (fabricmanager, nscq, imex, nvlsm, libnvsdm). It also downloads the driver metapackages,
the DKMS/kernel-source packages for both kernel module flavors, and nvidia-firmware (~185 MB
total) into a local file: apt repository baked into the image, then removes the CUDA network
repository so NVIDIA packages resolve only from the local one.

Container start installs the flavor-appropriate driver metapackage from the local
repository — nvidia-open for open, cuda-drivers for proprietary. At runtime, apt
skips everything already installed at build, so only the DKMS and kernel-source packages
are installed, and DKMS builds the modules against the host kernel headers.
nvidia-firmware is reinstalled at container start: the GPU operator mounts /lib/firmware in
the driver container from the host so the kernel can read the GSP firmware, and that mount
hides the copy installed at image build. The reinstall writes the files into the mounted host
directory, which the .run installer achieves by writing the firmware at install time. The fast path
mounts the driver rootfs before starting the daemons so that GPU initialization always finds the firmware.
dkms autoinstall -k $(uname -r) runs afterwards as a check: the DKMS package postinst
exits 0 even when it skips the build, so this rerun surfaces any build failure and fails init
instead of continuing without modules.

Fast path (driver already loaded, configuration unchanged) installs nothing: no apt, no
network. Runtime network access is needed only for kernel headers, same as the .run flow.

Package landscape

The two NVIDIA-documented install entry points are nvidia-open (open) and cuda-drivers
(proprietary). Both are empty metapackage shims whose dependency trees share the userspace
and differ only in the kernel-module rows at the bottom:

nvidia-open (shim)                          cuda-drivers (shim)
└── nvidia-driver-open (shim)               └── nvidia-driver (shim)
    ├── libnvidia-compute  ← nvidia-smi, NVML, libcuda, MPS ──┐
    │     └── nvidia-persistenced                             │
    ├── libnvidia-gl (273 MB: GLX/EGL/Vulkan ICDs)            │  identical
    ├── libnvidia-decode / libnvidia-encode / libnvidia-fbc1  │  userspace
    ├── nvidia-kernel-common                                  │  on both
    │     ├── nvidia-firmware  ← GSP firmware                 │  sides
    │     └── nvidia-modprobe ────────────────────────────────┘
    │
    ├── nvidia-kernel-source-open   |   ├── nvidia-kernel-source (~88 MB)
    └── nvidia-dkms-open            |   └── nvidia-dkms
          └── dkms (make, patch, …) |         └── dkms

nvidia-kernel-source[-open] is the module source tree under /usr/src; nvidia-dkms[-open]
registers it with DKMS. The two flavors are mutually exclusive (nvidia-kernel-source-open
declares Provides/Conflicts/Replaces: nvidia-kernel-source). This makes the runtime
kernel module type change just an apt-get install.

The same trees split by install time: the left side carries the open tree's dependencies,
the right side contains the proprietary tree's. The packages that come via
nvidia-headless-no-dkms-open are boxed.

CONTAINER START — installed from the local file: repo, per KERNEL_MODULE_TYPE
═══════════════════════════════════════════════════════════════════════════════════
 open/auto: 3 debs, <100 KB                proprietary: 4 debs, ~89 MB

 nvidia-open (shim)                        cuda-drivers (shim)
 └── nvidia-driver-open (shim)             └── nvidia-driver (shim)
     ├── nvidia-dkms-open                      ├── nvidia-dkms
     │                                         ├── nvidia-kernel-source (~88 MB)
     │                                         │     evicts nvidia-kernel-source-open
     │                                         └──────────────────────────────────┐
═════╪════════════════════════════════════════════════════════════════════════════╪
     │  BUILD TIME — userspace_install, baked into the image rootfs (~1 GB)       │
     │   ┌─ via nvidia-headless-no-dkms-open (shim) ──────────────────────────┐   │
     ├───┼─ libnvidia-compute          ← nvidia-smi, NVML, libcuda, MPS       ┼───┤
     │   │    └── nvidia-persistenced                                         │   │
     ├───┼─ nvidia-kernel-common                                              ┼───┤
     │   │    ├── nvidia-firmware       ← GSP firmware                        │   │
     │   │    └── nvidia-modprobe                                             │   │
     ├───┼─ nvidia-kernel-source-open  ← open module source; open tree only   │   │
     │   └────────────────────────────────────────────────────────────────────┘   │
     │   ┌─ added explicitly (the headless shim omits them) ──────────────────┐   │
     ├───┼─ libnvidia-gl (273 MB)      ← GLX/EGL/Vulkan ICDs                  ┼───┤
     ├───┼─ libnvidia-decode / libnvidia-encode / libnvidia-fbc1              ┼───┤
     │   │   libnvidia-extra            (outside both metapackage trees)      │   │
     └───┼─ dkms                       ← via nvidia-dkms[-open]               ┼───┘
         └────────────────────────────────────────────────────────────────────┘
 left side = nvidia-driver-open (open)
 right side = nvidia-driver (proprietary)

All userspace packages are installed at build time. Container start adds only the metapackage
shims and the DKMS packages that build the kernel modules.

Design decisions

  1. Install from packages, not the .run file (passthrough only). Removes the
    .run/nvidia-installer dependency for the passthrough type.

  2. Driver metapackage plus a version pinning package, not a curated package list. The build
    installs nvidia-driver-pinning-<version>, which pins the whole driver tree (driver
    metapackage, persistenced, fabricmanager, nscq, imex, libnvsdm) to the exact release, so builds
    are reproducible without hand-pinning each package. The metapackage is the NVIDIA-documented
    install path and needs no changes when package names change.

  3. Full driver, not headless. libnvidia-gl, libnvidia-decode/encode/fbc1 and the EGL
    libraries are kept for parity with the .run image. Graphics workloads consume them from the
    driver rootfs.

  4. KERNEL_MODULE_TYPE=auto resolves to open; no runtime detection. Every driver branch in
    the ubuntu2604 repository (>= 595) supports Turing+ GPUs only, where the open modules are the
    recommended default, so detection via nvidia-installer --print-recommended-kernel-module-type
    would always return "open". Proprietary remains an explicit opt-in (cuda-drivers).

  5. vGPU keeps the .run flow. vGPU guest drivers need to be installed using .run files from
    the licensing portal with no package equivalent. Only the install step differs between the two
    types; module load, daemons, and the rootfs mount are shared.

  6. Driver payload baked into the image; no network dependency at container start. Installing
    from the network CUDA repository at runtime would have made every container restart depend on
    repository access, which is a regression from the .run image where the fast path works offline.
    Cost: 0.84 GB compressed (amd64) vs 0.68 GB for the ubuntu24.04 .run image built from the
    same commit (arm64: 0.83 GB vs 0.60 GB).

CI

  • GitHub image build matrix includes ubuntu26.04, excluding the 580 driver (the ubuntu2604
    repository carries only branches >= 595).
  • Renovate pins the base image to resolute tags.
  • The QEMU emulator for arm64 cross-builds is now pinned per distribution. The previous
    tonistiigi/binfmt:qemu-v6.2.0 pin (December 2021) cannot run the Ubuntu 26.04 arm64
    userland: the build hangs indefinitely, with no output, at the first apt layer under
    emulation. qemu-v8.1.5 fixes that but segfaults in libc-bin on Ubuntu 22.04
    (tonistiigi/binfmt#240), so no single
    version covers both. ubuntu22.04 keeps qemu-v6.2.0; every other distribution uses
    qemu-v8.1.5 (all passed CI on it).

Known limitations

  • init -m / MAX_THREADS is not honored on the package path: the NVIDIA dkms.conf hardcodes
    -j$(nproc), so DKMS's own -j option has no effect. The vGPU .run path still honors it via
    --concurrency-level.

Testing

  • Local image build (amd64)
  • Container start on a GPU node: open flavor
  • Container start: proprietary flavor (KERNEL_MODULE_TYPE=proprietary)
  • Offline container start (no network): first boot and fast path
  • vGPU .run flow unchanged
  • arm64 (sbsa) build

@rajathagasthya
rajathagasthya force-pushed the worktree-driver-package-install branch 3 times, most recently from 55e4294 to 96850ac Compare July 8, 2026 15:14
The workflow pinned tonistiigi/binfmt:qemu-v6.2.0 (December 2021),
which cannot run the Ubuntu 26.04 arm64 userland: the ubuntu26.04
image build hangs indefinitely with no output when its first apt layer
starts under emulation, while the same layer completes natively and on
older Ubuntu releases under the same emulator. qemu-v8.1.5 fixes the
hang but segfaults in libc-bin on Ubuntu 22.04
(tonistiigi/binfmt#240): newer QEMU breaks the old jammy glibc, while
the old QEMU cannot run the new resolute userland.

No single version handles both, so select the emulator by
distribution: ubuntu22.04 keeps qemu-v6.2.0, everything else uses
qemu-v8.1.5. All other distributions passed CI on qemu-v8.1.5. v9.x is
avoided due to a reported apt regression on fresh Ubuntu images
(tonistiigi/binfmt#245).

Signed-off-by: Rajath Agasthya <ragasthya@nvidia.com>
For the passthrough driver type, install the NVIDIA driver from apt
packages in the ubuntu2604 CUDA repository and build the kernel modules
with DKMS, replacing the .run installer. The vGPU driver type keeps the
.run flow.

The image build preinstalls the driver userspace
(nvidia-headless-no-dkms-open plus the display/codec libraries it
omits, for .run parity), pinned to the exact release by
nvidia-driver-pinning-<version>, and downloads the metapackage shims,
the DKMS/kernel-source packages for both kernel module flavors, and
nvidia-firmware into a local file: apt repository baked into the image.
At container start the flavor-appropriate metapackage (nvidia-open for
open/auto, cuda-drivers for proprietary) installs offline from the
local repo and DKMS builds the modules against the host kernel headers;
dkms autoinstall -k reruns as a check since the package postinst exits
0 even when it skips the build. The fast path installs nothing, so
container restarts stay offline as with the baked .run file. Runtime
network access is needed only for kernel headers.

nvidia-firmware is reinstalled at container start: the GPU operator
mounts /lib/firmware in the driver container from the host so the
kernel can read the GSP firmware, and that mount hides the copy
installed at image build. The reinstall writes the files into the
mounted host directory, which the .run installer achieves by writing
the firmware at install time. The fast path mounts the driver rootfs
before starting the daemons so that GPU initialization always finds
the firmware.

Ubuntu 26.04 kernel debs ship their payload under usr/lib/modules
(merged-usr), so the kernel prerequisites step extracts module files
from there. zstd is installed at build so DKMS compresses built
modules to match the kernel's module compression.

KERNEL_MODULE_TYPE=auto resolves to open without runtime detection:
every driver branch in the ubuntu2604 repository (>= 595) supports
Turing+ GPUs only, where open is the recommended default.

CI builds ubuntu26.04 for 595.71.05; branch 580 is excluded since the
ubuntu2604 repository does not carry it. MAX_THREADS (init -m) is not
honored on the package path: the NVIDIA dkms.conf hardcodes -j$(nproc).
The compressed image is 0.84 GB on amd64, vs 0.68 GB for the
ubuntu24.04 .run image built from the same commit.

Signed-off-by: Rajath Agasthya <ragasthya@nvidia.com>
@rajathagasthya
rajathagasthya force-pushed the worktree-driver-package-install branch from 96850ac to e5b8c1c Compare July 8, 2026 22:01
@rajathagasthya
rajathagasthya marked this pull request as ready for review July 8, 2026 22:03
@rajathagasthya
rajathagasthya requested a review from tariq1890 July 8, 2026 22:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant