fix: Order container engines after nvidia-cdi-refresh.service

[ehfd]

Follow-up to #1979.

Reviewers: @cdesiniotis @henry118

Follow-up to #1979, split out at @cdesiniotis' request so that #1979 stayed scoped to device node creation.

Summary

nvidia-cdi-refresh.service is not ordered against any container engine, so a container started early at boot can reach the runtime before the service has created the control device nodes and generated the CDI specification. This adds a drop-in that orders docker, containerd and cri-o after the service, and bounds the service's start time now that engines wait on it.

Why This Exists

The service is Type=oneshot, WantedBy=multi-user.target, with no ordering against any engine. A GPU container with restart=always, or an early docker run, can therefore reach the runtime before the service's ExecStart has run: the first container of the boot gets a specification generated before the device nodes exist, and subsequent ones do not. @henry118 identified this ordering as the packaging-level fix for that race in #1979 (comment).

Why a drop-in rather than the unit

#1735 was about exactly this class of change: an ordering constraint shipped in the unit could not be removed by users, only worked around by overriding the whole unit. That is still true — verified on systemd 259, assigning an empty Before= in a drop-in does not reset an in-unit Before= list.

Shipping the ordering as a vendor drop-in gives it an off switch. A same-named empty file in /etc/systemd/system/nvidia-cdi-refresh.service.d/ takes precedence over the vendor copy and cancels it, with no need to override the unit.

The two cycle scenarios raised in review

@henry118 noted in #1979 that a Before= can only produce a cycle if either docker is ordered before us, or a third unit sits between us, and judged both unlikely. I built both on Ubuntu 26.04 / systemd 259 / Docker 29.7.2 to see what actually happens if they do occur.

A — docker.service has Before=nvidia-cdi-refresh.service:

systemd[1]: docker.service: Found ordering cycle: nvidia-cdi-refresh.service/start
            after docker.service/start - after nvidia-cdi-refresh.service
systemd[1]: docker.service: Job nvidia-cdi-refresh.service/start deleted to break
            ordering cycle starting with docker.service/start

B — a third unit has After=docker.service and Before=nvidia-cdi-refresh.service:

systemd[1]: docker.service: Found ordering cycle: nvidia-cdi-refresh.service/start
            after other.service/start after docker.service/start - after
            nvidia-cdi-refresh.service
systemd[1]: docker.service: Job nvidia-cdi-refresh.service/start deleted to break
            ordering cycle starting with docker.service/start

In both cases systemd resolves the cycle by deleting our job: docker and the third unit start normally, and nvidia-cdi-refresh.service does not run, leaving the CDI specification unrefreshed for that boot. Nothing fails loudly — the only signal is the journal line above.

That is milder than #1735, where the job that got deleted belonged to the user's own unit, but it is the same class of problem, and it is the reason the ordering is shipped as a drop-in. A host that hits either scenario can drop an empty same-named file into /etc/systemd/system/nvidia-cdi-refresh.service.d/ and get the previous behaviour back. Verified against scenario B: with the vendor drop-in cancelled, no cycle is reported, every unit starts, and the specification is regenerated.

Why TimeoutStartSec

Type=oneshot defaults to TimeoutStartSec=infinity, and the unit sets no override. Once engines are ordered after it, a nvidia-smi -L that hangs — a wedged GPU, a driver still initialising, a fabric manager not yet up — would delay them without bound. The drop-in therefore also sets TimeoutStartSec=90s, matching systemd's DefaultTimeoutStartSec. On expiry the engines start and the specification is refreshed by the next trigger; the ordering is Before=, not Requires=, so nothing depends on the service succeeding.

Both settings live in the same drop-in, so removing the drop-in restores the previous behaviour exactly.

Behavior Changes

• docker.service, containerd.service and crio.service gain an implicit After=nvidia-cdi-refresh.service when they are part of the same transaction, so the first container at boot no longer races node creation and CDI generation.
• The service's start is bounded at 90s instead of being unbounded.
• Both are removable by a same-named empty drop-in under /etc/systemd/system/.
• Hosts where the unit's ExecCondition fails (no NVIDIA kernel module) are unaffected: the job completes immediately and adds no measurable boot delay.

Implementation Summary

• Add deployments/systemd/10-container-engines.conf, installed to <unitdir>/nvidia-cdi-refresh.service.d/.
• Install it from the Debian and RPM packaging alongside the existing unit files.
• Extend the nvidia-cdi-refresh e2e suite to assert the drop-in is installed and that the resulting Before= includes docker.service.

Verification

Ubuntu 26.04, systemd 259, Docker 29.7.2, 2x Tesla P100. The file as shipped in this PR, installed to /usr/lib/systemd/system/nvidia-cdi-refresh.service.d/:

# before
Before=shutdown.target multi-user.target
TimeoutStartUSec=infinity

# with the drop-in
Before=containerd.service multi-user.target docker.service shutdown.target crio.service
TimeoutStartUSec=1min 30s
$ systemctl show docker.service -p After | tr ' ' '\n' | grep nvidia
nvidia-cdi-refresh.service

No ordering cycles are reported by systemd-analyze verify, and the unit still loads.

Cancelling it as a user would, with a same-named empty drop-in under /etc:

$ printf '# intentionally empty\n' | \
    sudo tee /etc/systemd/system/nvidia-cdi-refresh.service.d/10-container-engines.conf
$ sudo systemctl daemon-reload
$ systemctl show nvidia-cdi-refresh.service -p Before -p TimeoutStartUSec
Before=multi-user.target shutdown.target
TimeoutStartUSec=infinity

For scale, the service took 10.758s wall clock at boot on this two-GPU host, almost all of it the cold nvidia-smi -L; that is the delay the ordering introduces before the engines start.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.