diff --git a/gpu-operator/cdi.rst b/gpu-operator/cdi.rst index e92ea0d6a..5bfbb1c5f 100644 --- a/gpu-operator/cdi.rst +++ b/gpu-operator/cdi.rst @@ -244,6 +244,98 @@ Disable the NRI Plugin by modifying the cluster policy: After disabling the NRI Plugin, the ``nvidia`` runtime class will be created. +.. _nri-cdi-devices: + +****************************************** +Requesting CDI Devices with the NRI Plugin +****************************************** + +When the NRI Plugin is enabled, a pod can request that the NRI Plugin inject specific CDI devices into one of its containers. +Add an annotation of the following form to the pod specification: + +.. code-block:: yaml + + metadata: + annotations: + nvidia.cdi.k8s.io/container.: "" + +Replace ```` with the name of the container in the pod that the device is injected into, and replace ```` with the CDI device to request. +The NRI Plugin supports two kinds of CDI devices: non-management devices and management devices. + +Requesting a Non-Management CDI Device +************************************** + +A non-management CDI device injects a single GPU, identified by its GPU UUID, into a container. +Pods can request a non-management CDI device from **any** namespace. + +Retrieve the GPU UUID from the ``/var/run/cdi/k8s.device-plugin.nvidia.com-gpu.json`` file on the GPU node. + +Request the device with an annotation of the form ``k8s.device-plugin.nvidia.com/gpu=``. +The following pod specification requests a specific GPU for the ``ubuntu`` container: + +.. code-block:: yaml + + apiVersion: v1 + kind: Pod + metadata: + name: ubuntu + labels: + app: ubuntu + annotations: + nvidia.cdi.k8s.io/container.ubuntu: "k8s.device-plugin.nvidia.com/gpu=GPU-1d7f7c5b-b2f6-148d-ff9a-d8bc4941885c" + spec: + containers: + - image: ubuntu + name: ubuntu + command: + - "sleep" + - "604800" + imagePullPolicy: IfNotPresent + restartPolicy: Always + +Requesting a Management CDI Device +********************************** + +A management CDI device provides management-level access to all GPUs on the node, bypassing GPU allocation through the Device Plugin or DRA Driver for GPUs. +This is the access that a GPU Management Container requires. +Request a management device with the ``management.nvidia.com/gpu=all`` annotation value. + +By default, only pods in the namespace where the GPU Operator is installed, which is the namespace where the NVIDIA Container Toolkit runs, are permitted to request management CDI devices. +To permit pods in other namespaces to request management CDI devices, add those namespaces to the ``NRI_MANAGEMENT_CDI_DEVICE_NAMESPACES`` environment variable on the NVIDIA Container Toolkit. +The value is a comma-separated list of namespaces. + +Set the environment variable through the ``toolkit.env`` Helm value at installation time, or by modifying the cluster policy on an existing installation: + +.. code-block:: console + + $ kubectl patch clusterpolicies.nvidia.com/cluster-policy --type='json' \ + -p='[{"op": "replace", "path": "/spec/toolkit/env", "value": [{"name": "NRI_MANAGEMENT_CDI_DEVICE_NAMESPACES", "value": "test-namespace"}]}]' + +The following pod specification, in ``test-namespace``, requests a management CDI device for the ``ctr`` container. +For this pod to start, ``test-namespace`` must be included in the ``NRI_MANAGEMENT_CDI_DEVICE_NAMESPACES`` environment variable. + +.. code-block:: yaml + + apiVersion: v1 + kind: Pod + metadata: + name: cdi-nri-test + namespace: test-namespace + labels: + app: cdi-nri-test + annotations: + nvidia.cdi.k8s.io/container.ctr: "management.nvidia.com/gpu=all" + spec: + containers: + - name: ctr + image: ubuntu + imagePullPolicy: IfNotPresent + command: + - bash + - -c + - sleep infinity + + ************ Known Issues ************ diff --git a/gpu-operator/life-cycle-policy.rst b/gpu-operator/life-cycle-policy.rst index a0428a674..d7cd9ec46 100644 --- a/gpu-operator/life-cycle-policy.rst +++ b/gpu-operator/life-cycle-policy.rst @@ -53,13 +53,13 @@ The product life cycle and versioning are subject to change in the future. * - GPU Operator Version - Status - * - 26.3.x + * - 26.6.x - Supported - * - 25.10.x + * - 26.3.x - Deprecated - * - 25.3.x and lower + * - 25.10.x and lower - End of Support @@ -88,132 +88,62 @@ Refer to :ref:`Upgrading the NVIDIA GPU Operator` for more information. :header-rows: 2 * - :rspan:`1` Component - - :cspan:`3` GPU Operator Version - - * - v26.3.0 - - v26.3.1 - - v26.3.2 - - v26.3.3 - - * - NVIDIA GPU Driver |ki|_ - - | `595.71.05 `_ - | `595.58.03 `_ - | `590.48.01 `_ - | `580.173.02 `_ (**R**) - | `580.167.08 `_ - | `580.159.04 `_ - | `580.159.03 `_ - | `580.126.20 `_ (**D**) - | `570.211.01 `_ - | `535.309.01 `_ - | `535.288.01 `_ - - | `595.71.05 `_ - | `595.58.03 `_ - | `590.48.01 `_ - | `580.173.02 `_ (**R**) - | `580.167.08 `_ - | `580.159.04 `_ - | `580.159.03 `_ - | `580.126.20 `_ (**D**) - | `570.211.01 `_ - | `535.309.01 `_ - | `535.288.01 `_ - - | `595.71.05 `_ - | `595.58.03 `_ - | `590.48.01 `_ - | `580.173.02 `_ (**R**) - | `580.167.08 `_ - | `580.159.04 `_ - | `580.159.03 `_ - | `580.126.20 `_ (**D**) - | `570.211.01 `_ - | `535.309.01 `_ - | `535.288.01 `_ + - GPU Operator Version + + * - v26.7.0 + + * - NVIDIA GPU Driver - | `610.57.04 `_ - | `595.91.07 `_ - | `595.71.05 `_ - | `595.58.03 `_ - | `590.48.01 `_ - | `580.178.04 `_ (**R**) + | `595.91.07 `_ (**R**, **D**) | `580.173.02 `_ - | `580.167.08 `_ - | `580.159.04 `_ - | `580.159.03 `_ - | `580.126.20 `_ (**D**) - | `570.211.01 `_ | `535.309.01 `_ - | `535.288.01 `_ * - NVIDIA Driver Manager for Kubernetes - - :cspan:`1` `v0.10.0 `__ - - :cspan:`1` `v0.11.0 `__ + - `v0.12.0 `__ * - NVIDIA Container Toolkit - - :cspan:`1` `1.19.0 `__ - - :cspan:`1` `1.19.1 `__ + - `1.20.0 `__ - * - NVIDIA Kubernetes Device Plugin - - :cspan:`1` `0.19.0 `__ - - `0.19.2 `__ - - `0.19.3 `__ + * - NVIDIA Device Plugin for Kubernetes + - `0.20.0 `__ * - DCGM Exporter - - :cspan:`1` `v4.5.1-4.8.0 `__ - - :cspan:`1` `v4.5.3-4.8.2 `__ + - `v4.6.0-4.8.3 `__ * - Node Feature Discovery - - :cspan:`3` `v0.18.3 `__ + - `v0.19.0 `__ * - | NVIDIA GPU Feature Discovery | for Kubernetes - - :cspan:`1` `0.19.0 `__ - - `0.19.2 `__ - - `0.19.3 `__ + - `0.20.0 `__ * - NVIDIA MIG Manager for Kubernetes - - :cspan:`1` `0.14.0 `__ - - :cspan:`1` `0.14.2 `__ + - `0.15.0 `__ * - DCGM - - :cspan:`3` `4.5.2-1 `__ + - `4.6.0-1 `__ * - Validator for NVIDIA GPU Operator - - v26.3.0 - - v26.3.1 - - v26.3.2 - - v26.3.3 + - ${version} * - NVIDIA KubeVirt GPU Device Plugin - - :cspan:`3` `v1.5.0 `__ + - `v1.5.0 `__ * - NVIDIA vGPU Device Manager - - :cspan:`3` `v0.4.2 `__ + - `v0.4.2 `__ * - NVIDIA GDS Driver |gds|_ - - :cspan:`3` `2.27.3 `__ + - `2.29.4 `__ * - | NVIDIA Confidential Computing | Manager for Kubernetes - - `v0.3.0 `__ - - :cspan:`2` `v0.4.0 `__ + - `v0.4.3 `__ * - NVIDIA GDRCopy Driver - - `v2.5.1 `__ - - :cspan:`2` `v2.5.2 `__ + - `v2.6 `__ * - NVIDIA Kata Sandbox Device Plugin - - `v0.0.2 `__ - - :cspan:`2` `v0.0.3 `__ - -.. _known-issue: - - :sup:`1` - Known Issue: For drivers 570.124.06, 570.133.20, 570.148.08, and 570.158.01, - GPU workloads cannot be scheduled on nodes that have a mix of MIG slices and full GPUs. - This manifests as GPU pods getting stuck indefinitely in the ``Pending`` state. - NVIDIA recommends that you upgrade the driver to version 580.65.06 or later to resolve this issue. - For more detailed information, see GitHub issue https://github.com/NVIDIA/gpu-operator/issues/1361. - + - `v0.0.5 `__ .. _gds-open-kernel: diff --git a/gpu-operator/release-notes.rst b/gpu-operator/release-notes.rst index e4d47ad1d..b1cb90850 100644 --- a/gpu-operator/release-notes.rst +++ b/gpu-operator/release-notes.rst @@ -33,6 +33,148 @@ Refer to the :ref:`GPU Operator Component Matrix` for a list of software compone ---- +.. _v26.7.0: + +26.7.0 +======= + +New Features +------------ + +* Updated software component versions: + + - NVIDIA GPU Driver 595.91.07 + - NVIDIA Driver Manager for Kubernetes v0.12.0 + - NVIDIA Container Toolkit v1.20.0 + - NVIDIA Device Plugin for Kubernetes v0.20.0 + - NVIDIA DCGM Exporter v4.6.0-4.8.3 + - NVIDIA DCGM 4.6.0-1 + - NVIDIA MIG Manager for Kubernetes v0.15.0 + - Node Feature Discovery v0.19.0 + - NVIDIA GPU Feature Discovery for Kubernetes v0.20.0 + - NVIDIA GDS Driver v2.29.4 + - NVIDIA Confidental Computing Manager for Kubernetes v0.4.3 + - NVIDIA GDRCopy Driver v2.6 + - NVIDIA Kata Sandbox Device Plugin v0.0.5 + +* Added support for managing the DRA Driver for NVIDIA GPUs through the GPU Operator. + The new ``GPUCluster`` custom resource deploys and manages the DRA driver, ComputeDomain support for Multi-Node NVLink, DCGM, DCGM Exporter, and a DRA validation workload. + Workloads can allocate full GPUs and preconfigured MIG devices through Kubernetes ``ResourceClaim`` objects and select devices by attributes. + + The ``GPUCluster`` workflow uses an ``NVIDIADriver`` resource for an Operator-managed driver, or a driver that is preinstalled on the host. + A cluster cannot use ``GPUCluster`` and ``ClusterPolicy`` at the same time. + This workflow requires Kubernetes v1.34.2 or later, an NVIDIA GPU driver version 580 or later, and a CDI-compatible container runtime. + Some optional DRA capabilities remain alpha and are disabled by default. + + Refer to :doc:`DRA Driver for NVIDIA GPUs ` for prerequisites, capability maturity, limitations, and installation instructions. + (`PR #2571 `__, + `PR #2572 `__, + `PR #2762 `__) + +* Added support for assigning full NVIDIA GPUs to KubeVirt virtual machines with the DRA driver. + With the GPU Operator-managed ``GPUCluster``, a ``ResourceClaim`` selects GPUs and the DRA driver prepares them for VFIO passthrough. + This feature enables running container and virtual-machine GPU workloads on the same node. + On supported HGX and single-node NVL systems, optional Fabric Manager partitioning can constrain multi-GPU claims to a valid NVSwitch fabric partition. + + Refer to :ref:`GPU Operator with KubeVirt and DRA ` for prerequisites, limitations, and configuration. + +* Added a ``nvidia.com/gpu.deploy.client`` node label that lets the GPU Operator manage third-party GPU client pods during driver upgrades and MIG configuration changes. + Advanced users who run their own GPU client workloads that hold GPU device handles (for example, a standalone NVIDIA DRA driver) can add ``nvidia.com/gpu.deploy.client=true`` to the ``nodeSelector`` of the workload's DaemonSet. + The GPU Operator then automatically restarts these pods during a driver upgrade or a MIG configuration change, so the operation can proceed without manual pod eviction. + (`PR #2607 `__) + +* Added support for restarting driver pods in place during patch upgrades when the driver configuration is unchanged. + Previously, a chart upgrade that changed only cosmetic pod-template metadata, such as the ``helm.sh/chart`` label, evicted running GPU workloads and drained the node. + The driver-upgrade controller now compares the driver configuration digest between the running pod and the desired DaemonSet, and when they match, it cordons the node and restarts the driver pod in place without evicting workloads or draining the node. + (`PR #2527 `__) + +* Added an ``upgradePolicy`` field to the NVIDIA driver custom resource definition (CRD). + You can now define a driver upgrade policy per NVIDIADriver custom resource. + When the field is unset, the driver-upgrade controller falls back to the default upgrade policy that is defined in the Helm chart values. + (`PR #2582 `__) + +* Added the ``hostPaths.kubeletRootDir`` Helm value to configure a custom kubelet root directory. + When left empty, the GPU Operator uses ``/var/lib/kubelet`` as the default path. + (`PR #1384 `__) + +* Added the ``dcgmExporter.serviceMonitor.scrapeTimeout`` Helm value to configure the scrape timeout for the DCGM Exporter ``ServiceMonitor``. + (`PR #2404 `__) + +* Added support for configuring the namespace that the GPU Operator uses for its leader election ``Lease`` object. + When unset, the operator continues to use the namespace that it runs in. + (`PR #1333 `__) + +* Changed the RHEL driver container image tag selection to use only the operating system major version, matching the existing Rocky Linux behavior. + The Operator selects NVIDIA-published conventional driver images with major-version operating system tags for RHEL and Rocky Linux 8, 9, and 10, whether you configure the driver through ``ClusterPolicy`` or ``NVIDIADriver``. + (`PR #2497 `__) + +* Added the ``NRI_MANAGEMENT_CDI_DEVICE_NAMESPACES`` environment variable for the NVIDIA Container Toolkit. + When the NRI Plugin is enabled, this variable holds the list of namespaces whose pods are permitted to request management CDI devices. + By default, only pods in the namespace where the GPU Operator is installed can request management CDI devices. + Set the variable through the ``toolkit.env`` Helm value or the ClusterPolicy custom resource. + Refer to :ref:`Requesting CDI Devices with the NRI Plugin ` for more information. + +Fixed Issues +------------ + +* Fixed an issue where host driver validation could incorrectly report that ``nvidia-smi`` was missing when ``/usr/bin`` was an absolute symlink. + The validator now resolves the symlink within the mounted host file system. + (`PR #2464 `__, `Issue #1357 `__) + +* Fixed host driver and vGPU Manager validation so that the validator finds ``nvidia-smi`` in additional host locations, including ``/usr/sbin``, ``/opt/bin``, and the Windows Subsystem for Linux driver path. + Symlinks are resolved within the mounted host file system. + (`PR #2611 `__, `Issue #2506 `__) + +* Fixed an issue where vGPU Manager validation on non-SR-IOV GPUs could wait for unavailable SR-IOV virtual functions and time out. + The validator now waits for mediated-device parents on non-SR-IOV GPUs and for virtual functions on SR-IOV GPUs. + (`PR #2502 `__, `Issue #2365 `__) + +* Fixed an issue where the driver pod failed to start on kernels without ``CONFIG_MEMORY_HOTPLUG`` because the ``/sys/devices/system/memory/auto_online_blocks`` file was unavailable. + The driver pod now mounts the stable ``/sys/devices/system`` parent directory. + (`PR #2517 `__, `Issue #2463 `__) + +* Fixed an issue where the NVIDIA vGPU Device Manager did not start when the vGPU Manager driver was preinstalled on the host and ``driver.enabled=false``. + The readiness gate now accepts the status file from either a container-managed or host-installed vGPU Manager. + (`PR #2599 `__) + +* Fixed an issue where setting ``devicePlugin.config.create=true`` with empty configuration data could leave device plugin pods referencing a missing ConfigMap. + The Helm chart now rejects the configuration when ``devicePlugin.config.name`` or ``devicePlugin.config.data`` is empty. + (`PR #2642 `__, `Issue #2641 `__) + +* Fixed an issue where the ``ClusterPolicy`` status could fluctuate during an NVIDIADriver rolling upgrade. + The status now remains ``notReady`` while an NVIDIADriver-owned node has a pending, in-progress, or failed upgrade. + (`PR #2665 `__) + +* Fixed an issue where multiple NVIDIADriver custom resources with ``spec.default=true`` could incorrectly report a ``ready`` status. + The conflicting resources now report ``notReady`` with a ``ReconcileFailed`` condition until the conflict is resolved. + (`PR #2678 `__) + +* Updated the bundled vGPU Device Manager configuration for vGPU 20.0 so that PCI device ID ``0x318210DE`` maps to the ``B300X-269C`` profile. + (`PR #2625 `__, `Issue #2231 `__) + +* Fixed an issue where custom NVIDIADriver custom resources could remain stuck and fail to reconcile after a conflicting default NVIDIADriver custom resource was deleted. + The GPU Operator now re-evaluates all NVIDIADriver custom resources whenever any NVIDIADriver custom resource changes. + (`PR #2258 `__) + +* Fixed an issue where the ``ClusterPolicy`` status could briefly report ``Ready`` during a driver upgrade before all driver pods were upgraded. + (`PR #2510 `__, `Issue #1567 `__) + +* Fixed an issue where the driver container's containerd configuration mounts were set up even when the NRI plugin was enabled. + On immutable hosts with read-only containerd paths, this prevented the NRI plugin pod from starting. + The GPU Operator no longer configures these mounts when NRI is enabled. + (`PR #2514 `__) + +Known Issues +------------ + +* With DRA passthrough enabled, the driver initially advertises each eligible physical GPU as both a container GPU and a VFIO device. + If a container GPU claim and a VFIO passthrough claim for the same physical GPU are allocated before either device is prepared, the scheduler can allocate both claims. + The device that is prepared first succeeds, and preparation of the other device fails. + + To avoid conflicting allocations, submit the workloads serially and wait for device preparation to complete, or dedicate separate nodes to container and VFIO workloads. + +---- + .. _v26.3.3: 26.3.3 @@ -203,7 +345,7 @@ New Features - 580.126.20 (default) -* Added support for Node Resource Interface (NRI) Plugin. +* Added support for Node Resource Interface (NRI) Plugin. The NRI Plugin offers a new way of injecting GPUs into GPU management containers, without needing the ``nvidia`` runtime class. Enable by setting the ``cdi.nriPluginEnabled`` field to true in the ClusterPolicy custom resource or by setting the ``cdi.nriPluginEnabled`` flag in the Helm chart. @@ -219,7 +361,7 @@ New Features Enabling the NRI plugin is not supported with cri-o. * Added support for dynamic MIG config generation. - By default, the MIG Manager will automatically generate a per-node ConfigMap with the default MIG profiles for the available GPUs on the node. + By default, the MIG Manager will automatically generate a per-node ConfigMap with the default MIG profiles for the available GPUs on the node. This replaces the previous static ConfigMap. You are still able to use a custom MIG configuration if you have specific requirements. Refer to the :doc:`MIG Manager documentation ` for more information. @@ -234,7 +376,7 @@ New Features This feature does not support an upgrade from an earlier version of the NVIDIA GPU Operator or switching from ClusterPolicy to the NVIDIA Driver CRD. It is recommended that you only use this feature from new installations. -* Added support for KubeVirt with GPU passthrough on Ubuntu 24.04 LTS +* Added support for KubeVirt with GPU passthrough on Ubuntu 24.04 LTS * Added support for K3s. @@ -272,8 +414,8 @@ Improvements ------------ * Improved NVIDIA Driver resiliency when the driver container is removed. - In previous versions, the NVIDIA Driver would unload the kernel modules and perform the driver compilation process, which could take several minutes to complete, delaying the driver container from restarting. - In v26.3.0, if there is no change to the CUDA driver version (or other driver configuration) in the ClusterPolicy, the NVIDIA Driver will reuse the kernel modules that are available on the node. + In previous versions, the NVIDIA Driver would unload the kernel modules and perform the driver compilation process, which could take several minutes to complete, delaying the driver container from restarting. + In v26.3.0, if there is no change to the CUDA driver version (or other driver configuration) in the ClusterPolicy, the NVIDIA Driver will reuse the kernel modules that are available on the node. This reduces the time to recover from the driver container removal from minutes to seconds. * Reduced unnecessary API calls and decreased reconciliation time on large GPU clusters by improving node label logic (`PR #2113 `_). @@ -285,7 +427,7 @@ Improvements * Improved support for Kata Containers. Changes in this release include: - * Deprecating the NVIDIA Kata Manager. + * Deprecating the NVIDIA Kata Manager. You now use ``kata-deploy`` to install the Kata Container and the Kata runtime class * Adding support for the NVIDIA Kata Sandbox Device Plugin. * Configure ``sandboxWorkload.mode=kata`` during installation or in the ClusterPolicy to enable Kata Containers. @@ -326,12 +468,12 @@ Known Issues To work around this issue, set the ``FORCE_REINSTALL=true`` environment variable in the ClusterPolicy. - .. code-block:: console + .. code-block:: console $ kubectl patch clusterpolicy cluster-policy --type=json \ -p='[{"op": "add", "path": "/spec/driver/manager/env/-", "value": {"name": "FORCE_REINSTALL", "value": "true"}}]' - Setting ``FORCE_REINSTALL=true`` forces full driver recompilation, node drain, and GPU workload disruption on every restart. + Setting ``FORCE_REINSTALL=true`` forces full driver recompilation, node drain, and GPU workload disruption on every restart. Alternatively, rebooting the node clears the kernel state and allows the ``nvidia-peermem`` module to load successfully, though this may disrupt running workloads. * On RHEL 8 nodes with pre-installed NVIDIA drivers (``driver.enabled=false``), MIG configuration can fail when using NVIDIA MIG Manager v0.13.1 or later. @@ -344,30 +486,12 @@ Known Issues /usr/local/nvidia/mig-manager/nvidia-mig-parted: /lib64/libc.so.6: version `GLIBC_2.34' not found To work around this issue, downgrade the NVIDIA MIG Manager component to v0.12.3. - After downgrading, automatically generated per-node MIG configuration ConfigMaps will not be available. + After downgrading, automatically generated per-node MIG configuration ConfigMaps will not be available. MIG configuration information will be available in the ``default-mig-parted-config`` ConfigMap instead. Refer to the :doc:`MIG Manager documentation ` for more information on MIG configuration. Refer to the MIG Controller issue `#329 `_ for more information. -* After you delete the default NVIDIADriver custom resource, any custom NVIDIADriver - custom resources that you created might not become active automatically. - The custom resources remain in a pending state because the Operator controller - does not re-evaluate them after the conflicting default custom resource is removed. - - To work around this issue, restart the GPU Operator controller by deleting - the controller pod: - - .. code-block:: console - - $ kubectl delete pod -n gpu-operator -l app=gpu-operator - - Restarting the controller pod does not disrupt running GPU workloads or - driver pods on nodes. - - Refer to issue `#2259 `_ - for more information. - Removals and Deprecations ------------------------- @@ -388,7 +512,7 @@ New Features - NVIDIA Container Toolkit v1.18.1 - NVIDIA DCGM v4.4.2-1 - - NVIDIA DCGM Exporter v4.4.2-4.7.0 + - NVIDIA DCGM Exporter v4.4.2-4.7.0 - NVIDIA Kubernetes Device Plugin v0.18.1 - NVIDIA GPU Feature Discovery v0.18.1 - NVIDIA MIG Manager for Kubernetes 0.13.1 @@ -401,7 +525,7 @@ New Features * Add HPC job mapping support to DCGM Exporter to collect metrics for HPC jobs running on the cluster. Configure the HPC job mapping by setting the ``dcgmExporter.hpcJobMapping.enabled`` field to ``true`` in the ClusterPolicy custom resource. - Set ``dcgmExporter.hpcJobMapping.directory`` with the directory path where HPC job mapping files are created by the workload manager. + Set ``dcgmExporter.hpcJobMapping.directory`` with the directory path where HPC job mapping files are created by the workload manager. The default directory is ``/var/lib/dcgm-exporter/job-mapping``. * Improved the cluster policy reconciler to be more resilient to race conditions during node updates. @@ -415,13 +539,13 @@ Fixed Issues * NVIDIA Container Toolkit 1.18.0 overwrites the imports field in the top-level containerd configuration file, so any previously imported paths are lost. This was fixed in NVIDIA Container Toolkit v1.18.1. -* Fixed a race condition where user-supplied NVIDIA kernel module parameters were sometimes not being applied by the driver daemonset. +* Fixed a race condition where user-supplied NVIDIA kernel module parameters were sometimes not being applied by the driver daemonset. For more information, refer to `PR #1939 `__. -* Fixed a bug where driver images were being incorrectly assigned in multi-nodepool clusters. +* Fixed a bug where driver images were being incorrectly assigned in multi-nodepool clusters. For more information, refer to `Issue #1622 `__. * Fixed a bug where the GPU Operator Helm chart template was not assigning the correct namespace to resources it created. -* Fixed a bug where the k8s-driver-manager would wait indefinitely when MOFED is enabled and ``USE_HOST_MOFED`` is set to true despite the MOFED being pre-installed on the host. +* Fixed a bug where the k8s-driver-manager would wait indefinitely when MOFED is enabled and ``USE_HOST_MOFED`` is set to true despite the MOFED being pre-installed on the host. Known Issues @@ -469,7 +593,7 @@ New Features - 570.195.03 - 535.274.02 -* Container Device Interface (CDI) is now enabled by default when installing or upgrading (via helm) the GPU Operator to 25.10.0. +* Container Device Interface (CDI) is now enabled by default when installing or upgrading (via helm) the GPU Operator to 25.10.0. The ``cdi.enabled`` field in the ClusterPolicy is now set to ``true`` by default. The ``cdi.default`` field is now deprecated and will be ignored. @@ -519,7 +643,7 @@ New Features * ``1g.34gb`` :math:`\times` 2 * ``2g.67gb`` :math:`\times` 1 * ``3g.135gb`` :math:`\times` 1 - + * Added support for new MIG profiles with NVIDIA HGX GB300 NVL72. @@ -533,16 +657,16 @@ New Features * ``4g.139gb`` * ``7g.278gb`` - * Added an ``all-balanced`` profile that creates the following GPU instances: + * Added an ``all-balanced`` profile that creates the following GPU instances: * ``1g.35gb`` :math:`\times` 2 * ``2g.70gb`` :math:`\times` 1 - * ``3g.139gb`` :math:`\times` 1 + * ``3g.139gb`` :math:`\times` 1 Improvements ------------ -* The GPU Operator now configures containerd and cri-o to use drop-in files for container runtime config overrides by default. +* The GPU Operator now configures containerd and cri-o to use drop-in files for container runtime config overrides by default. As a consequence of this change, some of the install procedures for Kubernetes distributions that use custom containerd installations have changed. @@ -553,7 +677,7 @@ Improvements * Validator for NVIDIA GPU Operator is now included as part of the GPU Operator container image. It is no longer a separate image. -* The GPU Operator now supports passing the vGPU licensing token as a secret. +* The GPU Operator now supports passing the vGPU licensing token as a secret. It is recommended that you migrate to using secrets instead of a configMap for improved security. * Enhanced the driver pod to allow resource requests and limits to be configurable for all containers in the driver pod. @@ -565,7 +689,7 @@ Fixed Issues * Fixed an issue where the vGPU Manager pod was terminated before it finished disabling VFs on all GPUs. The terminationGracePeriodSeconds is now set to 120 seconds to ensure the vGPU Manager has enough time to finish its cleanup logic when the pod is terminated. - + * Added GDRCopy validation to validator daemonset. When GDRCopy is enabled, this ensures that the GDRCopy driver is loaded prior to the k8s-device-plugin from starting up. * Added required permissions when GPU Feature Discovery is configured to use the Node Feature API instead of feature files. @@ -574,20 +698,20 @@ Fixed Issues Known Issues ------------ -* When using cri-o as the container runtime, several of the GPU Operator pods may be stuck in the ``Init:RunContainerError`` or ``Init:CreateContainerError`` state during installation of GPU Operator, upgrade of GPU Operator, or upgrade of the GPU driver daemonset. +* When using cri-o as the container runtime, several of the GPU Operator pods may be stuck in the ``Init:RunContainerError`` or ``Init:CreateContainerError`` state during installation of GPU Operator, upgrade of GPU Operator, or upgrade of the GPU driver daemonset. The pods may be in this state for several minutes and restart several times. The pods will recover from this state as soon as the container toolkit pod starts running. * NVIDIA Container Toolkit 1.18.0 will overwrite the `imports` field in the top-level containerd configuration file, so any previously imported paths will be lost. -* When using MIG-backed vGPU on the RTX Pro 6000 Blackwell Server Edition, the vgpu-device-manager will fail to configure nodes with the default vgpu-device-manager configuration. +* When using MIG-backed vGPU on the RTX Pro 6000 Blackwell Server Edition, the vgpu-device-manager will fail to configure nodes with the default vgpu-device-manager configuration. To work around this, create a custom ConfigMap that adds the GFX suffix to the vGPU profile name. - All of the MIG-backed vGPU profiles are only supported on MIG instances created with the ``+gfx`` attribute. + All of the MIG-backed vGPU profiles are only supported on MIG instances created with the ``+gfx`` attribute. Refer to the following example: .. code-block:: yaml - + version: v1 vgpu-configs: DC-1-2Q: @@ -597,8 +721,8 @@ Known Issues Create the ConfigMap, then update the ClusterPolicy with the name of the configMap in the ``vgpuDeviceManager.config.name``, and restart the vgpu-device-manager pod. -- When using GKE 1.33+, there is a known issue where NVIDIA Container Toolkit will misconfigure the containerd `config.toml` file and prevent GPU Operator containers from starting up correctly. - To resolve this issue, set the ``RUNTIME_CONFIG_SOURCE=file`` environment variable in the toolkit container. +- When using GKE 1.33+, there is a known issue where NVIDIA Container Toolkit will misconfigure the containerd `config.toml` file and prevent GPU Operator containers from starting up correctly. + To resolve this issue, set the ``RUNTIME_CONFIG_SOURCE=file`` environment variable in the toolkit container. You can set this environment variable by setting the below in the ClusterPolicy CR: .. code-block:: yaml @@ -630,7 +754,7 @@ New Features - KubeVirt and OpenShift Virtualization: VM with GPU passthrough (Ubuntu 22.04 only) - KubeVirt and OpenShift Virtualization: VM with time-slice vGPU (Ubuntu 22.04 only) - - RTX Pro 6000D + - RTX Pro 6000D - KubeVirt and OpenShift Virtualization: VM with GPU passthrough (Ubuntu 22.04 only) @@ -672,7 +796,7 @@ New Features - 580.65.06 (recommended) - 570.172.08 (default) - - 535.261.03 + - 535.261.03 .. _v25.3.2-known-issues: @@ -681,20 +805,20 @@ Known Issues * Starting with version **580.65.06**, the driver container has **Coherent Driver Memory Management (CDMM)** enabled by default to support **GB200** on Kubernetes. For more information about CDMM, refer to the `release notes `__. - + .. note:: Currently, CDMM is not compatible with the **Multi-Instance GPUs (MIG)** sharing. CDMM is also not compatible with **GPU Direct Storage**. CDMM support for these features is planned for future driver updates. However, these limitations will remain in place until a future driver update removes them. - + CDMM enablement applies only to **Grace-based systems** such as **GH200** and **GB200** and is ignored on other GPU platforms. NVIDIA strongly recommends keeping CDMM enabled with Kubernetes on supported systems to prevent memory over-reporting and uncontrolled GPU memory access. * For drivers 570.124.06, 570.133.20, 570.148.08, and 570.158.01, - GPU workloads cannot be scheduled on nodes that have a mix of MIG slices and full GPUs. - This manifests as GPU pods getting stuck indefinitely in the ``Pending`` state. + GPU workloads cannot be scheduled on nodes that have a mix of MIG slices and full GPUs. + This manifests as GPU pods getting stuck indefinitely in the ``Pending`` state. NVIDIA recommends that you upgrade the driver to version 570.172.08 to avoid this issue. For more detailed information, see GitHub issue https://github.com/NVIDIA/gpu-operator/issues/1361. @@ -710,7 +834,7 @@ Fixed Issues ------------ * Fixed security vulnerabilities in NVIDIA Container Toolkit and related components. - This release addresses CVE-2025-23266 (Critical) and CVE-2025-23267 (High) that could allow + This release addresses CVE-2025-23266 (Critical) and CVE-2025-23267 (High) that could allow arbitrary code execution and link following attacks in container environments. For complete details, refer to the `NVIDIA Security Bulletin `__. @@ -744,13 +868,13 @@ New Features - 535.247.01 * Added support for Red Hat Enterprise Linux 9. - Non-precompiled driver containers for Red Hat Enterprise Linux 9.2, 9.4, 9.5, and 9.6 versions are available for x86 based platforms only. + Non-precompiled driver containers for Red Hat Enterprise Linux 9.2, 9.4, 9.5, and 9.6 versions are available for x86 based platforms only. They are not available for ARM based systems. * Added support for Kubernetes v1.33. * Added support for setting the internalTrafficPolicy for the DCGM Exporter service. - You can configure this in the Helm chart value by setting ``dcgmexporter.service.internalTrafficPolicy`` to ``Local`` or ``Cluster`` (default). + You can configure this in the Helm chart value by setting ``dcgmexporter.service.internalTrafficPolicy`` to ``Local`` or ``Cluster`` (default). Choose Local if you want to route internal traffic within the node only. .. _v25.3.1-known-issues: @@ -759,8 +883,8 @@ Known Issues ------------ * For drivers 570.124.06, 570.133.20, 570.148.08, and 570.158.01, - GPU workloads cannot be scheduled on nodes that have a mix of MIG slices and full GPUs. - This manifests as GPU pods getting stuck indefinitely in the ``Pending`` state. + GPU workloads cannot be scheduled on nodes that have a mix of MIG slices and full GPUs. + This manifests as GPU pods getting stuck indefinitely in the ``Pending`` state. NVIDIA recommends that you upgrade the driver to version 570.172.08 to avoid this issue. For more detailed information, see GitHub issue https://github.com/NVIDIA/gpu-operator/issues/1361. @@ -771,7 +895,7 @@ Known Issues Fixed Issues ------------ -* Fixed an issue where the NVIDIADriver controller may enter an endless loop of creating and deleting a DaemonSet. +* Fixed an issue where the NVIDIADriver controller may enter an endless loop of creating and deleting a DaemonSet. This could occur when the NVIDIADriver DaemonSet does not tolerate a taint present on all nodes matching its configured nodeSelector, or when none of the DaemonSet pods have been scheduled yet. Refer to GitHub `pull request #1416 `__ for more details. @@ -802,32 +926,32 @@ New Features * Added support for the NVIDIA GPU DRA Driver v25.3.0 component (coming soon) which enables Multi-Node NVLink through Kubernetes Dynamic Resource Allocation (DRA) and IMEX support. - This component can be installed alongside the GPU Operator. - It is supported on Kubernetes v1.32 clusters, running on NVIDIA HGX GB200 NVL, and with CDI enabled on your GPU Operator. + This component can be installed alongside the GPU Operator. + It is supported on Kubernetes v1.32 clusters, running on NVIDIA HGX GB200 NVL, and with CDI enabled on your GPU Operator. -* Transitioned to installing the open kernel modules by default starting with R570 driver containers. +* Transitioned to installing the open kernel modules by default starting with R570 driver containers. * Added a new parameter, ``kernelModuleType``, to the ClusterPolicy and NVIDIADriver APIs which specifies how the GPU Operator and driver containers will choose kernel modules to use. - + Valid values include: * ``auto``: Default and recommended option. ``auto`` means that the recommended kernel module type (open or proprietary) is chosen based on the GPU devices on the host and the driver branch used. - * ``open``: Use the NVIDIA Open GPU kernel module driver. + * ``open``: Use the NVIDIA Open GPU kernel module driver. * ``proprietary``: Use the NVIDIA Proprietary GPU kernel module driver. - Currently, ``auto`` is only supported with the 570.86.15 and 570.124.06 or later driver containers. + Currently, ``auto`` is only supported with the 570.86.15 and 570.124.06 or later driver containers. 550 and 535 branch drivers do not yet support this mode. - In previous versions, the ``useOpenKernelModules`` field specified the driver containers to install the NVIDIA Open GPU kernel module driver. + In previous versions, the ``useOpenKernelModules`` field specified the driver containers to install the NVIDIA Open GPU kernel module driver. This field is now deprecated and will be removed in a future release. - If you were using the ``useOpenKernelModules`` field, NVIDIA recommends that you update your configuration to use the ``kernelModuleType`` field instead. + If you were using the ``useOpenKernelModules`` field, NVIDIA recommends that you update your configuration to use the ``kernelModuleType`` field instead. * Added support for Ubuntu 24.04 LTS. * Added support for NVIDIA HGX GB200 NVL and NVIDIA HGX B200. Note that HGX B200 requires a driver container version of 570.133.20 or later. -* Added support for the NVIDIA Data Center GPU Driver version 570.124.06. +* Added support for the NVIDIA Data Center GPU Driver version 570.124.06. * Added support for KubeVirt and OpenShift Virtualization with vGPU v18 on H200NVL. @@ -877,7 +1001,7 @@ New Features * ``2g.47gb`` :math:`\times` 1 * ``3g.95gb`` :math:`\times` 1 -Improvements +Improvements ------------ * Improved security by removing unnecessary permissions in the GPU Operator ClusterRole. @@ -891,7 +1015,7 @@ Improvements Fixed Issues ------------ -* Removed default liveness probe from the ``nvidia-fs-ctr`` and ``nvidia-gdrcopy-ctr`` containers of the GPU driver daemonset. +* Removed default liveness probe from the ``nvidia-fs-ctr`` and ``nvidia-gdrcopy-ctr`` containers of the GPU driver daemonset. Long response times of the `lsmod` commands were causing timeout errors in the probe and unnecessary restarts of the container, resulting in the DaemonSet being in a bad state. * Fixed an issue where the GPU Operator failed to create a valid DaemonSet name on OpenShift Container Platform when using 64 kernel page size. @@ -909,7 +1033,7 @@ Fixed Issues New Features ------------ -* Added support for the NVIDIA Data Center GPU Driver version 570.86.15. +* Added support for the NVIDIA Data Center GPU Driver version 570.86.15. * The default driver in this version is now 550.144.03. Refer to the :ref:`GPU Operator Component Matrix` on the platform support page for more details on supported drivers. diff --git a/gpu-operator/versions1.json b/gpu-operator/versions1.json index 27150f84d..508fbb389 100644 --- a/gpu-operator/versions1.json +++ b/gpu-operator/versions1.json @@ -1,6 +1,10 @@ [ { "preferred": "true", + "url": "../26.7", + "version": "26.7" + }, + { "url": "../26.3/", "version": "26.3" }, diff --git a/openshift/versions1.json b/openshift/versions1.json index a6293db72..cde5ec46a 100644 --- a/openshift/versions1.json +++ b/openshift/versions1.json @@ -1,6 +1,10 @@ [ { "preferred": "true", + "url": "../26.7/", + "version": "26.7" + }, + { "url": "../26.3/", "version": "26.3" }, diff --git a/repo.toml b/repo.toml index 9816cc5de..29a62a6e6 100644 --- a/repo.toml +++ b/repo.toml @@ -176,7 +176,7 @@ docs_root = "${root}/gpu-operator" project = "gpu-operator" name = "NVIDIA GPU Operator" version = "26.7" # Update repo_docs.projects.openshift.version to match latest patch version maj.min.patch -source_substitutions = { minor_version = "26.7", version = "v26.7.0", recommended = "580.173.02", dra_version = "0.4.1", kata_version = "4.0.0" } +source_substitutions = { minor_version = "26.7", version = "v26.7.0", recommended = "595.91.07", dra_version = "0.5.0", kata_version = "4.0.0" } copyright_start = 2020 sphinx_exclude_patterns = [ "life-cycle-policy.rst", @@ -226,7 +226,7 @@ output_format = "linkcheck" docs_root = "${root}/openshift" project = "gpu-operator-openshift" name = "NVIDIA GPU Operator on Red Hat OpenShift Container Platform" -version = "26.3" # Check that this value matches maj.min version of repo_docs.projects.gpu-operator.version +version = "26.7" # Check that this value matches maj.min version of repo_docs.projects.gpu-operator.version copyright_start = 2020 sphinx_exclude_patterns = [ "get-entitlement.rst",