Dockerfile - Add cuda13.3.dockerfile on pytorch:26.07-py3 - #846
Draft
gusui-msft wants to merge 3 commits into
Draft
Dockerfile - Add cuda13.3.dockerfile on pytorch:26.07-py3#846gusui-msft wants to merge 3 commits into
gusui-msft wants to merge 3 commits into
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
This PR adds a new CUDA 13.3 container image definition based on nvcr.io/nvidia/pytorch:26.07-py3, updates the GitHub Actions image build/merge matrices to build/publish cuda13.3 instead of cuda13.0, and adjusts third-party build logic to select a newer CUTLASS for CUDA 13.3+.
Changes:
- Add
dockerfile/cuda13.3.dockerfile(new base image + updated HPC-X/UCX/Docker client handling). - Update
.github/workflows/build-image.ymlmatrix entries to build/merge/publishcuda13.3tags. - Update
third_party/MakefileCUTLASS selection to usev4.7.0whenCUDA_VER >= 13.3.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
third_party/Makefile |
Adds a CUDA 13.3+ conditional branch to select CUTLASS v4.7.0. |
dockerfile/cuda13.3.dockerfile |
Introduces the CUDA 13.3 image based on nvcr.io/nvidia/pytorch:26.07-py3 with updated dependency versions. |
.github/workflows/build-image.yml |
Switches CI build/merge matrices from cuda13.0 to cuda13.3 tags/dockerfile. |
Suppressed comments (1)
third_party/Makefile:47
git clone ... && cd cutlassis misleading/no-op here: each recipe line runs in its own shell, and the later build uses-S ./cutlassanyway, so thecd cutlasshas no effect on subsequent steps. Consider removing the trailing&& cd cutlassto avoid implying later commands run from inside the repo.
$(eval ARCHS := "100;103")
if [ -d cutlass ]; then rm -rf cutlass; fi
git clone --branch v4.7.0 --depth 1 https://github.com/NVIDIA/cutlass.git && cd cutlass
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 3 out of 3 changed files in this pull request and generated no new comments.
Suppressed comments (2)
third_party/Makefile:48
git clone ... && cd cutlassis misleading here: Make runs each recipe line in its own shell, so thecd cutlassdoes not affect subsequent commands (which already reference./cutlass). Consider dropping the&& cd cutlasssuffix to avoid implying that later steps run inside the repo directory.
ifeq ($(shell echo $(CUDA_VER)">=13.3" | bc -l), 1)
$(eval ARCHS := "100;103")
if [ -d cutlass ]; then rm -rf cutlass; fi
git clone --branch v4.7.0 --depth 1 https://github.com/NVIDIA/cutlass.git && cd cutlass
else ifeq ($(shell echo $(CUDA_VER)">=12.9" | bc -l), 1)
dockerfile/cuda13.3.dockerfile:162
- This Dockerfile changes the established pattern of pinning
setuptoolsto an exact version (e.g.,cuda13.0.dockerfileandcuda12.9.dockerfileusesetuptools==...). Using a lower-bound specifier (setuptools>=...) makes image builds non-reproducible because the resolved version can change over time. Consider pinning to a specific known-good version (e.g., the base image’s current setuptools) and bump it deliberately when needed.
RUN python3 -m pip install --upgrade "setuptools>=78.1.1" && \
python3 -m pip install --no-cache-dir .[nvworker] && \
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #846 +/- ##
=======================================
Coverage 86.02% 86.02%
=======================================
Files 103 103
Lines 7950 7950
=======================================
Hits 6839 6839
Misses 1111 1111
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Author
|
@microsoft-github-policy-service agree [company="{microsoft}"] |
|
gusui-msft the command you issued was incorrect. Please try again. Examples are: and |
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.
Suppressed comments (1)
third_party/Makefile:48
- The new CUDA_VER>=13.3 branch duplicates the same ARCHS/rm/clone recipe used for the >=12.9 branch, which increases maintenance cost and makes future tag/arch updates easy to miss. Consider setting CUTLASS_TAG (and ARCHS) inside the conditional branches, then running the shared rm/clone logic once after the conditional (or factoring a small helper target).
# The CUTLASS tag is picked per CUDA_VER because each release only supports the SM archs of its own CUDA generation.
cuda_cutlass:
ifeq ($(shell echo $(CUDA_VER)">=13.3" | bc -l), 1)
$(eval ARCHS := "100;103")
if [ -d cutlass ]; then rm -rf cutlass; fi
git clone --branch v4.7.0 --depth 1 https://github.com/NVIDIA/cutlass.git && cd cutlass
else ifeq ($(shell echo $(CUDA_VER)">=12.9" | bc -l), 1)
Comment on lines
+69
to
+74
| # Install Docker | ||
| ENV DOCKER_VERSION=29.7.2 | ||
| RUN TARGETARCH_HW=$(uname -m) && \ | ||
| wget -q https://download.docker.com/linux/static/stable/${TARGETARCH_HW}/docker-${DOCKER_VERSION}.tgz -O docker.tgz && \ | ||
| tar --extract --file docker.tgz --strip-components 1 --directory /usr/local/bin/ && \ | ||
| rm docker.tgz |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Adds a
cuda13.3image built onnvcr.io/nvidia/pytorch:26.07-py3, replacingcuda13.0(25.08-py3) in the CI matrix.Motivation is security. A Trivy 0.72.0 scan of
main-cuda13.0(2026-07-15) reports 58 Critical occurrences. 18 of them are inherited from the NVIDIA base image and cannot be fixed in our own layers.26.07-py3clears all 18:25.08-py326.07-py3shipslinux-libc-dev6.8.0-71.716.8.0-134.1346.8.0-136.136jupyter_server2.16.02.20.02.20.0efa_metrics/nic_samplerv1.23.41.26.0-rc.3go1.26.1Each of those three versions was read directly out of the
26.07-py3registry layers, not taken from release notes. The upgrade additionally bringswheel 0.45.1 -> 0.46.3(CVE-2026-24049, HIGH) andpip 25.2 -> 26.1.2(4 Medium, 1 Low).Follows the one-Dockerfile-per-CUDA-version convention, so this is a new file rather than an edit to
cuda13.0.dockerfile.Major Revision
dockerfile/cuda13.3.dockerfilebased onnvcr.io/nvidia/pytorch:26.07-py3.docker-buildanddocker-mergematrices atcuda13.3instead ofcuda13.0.CUDA_VER >= 13.3branch tothird_party/Makefileselecting CUTLASSv4.7.0, whose release notes state "Optimal code generation with CUDA toolkit versions 13.3". The existing>= 12.9branch would otherwise still pickv4.1.0, which predates CUDA 13.3.Minor Revision
HPCX_VERSIONv2.24.1->v2.50. Not cosmetic: the install block doesrm -rf /opt/hpcxand/etc/bash.bashrcsourceshpcx-init.sh, so keepingv2.24.1would replace the base image's HPC-X 2.50 with an older stack.UCX_VERSION1.18.0->1.21.0, matching the base image. Also not cosmetic:LD_LIBRARY_PATHputs/usr/local/libahead of the base libraries, so the source-built UCX shadows the base's. The URL is also corrected from the-rc1tag to the final release tag. Theconfigure-release-mtmulti-threaded build is kept.setuptools==78.1.0->setuptools==81.0.0. The old pin is a downgrade now (base ships81.0.0) and78.1.0is itself CVE-2025-47273 (HIGH), which the current scans report. Pinning to the version the base already carries keeps the line reproducible and makes it a no-op.DOCKER_VERSIONset to29.7.2so a brand-new image does not ship the 40 Critical occurrences that Dockerfile - Upgrade bundled Docker to 29.7.2 in cuda13.0 #845 removes fromcuda13.0.26.07-py3component versions.cuda_cutlassinthird_party/Makefile; it claimed 12.9+ buildsv3.9while the recipe already clonedv4.1.0, and adding a third branch made it more misleading.What changed in the base image
Unchanged, so no migration work needed: Python 3.12, Ubuntu 24.04,
CUDA_HOME,_CUDA_COMPAT_PATH.25.08-py326.07-py313.0.0.04413.3.1.0082.8.0a02.13.0a0+9186a089.12.0.469.24.0.4313.0.0.1913.6.0.22.27.72.30.72.52.174.1.75.0.102.242.501.19.01.21.056.063.02025.4.1.1362026.3.1.117Artifact availability, checked 2026-08-18
hpcx-v2.50-gcc-doca_ofed-ubuntu24.04-cuda13-{x86_64,aarch64}.tbz: both 200ucx-1.21.0.tar.gzat tagv1.21.0: 200MLNX_OFED_LINUX-24.10-1.1.4.0-ubuntu24.04-{x86_64,aarch64}.tgz: both 200NVIDIA/cuda-samplestagv13.3exists, so the unconditional clone incuda_bandwidthTestwill not breakNVIDIA/cutlasstagv4.7.0existsDraft, because this is unvalidated
This replaces CUDA, cuDNN, NCCL, PyTorch, HPC-X, and UCX at once, so it cannot be accepted on a scan result alone.
Risks, in the order I expect them to bite:
2.8.0a0->2.13.0a0, five minor releases. Affectsmegatron_lm,megatron_deepspeed, Apex, and TransformerEngine.setup.pyonly declarestorch>=1.7.0a0and there is notorch.__version__guard anywhere insuperbench/, so pip will not surface a break; it has to be found by building and running.python3-mpi4pycomes from apt and links the distro MPI, whilenccl-testsandperftestbuild againstMPI_HOME=/usr/local/mpi.v4.7.0has not been built here against CUDA 13.3 yet.MLNX_OFED 24.10-1.1.4.0user-space now sits on top of rdma-core 63.0 rather than 56.0. Pre-existing pattern, wider gap.Questions for reviewers
dockerfile/cuda13.0.dockerfilebe deleted in this PR? CI/CD - Clean up image builds, remove cuda 12.8/12.4/12.2 and add cuda13.0 merge #819 kept superseded Dockerfiles and only changed the matrix, so this PR follows that precedent and leaves the file in place.git clone ... && cd cutlassinthird_party/Makefileis a no-op, since Make runs each recipe line in its own shell. Raised by review, but it is pre-existing onmainin all three branches, so cleaning it up belongs in its own change rather than here.Relationship to #845
Independent. #845 bumps Docker to 29.7.2 in
cuda13.0.dockerfile; this PR creates a separate file that already carries 29.7.2, so there is no textual conflict. #845 should merge first because it closes 40 Critical occurrences on the image CI builds today, and it does not depend on any of the validation above.