From 7fb60b0058534b91f55eb1639de080c73b015203 Mon Sep 17 00:00:00 2001 From: shoumikhin Date: Tue, 11 Aug 2026 23:08:11 -0700 Subject: [PATCH 1/2] Build and publish CUDA wheels ## The problem The wheel can carry the CUDA delegate, but no published wheel contains one: there is no CUDA row in any workflow, so a GPU user has to build from source. ## The change Add the workflows that build and publish CUDA wheels for Linux x86_64 and aarch64, and a smoke test that checks each wheel from the artifact itself. The build machines for these rows have no GPU, so the smoke test does not execute a model; it verifies the CUDA libraries are present, that the declared runtime matches the wheel's CUDA version, that nothing resolves through the build machine's toolkit, and that the shipped device code covers every GPU architecture the row claims. ``` executorch-1.5.0-cp312-cp312-manylinux_2_28_x86_64.whl +cu130 ``` A release publishes CUDA 12.6, 13.0 and 13.2, for Python 3.10 through 3.13. A pull request builds a single row instead of all twelve, because a full matrix costs hours for little extra signal. Which GPU architectures each row compiles for is chosen per row rather than detected on the builder. Detecting it would produce a wheel carrying device code for whatever machine happened to build it, which installs fine and then fails at the first GPU call. The aarch64 CUDA 12.6 row also compiles for compute capability 8.7, which is an embedded module. Every other row lists only the architectures the published PyTorch build for that train covers, and by that rule 8.7 would be left out, because the generic aarch64 build of this train carries 8.0 and 9.0 only. It is included because this is the only row whose CUDA major version matches what that module's software release ships, and because this wheel declares no PyTorch dependency: a user there supplies the build that carries their architecture. Leaving 8.7 out does not protect them from a bad pairing, it only removes the device code they need. Without it, a model reaching one of the shipped optional operators, quantized matrix multiply, sort or random number generation, fails at the first launch on that device. Two guards keep a release honest: - if the shared matrix generator stops offering a combination this policy advertises, the step fails instead of quietly publishing fewer wheels. A missing job is otherwise a green check for a wheel that was never built. - if a row reaches the architecture list with no CUDA version, the build refuses rather than falling back to the builder's GPU. A TORCH_CUDA_ARCH_LIST that holds only named GPU families PyTorch accepts, such as "Hopper", now fails to configure instead of quietly leaving CMAKE_CUDA_ARCHITECTURES unset and taking the compiler default. The three named forms CMake itself understands, "native", "all" and "all-major", are rejected before this logic runs: torch resolves the list with its own bundled CUDA architecture module, which does not know those names and stops the configure. That is upstream behaviour, not something this change introduces or can work around, so a caller has to name architectures explicitly. Windows CUDA is deliberately absent. The separate shared libraries this wheel exists to ship are Linux only today, so a Windows CUDA wheel would carry a delegate a C++ application still could not link. ## Test plan - built the full release matrix, twelve wheels, and confirmed each one's contents match the row it claims: the CUDA libraries present, the CUDA runtime declared, and device code for every GPU architecture the row advertises. - ran a GPU model end to end from a CI-built wheel on three NVIDIA GPUs covering three device architectures, with output identical to eager PyTorch on each (largest absolute difference 0), and inspected the wheel for a fourth device it cannot execute on. - ran the matrix filter over generated inputs, including incomplete and malformed ones, and confirmed it refuses rather than publishing a partial release: a missing CUDA version, a missing python, or a python present on rows this policy does not build are each reported by name. - confirmed a CPU row still produces a CPU wheel on a builder that happens to have a CUDA toolkit installed. - the newest architecture also ships in its portable form, so a GPU newer than any in the row can still run by having the driver compile it at load time. Checked with `cuobjdump --list-ptx`, since `--list-elf` prints identical output whether or not the portable form is present. - every library that carries GPU device code covers the whole row on its own. - the declared CUDA packages are compared against the expected set in BOTH directions. A one-way comparison accepted a wheel that omitted required packages, and a name-suffix comparison accepted cross-train names because for CUDA 13 the suffix is empty. - the python axis is an allowlist, matching the CUDA axis. Testing only the disabled list let any python not on it through: a 3.9 row was emitted successfully. - `install_utils.py` is in both CUDA workflows' path filters. It owns the supported CUDA train list and the toolkit detection, so a change there previously ran no CUDA wheel job. - requesting the JetPack rows fails with its own reason instead of the generic empty-matrix message, since both of its lists are deliberately empty and no workflow asks for them. - torchao keeps its CUDA channel where that channel exists. Falling back to the plain nightly index was needed only on aarch64, where the CUDA channel publishes nothing, and doing it everywhere changed which torchao an x86_64 install resolves. - the CUDA smoke test now asserts the QnnBackend and OpenvinoBackend registrations that a CPU Linux row asserts. The CUDA build enables OpenVINO on every Linux architecture and downloads the QNN SDK on x86_64, so a CUDA wheel carries both backends; a previous premise that "a CUDA row is not built with them" was false, and dropping the checks meant those two backends were unverified on every CUDA wheel. Known gap: no automated job runs a CUDA model on real hardware before publication. Running a model on real hardware is a separate release-time step that a person owns today, not an automated job wired into these workflows. ghstack-source-id: 95d67c70e217a02053ec7683c0a0ec0709d0a9d7 ghstack-comment-id: 5220374521 Pull-Request: https://github.com/pytorch/executorch/pull/21668 --- .ci/scripts/wheel/cuda_arch_list.sh | 133 +++++++ .ci/scripts/wheel/envvar_cuda_linux.sh | 42 +++ .ci/scripts/wheel/test_cuda_linux.py | 353 ++++++++++++++++++ .ci/scripts/wheel/test_shared_libraries.py | 67 +++- .github/scripts/filter_cuda_matrix.py | 238 ++++++++++++ .../build-wheels-cuda-aarch64-linux.yml | 103 +++++ .github/workflows/build-wheels-cuda-linux.yml | 99 +++++ backends/cuda/CMakeLists.txt | 68 ++++ install_requirements.py | 15 +- 9 files changed, 1114 insertions(+), 4 deletions(-) create mode 100644 .ci/scripts/wheel/cuda_arch_list.sh create mode 100644 .ci/scripts/wheel/envvar_cuda_linux.sh create mode 100644 .ci/scripts/wheel/test_cuda_linux.py create mode 100644 .github/scripts/filter_cuda_matrix.py create mode 100644 .github/workflows/build-wheels-cuda-aarch64-linux.yml create mode 100644 .github/workflows/build-wheels-cuda-linux.yml diff --git a/.ci/scripts/wheel/cuda_arch_list.sh b/.ci/scripts/wheel/cuda_arch_list.sh new file mode 100644 index 00000000000..ed14bd9a2bd --- /dev/null +++ b/.ci/scripts/wheel/cuda_arch_list.sh @@ -0,0 +1,133 @@ +#!/usr/bin/env bash +# Copyright (c) Meta Platforms, Inc. and affiliates. +# All rights reserved. +# +# This source code is licensed under the BSD-style license found in the +# LICENSE file in the root directory of this source tree. + +# GPU architectures to compile device code for, chosen per release row rather than detected from +# the build machine. +# +# Without this nothing selects the architectures, so nvcc falls back to its own default and the +# wheel carries device code for that one architecture regardless of the builder's GPU. Measured: +# with the architecture list unset the compile line has no gencode flags at all. The wheel then +# installs on every machine the row claims and fails when a model runs on a different generation, +# with an error that looks like a model problem rather than a packaging one. Detection is the right +# default for a local build and the wrong one for a published artifact. +# +# The value is published as TORCH_CUDA_ARCH_LIST rather than CMAKE_CUDA_ARCHITECTURES, because +# PyTorch's CMake rejects the latter and overrides it, so setting only that reduces the build to a +# single detected architecture. + +# The architectures each row serves. Two rules decide the list, and they pull in opposite directions. +# +# The upper end follows the published PyTorch build for that train, read from its own library rather than +# chosen by reasoning about which GPUs matter. A delegate is only useful where torch already runs, and an +# architecture torch supports but this wheel omits produces a wheel that installs and then fails at the +# first kernel launch. Two omissions found that way were the GPU on the runner that tests these wheels, +# and a common desktop card. +# +# The lower end does NOT follow torch. It stops at 8.0 even though torch reaches further down, because one +# source here compiles an integer matrix-multiply path only at 8.0 and above. Below that a user gets a +# delegate that loads, runs most models, and fails on one needing that operator, which is worse than a row +# that never claimed the device. So these lists are narrower than torch at the bottom on purpose. +_cuda_arch_x86_64_cu130="8.0 8.6 8.9 9.0 10.0 12.0" +_cuda_arch_x86_64_cu132="${_cuda_arch_x86_64_cu130}" + +# The architectures the published aarch64 PyTorch CUDA build covers, read from its own library on an ARM +# machine, for the same reason as the x86_64 rows above. Includes the ARM module whose train matches. +_cuda_arch_aarch64_cu130="8.0 9.0 10.0 11.0 12.0" +_cuda_arch_aarch64_cu132="${_cuda_arch_aarch64_cu130}" + +# The older CUDA train. +# +# The two architectures do not carry identical lists, because each covers what the published PyTorch +# build for that architecture covers, and those differ. Matching them to each other instead would mean +# advertising a GPU on one architecture that PyTorch cannot serve there. +# +# The smaller embedded modules are deliberately absent, with one exception. An embedded-only +# architecture in a generic wheel would advertise a device the row cannot otherwise serve, since +# those devices also need the CUDA, TensorRT and PyTorch pinned by their own software release +# rather than the ones a generic wheel resolves. +# +# 8.7 is that exception. This is the only row whose CUDA major matches what that module's software +# release ships, and the wheel declares no PyTorch, so the user supplies the build that carries +# their architecture. Omitting it does not protect them from a bad pairing, it only removes the +# device code they need. +# +# The floor is 8.0 rather than the oldest architecture PyTorch still carries. One of these sources compiles +# an integer matrix-multiply path only at 8.0 and newer, so an older architecture would get a delegate that +# loads, runs most models, and fails on one that needs that operator. Claiming hardware the delegate only +# partly serves is the same problem the embedded modules have, so the row leaves it out for the same reason. +_cuda_arch_x86_64_cu126="8.0 8.6 8.9 9.0" +_cuda_arch_aarch64_cu126="8.0 8.7 9.0" + +# A CUDA train with no architecture list would leave the build detecting the builder's GPU, which is +# the failure this file exists to prevent. Adding a train to the release matrix without adding its +# architectures should fail loudly rather than silently produce a single-GPU wheel. +_executorch_unknown_train() { + echo "cuda_arch_list.sh: no GPU architecture list for CUDA train '$1' on $(uname -m)." >&2 + echo "Add one before building this row, or the wheel ships device code for one GPU only." >&2 + return 64 +} + +# The architectures for the current row, space separated in the dotted form PyTorch expects. +executorch_cuda_arch_list() { + local machine + machine="$(uname -m)" + # The wheel build exports the row's CUDA train as CU_VERSION. DESIRED_CUDA is the name of the + # matrix field rather than of the variable, so reading only that leaves every row falling back to + # detecting the builder's GPU. + local train="${CU_VERSION:-${DESIRED_CUDA:-}}" + # A CPU row names no CUDA train and needs no architectures, so it is not an error. + # + # A CUDA row always names one, so an empty value there means the row lost it. Treating that as a CPU + # row let the build fall back to detecting the builder's GPU, which produces a wheel carrying device + # code for whatever machine happened to build it while every check still reports green. + case "${train}" in + "" | cpu | CPU | none | NONE) + if [ "${EXECUTORCH_BUILD_CUDA:-}" = "1" ]; then + echo "this is a CUDA build but the row's CUDA version is '${train}', which names no CUDA" >&2 + echo "train. Refusing to detect the builder GPU instead." >&2 + return 65 + fi + return 0 + ;; + esac + # The value arrives as cu130, while some callers pass 13.0 instead. + train="${train#cu}" + train="${train//./}" + + case "${machine}" in + aarch64 | arm64) + case "${train}" in + 126) printf '%s' "${_cuda_arch_aarch64_cu126}" ;; + 130) printf '%s' "${_cuda_arch_aarch64_cu130}" ;; + 132) printf '%s' "${_cuda_arch_aarch64_cu132}" ;; + *) _executorch_unknown_train "${train}" ;; + esac + ;; + x86_64) + case "${train}" in + 126) printf '%s' "${_cuda_arch_x86_64_cu126}" ;; + 130) printf '%s' "${_cuda_arch_x86_64_cu130}" ;; + 132) printf '%s' "${_cuda_arch_x86_64_cu132}" ;; + *) _executorch_unknown_train "${train}" ;; + esac + ;; + *) _executorch_unknown_train "${train}" ;; + esac +} + +# The architecture list for a row. Every entry already carries the portable form that lets a newer +# GPU compile at load time: an unsuffixed architecture asks the compiler for both the compiled and +# the portable form, measured as code=[compute_120,sm_120] for a bare "120". Nothing extra is needed +# for forward compatibility. +executorch_cuda_arch_list_with_ptx() { + local dotted + # Propagate a failed lookup rather than reporting an empty list, since a caller cannot tell an + # unknown row from a CPU row and the unknown one must not pass silently. + dotted="$(executorch_cuda_arch_list)" || return $? + [ -n "${dotted}" ] || return 0 + printf '%s' "${dotted}" +} diff --git a/.ci/scripts/wheel/envvar_cuda_linux.sh b/.ci/scripts/wheel/envvar_cuda_linux.sh new file mode 100644 index 00000000000..d66ae3f2d22 --- /dev/null +++ b/.ci/scripts/wheel/envvar_cuda_linux.sh @@ -0,0 +1,42 @@ +# Copyright (c) Meta Platforms, Inc. and affiliates. +# All rights reserved. +# +# This source code is licensed under the BSD-style license found in the +# LICENSE file in the root directory of this source tree. + +# This file is sourced into the environment before building a pip wheel. It +# should typically only contain shell variable assignments. Be sure to export +# any variables so that subprocesses will see them. + +source "${GITHUB_WORKSPACE}/${REPOSITORY}/.ci/scripts/wheel/envvar_base.sh" + +# Ask for the CUDA delegate explicitly rather than letting the build detect a toolkit. A detected +# build is fine locally, but a release row states what it is producing, and a row that silently +# produced a CPU wheel because the toolkit was missing would publish under a CUDA name. +export EXECUTORCH_BUILD_CUDA=1 +export CMAKE_ARGS="${CMAKE_ARGS} -DEXECUTORCH_BUILD_CUDA=ON" + +# Fail the build if CUDA is not actually present. Without this the packaging step would look for +# CUDA libraries that were never built and report a confusing missing-file error several minutes +# after the real problem. +if [ ! -x "${CUDA_HOME:-/usr/local/cuda}/bin/nvcc" ]; then + echo "EXECUTORCH_BUILD_CUDA is set but no nvcc was found. This row cannot build a CUDA wheel." >&2 + exit 1 +fi + +# Compile device code for the GPUs this release row claims, rather than for whichever GPU the +# builder happens to have. A wheel built by detection alone installs on every machine the row covers +# and then fails when a model runs on a different generation. +source "${GITHUB_WORKSPACE}/${REPOSITORY}/.ci/scripts/wheel/cuda_arch_list.sh" +# The status is checked rather than only the output. An unrecognised row makes the lookup fail, and +# this file is sourced rather than run under a failing-command shell, so ignoring the status would +# leave the variable unset and let the build fall back to detecting the builder's own GPU. That is +# exactly the outcome this is meant to prevent, and it would ship quietly. +if ! _executorch_cuda_arch="$(executorch_cuda_arch_list_with_ptx)"; then + echo "could not resolve GPU architectures for CU_VERSION=${CU_VERSION:-unset}" >&2 + exit 1 +fi +if [ -n "${_executorch_cuda_arch}" ]; then + export TORCH_CUDA_ARCH_LIST="${_executorch_cuda_arch}" + echo "building device code for: ${TORCH_CUDA_ARCH_LIST}" +fi diff --git a/.ci/scripts/wheel/test_cuda_linux.py b/.ci/scripts/wheel/test_cuda_linux.py new file mode 100644 index 00000000000..44728a64860 --- /dev/null +++ b/.ci/scripts/wheel/test_cuda_linux.py @@ -0,0 +1,353 @@ +#!/usr/bin/env python +# Copyright (c) Meta Platforms, Inc. and affiliates. +# All rights reserved. +# +# This source code is licensed under the BSD-style license found in the +# LICENSE file in the root directory of this source tree. + +"""Smoke test for a CUDA wheel row. + +Runs the checks a GPU wheel needs, then the packaging, backend, and C++ SDK checks a CPU wheel +gets. The extra CUDA checks exist because a GPU wheel can install cleanly, import cleanly, and +still be unusable: + + the CUDA libraries can be absent while the wheel is still named as a CUDA build + the runtime dependency can be undeclared, so a user has nothing to resolve it from + the loader path can point at the build machine's toolkit, which no user has + the device code can cover no GPU the row claims, which only appears when a model runs + +This does not execute a model. The aarch64 rows have no GPU to execute one on, because their +validation runner has no accelerator, so for those rows inspection is all that is available +here. The x86_64 rows do run on a GPU runner, so a model-execution check is possible there and +its absence is a gap rather than a limit. What runs a model on real hardware before a +publication is a separate release-time step that a person owns today, not an automated job +wired into these workflows. +""" + +import os +import pathlib +import platform +import subprocess +import tempfile +from pathlib import Path + +import test_base +import test_cpp_sdk +import test_shared_libraries +from examples.models import Backend, Model + + +def _package_dir() -> Path: + import executorch + + return Path(executorch.__path__[0]) + + +def test_cuda_libraries_are_shipped() -> None: + """The row is named for CUDA, so the CUDA libraries have to be in it.""" + lib_dir = _package_dir() / "lib" + shipped = {path.name for path in lib_dir.iterdir()} if lib_dir.is_dir() else set() + expected = { + "libexecutorch_backend_cuda.so", + "libexecutorch_extension_cuda.so", + } + missing = sorted(expected - shipped) + assert not missing, ( + f"this is a CUDA row but {missing} are not in the wheel, so it would install as a " + f"CUDA build with no CUDA delegate. Shipped: {sorted(shipped)}" + ) + print(f"✓ the CUDA libraries ship ({len(expected)} of them)") + + +def test_cuda_runtime_is_declared() -> None: + """The wheel links the CUDA runtime without bundling it, so it must declare it. + + Without this a user installs the wheel and has nothing to resolve libcudart from, which + surfaces as a loader error at the first import rather than as a resolution failure at + install time. + """ + import importlib.metadata as metadata + + requirements = metadata.requires("executorch") or [] + cuda = [ + requirement + for requirement in requirements + if "nvidia" in requirement.lower() or "cuda" in requirement.lower() + ] + assert cuda, ( + "this is a CUDA row but the wheel declares no CUDA runtime dependency, so nothing " + "would install the libraries its delegate links" + ) + print(f"✓ the CUDA runtime is declared ({len(cuda)} requirements)") + + +def test_cuda_libraries_resolve_relatively() -> None: + """Each CUDA library must reach its runtime through a relative path. + + An absolute toolkit path names the machine that built the wheel. It resolves there and + nowhere else, so the wheel would work only on a builder. + + Every shipped library that links the CUDA runtime is inspected, wherever it lives. Naming + only the two in lib/ skipped libaoti_cuda_shims.so, which sits under backends/cuda/, links + cudart and curand, and carries the device code, so an absolute toolkit path on the library + that matters most shipped green. + """ + readelf = test_shared_libraries._tool("readelf") + assert readelf is not None, "readelf is required to inspect the wheel" + + package_dir = _package_dir() + libraries = sorted(test_shared_libraries._shipped_shared_objects(package_dir)) + # Without this the loop below finds nothing on a wheel that ships no CUDA library and + # reports a pass, which is the same as having no check at all. + assert libraries, f"no shared libraries found under {package_dir}" + + linked_to_cuda = [] + for library in libraries: + output = subprocess.run( + [readelf, "-d", str(library)], capture_output=True, text=True, check=True + ).stdout + if any("NEEDED" in line and "libcud" in line for line in output.splitlines()): + linked_to_cuda.append((library, output)) + + assert linked_to_cuda, ( + "no shipped library links the CUDA runtime, so this check inspected nothing. A CUDA " + "row must ship the libraries it is named for." + ) + for library, output in linked_to_cuda: + name = library.relative_to(package_dir) + entries: list[str] = [] + for line in output.splitlines(): + if "RPATH" in line or "RUNPATH" in line: + entries += line.split("[", 1)[1].rstrip("]").strip().split(":") + relative = [ + entry + for entry in entries + if entry.startswith("$ORIGIN") and "nvidia" in entry + ] + assert relative, ( + f"{name} links the CUDA runtime but has no relative path to the CUDA wheels " + f"installed beside it, so it can only resolve where the builder had a toolkit: " + f"{entries}" + ) + print(f"✓ {name} resolves the CUDA runtime relatively ({relative[0]})") + + +def _row_architectures() -> list[str]: + """The architectures this row claims, from the same script the build uses. + + A refusal from that script is a fault, not an absence. It returns non-zero when a CUDA row reaches it + with no version, which is precisely the case that would otherwise build device code for whatever GPU the + builder happens to have, so swallowing it here would hide the one failure this check exists to catch. + + EXECUTORCH_BUILD_CUDA is passed through because that is how the build invokes the script, and the + refusal is conditional on it. Without it the script returned an empty list on a CUDA row that had lost + its version, this check reported nothing to do, and the assertion below could never fire. + """ + script = pathlib.Path(__file__).parent / "cuda_arch_list.sh" + assert script.is_file(), f"the architecture script is missing at {script}" + result = subprocess.run( + ["bash", "-c", f"source {script}; executorch_cuda_arch_list"], + capture_output=True, + text=True, + check=False, + env={**os.environ, "EXECUTORCH_BUILD_CUDA": "1"}, + ) + assert result.returncode == 0, ( + f"the architecture script refused this row with exit {result.returncode}, so the build had no list " + f"to compile against: {result.stderr.strip()[:300]}" + ) + # "8.0 9.0" describes sm_80 and sm_90. + return ["sm_" + value.replace(".", "") for value in result.stdout.split()] + + +def test_device_code_covers_the_row() -> None: + """Every GPU the row claims must have device code in the shipped libraries. + + A row that promises a GPU it did not compile for produces a wheel that installs and then dies + at the first kernel launch, which is the worst failure to publish. + """ + expected = _row_architectures() + if not expected: + print("- this row claims no GPU architectures, nothing to check") + return + + cuobjdump = test_shared_libraries._tool("cuobjdump") + if cuobjdump is None: + raise AssertionError( + "cuobjdump is required to check device code, and this is a CUDA row. Without it a " + "wheel missing code for a claimed GPU would ship unnoticed." + ) + + # Searched across every shipped library rather than a named one. The kernels are compiled + # into their own library, not into the delegate, and which library holds them is an internal + # detail. What the row promises is that the wheel covers those GPUs. + present: set[str] = set() + inspected = [] + with_device_code: dict = {} + for library in sorted(_package_dir().rglob("*.so")): + listed = subprocess.run( + [cuobjdump, "--list-elf", str(library)], + capture_output=True, + text=True, + check=False, + ).stdout + found = { + token + for token in listed.replace(".", " ").split() + if token.startswith("sm_") + } + if found: + inspected.append(f"{library.name} ({', '.join(sorted(found))})") + present |= found + with_device_code[library.name] = found + + assert inspected, ( + "no shipped library contains any GPU device code, so this wheel cannot run a model on any " + f"GPU, while the row claims {expected}" + ) + missing = sorted(set(expected) - present) + assert not missing, ( + f"the row claims {expected} but the wheel carries no device code for {missing}. " + f"Found: {inspected}. A user with one of those GPUs would install this wheel and fail at " + "the first kernel launch." + ) + # The other direction matters just as much. Device code for an architecture the row does not + # claim means the build did not use the row's list, so whatever selected the architectures + # ignored it. That went unnoticed once already: a selection bug substituted a single default + # architecture and this check stayed green because it only looked for what was absent. + unexpected = sorted(present - set(expected)) + assert not unexpected, ( + f"the row claims {sorted(set(expected))} but the wheel also carries device code for " + f"{unexpected}. Found: {inspected}. The build did not use the row's list, so the artifact " + "does not match what the row published." + ) + # Every library that carries device code has to cover the row on its own. Unioning + # across libraries let a library with kernels cover only part of the row while an + # unrelated object supplied the rest, so on a GPU the first one did not compile for + # there was no executable kernel even though the union looked complete. + short = sorted(set(expected)) + for library in sorted(with_device_code): + library_missing = sorted(set(expected) - with_device_code[library]) + assert not library_missing, ( + f"{library} carries GPU device code but none for {library_missing}, while the row " + f"claims {short}. Checking the union across libraries hid this: another shipped " + "object supplied those architectures, and on such a GPU this library would have no " + "executable kernel." + ) + print(f"✓ device code covers the row in every library that has any: {inspected}") + + +def test_portable_device_code_is_present() -> None: + """The newest architecture must also ship in its portable form. + + The build appends "+PTX" for the top architecture so a GPU newer than any in the row can + still run, by having the driver compile that portable form at load time. Without it such a + GPU gets no usable code at all. + + Checked with --list-ptx rather than --list-elf. --list-elf prints byte-identical output for + a library built with or without the portable form, so it cannot see this. --list-ptx prints + an entry only for the library that has it. The entry is named for the target architecture, + "sm_90.ptx" rather than "compute_90.ptx", which is what the real tool prints. + """ + expected = _row_architectures() + if not expected: + print("- this row claims no GPU architectures, nothing to check") + return + + cuobjdump = test_shared_libraries._tool("cuobjdump") + assert ( + cuobjdump is not None + ), "cuobjdump is required to check the portable device code, and this is a CUDA row." + + # The newest architecture in the row, which is the one the build makes portable. + newest = max(expected, key=lambda name: int(name.removeprefix("sm_"))) + + found_in = [] + for library in sorted(_package_dir().rglob("*.so")): + listed = subprocess.run( + [cuobjdump, "--list-ptx", str(library)], + capture_output=True, + text=True, + check=False, + ).stdout + if newest in listed.replace(".", " ").split(): + found_in.append(library.name) + + assert found_in, ( + f"no shipped library carries portable device code for {newest}, the newest architecture " + f"in this row ({sorted(expected)}). A GPU newer than {newest} would install this wheel and " + "find no code it can run. The build appends the portable form for exactly this case, so " + "either it was dropped or the spelling in the architecture list is wrong." + ) + print(f"✓ portable device code for {newest} ships in {', '.join(found_in)}") + + +def test_the_delegate_registers() -> None: + """The delegate has to appear in the runtime's backend list, not merely be present as a file. + + Registration happens in a static initializer, which a normal link discards because nothing in the + program references it. Keeping it alive needs a linker option, and a wheel whose delegate ships but + does not register would load a delegated program and fail with an unregistered backend. That is the + failure this whole layout is most able to introduce, so it is worth asserting rather than assuming. + + Needs no GPU: registration is a link-time property, checked by importing. + """ + from executorch.extension.pybindings.portable_lib import ( + _get_registered_backend_names, + ) + + registered = _get_registered_backend_names() + assert "CudaBackend" in registered, ( + f"the wheel ships the CUDA delegate but CudaBackend is not registered: {registered}. " + "The library is present and its static initializer did not run, which means the option " + "that keeps it on the link line stopped working." + ) + print(f"✓ the delegate registers: CudaBackend among {len(registered)} backend(s)") + + +if __name__ == "__main__": + assert platform.system() == "Linux", "the CUDA rows are Linux only" + + test_cuda_libraries_are_shipped() + test_cuda_runtime_is_declared() + test_cuda_libraries_resolve_relatively() + test_device_code_covers_the_row() + test_portable_device_code_is_present() + test_the_delegate_registers() + + # The backend registrations a CPU Linux row asserts also apply here: the CUDA build enables + # OpenVINO on every Linux architecture and downloads the QNN SDK on x86_64, so a CUDA wheel + # carries both backends and needs both to register. + from executorch.extension.pybindings.portable_lib import ( + _get_registered_backend_names, + ) + + registered = _get_registered_backend_names() + if platform.machine() in ("x86_64", "amd64"): + assert ( + "QnnBackend" in registered + ), f"QnnBackend not found in registered backends: {registered}" + print("✓ QnnBackend is registered") + assert ( + "OpenvinoBackend" in registered + ), f"OpenvinoBackend not found in registered backends: {registered}" + print("✓ OpenvinoBackend is registered") + + test_base.test_cmsis_nn_install() + + # The packaging and linking checks a CPU wheel is held to still apply: one owner per + # component, no build-tree paths, and a C++ application able to link what the wheel + # ships. + with tempfile.TemporaryDirectory() as work_dir: + test_shared_libraries.run_tests(Path(work_dir)) + with tempfile.TemporaryDirectory() as work_dir: + test_cpp_sdk.run_tests(Path(work_dir)) + + test_base.run_tests( + model_tests=[ + test_base.ModelTest( + model=Model.Mv3, + backend=Backend.XnnpackQuantizationDelegation, + ), + ] + ) diff --git a/.ci/scripts/wheel/test_shared_libraries.py b/.ci/scripts/wheel/test_shared_libraries.py index 0484736f9f4..4fe89b47f28 100644 --- a/.ci/scripts/wheel/test_shared_libraries.py +++ b/.ci/scripts/wheel/test_shared_libraries.py @@ -440,6 +440,22 @@ def _wheel_cuda_train() -> str: _REQUIRED_ON_A_CUDA_WHEEL = "cuda-wheel-only" +# The exact dependency names packaging declares per CUDA train, mirroring +# _CUDA_RUNTIME_PACKAGES in setup.py. Listed here rather than imported because setup.py +# runs a build when imported, and duplicated deliberately so a rename on the packaging +# side has to be made here too rather than silently agreeing with itself. +_EXPECTED_CUDA_PACKAGES = { + "12": ( + "nvidia-cuda-runtime-cu12", + "nvidia-curand-cu12", + ), + "13": ( + "nvidia-cuda-runtime", + "nvidia-curand", + ), +} + + # Each component the wheel ships as its own library, the symbols that identify it, # and the library that must own them. `required` says whether the owner has to be # present: the optimized kernels are optional, because a wheel built without them @@ -1635,17 +1651,34 @@ def test_model_matches_eager_pytorch(work_dir: Path) -> None: def test_declared_dependencies_match_the_wheel_tag() -> None: - """A CPU wheel must not declare the CUDA runtime, and a CUDA wheel must declare it. + """A CPU wheel must not declare the CUDA runtime, and a CUDA wheel must declare its own train. The tag is what a user resolves against, so a mismatch is a promise the wheel cannot keep in either direction: a CPU wheel that pulls the CUDA packages costs a user hundreds of megabytes it never loads, and a CUDA wheel that declares nothing leaves the runtime unresolvable. + Declaring the wrong train is the quiet case, and the reason this checks the names rather than + only their presence. The CUDA 12 packages are published with a "-cu12" suffix and the CUDA 13 + ones without, so a cu130 wheel that asked for the cu12 packages would install a runtime its + libraries cannot load, while looking correctly specified. + This is metadata only, so no library check can see it. A CPU wheel that wrongly declared the CUDA runtime passed every other check in this file. """ requirements = importlib.metadata.requires("executorch") or [] - cuda = sorted(r.split()[0] for r in requirements if r.lower().startswith("nvidia")) + + # Split off any environment marker AND any version specifier. The name, the specifier + # and the marker can arrive as one token, so taking the first whitespace-separated + # word left "nvidia-cuda-runtime-cu12==12.6.77" as the name and made a correctly + # specified wheel fail the moment any CUDA dependency gained a pin. + def distribution_name(requirement: str) -> str: + return re.split(r"[\s;\[<>=!~(]", requirement.strip(), maxsplit=1)[0] + + cuda = sorted( + name + for name in (distribution_name(r) for r in requirements) + if name.lower().startswith("nvidia") + ) # The local version segment of the installed version states what the wheel was built for. version = importlib.metadata.version("executorch") @@ -1657,7 +1690,35 @@ def test_declared_dependencies_match_the_wheel_tag() -> None: f"version {version} says this is a CUDA wheel, but it declares no CUDA runtime " "packages, so nothing resolves the runtime it links" ) - print(f"✓ this CUDA wheel declares the runtime ({len(cuda)} packages)") + # Compared as sets in both directions rather than as a name suffix: for CUDA 13 the + # expected suffix is the empty string and every name ends with that, so a suffix test + # accepted a name from any train whose spelling happened not to be one of the two + # literals it also excluded. Measured: a cu130 wheel declaring nvidia-cuda-runtime-cu11 + # passed. The reverse check catches the other side of the same defect: a wheel that + # declares one package and omits the others still cannot load, and one-direction only + # would accept it. + train = local[len("cu") : len("cu") + 2] + expected = set(_EXPECTED_CUDA_PACKAGES.get(train, ())) + assert expected, ( + f"version {version} names CUDA train {train}, which this check has no expected " + f"package list for. Add it beside the packaging list it mirrors." + ) + actual = set(cuda) + wrong = sorted(actual - expected) + missing = sorted(expected - actual) + assert not wrong, ( + f"version {version} is a CUDA {train} wheel, but it declares {wrong}, which belong to " + f"another CUDA train. Expected only {sorted(expected)}. A user would install a runtime " + "this wheel's libraries cannot load." + ) + assert not missing, ( + f"version {version} is a CUDA {train} wheel, but it does not declare {missing} " + f"(expected {sorted(expected)}). A user installing this wheel would end up without part " + "of the CUDA runtime the wheel's libraries need." + ) + print( + f"✓ this CUDA {train} wheel declares its own runtime ({len(cuda)} packages)" + ) else: assert not cuda, ( f"version {version} is not a CUDA wheel, yet it declares {cuda}. A user installing it " diff --git a/.github/scripts/filter_cuda_matrix.py b/.github/scripts/filter_cuda_matrix.py new file mode 100644 index 00000000000..385ea9385cb --- /dev/null +++ b/.github/scripts/filter_cuda_matrix.py @@ -0,0 +1,238 @@ +#!/usr/bin/env python3 +# Copyright (c) Meta Platforms, Inc. and affiliates. +# All rights reserved. +# +# This source code is licensed under the BSD-style license found in the +# LICENSE file in the root directory of this source tree. + +"""Narrow the generated build matrix to the rows a GPU wheel can honestly support. + +The shared matrix generator emits every CUDA version and Python version it knows about. +Building all of them would publish wheels for combinations nothing can verify, and a GPU +wheel that installs and then cannot run is worse than one that does not exist: the failure +appears when a model runs, and it looks like a model problem rather than a packaging one. + +A row is kept only when both of these hold: + + a GPU exists that the row's device code covers + a PyTorch build is published for that CUDA version and architecture + +Running a real model before release is a separate gate, on hardware that has the matching +GPU, so a row can be published for a CUDA version no machine here can execute. + +The values below are the current answers to those questions. They are written out rather +than derived because each one is an external fact that can change independently. +""" + +import argparse +import json +import sys +from typing import Any, Dict, List + +# Python versions that are deliberately NOT published, with the reason, so a row naming one +# is rejected for a stated cause rather than for merely being absent from the supported list. +# 3.14 is excluded because the current CPU wheel rows already fail on it for an unrelated +# reason in the example requirements, so a GPU row would inherit a known-broken build. The +# free-threaded builds are excluded because the CUDA dependencies are not published for them. +# +# This is documentation, not the gate. The gate is SUPPORTED_PYTHON_VERSIONS below: anything +# not on that list is rejected whether or not it appears here. +DISABLED_PYTHON_VERSIONS: List[str] = ["3.13t", "3.14", "3.14t", "3.15", "3.15t"] + +# CUDA versions to publish. +# +# Chosen so that every consumer row can find a matching wheel rather than by what is +# convenient to verify. A delegate built against one of these has to be able to depend on an +# ExecuTorch wheel for the same CUDA version, and a missing version means that consumer has +# nothing to depend on: +# +# cu126 the floor, and what Jetson devices are limited to +# cu130 the generator's stable choice, and the default for accelerator consumers +# cu132 the newest, which consumers building against a current TensorRT need +# +# cu132 is included even though no machine here can execute it, because omitting it would +# leave a published consumer row with no ExecuTorch wheel to pair with. The packaging +# properties are checked on every row; executing a model is a release-gate step on hardware +# that has the matching GPU. +SUPPORTED_CUDA_VERSIONS: List[str] = ["cu126", "cu130", "cu132"] + +# Python versions to publish, stated rather than derived for the same reason the CUDA +# versions are. Deriving them from the rows that survived the filter made the release +# guard below unable to notice a python that disappeared from every supported train: with +# nothing left to compare, a release quietly published nine wheels instead of twelve. +# Keep in step with the python-versions list in the CUDA wheel workflows. +SUPPORTED_PYTHON_VERSIONS: List[str] = ["3.10", "3.11", "3.12", "3.13"] + +# The single row built for a pull request. A full matrix on every push would cost hours for +# little signal, and this pair is the one with a machine that can run a model on it. +PR_PYTHON_VERSION: str = "3.12" +PR_CUDA_VERSION: str = "cu130" + +# Jetson devices are their own row: a JetPack image, one Python version, and one CUDA +# version. Kept empty on purpose today, so no Jetson row is emitted. +# +# The generic aarch64 CUDA 12.6 wheel does compile sm_87 device code for one embedded +# module, so the wheel itself is not the blocker. What is: published PyTorch stopped +# shipping sm_87 device code after 2.8.0, so a Jetson row today would produce a wheel +# whose PyTorch dependency cannot execute on the device. Populate this when that +# changes. +# +# Because both lists are empty, asking for the JetPack rows can only produce an empty result. +# No workflow asks, and the request is rejected up front with that reason rather than left to +# surface as the generic "the filter produced no rows" message, which reads as a broken +# matrix rather than as a row that is deliberately not built yet. +JETPACK_PYTHON_VERSIONS: List[str] = [] +JETPACK_CUDA_VERSIONS: List[str] = [] +JETPACK_CONTAINER_IMAGE: str = "nvcr.io/nvidia/l4t-jetpack:r36.4.0" + + +def keep(item: Dict[str, Any], is_jetpack: bool) -> bool: + """Whether this row should be built, adjusting its container image where needed.""" + # An allowlist, the same shape as the CUDA test below. Testing only the disabled list + # let any python not on it through: passing a 3.9 row returned success and emitted it, + # and the only thing preventing that today is both workflows happening to pin the list + # they pass in. + if item["python_version"] not in SUPPORTED_PYTHON_VERSIONS: + return False + + if is_jetpack: + if ( + item["python_version"] in JETPACK_PYTHON_VERSIONS + and item["desired_cuda"] in JETPACK_CUDA_VERSIONS + ): + item["container_image"] = JETPACK_CONTAINER_IMAGE + return True + return False + + if item["desired_cuda"] not in SUPPORTED_CUDA_VERSIONS: + return False + + return True + + +def _version_rank(cuda: str) -> int: + """Where a CUDA version sits in the supported list, or -1 when it is not supported at all.""" + try: + return SUPPORTED_CUDA_VERSIONS.index(cuda) + except ValueError: + return -1 + + +def only_pull_request_row(items: List[Dict[str, Any]]) -> List[Dict[str, Any]]: + """One representative row, so a pull request does not build the whole matrix. + + Chosen by preference rather than exact match, so a request that does not appear in the + generated matrix degrades to the closest supported combination instead of falling off the + end. + """ + if not items: + return [] + + # Looked up once, and tolerantly: a PR_CUDA_VERSION that falls off SUPPORTED_CUDA_VERSIONS used to + # raise here and break every pull request while releases kept working, which is the wrong way round + # for a constant that only chooses which single row to build. + wanted = _version_rank(PR_CUDA_VERSION) + + def rank(item: Dict[str, Any]) -> tuple: + # Closeness peaks at the requested version, then falls off, and it outranks the python match. + # Ranking python first picked a wheel for a CUDA version nothing on hand can execute whenever the + # generator skewed the two axes, and the point of building one row is to get signal from it. + offered = _version_rank(item["desired_cuda"]) + # Negative above the requested version, so a newer one never outranks an older one a machine here + # can actually run. + closeness = offered if offered <= wanted else wanted - offered + return (closeness, item["python_version"] == PR_PYTHON_VERSION) + + return [max(items, key=rank)] + + +def main(argv: List[str]) -> None: + parser = argparse.ArgumentParser() + parser.add_argument("--matrix", required=True, help="the generated matrix, as JSON") + parser.add_argument( + "--jetpack", default="false", help="build the Jetson row instead" + ) + parser.add_argument("--limit-pr-builds", default="false", help="build one row only") + args = parser.parse_args(argv) + + try: + matrix = json.loads(args.matrix) + except json.JSONDecodeError as error: + print(f"could not parse the matrix: {error}", file=sys.stderr) + sys.exit(1) + + is_jetpack = args.jetpack.lower() == "true" + if is_jetpack and not (JETPACK_PYTHON_VERSIONS and JETPACK_CUDA_VERSIONS): + # Rejected here rather than allowed to fall through to an empty result, so the reason + # is the actual one. Nothing passes this flag today. + print( + "the JetPack rows are not published yet: JETPACK_PYTHON_VERSIONS and " + "JETPACK_CUDA_VERSIONS are empty because published PyTorch carries no device code " + "for that GPU architecture, so any wheel built here could not run on the device. " + "Populate both lists to enable this row.", + file=sys.stderr, + ) + sys.exit(1) + items = [item for item in matrix.get("include", []) if keep(item, is_jetpack)] + + if args.limit_pr_builds.lower() == "true" and items: + items = only_pull_request_row(items) + elif items and not is_jetpack: + # A release has to publish every combination this policy advertises. Comparing the result against + # what the generator offered cannot catch anything, because both sides apply the same conditions, so + # the difference is empty by construction and the check never fires. The policy's own list is the + # thing to compare against: a CUDA version the generator stopped offering otherwise disappears from + # the release silently, and a missing job is a green check for a wheel that was never built. + # + # The generic rows only. A JetPack release advertises the single pair its own lists name rather than + # every supported CUDA version, so checking it against this list would fail a correct release. + # + # Both axes come from this policy's own lists, not from the matrix. Reading the generator's python + # axis pulled in rows this policy never builds, and deriving it from the rows that survived went + # blind to a python that disappeared from every supported train. The generator lives in another + # repository and its axes move independently of what this policy promises to publish. + built = {(item["python_version"], item["desired_cuda"]) for item in items} + # A train that produced no row at all is missing for every python, so reporting it per python + # would read as a python problem. Named on its own instead, and first, because the per-pair + # report below would otherwise bury it. + absent_trains = sorted( + set(SUPPORTED_CUDA_VERSIONS) - {cuda for _, cuda in built} + ) + if absent_trains: + print( + f"this policy publishes {SUPPORTED_CUDA_VERSIONS}, but the generator offered no row " + f"this filter could keep for {absent_trains}, so a release would publish no wheel for " + "that CUDA version at all", + file=sys.stderr, + ) + sys.exit(1) + missing = sorted( + f"{python}/{cuda}" + for python in SUPPORTED_PYTHON_VERSIONS + for cuda in SUPPORTED_CUDA_VERSIONS + if (python, cuda) not in built + ) + if missing: + print( + f"this policy publishes {SUPPORTED_CUDA_VERSIONS} for each of " + f"{SUPPORTED_PYTHON_VERSIONS}, but {len(missing)} combination(s) produced no row, so a " + f"release would publish no wheel for them: {missing}", + file=sys.stderr, + ) + sys.exit(1) + + # Fail loudly on an empty result. A silently empty matrix produces a workflow with no + # build job, which shows up as a green check for a build that never happened. + if not items: + print( + "the filter produced no rows to build, so nothing would be verified. " + f"jetpack={is_jetpack}, supported CUDA={SUPPORTED_CUDA_VERSIONS}", + file=sys.stderr, + ) + sys.exit(1) + + print(json.dumps({"include": items})) + + +if __name__ == "__main__": + main(sys.argv[1:]) diff --git a/.github/workflows/build-wheels-cuda-aarch64-linux.yml b/.github/workflows/build-wheels-cuda-aarch64-linux.yml new file mode 100644 index 00000000000..bb20d815b62 --- /dev/null +++ b/.github/workflows/build-wheels-cuda-aarch64-linux.yml @@ -0,0 +1,103 @@ +# From https://github.com/pytorch/test-infra/wiki/Using-Nova-Reusable-Build-Workflows +name: Build Aarch64 Linux CUDA Wheels + +on: + pull_request: + paths: + - .ci/**/* + - .github/scripts/filter_cuda_matrix.py + - .github/workflows/build-wheels-cuda-aarch64-linux.yml + - '**/CMakeLists.txt' + - backends/cuda/**/* + - examples/**/* + - extension/cuda/**/* + - install_requirements.py + - install_utils.py + - pyproject.toml + - setup.py + - tools/cmake/**/* + push: + branches: + - nightly + - release/* + tags: + # NOTE: Binary build pipelines should only get triggered on release candidate builds + # Release candidate tags look like: v1.11.0-rc1 + - v[0-9]+.[0-9]+.[0-9]+-rc[0-9]+ + - ciflow/binaries/* + workflow_dispatch: + +jobs: + generate-matrix: + uses: pytorch/test-infra/.github/workflows/generate_binary_build_matrix.yml@main + with: + package-type: wheel + os: linux-aarch64 + test-infra-repository: pytorch/test-infra + test-infra-ref: main + with-cuda: enable + with-cpu: disable + with-rocm: disable + python-versions: '["3.10", "3.11", "3.12", "3.13"]' + + # The generator emits every CUDA version it knows about. Publishing all of them would ship + # wheels for combinations nothing can verify, so this keeps only the rows with a GPU to run + # them on. The script fails rather than emitting an empty matrix, because a workflow with no + # build job reads as a pass. + filter-matrix: + needs: generate-matrix + runs-on: ubuntu-latest + outputs: + matrix: ${{ steps.filter.outputs.matrix }} + steps: + - uses: actions/setup-python@v6 + with: + python-version: '3.12' + - uses: actions/checkout@v4 + - name: Filter the matrix + id: filter + run: | + set -eou pipefail + MATRIX_BLOB=${{ toJSON(needs.generate-matrix.outputs.matrix) }} + # Mirrors the shared generator, which drops its own row limit when this label is + # present. Clamping here regardless meant the label was accepted as a trigger and + # then ignored, so the full matrix could never be exercised before a release. + LIMIT_PR=${{ (github.event_name == 'pull_request' && !contains(github.event.pull_request.labels.*.name, 'ciflow/binaries/all')) && 'true' || 'false' }} + MATRIX_BLOB="$(python3 .github/scripts/filter_cuda_matrix.py \ + --matrix "${MATRIX_BLOB}" --limit-pr-builds "${LIMIT_PR}")" + echo "${MATRIX_BLOB}" + echo "matrix=${MATRIX_BLOB}" >> "${GITHUB_OUTPUT}" + + build: + needs: filter-matrix + permissions: + id-token: write + contents: read + strategy: + fail-fast: false + matrix: + include: + - repository: pytorch/executorch + pre-script: .ci/scripts/wheel/pre_build_script.sh + post-script: .ci/scripts/wheel/post_build_script.sh + smoke-test-script: .ci/scripts/wheel/test_cuda_linux.py + package-name: executorch + name: ${{ matrix.repository }} + uses: pytorch/test-infra/.github/workflows/build_wheels_linux.yml@main + with: + repository: ${{ matrix.repository }} + ref: "" + test-infra-repository: pytorch/test-infra + test-infra-ref: main + build-matrix: ${{ needs.filter-matrix.outputs.matrix }} + submodules: recursive + env-var-script: .ci/scripts/wheel/envvar_cuda_linux.sh + pre-script: ${{ matrix.pre-script }} + post-script: ${{ matrix.post-script }} + package-name: ${{ matrix.package-name }} + smoke-test-script: ${{ matrix.smoke-test-script }} + trigger-event: ${{ github.event_name }} + # Required for aarch64. Without it the shared build workflow prepares an x86_64 job + # and skips the aarch64 conda install, so the first build step fails on a missing + # conda. + architecture: aarch64 diff --git a/.github/workflows/build-wheels-cuda-linux.yml b/.github/workflows/build-wheels-cuda-linux.yml new file mode 100644 index 00000000000..5d948370583 --- /dev/null +++ b/.github/workflows/build-wheels-cuda-linux.yml @@ -0,0 +1,99 @@ +# From https://github.com/pytorch/test-infra/wiki/Using-Nova-Reusable-Build-Workflows +name: Build Linux CUDA Wheels + +on: + pull_request: + paths: + - .ci/**/* + - .github/scripts/filter_cuda_matrix.py + - .github/workflows/build-wheels-cuda-linux.yml + - '**/CMakeLists.txt' + - backends/cuda/**/* + - examples/**/* + - extension/cuda/**/* + - install_requirements.py + - install_utils.py + - pyproject.toml + - setup.py + - tools/cmake/**/* + push: + branches: + - nightly + - release/* + tags: + # NOTE: Binary build pipelines should only get triggered on release candidate builds + # Release candidate tags look like: v1.11.0-rc1 + - v[0-9]+.[0-9]+.[0-9]+-rc[0-9]+ + - ciflow/binaries/* + workflow_dispatch: + +jobs: + generate-matrix: + uses: pytorch/test-infra/.github/workflows/generate_binary_build_matrix.yml@main + with: + package-type: wheel + os: linux + test-infra-repository: pytorch/test-infra + test-infra-ref: main + with-cuda: enable + with-cpu: disable + with-rocm: disable + python-versions: '["3.10", "3.11", "3.12", "3.13"]' + + # The generator emits every CUDA version it knows about. Publishing all of them would ship + # wheels for combinations nothing can verify, so this keeps only the rows with a GPU to run + # them on. The script fails rather than emitting an empty matrix, because a workflow with no + # build job reads as a pass. + filter-matrix: + needs: generate-matrix + runs-on: ubuntu-latest + outputs: + matrix: ${{ steps.filter.outputs.matrix }} + steps: + - uses: actions/setup-python@v6 + with: + python-version: '3.12' + - uses: actions/checkout@v4 + - name: Filter the matrix + id: filter + run: | + set -eou pipefail + MATRIX_BLOB=${{ toJSON(needs.generate-matrix.outputs.matrix) }} + # Mirrors the shared generator, which drops its own row limit when this label is + # present. Clamping here regardless meant the label was accepted as a trigger and + # then ignored, so the full matrix could never be exercised before a release. + LIMIT_PR=${{ (github.event_name == 'pull_request' && !contains(github.event.pull_request.labels.*.name, 'ciflow/binaries/all')) && 'true' || 'false' }} + MATRIX_BLOB="$(python3 .github/scripts/filter_cuda_matrix.py \ + --matrix "${MATRIX_BLOB}" --limit-pr-builds "${LIMIT_PR}")" + echo "${MATRIX_BLOB}" + echo "matrix=${MATRIX_BLOB}" >> "${GITHUB_OUTPUT}" + + build: + needs: filter-matrix + permissions: + id-token: write + contents: read + strategy: + fail-fast: false + matrix: + include: + - repository: pytorch/executorch + pre-script: .ci/scripts/wheel/pre_build_script.sh + post-script: .ci/scripts/wheel/post_build_script.sh + smoke-test-script: .ci/scripts/wheel/test_cuda_linux.py + package-name: executorch + name: ${{ matrix.repository }} + uses: pytorch/test-infra/.github/workflows/build_wheels_linux.yml@main + with: + repository: ${{ matrix.repository }} + ref: "" + test-infra-repository: pytorch/test-infra + test-infra-ref: main + build-matrix: ${{ needs.filter-matrix.outputs.matrix }} + submodules: recursive + env-var-script: .ci/scripts/wheel/envvar_cuda_linux.sh + pre-script: ${{ matrix.pre-script }} + post-script: ${{ matrix.post-script }} + package-name: ${{ matrix.package-name }} + smoke-test-script: ${{ matrix.smoke-test-script }} + trigger-event: ${{ github.event_name }} diff --git a/backends/cuda/CMakeLists.txt b/backends/cuda/CMakeLists.txt index 05f238401a4..c56e955e8a1 100644 --- a/backends/cuda/CMakeLists.txt +++ b/backends/cuda/CMakeLists.txt @@ -42,6 +42,74 @@ if(NOT CMAKE_CUDA_COMPILER) check_language(CUDA) endif() +# Take the architectures from the release row when it names them, before the +# language is enabled, since CMake fixes them at that point. Without this the +# build uses CMake's default, which on some devices is older than the intrinsics +# these sources use, and the compile fails with an undefined identifier that +# looks like a source problem. +# +# TORCH_CUDA_ARCH_LIST is the variable the surrounding build environment already +# sets, in PyTorch's dotted form. CMake wants bare integers, so "9.0" becomes +# 90. A "+PTX" suffix asks for the portable form in addition to the compiled +# one, which is what PyTorch means by it, so it adds the -virtual kind rather +# than replacing the -real one. Torch sets this to OFF at root scope when it is +# defined, warning that it ignores the value, so a defined value does not mean a +# caller chose it. OFF is treated as absent here, otherwise asking for an +# architecture through the preset silently compiles for whatever torch's own +# gencode flags select instead. CMAKE_CUDA_ARCHITECTURES is deliberately not +# read here. torch's CMake rejects and overrides it, which is why the release +# rows publish TORCH_CUDA_ARCH_LIST instead, and CMake fills the cache entry +# with a default of its own once the CUDA language is enabled. torch enables +# that language before this directory is added, so the cache always holds a +# value and cannot be read as a caller's intent: doing so replaced torch's +# autodetected architecture with CMake's default and broke the build on a device +# the default does not cover. +if(DEFINED ENV{TORCH_CUDA_ARCH_LIST}) + string(REPLACE "." "" _executorch_cuda_arch_list "$ENV{TORCH_CUDA_ARCH_LIST}") + string(REPLACE " " ";" _executorch_cuda_arch_list + "${_executorch_cuda_arch_list}" + ) + # torch spells a request for the portable form as a "+PTX" suffix, which nvcc + # rejects literally. Drop the suffix rather than expanding it into a separate + # -virtual entry: an unsuffixed CMake architecture already asks for both the + # compiled and the portable form, measured as code=[compute_120,sm_120] for a + # bare "120", so the extra entry only duplicates a gencode that is already + # there. + set(_executorch_cuda_arch_resolved "") + foreach(_arch IN LISTS _executorch_cuda_arch_list) + string(REGEX REPLACE "\\+PTX$" "" _arch "${_arch}") + list(APPEND _executorch_cuda_arch_resolved "${_arch}") + endforeach() + # A row that names both "12.0" and "12.0+PTX" collapses to the same entry + # twice, which would duplicate a gencode on the compile line. + list(REMOVE_DUPLICATES _executorch_cuda_arch_resolved) + # torch also accepts family names such as Ampere or All. They survive the + # translation above and then fail deep in a CUDA compile as an unsupported + # architecture, so reject them here where the message can name the cause. + foreach(_arch IN LISTS _executorch_cuda_arch_resolved) + if(NOT _arch MATCHES "^[0-9]+[a-z]*(-real|-virtual)?$") + message( + FATAL_ERROR + "TORCH_CUDA_ARCH_LIST entry \"${_arch}\" is not a compute capability. " + "Name capabilities numerically, for example \"8.0 9.0\" or \"12.0+PTX\". " + "Family names such as Ampere or All are accepted by torch but not by nvcc." + ) + endif() + endforeach() + set(CMAKE_CUDA_ARCHITECTURES "${_executorch_cuda_arch_resolved}") + message( + STATUS + "CUDA architectures from TORCH_CUDA_ARCH_LIST: ${CMAKE_CUDA_ARCHITECTURES}" + ) +else() + # torch sets this to OFF on purpose and drives nvcc with its own gencode + # flags, so there is nothing to choose here and overriding it would replace a + # working set of architectures with one value. + message( + STATUS "CUDA architectures left to torch: ${CMAKE_CUDA_ARCHITECTURES}" + ) +endif() + if(CMAKE_CUDA_COMPILER) enable_language(CUDA) endif() diff --git a/install_requirements.py b/install_requirements.py index 1aedcf6f0f8..648da1df243 100644 --- a/install_requirements.py +++ b/install_requirements.py @@ -7,6 +7,7 @@ import argparse import os +import platform import subprocess import sys @@ -45,7 +46,19 @@ def install_requirements(use_pytorch_nightly): # Determine the appropriate PyTorch URL based on CUDA delegate status torch_url = determine_torch_url(TORCH_URL_BASE) - torchao_url = determine_torch_url(TORCHAO_URL_BASE) + # torchao's CUDA channel publishes x86_64 only, so asking for a CUDA build makes the pin + # unsatisfiable on aarch64. Only that case is special-cased: falling back everywhere would + # change which torchao a CPU x86_64 install resolves, and the CUDA build is genuinely wanted + # where it exists. Nothing in the wheel links or bundles torchao; it is a quantization + # workflow dependency of the examples and tests. + if platform.machine().lower() in ("aarch64", "arm64"): + # The cpu channel specifically, not the index root. The root carries every variant, and a + # pin without a local segment admits all of them while ordering a local segment highest, + # so the xpu channel's pure python wheel would win on version before pip compares wheel + # tags, silently replacing the compiled aarch64 build. + torchao_url = f"{TORCHAO_URL_BASE}/cpu" + else: + torchao_url = determine_torch_url(TORCHAO_URL_BASE) # pip packages needed by exir. TORCH_PACKAGE = [ From fac460cb9ff1a65f71a8e9c5c51bc1649044ad4f Mon Sep 17 00:00:00 2001 From: Anthony Shoumikhin Date: Wed, 12 Aug 2026 05:01:36 -0700 Subject: [PATCH 2/2] Ship the OpenVINO delegate as its own library The OpenVINO delegate was compiled into the Python extension, so only Python could use it. A C++ application had no way to link it from the wheel, which is the same gap the other delegates had before they were split out. It now builds as its own library and ships beside the runtime, so an application can ask for it: find_package(executorch REQUIRED COMPONENTS backend_openvino) target_link_libraries(my_app PRIVATE executorch::backend_openvino) Only the adapter ships here. The OpenVINO runtime itself is loaded at run time and is not part of the wheel, so the component's documentation says to install it separately, and the wheel does not grow meaningfully: the adapter is a few kilobytes. The library resolves the runtime from the shared runtime library rather than from the static core, so the delegate registers into the one registry the process has instead of a second private one. Test Plan: built a wheel with OpenVINO enabled on Linux x86_64 and inspected it. The library ships at executorch/lib/libexecutorch_backend_openvino.so. Its delegate symbols moved out of the Python extension, 13 in the library and 0 in the extension, so the process has one copy and one registration. Its only ExecuTorch dependency is the shared runtime library, and it records no link dependency on OpenVINO, confirming the runtime is still loaded at run time. Added an ownership row to the shared library test so a future change that puts the delegate back into the extension fails there. ghstack-source-id: b6b210a1f7e9f4184e1852c4673b815607c3a45a ghstack-comment-id: 5263017453 Pull-Request: https://github.com/pytorch/executorch/pull/21770 --- .ci/scripts/wheel/test_shared_libraries.py | 320 +++++++++++++++++---- CMakeLists.txt | 3 + backends/openvino/CMakeLists.txt | 27 +- docs/source/using-executorch-cpp.md | 1 + setup.py | 12 + tools/cmake/executorch-wheel-config.cmake | 1 + 6 files changed, 305 insertions(+), 59 deletions(-) diff --git a/.ci/scripts/wheel/test_shared_libraries.py b/.ci/scripts/wheel/test_shared_libraries.py index 4fe89b47f28..a2f433aef2b 100644 --- a/.ci/scripts/wheel/test_shared_libraries.py +++ b/.ci/scripts/wheel/test_shared_libraries.py @@ -34,6 +34,7 @@ import subprocess import sys import tempfile +import zipfile from pathlib import Path # Registry entry points. A second definer of any of these means a second @@ -104,7 +105,18 @@ # Checked separately from the wrapper symbols above because the wrappers can each # have exactly one owner while the bundled code underneath them does not. That is # the same failure the split exists to prevent, reached by a different route. -_BUNDLED_THREADPOOL_SYMBOLS = ("pthreadpool_create", "cpuinfo_initialize") +# pthreadpool is compiled with hidden visibility on Apple, deliberately, so that the +# copy inside libtorch_cpu cannot take precedence over the bundled one. Its symbols are +# present but not exported there, so only cpuinfo can serve as the sentinel on that +# platform. Both are checked elsewhere. +if sys.platform == "darwin": + _BUNDLED_THREADPOOL_SYMBOLS = ("cpuinfo_initialize",) +else: + _BUNDLED_THREADPOOL_SYMBOLS = ("pthreadpool_create", "cpuinfo_initialize") +# The delegate's own entry points. A second definer means the delegate is compiled +# into the Python extension as well, which would register it twice in one process. +_OPENVINO_BACKEND_SYMBOLS = ("executorch::backends::openvino::OpenvinoBackend",) + _BUNDLED_XNNPACK_SYMBOLS = ("xnn_create_runtime_v4",) # A representative symbol from the profiler. A second definer means two event @@ -181,11 +193,6 @@ def _declared_requirements() -> set: return names -def _nm_defined_args(): - """The nm flags that list what a library defines.""" - return ["-DC"] - - def _installed_package_dir() -> Path: """The installed executorch package, never the source checkout. @@ -233,6 +240,185 @@ def _tool(name: str): return str(beside) if beside.is_file() else None +def _dynamic_section(library) -> str | None: + """What a library records about its dependencies and search paths. + + Returns None when no tool can read it, so a caller can tell "nothing recorded" + apart from "could not look", which are different verdicts. + + readelf prints the ELF dynamic section. otool -l prints the Mach-O load commands, + which carry the same facts under different names: LC_LOAD_DYLIB for a dependency + where ELF has NEEDED, and LC_RPATH where ELF has RUNPATH. + """ + if sys.platform == "darwin": + tool, args = _tool("otool"), ["-l"] + else: + tool, args = _tool("readelf"), ["-d"] + if tool is None: + return None + return subprocess.run( + [tool, *args, str(library)], + capture_output=True, + text=True, + check=False, + ).stdout + + +def _linked_libraries(library) -> str | None: + """The libraries this one resolves at load time, as text to search. + + ldd resolves an ELF library's dependencies transitively. otool -L lists a Mach-O + library's direct dependencies without resolving them, which is weaker, so a macOS + result says which names are recorded rather than whether each one was found. + """ + if sys.platform == "darwin": + tool, args = _tool("otool"), ["-L"] + else: + tool, args = _tool("ldd"), ["-r"] + if tool is None: + return None + result = subprocess.run( + [tool, *args, str(library)], + capture_output=True, + text=True, + check=False, + ) + return result.stdout + result.stderr + + +def _runtime_search_paths(library) -> list | None: + """The runtime search path entries recorded in a shipped library. + + Returns None when the file cannot be read, so a caller can tell "records nothing" + apart from "could not look", which are different verdicts. + + patchelf prints an ELF RPATH as one colon separated string. Mach-O keeps each entry + in its own LC_RPATH load command, so otool is parsed for those instead. patchelf + cannot read Mach-O at all, which is why it is not simply reused here. + """ + if sys.platform == "darwin": + tool = _tool("otool") + if tool is None: + return None + result = subprocess.run( + [tool, "-l", str(library)], capture_output=True, text=True, check=False + ) + if result.returncode != 0: + return None + entries = [] + lines = result.stdout.splitlines() + for index, line in enumerate(lines): + if "LC_RPATH" not in line: + continue + # The path sits a couple of lines below its command, followed by the + # offset otool appends, which is not part of the value. + for following in lines[index + 1 : index + 4]: + stripped = following.strip() + if stripped.startswith("path "): + entries.append(stripped.split(" (offset", 1)[0][len("path ") :]) + break + return entries + tool = _tool("patchelf") + if tool is None: + return None + result = subprocess.run( + [tool, "--print-rpath", str(library)], + capture_output=True, + text=True, + check=False, + ) + if result.returncode != 0: + return None + return [entry for entry in result.stdout.strip().split(":") if entry] + + +def _assert_mach_o_architecture_matches(wheel: Path) -> None: + """Fail if a macOS wheel's declared architecture is not what its binaries contain. + + auditwheel answers this on Linux by classifying against manylinux, which has no macOS + equivalent, so lipo is asked directly instead. It reports the architectures present in + a Mach-O file, and every shipped binary has to be one the tag promises. + + A universal binary lists several architectures, so containing the declared one is the + test rather than equalling it. + """ + lipo = _tool("lipo") + assert lipo is not None, ( + "lipo is required to check that a macOS wheel's contents match the architecture " + "it claims, and it was not found" + ) + claimed = wheel.name.split("-")[-1].removesuffix(".whl") + # macosx_14_0_arm64 and macosx_11_0_x86_64 both end in the architecture. + declared = claimed.split("_")[-1] + if declared == "64" and claimed.endswith("x86_64"): + declared = "x86_64" + + with tempfile.TemporaryDirectory() as unpacked: + with zipfile.ZipFile(wheel) as archive: + archive.extractall(unpacked) + root = Path(unpacked) + binaries = [ + path + for path in sorted(root.rglob("*")) + if path.is_file() + and not path.is_symlink() + and path.suffix in (".dylib", ".so") + ] + assert binaries, ( + f"the wheel {wheel.name} contains no binaries, so the architecture it claims " + "cannot be checked against anything" + ) + mismatched = [] + for binary in binaries: + result = subprocess.run( + [lipo, "-archs", str(binary)], + capture_output=True, + text=True, + check=False, + ) + if result.returncode != 0: + continue + present = result.stdout.split() + if declared not in present: + mismatched.append(f"{binary.relative_to(root)} is {' '.join(present)}") + assert not mismatched, ( + f"the wheel claims architecture {declared} but these binaries are built for " + f"something else, so it would install where it cannot run: {mismatched}" + ) + print(f"\u2713 the wheel is tagged for the architecture it contains ({declared})") + + +def _dynamic_lib_suffix() -> str: + """The loadable library suffix on this platform, including the dot.""" + return ".dylib" if sys.platform == "darwin" else ".so" + + +def _library_file_name(base_name: str) -> str: + """The file name a component's library has on this platform. + + The component table names libraries without a suffix so one table serves both + platforms. + """ + return f"{base_name}{_dynamic_lib_suffix()}" + + +def _nm_defined_args(): + """The nm flags that list what a library defines. + + GNU nm reads the dynamic symbol table with -D. Mach-O has no separate dynamic + symbol table, so that flag fails outright there and -gU, global and defined, is + the equivalent question. + """ + return ["-gU", "-C"] if sys.platform == "darwin" else ["-DC"] + + +def _nm_undefined_args(): + """The nm flags that list what a library needs from elsewhere.""" + if sys.platform == "darwin": + return ["-gu", "-C"] + return ["-DC", "--undefined-only"] + + def _shipped_shared_objects(package_dir: Path): """Every shared object the wheel installed. @@ -241,7 +427,7 @@ def _shipped_shared_objects(package_dir: Path): """ found = [ path - for path in sorted(package_dir.rglob("*.so*")) + for path in sorted(package_dir.rglob(f"*{_dynamic_lib_suffix()}*")) if path.is_file() and not path.is_symlink() ] assert ( @@ -265,7 +451,7 @@ def _shipped_runtime_libraries(package_dir: Path): return [] return [ path - for path in sorted(lib_dir.glob("lib*.so*")) + for path in sorted(lib_dir.glob(f"lib*{_dynamic_lib_suffix()}*")) if path.is_file() and not path.is_symlink() ] @@ -285,7 +471,10 @@ def _defines_symbol(library: Path, symbol: str) -> bool: process, which counts what actually registered rather than what is visible. """ result = subprocess.run( - [_tool("nm"), "-DC", str(library)], capture_output=True, text=True, check=False + [_tool("nm"), *_nm_defined_args(), str(library)], + capture_output=True, + text=True, + check=False, ) if result.returncode != 0: # A file that is not an object file at all is not this check's concern: something whose @@ -300,13 +489,17 @@ def _defines_symbol(library: Path, symbol: str) -> bool: f"checks cannot be trusted: {result.stderr.strip()[:200]}" ) return False + # Mach-O prefixes a C symbol with an underscore, so nm prints _cpuinfo_initialize + # where ELF prints cpuinfo_initialize. A C++ name demangles to the same text on both, + # so accepting the prefix is enough and no per-symbol spelling is needed. + accepted = (symbol, f"_{symbol}") if sys.platform == "darwin" else (symbol,) for line in result.stdout.splitlines(): if symbol not in line: continue match = _DEFINED.match(line) if ( match - and match.group("name").startswith(symbol) + and match.group("name").startswith(accepted) and match.group("kind") in _OWNING_KINDS ): return True @@ -333,17 +526,12 @@ def _is_export_only(library: Path) -> bool: """ if ".cpython-" in library.name or library.name.endswith(".pyd"): return False - if library.name.endswith("_aot_lib.so"): + if library.name.endswith(f"_aot_lib{_dynamic_lib_suffix()}"): return True - if _tool("readelf") is None: + dynamic = _dynamic_section(library) + if dynamic is None: return False - dynamic = subprocess.run( - [_tool("readelf"), "-d", str(library)], - capture_output=True, - text=True, - check=False, - ).stdout - return "libtorch.so" in dynamic + return _library_file_name("libtorch") in dynamic def _assert_single_definer( @@ -466,26 +654,36 @@ def _wheel_cuda_train() -> str: # one of them drift: it looked up its library with its own glob, which silently # stopped matching when the libraries were renamed while the others kept working. _OWNED_COMPONENTS = ( - ("backend registry", _REGISTRY_SYMBOLS, "libexecutorch.so", True), - ("operator registry", _KERNEL_REGISTRY_SYMBOLS, "libexecutorch.so", True), - ("thread pool", _THREADPOOL_SYMBOLS, "libexecutorch_threadpool.so", True), - ("profiler", _ETDUMP_SYMBOLS, "libexecutorch_etdump.so", True), + ("backend registry", _REGISTRY_SYMBOLS, _library_file_name("libexecutorch"), True), + ( + "operator registry", + _KERNEL_REGISTRY_SYMBOLS, + _library_file_name("libexecutorch"), + True, + ), + ( + "thread pool", + _THREADPOOL_SYMBOLS, + _library_file_name("libexecutorch_threadpool"), + True, + ), + ("profiler", _ETDUMP_SYMBOLS, _library_file_name("libexecutorch_etdump"), True), ( "XNNPACK delegate", _XNNPACK_SYMBOLS, - "libexecutorch_backend_xnnpack.so", + _library_file_name("libexecutorch_backend_xnnpack"), True, ), ( "set of CPU kernels", _KERNEL_SYMBOLS, - "libexecutorch_kernels_optimized.so", + _library_file_name("libexecutorch_kernels_optimized"), False, ), ( "set of quantized kernels", _QUANTIZED_KERNEL_SYMBOLS, - "libexecutorch_kernels_quantized.so", + _library_file_name("libexecutorch_kernels_quantized"), True, ), # The CUDA components. Required exactly when the wheel says it is a CUDA wheel, @@ -496,19 +694,19 @@ def _wheel_cuda_train() -> str: ( "CUDA delegate", _CUDA_BACKEND_SYMBOLS, - "libexecutorch_backend_cuda.so", + _library_file_name("libexecutorch_backend_cuda"), _REQUIRED_ON_A_CUDA_WHEEL, ), ( "CUDA stream helper", _CUDA_STREAM_SYMBOLS, - "libexecutorch_extension_cuda.so", + _library_file_name("libexecutorch_extension_cuda"), _REQUIRED_ON_A_CUDA_WHEEL, ), ( "AOTI shim layer", _AOTI_SHIM_SYMBOLS, - "libaoti_cuda_shims.so", + _library_file_name("libaoti_cuda_shims"), _REQUIRED_ON_A_CUDA_WHEEL, ), # The third-party code these libraries bundle, checked separately from the @@ -525,15 +723,21 @@ def _wheel_cuda_train() -> str: ( "bundled thread pool implementation", _BUNDLED_THREADPOOL_SYMBOLS, - "libexecutorch_threadpool.so", + _library_file_name("libexecutorch_threadpool"), True, ), ( "bundled XNNPACK runtime", _BUNDLED_XNNPACK_SYMBOLS, - "libexecutorch_backend_xnnpack.so", + _library_file_name("libexecutorch_backend_xnnpack"), True, ), + ( + "OpenVINO delegate", + _OPENVINO_BACKEND_SYMBOLS, + _library_file_name("libexecutorch_backend_openvino"), + False, + ), ) # The one component that legitimately exists twice. The quantized kernels are compiled into the runtime @@ -995,7 +1199,7 @@ def test_custom_op_compiles(work_dir: Path) -> None: "a custom operator does not compile or link against the shipped extension: " f"{(compiled.stderr or compiled.stdout).strip()[-800:]}" ) - produced = list(build_dir.rglob("libcustom_op_check.so")) or list( + produced = list(build_dir.rglob(_library_file_name("libcustom_op_check"))) or list( build_dir.rglob("custom_op_check.dll") ) assert produced, "the custom operator library was not produced" @@ -1148,6 +1352,9 @@ def test_wheel_platform_tag() -> None: print("- no wheel file to inspect, skipping the platform tag check") return + if sys.platform == "darwin": + _assert_mach_o_architecture_matches(wheels[-1]) + return result = subprocess.run( [sys.executable, "-m", "auditwheel", "show", str(wheels[-1])], capture_output=True, @@ -1204,7 +1411,9 @@ def test_no_absolute_runtime_paths() -> None: # guarantee patchelf on PATH, so this is the only place the guarantee can be # enforced. If both went quiet on the same missing tool, a wheel carrying the # build machine's directories would ship looking correct. - if _tool("patchelf") is None: + # Mach-O keeps its search path in load commands that otool reads, and otool comes + # with the developer tools, so only the ELF side needs an install step. + if sys.platform != "darwin" and _tool("patchelf") is None: print("- patchelf not present, installing it so this check can run") subprocess.run( [sys.executable, "-m", "pip", "install", "--quiet", "patchelf"], @@ -1212,10 +1421,10 @@ def test_no_absolute_runtime_paths() -> None: text=True, check=False, ) - patchelf = _tool("patchelf") - assert patchelf is not None, ( - "patchelf is required to check the shipped runtime paths and could not be " - "installed. Packaging uses it to strip build-tree directories, and without it " + reader = _tool("otool") if sys.platform == "darwin" else _tool("patchelf") + assert reader is not None, ( + "a runtime path reader is required to check the shipped runtime paths and could " + "not be found. Packaging strips build-tree directories, and without a reader " "here neither side would notice that they were left in place." ) @@ -1257,28 +1466,21 @@ def names_a_build_directory(entry: str) -> bool: offenders = {} inspected = 0 with_a_runtime_path = 0 - for library in sorted(package_dir.rglob("*.so*")): + for library in sorted(package_dir.rglob(f"*{_dynamic_lib_suffix()}*")): if not library.is_file() or library.is_symlink(): continue - result = subprocess.run( - [patchelf, "--print-rpath", str(library)], - capture_output=True, - text=True, - check=False, - ) - if result.returncode != 0: + entries = _runtime_search_paths(library) + if entries is None: continue inspected += 1 - # An absent RPATH and one containing a single empty entry both print as an - # empty string, so treat empty output as "no runtime path" rather than as an - # empty entry. A library with nothing to search is fine; the defect is - # searching somewhere unusable. - raw = result.stdout.strip() - if not raw: + # A library with nothing to search is fine; the defect is searching somewhere + # unusable. An absent search path and one holding a single empty entry are the + # same thing here, and the reader reports both as an empty list. + if not entries: continue with_a_runtime_path += 1 bad = [] - for entry in raw.split(":"): + for entry in entries: if not entry: bad.append("") elif ( @@ -1425,7 +1627,7 @@ def test_extension_contains_no_component() -> None: # UNDEFINED reference cannot be faked that way: it says the definition is not # here and has to come from a dependency. undefined = subprocess.run( - [_tool("nm"), "-DC", "--undefined-only", str(extension)], + [_tool("nm"), *_nm_undefined_args(), str(extension)], capture_output=True, text=True, check=False, @@ -1482,7 +1684,9 @@ def test_shipped_library_names_are_expected() -> None: # stale-artifact case this check is about, and a leftover from an earlier build is # a real file, so it is still caught. shipped = sorted( - p for p in lib_dir.glob("*.so*") if p.is_file() and not p.is_symlink() + p + for p in lib_dir.glob(f"*{_dynamic_lib_suffix()}*") + if p.is_file() and not p.is_symlink() ) assert shipped, f"the wheel ships a lib directory with no libraries: {lib_dir}" @@ -1506,12 +1710,14 @@ def test_shipped_library_names_are_expected() -> None: # dependency, so both have to ship and both are expected here. "libextension_cuda", "libexecutorch_backend_xnnpack", + "libexecutorch_backend_openvino", "libexecutorch_threadpool", "libexecutorch_etdump", ) - # A plain .so, because the wheel build does not version these. A trailing - # .so. would also be a name packaging did not produce here. - permitted = re.compile(rf"(?:{'|'.join(known)})\.so") + # Unversioned, because the wheel build does not version these. A trailing + # . would also be a name packaging did not produce here. These are + # libraries, so the suffix follows the platform and macOS spells them .dylib. + permitted = re.compile(rf"(?:{'|'.join(known)})\{_dynamic_lib_suffix()}") unknown = sorted(p.name for p in shipped if not permitted.fullmatch(p.name)) assert not unknown, ( f"the wheel ships {unknown} under lib/, which packaging does not produce. " diff --git a/CMakeLists.txt b/CMakeLists.txt index 5511fab231e..d51fcbc25fa 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -1236,6 +1236,9 @@ if(EXECUTORCH_BUILD_PYBIND) endif() if(EXECUTORCH_BUILD_OPENVINO) + # Under a shared build this resolves to the shipped library rather than a + # static archive, so the delegate is not copied into the extension and the + # process registers it once. list(APPEND _dep_libs openvino_backend) endif() diff --git a/backends/openvino/CMakeLists.txt b/backends/openvino/CMakeLists.txt index 5b7a1349bf5..ebb11c096ea 100644 --- a/backends/openvino/CMakeLists.txt +++ b/backends/openvino/CMakeLists.txt @@ -32,7 +32,12 @@ include(${EXECUTORCH_ROOT}/tools/cmake/Utils.cmake) # there is no build-time dependency on the OpenVINO SDK. # Define OpenVINO backend as a static library -add_library(openvino_backend STATIC) +if(EXECUTORCH_BUILD_SHARED) + set(_openvino_backend_library_type SHARED) +else() + set(_openvino_backend_library_type STATIC) +endif() +add_library(openvino_backend ${_openvino_backend_library_type}) # Enable exceptions and RTTI for OpenVINO backend target_compile_options(openvino_backend PRIVATE -frtti -fexceptions) @@ -47,7 +52,25 @@ target_sources( target_include_directories(openvino_backend PRIVATE ${COMMON_INCLUDE_DIRS}) # Link ExecuteTorch core and dynamic loading libraries -target_link_libraries(openvino_backend PRIVATE executorch_core ${CMAKE_DL_LIBS}) +target_link_libraries(openvino_backend PRIVATE ${CMAKE_DL_LIBS}) + +if(EXECUTORCH_BUILD_SHARED) + # Named after what the library provides rather than after the target that + # produces it, matching the other shipped delegates, so the file reads as + # libexecutorch_backend_openvino.so. + set_target_properties( + openvino_backend PROPERTIES OUTPUT_NAME executorch_backend_openvino + ) + executorch_target_soname_policy(openvino_backend) + # The shared runtime, never also the static core: linking both would compile + # the backend registry into this library as well and give the process two of + # them. + target_link_libraries(openvino_backend PUBLIC executorch_shared) + # Ships beside the runtime in the wheel's lib/ directory. + executorch_target_shipped_runtime_path(openvino_backend) +else() + target_link_libraries(openvino_backend PUBLIC executorch_core) +endif() executorch_target_link_options_shared_lib(openvino_backend) diff --git a/docs/source/using-executorch-cpp.md b/docs/source/using-executorch-cpp.md index b6187490bd8..b6433f3d8ec 100644 --- a/docs/source/using-executorch-cpp.md +++ b/docs/source/using-executorch-cpp.md @@ -110,6 +110,7 @@ reported while CMake configures, rather than failing later at link time. | `executorch::kernels_optimized` | CPU operator kernels. Needed for any operator a delegate does not claim. | | `executorch::kernels_quantized` | quantized operator kernels, for a quantized model. Link it only when you need it: see the note below. | | `executorch::backend_xnnpack` | the XNNPACK delegate. | +| `executorch::backend_openvino` | the OpenVINO delegate. Present only in a wheel built with OpenVINO, and it loads the OpenVINO runtime at run time, so install that separately. | | `executorch::backend_cuda` | the CUDA delegate, in a CUDA wheel. | | `executorch::extension_cuda` | the CUDA stream helper, in a CUDA wheel. Lets you pick the CUDA stream a model runs on. | | `executorch::threadpool` | the shared thread pool. | diff --git a/setup.py b/setup.py index bf2fec8c148..d7ff2a37ed3 100644 --- a/setup.py +++ b/setup.py @@ -1895,6 +1895,18 @@ def run(self): # noqa C901 "EXECUTORCH_BUILD_KERNELS_QUANTIZED", ], ), + # The OpenVINO delegate, so a C++ application can link it from the + # wheel. Only the adapter ships here: the OpenVINO runtime itself is + # loaded at run time and comes from the openvino extra. + BuiltFile( + src_dir="%CMAKE_CACHE_DIR%/backends/openvino/%BUILD_TYPE%/", + src_name="*executorch_backend_openvino" + _dynamic_lib_suffix(), + dst="executorch/lib/", + dependent_cmake_flags=[ + "EXECUTORCH_BUILD_SHARED", + "EXECUTORCH_BUILD_OPENVINO", + ], + ), # Install the XNNPACK delegate beside them, so a process has one # copy of it instead of one per component that uses it. BuiltFile( diff --git a/tools/cmake/executorch-wheel-config.cmake b/tools/cmake/executorch-wheel-config.cmake index d34bf4198d6..e11c58ffac0 100644 --- a/tools/cmake/executorch-wheel-config.cmake +++ b/tools/cmake/executorch-wheel-config.cmake @@ -594,6 +594,7 @@ if(TARGET executorch::runtime AND TARGET executorch::threadpool) endif() _executorch_define_component(backend_xnnpack executorch_backend_xnnpack) +_executorch_define_component(backend_openvino executorch_backend_openvino) # The CUDA delegate and its stream helper, present only in a wheel built from a # CUDA index. A CPU wheel defines neither, so a consumer asking for one is told # while configuring.