POC fast_llvm toolchain - #973
Conversation
| # Download the archive and verify its SHA-256 before using it. | ||
| _log("downloading LLVM %s for %s" % (ctx.attr.llvm_version, host_arch)) | ||
| ctx.download( | ||
| url = dist["url"], | ||
| output = archive, | ||
| sha256 = dist["sha256"], | ||
| ) | ||
| _log("download completed") | ||
|
|
||
| # Use the host's `tar` executable to unpack the xz archive. `ctx.which` | ||
| # returns None instead of guessing when `tar` is unavailable. | ||
| tar = ctx.which("tar") | ||
| if not tar: | ||
| fail("tar not found in PATH") | ||
|
|
||
| # LLVM's archive has many independently decompressible XZ blocks. Let xz | ||
| # decode them in parallel, then stream the resulting tar file directly to | ||
| # tar; materializing the 8+ GiB uncompressed archive would be wasteful. | ||
| # Keep the regular tar invocation for minimal Linux hosts without xz. | ||
| xz = ctx.which("xz") | ||
| if xz: | ||
| _log("extracting LLVM with xz -T0 and tar") | ||
| sh = ctx.which("sh") | ||
| if not sh: | ||
| fail("sh not found in PATH") | ||
| result = ctx.execute( | ||
| [ | ||
| sh, | ||
| "-c", | ||
| "set -e; \"$1\" -T0 -dc \"$2\" | \"$3\" -xf - --strip-components=1 -C \"$4\"", | ||
| "fast_llvm_repo", | ||
| xz, | ||
| archive, | ||
| tar, | ||
| ctx.path("."), | ||
| ], | ||
| # Allow the relatively large archive up to 30 minutes to unpack. | ||
| timeout = 1800, | ||
| # Do not suppress command output from Bazel's repository-rule log. | ||
| quiet = False, | ||
| ) | ||
| else: | ||
| _log("xz not found; extracting LLVM with tar fallback") | ||
| result = ctx.execute( | ||
| [ | ||
| tar, | ||
| "-xf", | ||
| archive, | ||
| "--strip-components=1", | ||
| # Extract into the external repository represented by `ctx`. | ||
| "-C", | ||
| ctx.path("."), | ||
| ], | ||
| timeout = 1800, | ||
| quiet = False, | ||
| ) |
There was a problem hiding this comment.
There was a problem hiding this comment.
Thanks for pointing this out — it seems I forgot to include any documentation for this POC.
Functionally, http_archive would be a valid solution. The LLVM toolchain internally creates a repository for the LLVM distribution and uses Bazel’s standard download-and-extract mechanism for it. That download_and_extract() step is exactly the slow part this POC is trying to improve.
The custom rule downloads the pinned archive with ctx.download() and extracts it using xz -T0 | tar, allowing XZ decompression to run in parallel. In my measurements, this reduced the LLVM setup time from roughly 220 seconds to 70 seconds.
So the custom rule is not required for correctness. It replaces only the internally generated LLVM distribution repository, while keeping the existing toolchain configuration and BUILD layout intact. The trade-off is additional maintenance and currently limited platform coverage, which is why this is still a POC.
If the additional extraction time is acceptable, sticking with the original toolchains_llvm setup is indeed the simpler option.
2 participants
extraction of llvm tar: