Repository navigation
Conversation
ptxas 13.0.88 segfaults on aarch64-linux when compiling large sm90 kernels, e.g. the flash-attn3 hdimall bf16 instantiations (`'ptxas' died due to signal 11`). x86_64 is not affected. - Allow `ptxasVersion` in versions.nix to be a per-system attrset, so the override can be limited to aarch64-linux. - Set `ptxasVersion.aarch64-linux = "13.4"` for the CUDA 13.0 entries. - Fix the ptxas override for CUDA 13: nvvm is no longer part of cuda_nvcc, so only copy nvvm/bin when it exists.
danieldk
reviewed
Oct 29, 2026
| @@ -30,6 +30,8 @@ | |||
| { | |||
| torchVersion = "2.10"; | |||
| cudaVersion = "13.0"; | |||
Member
There was a problem hiding this comment.
This should not be done. This switches all 13.0 builds to ptxas 13.4. It is a hack and should only be done on kernels that really need it. See
Collaborator
Author
|
closing in favor of a more targeted fix huggingface/kernels-community#1285 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
ptxas 13.0.88 segfaults on aarch64-linux when compiling large sm90 kernels. flash-attn3's cu130 aarch64 build has failed on every run since 08-27 with
'ptxas' died due to signal 11 (Invalid memory reference)onflash_fwd_hdimall_bf16_sm90.cuandflash_fwd_hdimall_bf16_softcap_sm90.cu, while x86_64 builds the same variant fine, e.g.this pr swaps in ptxas 13.4 for CUDA 13.0 builds on aarch64-linux only, using the existing
ptxasVersionoverride (#671)ptxasVersioninversions.nixto be a per-system attrset, e.g.ptxasVersion.aarch64-linux = "13.4". a plain string still applies to all systemsnvvmis no longer incuda_nvcc(it moved tolibnvvm), so thecp .../nvvm/bin/*failed and broke the patched nvcc. now it only copiesnvvm/binwhen present, so for CUDA 13 justptxasandnvlinkare swapped and cicc stays at 13.0ptxasVersionforms inchecks.nixtesting
compiled the two crashing flash-attn3 files with the CI nvcc flags on native aarch64-linux, using the builder's own package sets, where ptxas is the only difference between the two runs:
flash_fwd_hdimall_bf16_sm90.cuptxas died due to signal 11flash_fwd_hdimall_bf16_softcap_sm90.cuptxas died due to signal 11nix eval .#checks.x86_64-linux.default.drvPathpasses (includes the new tests).not yet tested: a full flash-attn3 build and GPU tests with this builder rev, I'll run a kernels-community stage build against this branch.
note: this rebuilds the aarch64 cu130 variant of every kernel with ptxas 13.4. if we'd rather limit the blast radius, the override could go only on the entries that kernels hitting this actually build