ci: enable linux_amd64_musl extension build - #124
joseph-isaacs wants to merge 3 commits into
Conversation
Remove linux_amd64_musl from exclude_archs now that vortex builds for x86_64-unknown-linux-musl in its own CI. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
linux_amd64_musl is marked opt_in in extension-ci-tools' distribution_matrix.json, so dropping it from exclude_archs alone does not build it — the matrix generator skips opt_in archs unless they are also listed in opt_in_archs. Add the opt_in_archs input so the musl extension build is actually generated and runs in CI. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
The linux_amd64_musl build fails in the build scripts of custom-labels and vortex-duckdb: both run bindgen, which dlopen()s libclang, and the rustup musl toolchain links host binaries statically by default, so dlopen fails with "Dynamic loading not supported". The Rust staticlib also needs PIC codegen because it is folded into the extension's shared object. Neither RUSTFLAGS nor a [target.x86_64-unknown-linux-musl] cargo config table can fix the build scripts: corrosion always passes `--target`, and under `--target` cargo forwards rustflags from any source only to target artifacts, never to host build scripts. A RUSTC_WRAPPER sees every rustc invocation, so add scripts/musl-rustc-wrapper.sh, which appends -Ctarget-feature=-crt-static and -Crelocation-model=pic to each compilation, and have CMakeLists.txt set it as RUSTC_WRAPPER for the cargo build when corrosion reports a *-musl Rust target. Other platforms are unaffected. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk> Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Lw7eggRzXD89U83GW2AmFR
| # rustc wrapper for musl (Alpine) builds of the extension. CMakeLists.txt sets | ||
| # it as RUSTC_WRAPPER for the cargo build when the Rust target is *-musl. | ||
| # | ||
| # The rustup musl toolchain links binaries statically by default (crt-static), |
There was a problem hiding this comment.
If it already builds crt-static by default, why do we need to add crt-static in target-features?
There was a problem hiding this comment.
The flag is -C target-feature=-crt-static: the leading - on -crt-static disables the feature (+crt-static would enable it). The rustup musl target defaults to +crt-static, i.e. fully static binaries, and that is exactly what breaks the build scripts: a static musl binary cannot dlopen() libclang. Passing -crt-static switches those binaries to the dynamic CRT so bindgen can load libclang. It is the same flag vortex's own musl.yml workflow sets via RUSTFLAGS.
Generated by Claude Code
| # | ||
| # The rustup musl toolchain links binaries statically by default (crt-static), | ||
| # which breaks the build scripts of custom-labels and vortex-duckdb: they run | ||
| # bindgen, which dlopen()s libclang, and a static musl binary cannot dlopen |
There was a problem hiding this comment.
Another solution to this issue: we can git checkout generated cpp.rs to vortex-duckdb and run bindgen only on demand. We can even remove it as a crate dependency and build as a binary. Given changes to these functions are rare, it's reasonable to make the users (i.e. me) run another program. That's how we already regenerate vortex.h for cbindgen
There was a problem hiding this comment.
Checking in the generated bindings for vortex-duckdb would remove its own bindgen run, but it would not be enough on its own: custom-labels (a crates.io dependency of vortex-io and vortex-duckdb) also runs bindgen unconditionally in its build script, and that is the first place the musl build fails. So the wrapper would still be needed unless custom-labels becomes optional in vortex or its bindings get vendored upstream. The staticlib also still needs -crt-static/PIC codegen for the extension shared object regardless of how bindings are produced.
On-demand bindgen in vortex-duckdb (mirroring how vortex.h is regenerated for cbindgen) sounds like a good follow-up in the vortex repo; I'd keep this PR as the CI enabler.
Generated by Claude Code
2 participants
What
Enables the
linux_amd64_muslbuild in CI so thevortexextension is built and tested against musl libc (Alpine).Supersedes #87, which has a merge conflict with
mainand whose last approach could not work (see below). The two workflow commits from that PR are carried over unchanged.Changes
.github/workflows/MainDistributionPipeline.yml: droplinux_amd64_muslfromexclude_archsand list it inopt_in_archs(the arch isopt_inin extension-ci-tools'distribution_matrix.json, so removing the exclusion alone does not build it).scripts/musl-rustc-wrapper.sh+CMakeLists.txt: when corrosion reports a*-muslRust target, set the script asRUSTC_WRAPPERfor the cargo build. It appends-C target-feature=-crt-static -C relocation-model=picto every rustc compilation.Why a rustc wrapper
The musl job fails in the build scripts of
custom-labelsandvortex-duckdb: both run bindgen, whichdlopen()s libclang, and the rustup musl toolchain links host binaries statically, so dlopen fails withDynamic loading not supported. The staticlib also needs PIC codegen since it is folded into the extension shared object.RUSTFLAGS(PR 87's third commit) and a[target.x86_64-unknown-linux-musl].rustflagscargo config table (its last commit) cannot reach the build scripts: corrosion always passes--target, and under--targetcargo forwards rustflags from any source only to target artifacts, never to host build scripts. Verified locally with cargo 1.94 on a--target <host>build: a[target.<host>].rustflags --cfg foois invisible to the build script, while aRUSTC_WRAPPERappending the same flag reaches it. A wrapper sees every rustc invocation, so it is the one stable mechanism that covers build scripts.Verification
linux_amd64_musljob builds and tests the extension successfully (https://github.com/vortex-data/duckdb-vortex/actions/runs/34229876917/job/102073104381), and the existing linux_amd64, linux_arm64, osx_amd64 and osx_arm64 jobs stay green.build_script_buildand the crate under--target,rustc -vVpasses through untouched); a standalone CMake configure against corrosion v0.5.2 confirmsRust_CARGO_TARGET_ENV/Rust_CARGO_TARGET_CACHEDare available in the extension's scope.🤖 Generated with Claude Code
https://claude.ai/code/session_01Lw7eggRzXD89U83GW2AmFR