Initial support for lhc4codec - #23391
Open
apeters1971 wants to merge 2 commits into
Open
apeters1971 wants to merge 2 commits into
apeters1971 wants to merge 2 commits into
Conversation
Integrate optional lhc4codec into ROOT compression and RNTuple writing, including Auto and the Beam/Mosaic/Oracle/Crystal backends, builtin fetch with bundled bzip3, and tree2ntuple/rntuple2rntuple utilities to compare native and LHC4 codecs with verify, no-out, and compression diagnostics. Builtin pin is 922d272 (GitLab master). Auto races the ROOT default set (zstd|beam|crystal) via R__SetLHC4AutoCodecs / --auto-codecs; use all or a comma list to change it.
---
Build with a system package (searches /usr and Homebrew /opt/homebrew, /usr/local):
cmake -S /path/to/root -B /path/to/root-build -Dlhc4codec=ON
cmake --build /path/to/root-build -j
# AlmaLinux 9 / 10
sudo curl -fsSL -o /etc/yum.repos.d/lhc4codec.repo \
https://gitlab.cern.ch/apeters/lhc4codec-bin/-/raw/master/lhc4codec-el9.repo
sudo dnf install lhc4codec lhc4codec-devel
# macOS Apple silicon
brew tap apeters/lhc4codec https://gitlab.cern.ch/apeters/lhc4codec-bin.git
brew install apeters/lhc4codec/lhc4codec
If the library is missing, CMake prints these install steps and stops. Use -DLHC4CODEC_ROOT=/prefix for a custom layout.
Or let ROOT clone https://gitlab.cern.ch/apeters/lhc4codec.git (do not set LHC4CODEC_SOURCE_DIR):
cmake -S /path/to/root -B /path/to/root-build \
-Dlhc4codec=ON \
-Dbuiltin_lhc4codec=ON
cmake --build /path/to/root-build -j
CERN GitLab credentials must work non-interactively. The commit is pinned in builtins/lhc4codec/CMakeLists.txt. If a previous configure used LHC4CODEC_SOURCE_DIR, clear it (-ULHC4CODEC_SOURCE_DIR) and remove <build>/builtins/lhc4codec-prefix before reconfiguring.
---
After each page is compressed, --verify decompresses it and compares the result with the raw input buffer (catches codec round-trip corruption). Example:
tree2ntuple --no-out --verify --jobs 1 input.root
tree2ntuple --no-out --verify --auto-codecs all input.root
--no-out writes each variant to a temp file and deletes it after measuring size. --jobs 1 runs all default formats sequentially (native_zstd, native_lzma, and the lhc4_* variants). --auto-codecs selects the Auto race (root by default, or all / zstd,beam,...). LHC4 zip/unzip failures dump page buffers under /tmp/root-lhc4-failure-<pid>/.
GetConstSubfields() was allocating a new vector on every TTree entry, which made tree2ntuple and other imports much slower than they should be. Caching the leaf-count layout once removes that bottleneck and makes the conversion run much quicker.
apeters1971
requested review from
bellenot,
dpiparo,
jblomer,
pcanal and
silverweed
as code owners
September 16, 2026 17:38
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This Pull request:
Changes or fixes:
Checklist:
This PR fixes #