[Python] show_info() reports one codec twice and omits lz4_raw #52024
Copy link
Copy link
Open
Description
Describe the bug, including details regarding any error messages, version, and platform.
pa.show_info() prints its Compression Codecs section from a hardcoded list in python/pyarrow/__init__.py:
codecs = ["brotli", "bz2", "gzip", "lz4_frame", "lz4", "snappy", "zstd"]That list has drifted from _ensure_compression() in io.pxi, which is what actually decides which codec names are accepted. Two things follow from it:
lz4andlz4_frameboth map toCCompressionType_LZ4_FRAME, so the same codec is reported twice under two names.lz4_rawmaps toCCompressionType_LZ4and is a separate codec, but it is not listed at all, even thoughpa.Codec.is_available('lz4_raw')returns True.
So seven lines are printed for six codecs, and the codec that is missing is a real one.
Observed, on pyarrow 25.0.0:
>>> import pyarrow as pa
>>> pa.show_info()
...
Compression Codecs:
brotli : Enabled
bz2 : Enabled
gzip : Enabled
lz4_frame : Enabled
lz4 : Enabled
snappy : Enabled
zstd : Enabled
Expected: each supported codec listed once, including lz4_raw.
Walking every name _ensure_compression() accepts and the enum value each maps to:
bz2 -> BZ2
gzip -> GZIP
brotli -> BROTLI
lz4 -> LZ4_FRAME
lz4_frame -> LZ4_FRAME # same codec as lz4
lz4_raw -> LZ4 # distinct, not reported by show_info()
snappy -> SNAPPY
zstd -> ZSTD
Platform: Windows 11, CPython 3.12, pyarrow 25.0.0. The list is unchanged on current main, so it affects main too.
A fix is proposed in #51753.
Component(s)
Python
Reactions are currently unavailable
Activity
apacheGH-52024: [Python] Fix the codec list in show_info()