Repository navigation
Conversation
github-actions
Bot
added
awaiting review
prtkgaur
force-pushed
the
gh51268-dbp-vector-prefix-sum
branch
6 times, most recently
from
September 18, 2026 21:09
83a5289 to
5f91b9b
Compare
prtkgaur
force-pushed
the
gh51268-dbp-vector-prefix-sum
branch
2 times, most recently
from
October 4, 2026 02:36
4e3aaeb to
3e46ef0
Compare
prtkgaur
changed the title
|
Thanks for opening a pull request! This pull request has been automatically converted to a draft because its title doesn't match Arrow's required format. If this is not a minor PR, could you open an issue for this pull request on GitHub? https://github.com/apache/arrow/issues/new/choose Opening GitHub issues ahead of time contributes to the Openness of the Apache Arrow project. Then could you also rename the pull request title in the following format? or After updating the title, you can mark the pull request as ready for review. See also: |
The reconstruction loop stores to an output buffer of the same type as the frame and the running value, so the compiler has to assume a store can change either member and reloads both for every value. The loop now copies them to locals and writes the running value back once at the end. The arithmetic stays unsigned, so overflow still wraps.
The decoder unpacked one miniblock per call, which at the writer's default block sizes is a call for every 32 int32 or 64 int64 values, and at narrow widths the per-call setup is a large share of the work. Each miniblock holds a multiple of 32 values, so adjacent miniblocks of one width form a single bitstream. The decoder now unpacks such a run in one call, bounded by the current block and by the number of values requested. Zero-width miniblocks keep their own path. A new test covers where runs start and stop, and a read batch of 100 values ends partway through one.
Rebuilding values from deltas is a chain of dependent additions. An inclusive scan does a register's worth in log2(lanes) shifted additions, and adding the frame before the scan makes its running multiple come out of the scan instead of costing a multiply per lane. The running total moves between batches in a vector register, because reading it out to a general-purpose register would put a round trip on the dependency chain. Registers of fewer than four lanes keep the scalar loop, where the scan is slower; at the 128-bit baseline that includes all of int64. A new benchmark input holds non-decreasing values, and a new test decodes every delta width through the vector loop, the scalar remainder and the hand-off between them.
prtkgaur
force-pushed
the
gh51268-dbp-vector-prefix-sum
branch
from
October 8, 2026 03:33
3e46ef0 to
642c05e
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rationale for this change
Prefix sum: a chain of dependent additions. A log-step inclusive scan computes it a register at a
time, in log2(lanes) shifted additions. Adding the frame of reference before the scan turns
its running multiple into a term the scan produces, rather than a multiply per lane.
What changes are included in this PR?
I replaced the sequential addition loop with a vectorized prefix sum (scan) to speed up decoding.
Are these changes tested?
Yes. I added a new typed test that explicitly forces the decoder through the vector loop, the scalar remainder loop, and the hand-off between them to ensure totals and wrapping behave correctly.
Benchmark
Same setup as #51250.
NarrowSortedis added by this PR and holds non-decreasing values,the shape DELTA_BINARY_PACKED is usually chosen for.
Decode_Int32_NarrowSortedDecode_Int32_NarrowDecode_Int32_WideDecode_Int64_NarrowSortedDecode_Int64_NarrowDecode_Int64_WideDecode_Int32_FixedDecode_Int64_FixedAre there any user-facing changes?
No. No API change, no format change, and decoded values are identical.