Repository navigation
[vortex] Upgrade vortex to 0.89.0 and rework integration for better performance - #10723
Open
robert3005 wants to merge 3 commits into
Open
robert3005 wants to merge 3 commits into
robert3005 wants to merge 3 commits into
Conversation
Signed-off-by: Robert Kruszewski <github@robertk.io>
robert3005
force-pushed
the
rk/vortexbump
branch
from
October 21, 2026 12:24
e95a951 to
36d14f0
Compare
Add Vortex cases to TableReadBenchmark and TableWriterBenchmark next to Parquet, ORC and Avro. Speed up reading Vortex string and binary columns: - Copy inline Arrow view values with two fixed-size stores, copy the validity bitmap in bulk and write offsets directly. - Read string and binary values as slices of one heap copy of each column's data buffer instead of allocating per value. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Robert Kruszewski <github@robertk.io>
Reads: - Keep each Vortex batch's native Arrow memory alive until the batch is released, and read top-level string and binary view columns lazily: the first access copies the views straight into one heap array. These columns are left out of the batch's Arrow root. - Read fixed-width numbers straight from the data buffer and check null bits only when the vector holds a null. - Prefetch the next batch on a shared pool, starting when the reader opens, so the files of a merge decode in parallel. Writes: - Size the next batch's Arrow vectors from the previous batch instead of regrowing them from empty. - Read the field count once per row in ArrowFormatWriter, and skip setSafe's capacity checks for in-order string and binary values that fit the current buffers. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Robert Kruszewski <github@robertk.io>
robert3005
changed the title
Author
|
@JingsongLi I have made a pr to bump vortex here and improve its performance quite a bit. I'm on the TSC of Vortex and worked on the Iceberg integration. Would love to get this merged so Paimon can benefit and also happy to have a chat anytime if you think there's features paimon could use in vortex |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Upgrade to latest version of vortex and rework vortex scans for better
performance. We follow the model from apache iceberg and with these changes
paimon vortex outperforms iceberg vortex. Local synthetic benchmark show great
read, scan and write performance. Happy to include any additional validation
required
TableReadBenchmark and TableWriterBenchmark below
Tests
Added additional test coverage