Repository navigation
[core] Add schema-based bucket routing and postpone input planning - #1207
Merged
JingsongLi merged 2 commits intoOct 23, 2026
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Move distributed input bucket computation and POSTPONE input statistics/inference into Rust Core so PyPaimon/Ray can use the same implementation as local writers without creating file writers for preclustering.
Changes
BucketRouter::new(&TableSchema),with_bucket_planand input-order ArrowrouteAPIs for fixed and POSTPONE buckets. Reuse existing bucket functions and write-array normalization; payload columns are not needed for routing.PostponeBucketPlannerwithinput_statistics,planandcurrent_plan. Worker statistics can be concatenated and duplicate partitions are aggregated before inference; known layouts are preserved and invalid statistics leave the plan unchanged.PostponeBucketPlanArrow codecs depend onTableSchemainstead of a liveTable. Add typed Python binding classes and SDK docs; no new dependencies.Java semantics
Reference fixtures were generated from current Apache Paimon Java
InternalRowSerializer,RowHelperandBinaryRowWriteron paimon master7befc15f18d81bc406679e14b48aa92cfe863335. Tests include precision boundaries, a following variable-length string, fixed/mod/Hive buckets, nullable/sliced FixedSizeBinary partitions, option precedence, explicit-plan replacement, overflow/cap behavior, and normalized PK event inputs.Validation
nullable_primary_key_testintegration suite.write_routingAPI test.Paired PyPaimon PR apache/paimon#10770 uses these APIs for Ray preclustering and POSTPONE statistics/planning. PyPaimon public APIs remain unchanged.
CI follow-up
table.schema()after the Core Arrow codec signature change. This fixes the workspace MSRV and Windows build type error.fulltext,internal-api) 4629 passed, 7 ignored; Rust 1.94.0 workspace/all-targets/all-features check; Rust 1.98.0 workspace Clippy withfulltext,vortexand warnings denied; formatting and diff checks.