Skip to content

[core] Add schema-based bucket routing and postpone input planning - #1207

Merged
JingsongLi merged 2 commits into
apache:mainfrom
JingsongLi:codex/native-ray-bucket-routing
Oct 23, 2026
Merged

JingsongLi merged 2 commits into
apache:mainfrom
JingsongLi:codex/native-ray-bucket-routing

Conversation

@JingsongLi

@JingsongLi JingsongLi commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

Purpose

Move distributed input bucket computation and POSTPONE input statistics/inference into Rust Core so PyPaimon/Ray can use the same implementation as local writers without creating file writers for preclustering.

Changes

  • Add schema-based BucketRouter::new(&TableSchema), with_bucket_plan and input-order Arrow route APIs for fixed and POSTPONE buckets. Reuse existing bucket functions and write-array normalization; payload columns are not needed for routing.
  • Expose pure PostponeBucketPlanner with input_statistics, plan and current_plan. Worker statistics can be concatenated and duplicate partitions are aggregated before inference; known layouts are preserved and invalid statistics leave the plan unchanged.
  • Keep buffered automatic-writer ownership separate and reuse the same planner/router. Resolve write metadata through the existing write builder, preserving REST write authorization.
  • Make PostponeBucketPlan Arrow codecs depend on TableSchema instead of a live Table. Add typed Python binding classes and SDK docs; no new dependencies.
  • Match current Java's NULL BinaryRow layout for noncompact DECIMAL/TIMESTAMP/LTZ, including reserved variable slots and offsets. Cover exact Java bytes/hashes, nullable-key writer restarts, and exact NULL partition filtering. Existing format partition reconstruction uses the same typed-null helper to preserve its filtering behavior.

Java semantics

Reference fixtures were generated from current Apache Paimon Java InternalRowSerializer, RowHelper and BinaryRowWriter on paimon master 7befc15f18d81bc406679e14b48aa92cfe863335. Tests include precision boundaries, a following variable-length string, fixed/mod/Hive buckets, nullable/sliced FixedSizeBinary partitions, option precedence, explicit-plan replacement, overflow/cap behavior, and normalized PK event inputs.

Validation

  • Core focused suites: POSTPONE, BinaryRow, BucketRouter, partition filtering and partition reconstruction.
  • nullable_primary_key_test integration suite.
  • Registered Python SDK write_routing API test.
  • Joint PyPaimon execution against the registered current SDK: 116 passed, including all 8 new real Native routing/planning cases and existing writer/authorization regressions; 36 Native plans exercised. Used a temporary embedded test executable, without installing or replacing a Python extension.
  • Rust 1.98.0 Clippy and formatting checks.

Paired PyPaimon PR apache/paimon#10770 uses these APIs for Ray preclustering and POSTPONE statistics/planning. PyPaimon public APIs remain unchanged.

CI follow-up

  • Migrate the C binding bucket-plan import to table.schema() after the Core Arrow codec signature change. This fixes the workspace MSRV and Windows build type error.
  • Re-encode logical partition keys with Java's typed NULL layout. Source Arrow partitions and planned split partitions now share the same identity, so NULL DECIMAL/TIMESTAMP/LTZ upserts update the existing row instead of appending a duplicate.
  • Replace the old untyped-NULL bucket expectation with independently generated current-Java hashes/buckets, and cover two consecutive upserts while preserving the original row ID.
  • Validation: Core library suite (fulltext,internal-api) 4629 passed, 7 ignored; Rust 1.94.0 workspace/all-targets/all-features check; Rust 1.98.0 workspace Clippy with fulltext,vortex and warnings denied; formatting and diff checks.

@leaves12138 leaves12138 left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@JingsongLi
JingsongLi merged commit 9b094f5 into apache:main Oct 23, 2026
15 of 16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants