Skip to content

[Python] Execute Ray bucket routing and postpone planning in Rust - #10770

Merged
JingsongLi merged 2 commits into
apache:masterfrom
JingsongLi:codex/native-ray-bucket-routing
Oct 23, 2026
Merged

JingsongLi merged 2 commits into
apache:masterfrom
JingsongLi:codex/native-ray-bucket-routing

Conversation

@JingsongLi

@JingsongLi JingsongLi commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

Purpose

Use Native execution for Ray fixed/POSTPONE input bucket routing and POSTPONE input statistics/inference, which previously still ran Python row hashing and sizing before the Native writers received input.

Changes

  • Select a Native input router through dispatch before distributed tasks consume rows. Both Python and Native routers expose route(batch) -> Int32Array; Ray directly uses that interface, without method probing. Internal arguments consistently use router/bucket_router. Ray scheduling/grouping and public APIs remain unchanged.
  • Keep Native adapters limited to Arrow/partition-plan transport and serializable configuration. The Native planner no longer inherits Python input statistics or inference; Rust Core owns these operations.
  • Drop SDK objects during pickle and reconstruct from the shared schema/plan in workers. Runtime failures after backend selection propagate without replaying input through Python.
  • Preserve positional statistics columns when partition keys are named row_count, data_size or total_buckets.
  • Match current Java's NULL noncompact DECIMAL/TIMESTAMP/LTZ BinaryRow bytes, including decimal-overflow-to-NULL serialization, so Python hashing agrees with Native writers.
  • Add backend-isolation, worker pickle, reordered/subset input, chunked/sliced Arrow, nullable FixedSizeBinary partition and exact Java hash regressions; update Ray docs.

Validation

  • Focused Ray/router/dispatch regression suite: 49 passed, including the existing shuffle helper, chunked/empty inputs and both Python router modes.
  • Serializer/routing/dispatch focused suites: 145 passed.
  • Ray/postpone/table-write selection: 96 passed, 9 skipped.
  • Joint current-SDK run: 139 passed, including the 8 new actual Native routing/planning cases, existing write/authorization tests and NULL-layout regressions; 36 Native plans exercised.
  • Scoped flake8 and diff checks passed. No native extension was installed or replaced.

Dependency

Requires apache/paimon-rust#1207 and its BucketRouter and PostponeBucketPlanner SDK APIs. This PR is draft until that Rust change is merged. Native CI continues building Rust main; no SDK-version compatibility probes or fallback are introduced.

@leaves12138 leaves12138 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@JingsongLi
JingsongLi marked this pull request as ready for review October 23, 2026 09:44
@JingsongLi
JingsongLi merged commit c0a939a into apache:master Oct 23, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants