Skip to content

[Python] Execute captured Ray mutation and index plans through native core - #10757

Merged
JingsongLi merged 2 commits into
apache:masterfrom
JingsongLi:codex/native-ray-planned-execution
Oct 23, 2026
Merged

JingsongLi merged 2 commits into
apache:masterfrom
JingsongLi:codex/native-ray-planned-execution

Conversation

@JingsongLi

@JingsongLi JingsongLi commented Oct 22, 2026 •

Copy link
Copy Markdown
Contributor

Purpose

Complete the plan-execution and structured-conflict interfaces needed to run existing Ray row-ID mutations and vector index builds through Rust Core, while preserving Python public APIs.

Depends on apache/paimon-rust#1202. This PR remains draft until that dependency lands. Native CI continues to build Rust main; no SDK compatibility probes, Python replay, or CI branch override is added.

Changes

  • Capture the resolved snapshot and complete logical file groups once. Ray update, delete, self-MERGE update and row-ID read routing share neutral serializable metadata, instead of explicitly constructing Python file indexes in the Native path.
  • Route captured update/delete plans through dispatch to the new typed SDK Plan / with_plan interfaces. Preserve the worker's current environment and the captured read view. Preparation/execution authorization failures propagate without replay.
  • Keep full physical groups for mutations when scalar-index planning returns an IndexedSplit. Preserve QueryAuth wrappers, deletion vectors and canonical groups. Fill only missing canonical bucket paths while retaining external file locations, matching Java's split/file path model.
  • Execute Core-planned vindex shards with Ray. Each worker builds exactly one captured shard; publication stays with the existing caller. Collect messages in plan order. On worker or result-delivery failure, drain dispatched tasks and retain their outputs.
  • Share structured row-ID conflicts between execution backends, including pickle support and current DELETE layout fields. Native translates only the actual typed layout conflict; opaque/unknown publication errors remain errors.
  • Delegate the pre-rewrite logical history check to the selected backend without creating a Python publisher. Preserve Java's distinction between compactions and logical DML conflicts.
  • Keep existing public constructors and methods, explicit Python execution, and the existing commit ownership rules. No new dependencies.

Verification

  • Python 3.13 / Arrow 19: affected Ray, protocol, authorization and execution-boundary aggregate: 267 passed, 3 skipped.
  • Final plan/wire/conflict protocol suite after external-path coverage: 71 passed, 3 skipped; backend-selection suite: 14 passed.
  • Arrow 16 protocol checks passed; changed Python files pass flake8 and Python 3.6 grammar checks.
  • Paired Rust Core tests, actual PyO3 interface/error tests, and Rust 1.98 Core/binding Clippy passed.

The three skipped cases are real SDK integration tests for new plan interfaces. The local Python environment has no new SDK, and the extension was not rebuilt or installed. These cases are included for Native CI after the Rust dependency merges. Existing real Ray regressions and backend-independent protocol tests ran locally; mocked transport tests are not counted as real SDK execution.

Follow-up review fixes

  • Restore preexecution eligibility for query-authorized index builds: current Core index builders cannot enforce query grants, so these configurations retain the existing Python planner/executor. Actual permission/transport errors are never replayed across backends. Real REST regressions cover unrestricted, filtered, masked and denied policies with Native enabled and disabled.
  • Keep the self-MERGE shared context small and pass only each task's complete canonical group plan. Ray deserializes shared Python objects for every task; shipping the whole table plan introduced quadratic metadata work. Preserve snapshots, DVs, grant metadata, callable groups and failure retention.
  • Final affected Ray/protocol/authorization aggregate: 218 passed. Real REST policy tests plus an empty-plan Native control: 5 passed. Changed Python files pass flake8 and Python 3.6 grammar checks; paired Rust Core/binding strict Clippy passed.

@JingsongLi
JingsongLi marked this pull request as ready for review October 23, 2026 00:47

@leaves12138 leaves12138 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@JingsongLi
JingsongLi merged commit 18613c8 into apache:master Oct 23, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants