Skip to content

[core] Share MERGE action evaluation for distributed execution - #1203

Merged
JingsongLi merged 1 commit into
apache:mainfrom
JingsongLi:codex/native-merge-execution
Oct 23, 2026
Merged

JingsongLi merged 1 commit into
apache:mainfrom
JingsongLi:codex/native-merge-execution

Conversation

@JingsongLi

@JingsongLi JingsongLi commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • Extract engine-independent evaluation of ordered MERGE clauses into Core and reuse it from the existing complete MERGE operation.
  • Add MergeBatchExecution for Arrow batches that have already been joined by an execution engine. Rust compiles SQL conditions once and owns action selection, assignment casting, target passthrough and typed insert nulls.
  • Keep update, delete and insert output selection explicit, so extracting delete rows never runs unrelated UPDATE callbacks.
  • Preserve complete selected callback groups across Arrow chunks, exact output schemas, Python exception identity and serialized access to cached callback errors.

The action order and target passthrough follow the current Spark MergeIntoPaimonDataEvolutionTable implementation. The new evaluator has no Ray or table I/O dependency; callers still own joins, planning and authorization. No dependencies are added.

Verification

  • 24 Core MERGE tests passed, including all 18 existing complete MERGE tests and 6 new action tests. The original 5 new behavioral tests were first observed failing against an empty evaluator.
  • 7 real PyO3 MERGE tests passed with Arrow 19 and real Core execution (4 new action tests and 3 existing full-MERGE binding regressions).
  • An additional local PyO3 harness executed all 4 companion Python Native adapter integration cases against the real binding and Core, without installing an extension. The temporary cross-checkout harness is not part of this PR.
  • Core and binding Clippy passed with Rust 1.98.0 and warnings denied; formatting and diff checks passed.

Companion Python PR apache/paimon#10761 remains draft until this API is on Rust main; Python Native CI continues to build main.

@leaves12138 leaves12138 left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@JingsongLi
JingsongLi merged commit 3def470 into apache:main Oct 23, 2026
16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants