Skip to content

[python] Use typed MERGE clauses and named execution factories - #1204

Merged
JingsongLi merged 1 commit into
apache:mainfrom
JingsongLi:codex/typed-merge-execution
Oct 23, 2026
Merged

JingsongLi merged 1 commit into
apache:mainfrom
JingsongLi:codex/typed-merge-execution

Conversation

@JingsongLi

@JingsongLi JingsongLi commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • Replace MERGE dictionary/tuple descriptions with immutable SDK MergeAssignment, WhenMatched, and WhenNotMatched objects.
  • Replace the public MergeBatchExecution(..., clauses, mode) constructor with for_updates, for_deletes, and for_inserts; retain execute(Table | RecordBatch) -> Table.
  • Reuse the same typed clause conversion for batch/stream merge_into and joined-batch execution, delegating selection and casts to existing Core logic.
  • Preserve ordered first-match behavior, NULL fallthrough, callback grouping, lazy scalar conversion, original callback exceptions and exact output schemas. Delete factories retain every matched condition and skip UPDATE assignment conversion/evaluation.
  • Make explicit literals authoritative even when their Python value is callable; keep Arrow array/chunked-array values supported.
  • Document the factories and migrate SDK integration tests. Old dictionary descriptions and the old mode constructor are rejected. No dependencies are added.

Verification

  • 13 embedded PyO3 MERGE tests passed, including strict clause families, mapping snapshots, lazy literals, delete-only evaluation, exception identity, concurrent callers and batch/stream source handling.
  • 156 real SDK/PyPaimon Native integration cases passed across SDK update/Blob tests and PyPaimon MERGE/adapter/source-metadata tests. These ran against the real registered native module inside a Cargo test executable; no Native extension was installed or rebuilt into a Python environment. The temporary integration entry point is not included in this PR.
  • Rust 1.98.0 all-targets Clippy with warnings denied, rustfmt and diff checks passed.
  • Companion Python protocol/public-API checks: 105 passed, 16 skipped; Arrow 16: 35 passed, 1 skipped. Skipped Native cases were separately covered by the real-module integration run.

The companion Python PR apache/paimon#10764 remains draft until this SDK API is merged. Its Native CI continues to build Rust main; no old-SDK compatibility protocol is retained.

@leaves12138 leaves12138 left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@JingsongLi
JingsongLi merged commit ca0f40d into apache:main Oct 23, 2026
16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants