Skip to content

[core] Add captured mutation and index build plans with structured row-id conflicts - #1202

Merged
JingsongLi merged 2 commits into
apache:mainfrom
JingsongLi:codex/native-ray-planned-execution
Oct 23, 2026
Merged

JingsongLi merged 2 commits into
apache:mainfrom
JingsongLi:codex/native-ray-planned-execution

Conversation

@JingsongLi

@JingsongLi JingsongLi commented Oct 22, 2026 •

Copy link
Copy Markdown
Contributor

Purpose

Allow distributed Python execution to reuse Core plans and conflict semantics without exposing internal file indexes or re-planning in every worker. Rust Core remains independent of Ray.

Changes

  • Add typed Plan::from_splits and RowIdPlan routing metadata for complete data-evolution file groups. TableUpdate::with_plan reuses a captured snapshot for row-ID updates and deletes. Other operations explicitly reject that configuration before consuming inputs.
  • Reauthorize mutation workers, validate snapshot/table paths and complete groups, and reject deleted row IDs before producing any update file. Preserve captured deletion vectors and conflict baselines. Deletes still load the captured index manifest to preserve shared bucket DV metadata, and must be scheduled with one worker per partition/bucket.
  • Expose snapshot-bound GlobalIndexBuildPlan / GlobalIndexBuildShard through GlobalIndexBuildBuilder::plan and build_shard, covering the vindex types already supported by Python Ray builds. Each worker executes one captured shard with its own table/I/O environment; source metadata still protects final publication.
  • Add structured RowIdExistenceConflict details in Core and Python bindings. Align stale DELETE metadata checks with current Java DataEvolutionConflictDetection.
  • Add an independent TableCommit::check_row_id_conflicts and batch binding for history checks before a rewrite. Logical DML conflicts remain errors; historical compactions are ignored as in Java. Checking does not publish, abort, or consume the batch commit boundary.
  • Add bindings and type stubs for the new typed interfaces. No new dependencies.

Verification

  • Rust 1.98.0: row-ID tests (79), captured-plan misuse test (1), vindex tests (32).
  • Actual PyO3 harness: error tests (10), planned input rejection (1), independent conflict check followed by a real commit (1).
  • Core and Python binding Clippy with --all-targets -- -D warnings.
  • cargo fmt --all -- --check.

Regression cases include fixed snapshots after later appends, worker execution without data-manifest replanning, fresh authorization and masked/restricted grants, complete groups and DV checks, invalid/foreign plans, shard coverage and incremental builds, exception serialization, and preserving ordinary/unknown publication errors.

Paired Python PR: apache/paimon#10757 (draft, depends on this PR). It wires Ray scheduling to these APIs. Python's existing public API is unchanged.

Follow-up review fixes

  • Preserve the captured REST snapshot after a concurrent append even when the worker table has no manually pinned view. Reuse the REST snapshot when its ID matches, otherwise resolve the exact captured snapshot; expired baselines fail explicitly.
  • Clarify that captured plans avoid data-manifest replanning, while deletes still merge the captured bucket index. Add a subset-plan regression proving that an unselected anchor's DV is retained.
  • Red-to-green REST update/delete regression; planned mutation tests (7) and authorized mutation tests (14) passed. Core and binding Rust 1.98 Clippy, rustfmt and diff checks passed.

@leaves12138 leaves12138 left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@JingsongLi
JingsongLi merged commit ecbf580 into apache:main Oct 23, 2026
16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants