Skip to content

[core] Expose postpone bucket plans through the write builder - #1200

Merged
JingsongLi merged 3 commits into
apache:mainfrom
JingsongLi:codex/native-write-execution-coverage
Oct 22, 2026
Merged

JingsongLi merged 3 commits into
apache:mainfrom
JingsongLi:codex/native-write-execution-coverage

Conversation

@JingsongLi

@JingsongLi JingsongLi commented Oct 22, 2026 •

Copy link
Copy Markdown
Contributor

Purpose

Use the writer's metadata boundary for PyPaimon postpone bucket planning. The previous Native adapter derived existing bucket counts from a query plan, which requires unrestricted SELECT authorization and can combine entries carrying different total bucket counts. Java's PostponeUtils.getKnownNumBuckets reads writer/store metadata directly.

Brief change log

  • Add PostponeFixedBucketWriteBuilder.resolve_bucket_plan() to resolve known partition layouts from the latest writer baseline, independently of historical read selectors.
  • Reuse the existing per-entry bucket validation and conflict checks. Pending buckets do not define an existing layout. Explicit plans take precedence; overwrite otherwise starts with an empty layout. Configured default counts do not create fictional partition entries.
  • Require a catalog-loaded write identity for guarded tables without issuing SELECT requests. Reuse the REST snapshot response directly, including when its snapshot JSON file is unavailable in storage.
  • Add PostponeBucketPlan.to_arrow() using the same positional partition-columns + total_buckets schema accepted by from_arrow(). Bind the getter as a PyArrow RecordBatch and update the Python type stub.
  • Move a test-only BinaryRow import into the tests module to fix an existing main Clippy warning; no suppression or behavior change.
  • Publish the Python UriReader/UriReaderFactory protocols as runtime imports, use public with_blob_uri_reader_factory() setters on batch/stream/row-ID updates with chainable returns, and expose with_commit_user() consistently on batch/stream/postpone builders. Update stubs, callers, SDK documentation and public API tests; no compatibility aliases.

Tests

  • Rust 1.98.0: 9 relevant Core tests and 1 PyArrow binding test passed for bucket planning.
  • Additional public-API regressions: actual PyO3 binding harness passed for chainable setters, clearing factories, invalid configuration and public commit identities; the Python package-bootstrap test passed for real protocol exports without loading or rebuilding an extension.
  • Regressions cover explicit/overwrite/default precedence, nullable partition round trips, pending buckets, latest writer baseline, conflicting layouts within one bucket, REST write identity, denied SELECT, and missing filesystem snapshot JSON.
  • cargo +1.98.0 clippy --locked --all-targets --workspace --features fulltext,vortex -- -D warnings: passed.
  • cargo +1.98.0 fmt --all -- --check and git diff --check: passed. Python type stub parses with Python 3.6 grammar.
  • Getter rename verified with 9 Core tests, 1 PyO3/PyArrow round-trip test, strict workspace Clippy and fmt checks on Rust 1.98.0.
  • No Native Python extension was installed or rebuilt; the Rust binding test harness was compiled and run normally.

API and Format

The metadata resolver is named resolve_bucket_plan() consistently in Core, the Python binding and PyPaimon. with_bucket_plan() continues to configure an explicit plan; no old getter alias is retained.

Adds the Core builder getter, the symmetric Arrow plan export, and the matching binding. Promotes the existing Python stream/commit-identity configuration contracts to public names, including importable protocol definitions. No new dependency, table option, or storage-format change.

Documentation

Rust API comments and the Python type stub document the writer authorization boundary and plan precedence. The Python SDK guide describes public stream protocols, chainable setters and commit identities. apache/paimon#10753 integrates the getter and cleans the write execution boundary.

@leaves12138 leaves12138 left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@JingsongLi
JingsongLi merged commit c60d70e into apache:main Oct 22, 2026
16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants