Skip to content

perf(manifest): write bucket-first sorted entries - #852

Closed
jianguotian wants to merge 2 commits into
apache:mainfrom
jianguotian:codex/bucket-first-manifest-sort
Closed

jianguotian wants to merge 2 commits into
apache:mainfrom
jianguotian:codex/bucket-first-manifest-sort

Conversation

@jianguotian

Copy link
Copy Markdown
Contributor

Purpose

Keep entries for fixed-bucket and postponed-bucket tables close by bucket when
Rust writes a manifest. This produces tighter _MIN_BUCKET / _MAX_BUCKET
envelopes after rolling, so scans can skip more manifest files before fetching
and decoding their contents.

This is the write-side counterpart to the bucket metadata pruning added in
#850 and follows the bucket-first layout introduced by Apache Paimon Java in
apache/paimon#9792 and extended to non-partitioned fixed/postponed-bucket tables
in apache/paimon#9808.

Changes

  • add the Java-compatible manifest-sort.enabled table option (default false)
  • for non-data-evolution fixed/postponed-bucket tables, sort entries by:
    1. bucket
    2. the first partition field, when present
    3. file kind
    4. file name
  • apply the order to newly written delta manifests and manifests rewritten by
    the existing Rust minor/full merge paths
  • leave data files and row contents unchanged

The implementation intentionally does not claim parity with Java's explicit
compact_manifest action (apache/paimon#9802): paimon-rust does not currently
have the corresponding action/procedure API. Existing one-file manifest lists
are therefore not force-rewritten solely by this option; adding that API can be
reviewed independently.

Tests

  • cargo test -p paimon --lib (2818 passed, 2 ignored)
  • cargo clippy -p paimon --lib --tests -- -D warnings
  • cargo fmt --check

@JingsongLi JingsongLi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you want to get this done, I suggest you complete all the sorting tasks.

Not just bucket.

@jianguotian

Copy link
Copy Markdown
Contributor Author

Superseded by the complete manifest-sorting series requested in review. I will recreate the work in merge order: bounded generic manifest sorting, RowID sorting, bucket-first sorting, compact-manifest API, then explicit full sort. The bucket-first change from this PR will return only after its shared sorting foundation is in place.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants