Skip to content

feat(manifest): sort bucketed tables by bucket first - #856

Closed
jianguotian wants to merge 3 commits into
apache:mainfrom
jianguotian:codex/manifest-sort-bucket-first
Closed

jianguotian wants to merge 3 commits into
apache:mainfrom
jianguotian:codex/manifest-sort-bucket-first

Conversation

@jianguotian

Copy link
Copy Markdown
Contributor

Purpose

Add Java-compatible bucket-first manifest sorting for fixed and postponed bucket tables, after the shared sorter (#854) and RowID support (#855).

Changes

  • select bucket-first sorting for bucket > 0 and postponed bucket mode (bucket = -2)
  • keep Data Evolution RowID sorting higher priority when complete RowID bounds exist
  • sort entries by bucket, configured/default partition field, file kind, and file name
  • use manifest bucket bounds when every input manifest provides them
  • conservatively fall back to partition overlap when legacy manifests lack bucket bounds
  • support non-partitioned fixed/postponed bucket tables

Merge order

Merge after #855. This replaces the closed bucket-only PR #852. The next PR will add the explicit manifest-compaction API.

Verification

  • cargo fmt --check
  • cargo test -p paimon --lib — 2,847 passed, 2 ignored
  • dedicated tests for bucket-first entry order, legacy metadata fallback, RowID precedence, and non-partitioned bucket schemas

Related Java work

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant