Skip to content

perf(core): cache ROW footer and block index metadata - #1196

Open
XiaoHongbo-Hope wants to merge 1 commit into
apache:mainfrom
XiaoHongbo-Hope:codex/row-metadata-cache-20261022
Open

XiaoHongbo-Hope wants to merge 1 commit into
apache:mainfrom
XiaoHongbo-Hope:codex/row-metadata-cache-20261022

Conversation

@XiaoHongbo-Hope

Copy link
Copy Markdown
Contributor

A repeated ROW point read currently fetches and parses the footer and block index on every query against the same immutable file. With a one-block fixture, that makes three range reads per query (footer, index, block).

Cache the parsed footer and block index within the FileIO metadata-cache context. The second read of the same file then needs only the data-block read. The cache is enabled by default when file-format.metadata-cache.max-size is nonzero, bounded to 32 MiB and 4,096 entries, keyed by the storage-namespaced immutable file identity and size, and bypassed when the reader has no stable key or after fork. No new option or file-format change is introduced.

This is separate from #1195, which caches the ROW sidecar file size (stat); the two optimizations are independent. Both PRs touch the FileIO cache context, so this branch will need a trivial rebase if #1195 merges first.

Validation:

  • Counted ROW reader test: first query reads footer/index/block (3); repeated query reads only block (1); disabled cache and readers without stable keys still perform all reads.
  • cargo test --locked -p paimon --features storage-memory row_reader_ --lib (7 passed)
  • cargo test --locked -p paimon --features storage-memory row_metadata_cache_has_separate_type_and_bypasses_inherited_fork_state --lib (1 passed)
  • cargo test --locked -p paimon --test data_evolution_row_sidecar_test (14 passed)
  • cargo clippy --locked -p paimon --lib --features storage-memory -- -D warnings -A clippy::nonminimal_bool (passed; the suppressed lint is present in unrelated main-branch files)
  • cargo fmt --all; git diff --check

@JingsongLi JingsongLi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Requirement fit: SUPPORTED. Implementation: FINDINGS at ad1c207. Repeated sparse ROW reads have end-to-end value, but the new cache must respect the existing metadata memory limit.

Local worktree validation: 537 related core tests passed (1 ignored), all 14 data_evolution_row_sidecar_test write/commit/read regressions passed, and formatting/diff checks passed. A temporary single-FileIO probe with file-format.metadata-cache.max-size=64 kb confirmed that Parquet and ROW are each given a separate 64 KiB retention budget (128 KiB combined). The probe was removed afterward. CI runtime/integration checks passed; the failing check job reports baseline TLS dependency advisories, already addressed by main's #1194.

if self.max_bytes == 0 || self.process_id != std::process::id() {
return None;
}
let cache = Arc::clone(self.row_metadata_cache.get_or_init(|| {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Keep ROW and Parquet within the configured metadata-cache budget

This creates an additional cache with min(max_bytes, 32 MiB), while get_or_init still gives Parquet the full max_bytes. The existing file-format.metadata-cache.max-size option promises the maximum parsed file-format metadata cache size. A Data Evolution table can warm both pools through ordinary Parquet scans and sparse ROW sidecar reads in the same FileIO, so the setting no longer bounds their combined retention. A local probe with one FileIO configured to 64 KiB receives 64 KiB for each pool; at the default, their budgets total 82 MiB against the configured 50 MiB. Please share accounting or divide the configured budget between formats; an intentionally additive ROW budget needs its own explicit setting.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants