Repository navigation
perf(core): cache ROW footer and block index metadata - #1196
XiaoHongbo-Hope wants to merge 1 commit into
Conversation
JingsongLi
left a comment
There was a problem hiding this comment.
Requirement fit: SUPPORTED. Implementation: FINDINGS at ad1c207. Repeated sparse ROW reads have end-to-end value, but the new cache must respect the existing metadata memory limit.
Local worktree validation: 537 related core tests passed (1 ignored), all 14 data_evolution_row_sidecar_test write/commit/read regressions passed, and formatting/diff checks passed. A temporary single-FileIO probe with file-format.metadata-cache.max-size=64 kb confirmed that Parquet and ROW are each given a separate 64 KiB retention budget (128 KiB combined). The probe was removed afterward. CI runtime/integration checks passed; the failing check job reports baseline TLS dependency advisories, already addressed by main's #1194.
| if self.max_bytes == 0 || self.process_id != std::process::id() { | ||
| return None; | ||
| } | ||
| let cache = Arc::clone(self.row_metadata_cache.get_or_init(|| { |
There was a problem hiding this comment.
[P2] Keep ROW and Parquet within the configured metadata-cache budget
This creates an additional cache with min(max_bytes, 32 MiB), while get_or_init still gives Parquet the full max_bytes. The existing file-format.metadata-cache.max-size option promises the maximum parsed file-format metadata cache size. A Data Evolution table can warm both pools through ordinary Parquet scans and sparse ROW sidecar reads in the same FileIO, so the setting no longer bounds their combined retention. A local probe with one FileIO configured to 64 KiB receives 64 KiB for each pool; at the default, their budgets total 82 MiB against the configured 50 MiB. Please share accounting or divide the configured budget between formats; an intentionally additive ROW budget needs its own explicit setting.
A repeated ROW point read currently fetches and parses the footer and block index on every query against the same immutable file. With a one-block fixture, that makes three range reads per query (footer, index, block).
Cache the parsed footer and block index within the FileIO metadata-cache context. The second read of the same file then needs only the data-block read. The cache is enabled by default when
file-format.metadata-cache.max-sizeis nonzero, bounded to 32 MiB and 4,096 entries, keyed by the storage-namespaced immutable file identity and size, and bypassed when the reader has no stable key or after fork. No new option or file-format change is introduced.This is separate from #1195, which caches the ROW sidecar file size (
stat); the two optimizations are independent. Both PRs touch the FileIO cache context, so this branch will need a trivial rebase if #1195 merges first.Validation:
cargo test --locked -p paimon --features storage-memory row_reader_ --lib(7 passed)cargo test --locked -p paimon --features storage-memory row_metadata_cache_has_separate_type_and_bypasses_inherited_fork_state --lib(1 passed)cargo test --locked -p paimon --test data_evolution_row_sidecar_test(14 passed)cargo clippy --locked -p paimon --lib --features storage-memory -- -D warnings -A clippy::nonminimal_bool(passed; the suppressed lint is present in unrelated main-branch files)cargo fmt --all;git diff --check