Skip to content

perf(core): reuse immutable ROW sidecar file sizes - #1195

Open
XiaoHongbo-Hope wants to merge 3 commits into
apache:mainfrom
XiaoHongbo-Hope:codex/row-sidecar-size-cache-20261022
Open

XiaoHongbo-Hope wants to merge 3 commits into
apache:mainfrom
XiaoHongbo-Hope:codex/row-sidecar-size-cache-20261022

Conversation

@XiaoHongbo-Hope

Copy link
Copy Markdown
Contributor

Problem

Each sparse Data Evolution read that selects an immutable ROW sidecar calls object storage stat to learn the sidecar length, even when the same file is queried repeatedly. This adds a remote request before the ROW reader can seek to its footer.

Change

Cache the size of registered ROW sidecars in the existing FileIO metadata-cache context. Keys use the storage namespace and canonical object path, so distinct storage targets cannot share a cached size. The cache is bounded to 4,096 entries, is disabled when the metadata cache is disabled or the provider has no stable namespace, bypasses inherited state after fork, and is invalidated on FileIO file/directory deletion. No public option or file-format change is introduced.

Validation

  • cargo test --locked -p paimon test_immutable_row_sidecar_size_cache_is_scoped_and_disabled_with_metadata_cache --features storage-memory --lib
  • cargo test --locked -p paimon --test data_evolution_row_sidecar_test (14 passed)
  • cargo clippy --locked -p paimon --lib --features storage-memory -- -D warnings
  • cargo fmt --all -- --check; git diff --check

This PR optimizes the existing ROW sidecar path. It does not add the separate scalar .srow format used in the DROID prototype.

@JingsongLi

Copy link
Copy Markdown
Contributor

How does this relate to #1196? Why is it necessary to handle this separately?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants