[Feature] Prune manifest partitions before metadata decoding #433

Search before asking

• I searched the existing issues for manifest partition pruning and found no matching issue.

Motivation

Partition-selective scans currently construct complete manifest entries and file metadata before discarding entries outside the requested partition. This also affects repeated reads from the bucket-scoped snapshot manifest cache, which can contain live files from many partitions.

When a query needs only a small subset of those entries, decoding unrelated file metadata and statistics adds avoidable allocation and planning work. The optimization should apply consistently to ordinary manifest reads, manifest Arrow-cache reads, and exact snapshot-cache hits.

Solution

Evaluate partition predicates before constructing full ManifestEntry and DataFileMeta objects, reusing the existing manifest batch and row-range reading paths. On snapshot-cache hits, select the exact retained snapshot and skip other snapshots without materializing their entries.

Preserve the existing lazy-decode option, ADD/DELETE merge semantics, remaining scan filters, retention limits, and metrics. Keep cached values complete across different partition predicates. Predicate evaluation errors must propagate; malformed snapshot-cache values should fall back to rebuilding from manifests. No table or cache format change is required.

Anything else?

Implementation: #430

Are you willing to submit a PR?

• I'm willing to submit a PR (PR perf(scan): prune manifest partitions before metadata decoding #430 is already open).