feat(index): apply shared FRI remapping to scalar and vector queries

[LuQQiu]

What this adds

This PR connects scalar and vector index loading to the common version-1 FRI reader. It applies the coverage and translation plan produced for each index segment while keeping version-0 FRI on its existing synchronous path.

stored index segment
        │
        ▼
snapshot FRI plan
        ├── Identity     → use the existing plugin load path
        ├── Translate    → load through asynchronous batch remapping
        └── No coverage  → omit the segment and scan

An unaffected segment receives no remapper, so existing and external plugins continue using their normal loader. A segment that requires translation is retained only when its consumer supports batch row-ID remapping; otherwise its destination coverage falls back to scanning.

Index consumers

The shared batch-remapping path is used by:

• B-tree, bitmap, zone map, Bloom filter, RTree, label-list, NGram, and inverted/FTS scalar indices.
• Vector partition storage, including PQ-backed storage.

Consumers translate physical row IDs in batches, preserve result order and deletion handling, and avoid building a partition-sized per-row lookup table. Vector loading preserves its existing partition and quantization layouts.

Version-0 FRI continues to use the original synchronous remapper and plugin APIs. A separate version-1 identity state ensures an unaffected segment never receives a synthetic legacy remapper merely because the dataset contains version-1 history.

Query planning and caching

The version-1 plan is built once per dataset snapshot and cached by manifest identity. It records each segment’s derived coverage and the fragments owned by newer selected segments. Warm segment opens reuse this plan instead of recalculating whole-dataset coverage.

Direct destination segments take precedence over older translated segments. When a logical index needs multiple source segments to cover a destination, all required contributors are selected; omitting one forces scan fallback for that destination. Fragment-restricted vector queries also scan when their selected segments do not provide complete coverage.

Cached objects are keyed by what they depend on. Content decoded without translation (identity segments, version-0 histories, no history, the IVF state entry) lives in the plain per-index namespace and stays warm across commits. Content that embeds translated addresses (pages, postings, partitions loaded through a translating remapper) lives under a namespace named by the segment's translation identity: a hash of its live coverage, its sibling exclusions on its path's destinations, the fingerprints of the transitions on its path, and whether the ledger dropped unsupported records. Such an entry goes cold exactly when that state changes (a transition on the path trimmed or replaced, a fragment on the path dropped, a sibling taking direct ownership of a path destination) and survives appends and rewrites of fragments the segment never covered. The FRI entry's UUID is not part of the key: every rewrite and trim mints a new one.

Staged segments. A segment built but not yet committed (the distributed build flow: execute_uncommitted per fragment, then merge, then commit) is unknown to the snapshot plan. open_row_id_remapping_with_plan / open_scalar_index_with_plan open one segment through a caller-supplied plan; the existing entry points are the None case. The cache namespace still follows the resolved remapping. The planner that builds such plans for a staged group, and the staged merge itself, land in #9161.

Two boundaries. A legacy-layout full-text (inverted) segment, whose per-document lengths cannot be translated, is excluded from coverage on a tagged table and its fragments are scanned; a full-text query must then name its column, since a column-less query discovers columns from the usable segments. Segment-only selection (with_index_segments without a fragment list) answers from the selected segments only, by contract, where the pre-FRI path fell back to a flat search; pass the fragment list to get the coverage guard and the flat fallback.

Validation

End-to-end tests cover all supported scalar and vector consumers, multi-segment source coverage, two-source/two-destination rewrites, deleted rows, partial and filtered queries, direct destination precedence, reconverging lineage, unsupported plugins and index versions, scan fallback, cache reuse, prewarm, append/delete/checkout, and eager/deferred compaction.

A version-0 lifecycle suite covers scans, filtered queries, writes, compaction, optimization, historical checkout, remapping, and cleanup while asserting that version-0 datasets never enter the new asynchronous path.

Validated with workspace Clippy and cargo fmt --all. The python and java lockfiles carry the new blake3 dependency (CI runs --locked); the staged-group planner that an earlier revision of this PR carried moved to #9161 with its consumers.

🤖 Generated with Claude Code

[lance-gatekeeper]

✅ Gate recommendation: approve.

The unused staged-group planner moved to the downstream tagged-maintenance PR, while the opt-in loader hook stays here. Default scalar and vector opens still use the reviewed snapshot plan; the version-0 path, conservative scan fallback, and translation-scoped caches are unchanged. The excluded Python and Java lockfiles now include the same direct blake3 dependency as the root lockfile.

[jackye1995]

looks good!