fix(index): compact across compatible vector index segments

[kamronis]

fix(index): compact across compatible vector index segments

Problem

File compaction treats each physical index segment as an independent coverage boundary.

For a logical vector index whose physical segments cover interleaved fragments, adjacent fragments therefore appear to have different index sets. The compaction planner splits them into single-fragment bins, which are no-ops, and compaction makes no progress.

Grouping segments by logical index name alone is not safe. Vector segments may be query-compatible while using independently trained IVF or quantizer models, in which case they cannot be physically merged.

Compaction also cannot remap multiple segments independently when their source fragments become one output fragment. Fragment-level index metadata cannot represent partial ownership of that output fragment, and the transaction correctly rejects such a rewrite as mixed indexed and non-indexed data.

Changes

• Treat physical vector segments as one compaction coverage boundary only when they share a merge-compatible model.
• Preserve the existing physical-segment boundaries for independently trained or otherwise incompatible vector segments.
• Remap each affected compatible segment and merge the remapped outputs into one physical index segment.
• Extend rewrite transaction handling to atomically replace multiple source index UUIDs with one merged UUID.
• Union the source fragment coverage before applying rewrite groups and reject incomplete or inconsistent many-to-one replacements.
• Remove temporary remapped segment directories after a successful merge.

Correctness

The planner uses the existing vector-segment compatibility validation, so compaction only crosses segment boundaries when the same segments can be safely passed to merge_existing_index_segments.

Segments with different IVF centroids, quantizer metadata, or storage formats remain separate and retain the existing compaction behavior.

Tests

The focused compaction test creates three fragments with three compatible physical segments belonging to one logical vector index and verifies that:

• the planner produces one task covering all three fragments;
• compaction produces one data fragment and one physical index segment;
• the merged segment covers the new fragment;
• dataset validation succeeds;
• post-compaction KNN overlap remains at least 0.5.

The transaction test verifies that two source segment UUIDs can be replaced atomically by one merged UUID with the correct unioned fragment coverage.

Validation

• cargo test -p lance-table test_handle_rewrite_indices_merges_source_segments --no-fail-fast
• cargo test -p lance test_compaction_merges_compatible_vector_segments --no-fail-fast -- --nocapture
• cargo check -p lance --lib
• cargo fmt --all
• cargo clippy -p lance -p lance-table --lib --tests -- -D warnings
• git diff --check

[lance-gatekeeper]

✅ Gate recommendation: approve.

The merge from main is mechanically clean and leaves all seven reviewed PR commits unchanged. #9421 repairs scalar coverage after compaction; vector merging still needs the remap and staged merge in this PR. Definition-only vector indices carry empty coverage, so they do not create a spurious compaction boundary.

The focused compaction, cross-mode publication, unsupported-index boundary, compact ownership remap, and many-to-one transaction regressions pass on this head.