feat(index): maintain tagged fragment reuse histories

[LuQQiu]

Maintenance for tagged fragment reuse histories: trimming drained records, pruning superseded index segments, garbage-collecting unreferenced row maps, and remapping indices through compaction-only chains. Legacy (version 0) FRI maintenance keeps its existing path untouched.

Trim

cleanup_frag_reuse_index on a tagged table derives which transitions are still needed and splices the survivors byte-verbatim (no re-encode of retained records). A transition is retained while some index still translates through it: its provenance intersects the index's stored coverage and at least one destination is not directly covered, plus the downstream producer-to-consumer closure so chains never break mid-path. A fully drained entry is deleted outright. The trim intent is re-derived against the current manifest on every commit attempt; if a rebase leaves nothing to trim, the operation aborts instead of committing an empty version.

Superseded segment pruning

Before trimming, cleanup removes index segments whose entire valid coverage (stored bitmap plus reader-derived translated coverage) is directly covered by kept newer siblings, newest-first to a fixed point, as a separate removal-only commit. Only segments the reader considers usable count as covering; an unusable segment can neither justify pruning a sibling nor block a trim. The removal set is re-derived against the current manifest on every commit attempt, including commit-path rebases: prune transactions carry an explicit property, the conflict resolver replaces a rebased removal set wholesale, and a rebase that leaves nothing to remove aborts instead of committing (so a concurrent replacement that narrows a justifying sibling's coverage can never make a stale removal land). Pruning also refuses histories it cannot fully interpret, same as trim.

Row-map garbage collection

_fri/<map_id>/ directories follow the same three states as _indices: referenced by a retained manifest, unreferenced but young, unreferenced and aged out (deleted). References are collected from every retained manifest version's FRI entry. If any history record cannot be interpreted (unknown envelope records included), row-map GC disables itself for the run rather than guessing.

Index remapping

Indices whose provenance runs through ordered-compaction-only chains are remapped and swapped (each hop requires the transition's sources to be within the segment's current coverage); any chain touching a stable partition is skipped with a log line, since translation stays available at query time. Two correctness details: the published coverage is intersected with what the remapped file actually contains (the bitmap-index remap path excludes rows already ceded to direct siblings), and the metadata-only "keep" outcome preserves the original base_id so indices inherited through a shallow clone keep pointing at their real files.

Unknown-record consistency

Envelope-level unknown records flag the decoded ledger as unsupported, giving every consumer the same conservative answer: GC disables, trim, remap, and superseded-segment pruning all refuse, appends onto such histories are rejected, reads are unaffected. The pruning refusal sits inside the per-attempt derivation, so no removal commit can be built from a history the client cannot fully interpret.

Merging segments on a tagged table

optimize_indices merge opens the source segments through the translating loader, so the merged files hold live addresses. The merged segment claims the union of the merged segments' stored provenance plus the newly indexed fragments (provenance, not the derived live coverage), so translated coverage is kept, an unselected sibling's coverage is untouched, and the reader keeps translating the merged segment while its provenance runs through retained transitions. Vector merges classify dormant and live segments on the same derived coverage. Draining also works through rebuilds and delta builds.

Staged segments (merge_existing_index_segments over segments built with execute_uncommitted) are not in the manifest, so the merge plans them as one group with the reader's own steps (plan_staged_segments: may_need_translation, segment_plans, translation_fingerprint, the same ones the snapshot plan runs per committed group) and threads that plan through every scalar merge, the per-segment filters and the opens (the *_with_plan loaders from #9107). A segment that is neither staged nor listed is an error, never empty coverage. Here coverage follows the destination model: each staged segment's bitmap becomes the live coverage its plan derives before the merge runs, because the per-type merges emit the live-domain union and a provenance bitmap would be pruned to nothing at commit. A group that covers no rewrite destination completely claims nothing and those rows stay on the scan path. The NGram merge translates only through the version-0 remapper, so a staged NGram segment whose plan translates is refused rather than merged with stale addresses; the FM merge rebuilds over the derived coverage. The version-0 handle is never consulted on a tagged table. Staged vector shards never open the dataset: the merge filters each by its stored bitmap in the raw address domain and the merged segment keeps the provenance union, so it translates at query time like a committed vector merge (pinned by an IVF_FLAT case across a stable partition). For that to survive the commit, commit_existing_index_segments keeps a retired fragment that the tagged entry's lineage records in a new segment's bitmap; a retired fragment off the lineage is still pruned. Translating is not enough on its own, though: the values a segment holds are the ones it read at its own dataset version, and every commit since then may have moved its rows (a recorded transition: the provenance stays and translates) or rewritten an indexed column in place (the values are stale: that transition's coverage must go). So replay_staged_segments validates the staged segments up to the snapshot being merged against with the conflict resolver's own rules, as a CreateIndex built at the oldest build version and rebased across every transaction up to that snapshot: a transition keeps the provenance, an in-place rewrite withdraws the transition's sources (the withdrawal from #9139), and a bare compaction of a covered fragment, a partly covered rewrite group or a version in the window that can no longer be read is an error pointing at a rebuild. The merge runs after the replay, over what is still valid, and stamps the merged segment with the validated snapshot; the commit reads from the oldest validated version, so the commit loop carries the segments across anything newer through the same rules. The cost is proportional to the commits crossed between the build and the merge, and segments are replayed per build version, so a segment built after a rewrite is not withdrawn for it again. The history has to be complete: the transaction listing only knows the manifests that still exist, so the versions listed must be exactly build+1 through the snapshot, else the segments are refused with the rebuild error (a build version kept by a tag with the versions after it cleaned up would otherwise skip the rewrite in between). An in-place rewrite of an indexed column on a stable-partitioned table costs the whole transition's coverage for every segment still translating through it, so remap (#9187) after a stable partition before rewriting columns in place; optimize_indices restores what was withdrawn: maintenance opens through open_*_index_for_maintenance, which reaches a registered segment the listing omits and reads it as contributing nothing, so a rebuild over the live fragments replaces it. The query entries (open_scalar_index, open_vector_index, open_generic_index by uuid) see the listing only and refuse such a segment, since no scan was scheduled for its rows and an empty answer would drop them. The snapshot plan records why the reader derives no coverage for a registered segment (withdrawn, no derived coverage because superseded or missing contributors, unsupported by this build, corrupt metadata): maintenance reads the first two as empty translations and replaces them, skips the third, fails on the fourth; an empty bitmap is never planned as identity, so a merge never reads the withdrawn pages raw. The last segment of a name emptied by a withdrawal stays as the record of what to build: queries scan, optimize_indices reads its parameters through the maintenance entry and rebuilds it from the live fragments (scalar and vector; a vector segment with files and no live coverage is dormant, a definition without files is still trained from its definition), and it takes part in neither remap nor merge as a normal segment. Under a history this build cannot interpret optimize is a no-op with a warning, like trim, superseded pruning and remap. The default optimize's steady-state vector rebalance and the inverted append/merge publish the stored provenance and read old data through the reader's derived coverage, exactly as the scalar merge does, so neither turns indexed rows of a partition's destinations into scans. In the staged replay a same-name CreateIndex in the window (routine optimize_indices on the index a distributed rebuild is replacing) is no conflict unless it changed the index's fields or type; the overlap rules settle it at commit. Staged NGram segments cannot be carried across a stable partition (the commit-time NGram check and the merge both refuse a translating NGram segment): rebuild them. remap_column_index remaps the first segment of the name it is given, one segment per call. A user index built before a stable partition and committed after it lands the same way, translating, through the resolver rule in #9139, and one committed after a concurrent in-place rewrite lands with that transition withdrawn; the cases live here because this PR is where a user CreateIndex is admitted on a tagged table. The staged plan looks only at the staged group, not at committed same-name siblings: if a committed delta already covers a destination directly, the staged group still claims it and the commit displaces that delta, so the delta's other fragments fall back to scan. That is a coverage regression, not a wrong result.

Validation

On the current #9139 head: lance-table 485, lance lib whole suite green (staged-merge tests in index/create/tests/staged_tagged.rs, each proven to fail without its fix); fmt and clippy clean; python and java bindings compile. Lifecycle tests cover partial and full draining, retained-version file ownership, trim retries, and trim x trim conflicts.

Stack: 8 of 11, follows #9139 (synchronized), #9162 builds on this.

🤖 Generated with Claude Code

[LuQQiu]

Adjudicated both rounds against the code; the pruning concerns were real and are fixed, the trim concerns were already structural.

Pruning validated nothing before removing: fixed in 5db67d4. Removals were derived through the lenient reader path, which only logs on unsupported records, so an old client could prune a segment whose coverage flows through a transition it cannot interpret, and commit that removal before trim's refusal fired. The strict ledger check now sits inside derive_superseded_segments, so every attempt refuses (NotSupported, message consistent with trim, remap, and GC) before any removal-only commit is built.

Pruning race on rebase: confirmed and fixed in 867ae45. The removal-only shape fell through every re-derivation branch in finish_create_index, and a concurrent replacement of a justifying sibling (for example this PR's own remap swap, which publishes narrowed coverage) escaped both has_append_drop_conflict (replacement-style is exempt) and has_replaced_identity_conflict (keys on our removed set, not our justification). A stale removal set could therefore land after concurrent coverage withdrawal. Prune transactions now carry an explicit transaction property; on rebase the resolver re-runs derive_superseded_segments against the rebased manifest and replaces the removal set wholesale, aborting via a no-op marker (treated as success) when nothing remains, mirroring the tagged-trim mechanism. The regression test drives exactly the claimed race (stale prune rebasing over a same-uuid narrowed replacement) and fails without the fix.

Trim from a stale view: already handled. The trim is re-derived against the manifest actually being committed onto during every rebase (finish_create_index): the stale outcome is discarded wholesale, and a rebase that leaves nothing to trim aborts via the TAGGED_TRIM_REBASED_TO_NOOP marker instead of committing an empty version. With 867ae45 the prune-then-trim split is now symmetric: both commits re-derive on every attempt and only consume committed state.

[lance-gatekeeper]

⚠️ Gate recommendation: approve with a non-blocking risk.

The two remaining optimizer coverage regressions are fixed: vector rebalance retains source provenance, and Inverted merge uses the reader-derived tagged coverage. The focused regressions and original Inverted merge probe now pass.

I also reproduced a separate partial-withdrawal ANN mismatch in the tagged write-path prerequisite: after an indexed-column rewrite, a surviving segment can expose retired rows to search, even after default optimization. Its withdrawal, identity-planning, and query-prefilter paths are unchanged by this maintenance patch, so I am not requesting a change here. This prerequisite risk needs attention before the stacked feature is rolled out.

Please mark this PR with the breaking-change label.