feat(format): define a unified tagged fragment reuse history

[LuQQiu]

FRI keeps existing indices usable after fragment rewrites by translating old physical row addresses to new ones. Today it supports order-preserving compaction. This proposal extends the same system index to support stable partitioning, with both mapping types sharing one fragment-lineage history.

Following the unified FRI proposal, source/destination lineage stays in FRI details, while large mapping payloads remain in separate immutable files.

One history, multiple mappings

Continue storing FRI information in a single __lance_frag_reuse system-index entry. Keep the existing InlineContent / ExternalFile envelope and the original field number for legacy versions. Add tagged transitions alongside them:

FragmentReuseIndexDetails
└── InlineContent, stored inline or in external details.binpb
    ├── legacy_versions[]
    └── transitions[]
        ├── ordered sources[]
        ├── ordered destinations[]
        └── mapping
            ├── OrderedCompaction: surviving-row bitmap
            └── StablePartition: immutable row-map reference

Sources and destinations define the common rewrite graph. Each mapping defines how to translate row offsets. Legacy groups can be read as ordered-compaction transitions; mixed histories follow fragment lineage, not the order of records or dataset version numbers.

Lightweight metadata, external row maps

Ordered compaction retains its compact bitmap representation. Stable partition assigns each physical source row a nullable uint16 destination label, preserving source order within each destination. A null label means the row was deleted. A counts matrix lets readers reconstruct destination offsets without reading all preceding labels.

Stable-partition metadata records map_id, map_size_bytes, and optional base_id. The labels and counts are stored in _fri/<map_id>/stable_partition.lance. Mapping identity is independent of the FRI index UUID: updating the history rewrites its metadata, but does not rewrite existing row-map files. The history can be opened without loading labels; address translation reads the required blocks.

Publication and compatibility

AppendFragmentReuseTransitions expresses a transition delta. Combined atomically with a fragment rewrite, it lets the commit apply the delta to the current history and publish destination fragments and their mappings together. The persisted FRI details remain a snapshot of that history.

• Index version 0: existing compaction format and read/write behavior remain unchanged.
• Index version 1: supports legacy groups and tagged transitions in one history.
• The first commit publishing index version 1 sets reader and writer flag 512. The reader flag prevents old clients from partially interpreting the history; the writer flag prevents them from dropping mappings during metadata maintenance. Subsequent manifests retain both bits.

Scope and validation

This PR contains protobuf definitions, the corresponding format documentation, the proposed flag constant, and minimal compile adapters. It does not enable tagged-history reads or writes. Mapping implementations and reader integration follow in #9106 → #9064 → #9067 → #9068 → #9107.

Replaces #9065 as the standalone spec at the bottom of native stack #9137, based on main 31d78d170.

cargo fmt --all and whitespace checks pass. Clippy and tests are blocked by dependency resolution: main requires object_store_opendal 0.60.1, while the crates.io index currently offers only up to 0.60.0.

[github-actions]

Important

Format specification vote

This PR modifies the Lance format specification, so it requires 3 binding +1 votes from PMC members (excluding the proposer) and a minimum 72-hour voting period, weekends excluded, before it can merge. Vote by approving this PR (+1) or requesting changes (−1, a veto). See the voting process.

Status: ✅ Vote passed — 3 PMC approvals, voting period elapsed

Approvals (this commit)	@Xuanwo, @BubbleCal, @jackye1995 (3/3)
Vetoes	none
Voting period	elapsed — ended Tue 2026-09-15 18:18 UTC (11:18 PDT)

_{Updated automatically by the format-spec vote gate, which re-checks every 15 minutes — just voted? Re-check now (press Run workflow; leave the input blank to re-check every open format PR). A PMC member may apply the format-waived label to waive the vote for a trivial edit (typo, wording, formatting).}

[Xuanwo]

The standalone AppendFragmentReuseTransitions operation seems premature here, since Rewrite can already publish the FRI entry atomically. Could we defer it until there is a concrete caller for separate mapping publication or composite transactions? That would let us review the durable operation’s semantics alongside its actual use, while keeping this PR focused on the unified history format.

[LuQQiu]

This is pre-added for composite transaction V2 at the assumption that transaction V2 will be landed soon. In that case, having a separate AppendFragmentReuseTransitions can help make the transaction more clear. I can remove it now and add it back later when transaction V2 is ready

[LuQQiu]

Agreed, removed in 66232d4: the operation, its oneof entry, and the related doc sections (the Appending FRI Transitions section and the separate-commit wording). Rewrite remains the only publication path, atomic with its own commit. We can bring a durable append back together with a concrete caller or composite transactions.

[lance-gatekeeper]

✅ Gate recommendation: approve.

Deferring the standalone append operation keeps this format change focused on the concrete atomic path: a rewrite publishes its FRI mapping in the same commit as fragment replacement. The unified tagged history, compatibility fence, and fail-closed adapter behavior remain coherent, and the removed field had no released compatibility obligation.

Please mark this PR with the breaking-change label.

[Xuanwo]

+1 binding from me

[BubbleCal]

+1

[LuQQiu]

Synced with main to clear the merge conflict (5410084). The conflict was the flag table in versioning.md: main landed FLAG_MIXED_DATA_FILE_VERSIONS on bit 256, which this PR's text had listed as reserved. Resolution keeps both rows (256 from main, 1024 from this PR), reserves only bit 512 going forward, and keeps main's paired-bits sentence. No FRI spec content changed; feature_flags.rs and table.proto merged mechanically, and this build still does not claim the 1024 capability (support lands in the follow-up PRs, matching the spec-first posture).

@Xuanwo @BubbleCal apologies for the churn: the vote gate binds approvals to the commit, so this sync dropped your +1s. If the resolution looks right, a re-approve would be appreciated; the FRI text you reviewed is byte-identical.

[BubbleCal]

+1

[jackye1995]

great work!!!