feat(format): separate index keys from covering fields

[Ali2Arslan]

Summary

• define the new IndexMetadata contract so fields contains key fields only and covering_fields is an independent declaration that may overlap it
• define the index dependency set as the union of fields and covering_fields
• reserve FLAG_INDEPENDENT_COVERING_FIELDS (bit 11 / 2048) as a paired reader/writer fence and keep it unsupported until the implementation lands
• specify activation: setting the flag atomically normalizes every legacy-form entry of the index section in the same manifest (or requires that none declares covering fields), a retry normalizes against the index section it is retrying against, and derived manifests retain the bit
• update index lifecycle, overlay-staleness, and feature-flag specifications to use the independent declarations

Compatibility

The subset contract for covering fields (added in #8535, refined by the now-merged #8856) shipped in Lance 11.0.0, so the same protobuf bytes cannot be reinterpreted unconditionally. A manifest using the new contract must set both FLAG_COVERED_INDEX_METADATA and FLAG_INDEPENDENT_COVERING_FIELDS. A manifest with only FLAG_COVERED_INDEX_METADATA retains the stable legacy interpretation, where covering_fields is a subset of fields with no id repeated.

The new bit makes older releases reject independent declarations instead of treating covering fields as keys. No protobuf field number or wire type changes. Bit 11 avoids the reserved bit 9 and the bit 10 FLAG_FRAGMENT_REUSE_INDEX from #9136.

This PR intentionally reserves and specifies the contract without implementing readers or writers, following the format-change process. Bit 11 sits above FLAG_UNKNOWN (1 << 9) and supported_flags_when starts from FLAG_UNKNOWN - 1, so no build can open or write a manifest that sets it; FLAG_FRAGMENT_REUSE_INDEX is in the same reserved state on main today. Consequently the existing keyed_fields derivation, which subtracts the carried count from fields, can never observe an independent declaration and is left for the implementation PR. This PR layers on the carried-column storage contract from #8856 so physical storage and query behavior can follow the approved metadata contract.

Testing

• cargo fmt --all
• cargo test -p lance-table feature_flags --lib
• cargo clippy --all --tests --benches -- -D warnings
• RUSTDOCFLAGS='-D warnings' cargo doc -p lance-table --no-deps
• uv run --project docs python ci/check_proto_comments.py
• cd docs && uv run mkdocs build

[github-actions]

Important

Format specification vote

This PR modifies the Lance format specification, so it requires 3 binding +1 votes from PMC members (excluding the proposer), at least one of them on the latest commit, and a minimum 72-hour voting period, weekends excluded, before it can merge. Vote by approving this PR (+1) or requesting changes (−1, a veto). See the voting process.

Approvals carry over across pushes, so a rebase or a typo fix does not send everyone back to re-vote. Whoever approves the latest commit is vouching that nothing substantive has changed since the earlier approvals; if something has, ask for fresh votes.

Status: ✅ Vote passed — 3 PMC approvals, voting period elapsed

Approvals	@wjones127, @westonpace, @BubbleCal (3/3)
Latest commit approved by	@BubbleCal
Vetoes	none
Voting period	elapsed — ended Wed 2026-09-16 19:27 UTC (12:27 PDT)

_{Updated automatically by the format-spec vote gate, which re-checks every 15 minutes — just voted? Re-check now (press Run workflow; leave the input blank to re-check every open format PR). A PMC member may apply the format-waived label to waive the vote for a trivial edit (typo, wording, formatting).}

[wjones127]

Need to validate my assumption, but if we can I'm fine with making a breaking change to the existing spec so we avoid having these two separate flags.

[Ali2Arslan]

I've merged #8856 which added (a different) concept of "this is how you specify a key column is also covered" since it already had 3 votes. Let's go ahead and rebase this PR on top of the new wording (since it seems we have alignment on this approach over the other approach). Then we can merge this PR too.

@Ali2Arslan let me know if you need any help. Thanks!

Thanks @westonpace , I updated the wordage if you guys want to take another look.

[lance-gatekeeper]

✅ Gate recommendation: approve.

The rebase preserves the accepted independent-declaration contract while fitting it after newly occupied feature-flag bits 9 and 10: bit 11 remains explicitly rejected by current readers and writers, and the unknown boundary moves to bit 12. The atomic activation, current-index retry, and sticky-retention rules remain intact, so released bit-7 manifests and #8856ʼs carried-column storage contract stay unambiguous.

[wjones127]

I'm a little sad about the two flags, but I'll accept it.

[westonpace]

Thanks for the rebase and addressing the feedback, I'm +1 on the new wording.

[BubbleCal]

The independent key/covering declarations read much clearer to me, and since nothing writes covering indexes yet this is a good time to make the switch. Left a few small notes, mainly the bit collision after rebase. Approving assuming those get sorted out.