feat(format): spill row lineage sequences to hidden data file columns

[BubbleCal]

§5.3 of the Stable Row ID GA design (#8931, tracked as #9250): a fragment's row lineage sequences -- its row ids and its created-at and last-updated-at versions -- can leave the manifest and live as hidden uint64 columns of a Lance data file.

Builds on the prototype in #8953 (kept as its own commit, with authorship) and extends it to all three sequences, following the design write-up agreed on 2026-09-16.

Problem

Each sequence is stored inline in the fragment's manifest entry. An appended fragment's sequences are single runs and cost a few dozen bytes, but once compaction merges fragments whose rows came from many places the row id sequence degrades to 4-8 bytes per row and the version sequences to a run per row. The manifest then grows with the table's row count and every commit rewrites all of it (#8621).

Format

• DataFragment gains a DataFile arm on each of its three lineage oneofs: column_row_ids = 12, column_last_updated_at_versions = 13, column_created_at_versions = 14. The column is located by the file's fields/column_indices pair like a user column, at the reserved field ids -3 (_rowid), -4 (_row_created_at_version) and -5 (_row_last_updated_at_version).
• The spec now reserves every negative field id for system use, and readers skip any negative id in files[].fields. This is what lets a later writer put the lineage columns in the fragment's main data file instead of a separate one; this PR writes the separate-file form only.
• New feature flag FLAG_UNSTABLE_SPILLED_ROW_LINEAGE = 1 << 11 (bit 9 is reserved and bit 10 is the tagged FRI flag from feat(format): define a unified tagged fragment reuse history #9136), above FLAG_UNKNOWN the way that flag is. Every released build (v11.0.0 boundary 256, v12/v13 pre-releases 512) already refuses such a dataset. Like data overlay files, release builds understand the bit only with LANCE_ENABLE_UNSTABLE_SPILLED_ROW_LINEAGE=1; debug builds always do.
• The external_* arms are untouched: external_row_ids stays readable, the version arms stay a valid encoding nothing reads, and none is written.

Placement rule

Where a sequence may live follows from when its values are known. A value the commit assigns -- an appended fragment's row ids, an inserted row's created-at, every row's last-updated-at -- can change when a commit conflict is retried, so it stays inline where the retry can rewrite it. A value carried over from existing rows is fixed before the commit and may go to a data file. Compaction is the only such writer here.

Behavior

• Read: load_row_id_sequence gains a Column arm; a new load_row_version_sequence loads either version sequence wherever it is stored, with spilled ones cached per fragment and file. FileFragment::open now loads the version sequences asynchronously alongside the row ids instead of decoding them synchronously in the reader builder. Cleanup, shallow-clone base_id rewriting, file listing and Dataset::validate reach the spilled file through Fragment::referenced_lance_files.
• Write: compaction rechunks all three sequences, then places each one inline or in one shared lineage file per fragment, per sequence, when its encoding exceeds the inline budget. Spilling is opt-in per table: lance.row_lineage.spill=true, with lance.row_lineage.inline_max_bytes overriding the 200 KiB default. A table that never sets it is unchanged.
• Commit: the two commit-time paths in lance-table that read existing lineage (resolve_update_version_metadata for row rewrites, refresh_row_latest_update_meta_for_partial_frag_rewrite_cols for partial column rewrites) cannot read a data file. The commit path in lance now reads every spilled sequence of the current manifest ahead of each build attempt (load_spilled_row_lineage, cached per fragment and file) and hands them over in ManifestBuildConfig::spilled_row_lineage, so UpdateBuilder, merge_insert in both write modes, and externally assembled Operation::Updates keep every row's lineage on a spilled table exactly as on an inline one. A build that needs a spilled sequence it was not given returns NotSupported instead of the prototype's warn-and-default, which silently rewrote lineage.
• A new fragment that arrives with spilled row ids keeps the created-at versions its writer placed and gets its last-updated-at stamped with the commit version, which is the contract a writer that spills at update time needs: it owns the values a conflict retry cannot change, the commit owns the one it can.
• RowDatasetVersionMeta::load_sequence returns NotSupported for the external and column arms instead of todo!().

Known gaps

• The updaters in this crate place a new fragment's lineage inline, so an update-heavy workload regrows the manifest between compactions; the next compaction spills it again. A writer that wants the manifest flat under such a workload builds the fragment's lineage with place_row_lineage and commits it with created-at placed, as the contract above allows.
• Compaction writes the lineage columns to a separate file. Writing them into the fragment's main data file, which the format now allows, needs the write path to accept hidden columns (Schema::validate rejects negative ids) and is a follow-up.
• Only the compaction write path spills. Binary-copy compaction takes the same separate-file path.
• FileFragment::open still loads the row id sequence eagerly on stable-row-id tables; with a spilled sequence that is one file read per fragment per process before the cache is warm.

Benchmarks

LANCE_ENABLE_UNSTABLE_SPILLED_ROW_LINEAGE=1 cargo bench --bench rowid_spill, the benchmark from #8953 extended to this branch: 8 fragments x 1,000,000 rows, local NVMe (macOS), one run per arm, measured on this branch just before its rebase onto #9136 (the rebase only moved the feature flag bit and touched docs). The two arms differ only in the table config: the inline arm never opts in (today's behavior), the spilled arm sets lance.row_lineage.spill=true with the default 200 KiB budget. Byte counts are deterministic; latencies are single samples (cold open averaged over 10, sequence loads over 3), so treat differences under about 2x as noise. Lower is better for every row.

deleted: 30% of rows deleted, then compacted. The row id sequences still run-encode as range plus bitmap, so this is the workload where spilling is a bad trade on bytes.

Scenario / metric	Baseline (inline)	This PR (spilled)	Benefit
manifest size	5.73 MiB	< 0.01 MiB	>500x smaller
compaction transaction file	2.86 MiB	< 0.01 MiB	>500x smaller
cold dataset open	1.45 ms	0.55 ms	2.6x speedup
append commit (mean of 5)	3.65 ms	1.46 ms	2.5x speedup
load one fragment's sequence (cold)	0.15 ms	4.88 ms	33x slowdown
row id index build (cold)	0.11 ms	16.34 ms	150x slowdown
take by row id, index built	0.54 ms	4.55 ms	8x slowdown (see below)
compaction	221 ms	258 ms	1.2x slowdown
data files on disk	74.4 MiB	89.2 MiB	1.2x larger

shuffled: every row rewritten in random order, then compacted. No run structure survives, which is the workload the design is for.

Scenario / metric	Baseline (inline)	This PR (spilled)	Benefit
manifest size	30.52 MiB	< 0.01 MiB	>3000x smaller
compaction transaction file	61.04 MiB	30.52 MiB	2x smaller
cold dataset open	5.54 ms	0.20 ms	28x speedup
append commit (mean of 5)	15.73 ms	1.34 ms	12x speedup
load one fragment's sequence (cold)	2.35 ms	2.30 ms	1.0x
row id index build (cold)	142 ms	165 ms	1.2x slowdown
take by row id, index built	2.65 ms	2.75 ms	1.0x
compaction	305 ms	296 ms	1.0x
data files on disk	136.0 MiB	158.1 MiB	1.2x larger

The 30.52 MiB transaction file in the spilled arm is the old fragments' inline sequences that the setup rewrite created; the new fragments contribute only file references. The version sequences are zero in both arms because the benchmark's rewrite leaves version metadata unset.

The deleted take row: FileFragment::open loads the fragment's row id sequence whenever the table uses stable row ids, whether or not _rowid is projected, and the index build reads sequences uncached, so the first take after it pays one spilled-file read per fragment. That eager load predates this PR and is nearly free for inline sequences; making it conditional on the projection is a follow-up.

Validation

• cargo test -p lance-table
• cargo test -p lance --lib -- rowid row_version stable_row optimize::tests dataset_transactions fragment::tests cleanup::tests feature_flag update::tests merge_insert::tests dataset_io mem_wal
• cargo clippy --all --tests --benches -- -D warnings, cargo fmt --all
• cargo check --manifest-path python/Cargo.toml, cargo check --manifest-path java/lance-jni/Cargo.toml

Refs #8931, closes #9250

🤖 Generated with Claude Code