feat(core): define mapping readers with ordered compaction support - #9106
Merged
Merged
Conversation
This was referenced Sep 9, 2026
LuQQiu
added this pull request to stack #9108
September 9, 2026 19:16
LuQQiu
removed this pull request from stack #9108
September 10, 2026 03:15
LuQQiu
added this pull request to stack #9117
September 10, 2026 03:15
LuQQiu
removed this pull request from stack #9117
September 10, 2026 18:17
LuQQiu
added this pull request to stack #9137
September 10, 2026 18:17
This was referenced Sep 10, 2026
LuQQiu
marked this pull request as ready for review
September 10, 2026 18:21
LuQQiu
added a commit
that referenced
this pull request
Sep 15, 2026
FRI keeps existing indices usable after fragment rewrites by translating old physical row addresses to new ones. Today it supports order-preserving compaction. This proposal extends the same system index to support stable partitioning, with both mapping types sharing one fragment-lineage history. Following [the unified FRI proposal](#8972 (comment)), source/destination lineage stays in FRI details, while large mapping payloads remain in separate immutable files. ### One history, multiple mappings Continue storing FRI information in a single `__lance_frag_reuse` system-index entry. Keep the existing `InlineContent` / `ExternalFile` envelope and the original field number for legacy versions. Add tagged transitions alongside them: ```text FragmentReuseIndexDetails └── InlineContent, stored inline or in external details.binpb ├── legacy_versions[] └── transitions[] ├── ordered sources[] ├── ordered destinations[] └── mapping ├── OrderedCompaction: surviving-row bitmap └── StablePartition: immutable row-map reference ``` Sources and destinations define the common rewrite graph. Each mapping defines how to translate row offsets. Legacy groups can be read as ordered-compaction transitions; mixed histories follow fragment lineage, not the order of records or dataset version numbers. ### Lightweight metadata, external row maps Ordered compaction retains its compact bitmap representation. Stable partition assigns each physical source row a nullable `uint16` destination label, preserving source order within each destination. A null label means the row was deleted. A counts matrix lets readers reconstruct destination offsets without reading all preceding labels. Stable-partition metadata records `map_id`, `map_size_bytes`, and optional `base_id`. The labels and counts are stored in `_fri/<map_id>/stable_partition.lance`. Mapping identity is independent of the FRI index UUID: updating the history rewrites its metadata, but does not rewrite existing row-map files. The history can be opened without loading labels; address translation reads the required blocks. ### Publication and compatibility `AppendFragmentReuseTransitions` expresses a transition delta. Combined atomically with a fragment rewrite, it lets the commit apply the delta to the current history and publish destination fragments and their mappings together. The persisted FRI details remain a snapshot of that history. - **Index version 0:** existing compaction format and read/write behavior remain unchanged. - **Index version 1:** supports legacy groups and tagged transitions in one history. - The first commit publishing index version 1 sets reader and writer flag **512**. The reader flag prevents old clients from partially interpreting the history; the writer flag prevents them from dropping mappings during metadata maintenance. Subsequent manifests retain both bits. ### Scope and validation This PR contains protobuf definitions, the corresponding format documentation, the proposed flag constant, and minimal compile adapters. It does not enable tagged-history reads or writes. Mapping implementations and reader integration follow in #9106 → #9064 → #9067 → #9068 → #9107. Replaces #9065 as the standalone spec at the bottom of native stack #9137, based on main `31d78d170`. `cargo fmt --all` and whitespace checks pass. Clippy and tests are blocked by dependency resolution: main requires `object_store_opendal 0.60.1`, while the crates.io index currently offers only up to 0.60.0. --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
LuQQiu
force-pushed
the
lu/fri-mapping-reader
branch
from
September 15, 2026 18:46
8810c5b to
fd60a5e
Compare
Contributor
There was a problem hiding this comment.
✅ Gate recommendation: approve.
The mapping seam remains appropriately scoped: ordered compaction preserves the existing bitmap/rank behavior, while the async contract lets external row-map implementations override batch reads for efficient I/O. The tests cover deletion, ordering, duplicates, invalid addresses, and single/batch equivalence.
With #9136 now merged, the intended dependency order is satisfied.
LuQQiu
added a commit
that referenced
this pull request
Sep 16, 2026
## What this adds A stable-partition rewrite reads source fragments in scan order and distributes their live rows across an ordered list of destination fragments. Row order is preserved within each destination, but the destination cannot be inferred from source row order as it can for ordered compaction. This PR implements the mapping as one immutable Lance file and exposes it through the `MappingReader` interface introduced in #9106. It does not integrate dataset lineage traversal or change the legacy compaction reader/writer. ## Encoding The row-map file contains one entry for every physical source row, ordered by source fragment and then row offset: ```text stable_partition.lance ├── label: nullable UInt16 │ ├── 0..N-1 → position in the ordered destination list │ └── null → source row was deleted └── global buffer: cumulative counts matrix ``` For example, a label of `2` selects `destinations[2]`; it is not a fragment ID or destination row offset. The writer receives labels for live rows in source scan order and inserts nulls from the source deletion vectors. This preserves one file row per physical source row, including rows already deleted when the rewrite ran. The counts matrix divides the label column into 65,536-row blocks. Its on-disk layout is: ```text 28-byte header ├── magic: "LSPC" ├── version: u32 ├── representation: u32 (0 = dense) ├── num_destinations: u32 ├── rows_per_block: u32 └── total_source_rows: u64 dense grid └── cumulative u32 count for each (block, destination), block-major ``` The Lance schema metadata key `lance:stable_partition:counts_buffer_index` identifies this global buffer. ## Address translation For source row `(fragment_id, row_offset)`, the reader first converts it to its position in the concatenated source layout. A null label returns a deleted row. Otherwise: ```text destination fragment = destinations[label] destination offset = count(label before this block) + rank(label earlier in this block) ``` Because each destination is written in source scan order, this reconstructs its physical destination offset without storing a complete source-to-destination address table. Reordering rows within a destination is outside this encoding and requires a different mapping type. Opening the mapping reads and validates the counts metadata without reading labels. Point and batch translations lazily read only the touched label blocks; batch requests sweep each touched block once. ## Validation The reader validates the label schema, counts header and dimensions, monotonic cumulative counts, source and destination row totals, label bounds, and per-block label/count agreement. Validated with the stable-partition row-map and mapping-reader tests, workspace Clippy, and `cargo fmt --all`. --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this adds
This PR defines the common read contract for one fragment-rewrite mapping:
A row ID is a physical row address containing a fragment ID and row offset. A mapped row ID identifies the rewritten destination row;
Nonemeans the source row was deleted.remap_row_idis the required primitive. The default batch implementation calls it for each input in order. Mapping implementations can override the batch method to coalesce IO or translate more efficiently.The contract requires batch results to preserve input order and duplicates and return one result per input. Addresses outside the mapping's source fragments return
InvalidInput; corrupt persisted mappings returnCorruptFile; storage errors propagate unchanged.Ordered compaction
OrderedCompactionMappingadapts the existingRowAddrRemapbitmap/rank implementation to this contract:It reuses the existing compaction mapping and validated fragment layout without duplicating source or destination metadata. The legacy FRI reader and writer remain unchanged.
This interface represents one mapping only. FRI lineage traversal, coverage derivation, and planner integration remain responsibilities of the common FRI reader. Stable-partition mappings implement the same interface in #9064.
Validation
Tests cover default batch delegation and error propagation, deleted rows, duplicate and out-of-order inputs, empty batches, single/batch equivalence, invalid source fragments, and out-of-range source offsets.
Validated with workspace Clippy and
cargo fmt --all.