Skip to content

feat(core): define mapping readers with ordered compaction support - #9106

Merged
LuQQiu merged 4 commits into
mainfrom
lu/fri-mapping-reader
Sep 16, 2026
Merged

LuQQiu merged 4 commits into
mainfrom
lu/fri-mapping-reader

Conversation

@LuQQiu

@LuQQiu LuQQiu commented Sep 9, 2026 •

Copy link
Copy Markdown
Contributor

What this adds

This PR defines the common read contract for one fragment-rewrite mapping:

async fn remap_row_id(&self, row_id: u64) -> Result<Option<u64>>;

async fn remap_row_ids(
    &self,
    row_ids: &[u64],
) -> Result<Vec<Option<u64>>>;

A row ID is a physical row address containing a fragment ID and row offset. A mapped row ID identifies the rewritten destination row; None means the source row was deleted.

remap_row_id is the required primitive. The default batch implementation calls it for each input in order. Mapping implementations can override the batch method to coalesce IO or translate more efficiently.

The contract requires batch results to preserve input order and duplicates and return one result per input. Addresses outside the mapping's source fragments return InvalidInput; corrupt persisted mappings return CorruptFile; storage errors propagate unchanged.

Ordered compaction

OrderedCompactionMapping adapts the existing RowAddrRemap bitmap/rank implementation to this contract:

source physical row ID
        │
        ▼
existing RowAddrRemap
        │
        ├── destination physical row ID
        └── None when deleted

It reuses the existing compaction mapping and validated fragment layout without duplicating source or destination metadata. The legacy FRI reader and writer remain unchanged.

This interface represents one mapping only. FRI lineage traversal, coverage derivation, and planner integration remain responsibilities of the common FRI reader. Stable-partition mappings implement the same interface in #9064.

Validation

Tests cover default batch delegation and error propagation, deleted rows, duplicate and out-of-order inputs, empty batches, single/batch equivalence, invalid source fragments, and out-of-range source offsets.

Validated with workspace Clippy and cargo fmt --all.

@github-actions github-actions Bot added the enhancement New feature or request label Sep 9, 2026
@LuQQiu
LuQQiu added this pull request to stack #9108 September 9, 2026 19:16
@LuQQiu
LuQQiu removed this pull request from stack #9108 September 10, 2026 03:15
@LuQQiu
LuQQiu added this pull request to stack #9117 September 10, 2026 03:15
@LuQQiu
LuQQiu removed this pull request from stack #9117 September 10, 2026 18:17
@LuQQiu
LuQQiu changed the base branch from main to lu/fri-spec-first September 10, 2026 18:17
@LuQQiu
LuQQiu added this pull request to stack #9137 September 10, 2026 18:17
@LuQQiu
LuQQiu marked this pull request as ready for review September 10, 2026 18:21
lance-gatekeeper[bot]

This comment was marked as outdated.

@lance-gatekeeper lance-gatekeeper Bot added the K-approved Latest Gatekeeper recommendation permits acceptance. label Sep 10, 2026
Base automatically changed from lu/fri-spec-first to main September 15, 2026 18:46
LuQQiu added a commit that referenced this pull request Sep 15, 2026
FRI keeps existing indices usable after fragment rewrites by translating
old physical row addresses to new ones. Today it supports
order-preserving compaction. This proposal extends the same system index
to support stable partitioning, with both mapping types sharing one
fragment-lineage history.

Following [the unified FRI
proposal](#8972 (comment)),
source/destination lineage stays in FRI details, while large mapping
payloads remain in separate immutable files.

### One history, multiple mappings

Continue storing FRI information in a single `__lance_frag_reuse`
system-index entry. Keep the existing `InlineContent` / `ExternalFile`
envelope and the original field number for legacy versions. Add tagged
transitions alongside them:

```text
FragmentReuseIndexDetails
└── InlineContent, stored inline or in external details.binpb
    ├── legacy_versions[]
    └── transitions[]
        ├── ordered sources[]
        ├── ordered destinations[]
        └── mapping
            ├── OrderedCompaction: surviving-row bitmap
            └── StablePartition: immutable row-map reference
```

Sources and destinations define the common rewrite graph. Each mapping
defines how to translate row offsets. Legacy groups can be read as
ordered-compaction transitions; mixed histories follow fragment lineage,
not the order of records or dataset version numbers.

### Lightweight metadata, external row maps

Ordered compaction retains its compact bitmap representation. Stable
partition assigns each physical source row a nullable `uint16`
destination label, preserving source order within each destination. A
null label means the row was deleted. A counts matrix lets readers
reconstruct destination offsets without reading all preceding labels.

Stable-partition metadata records `map_id`, `map_size_bytes`, and
optional `base_id`. The labels and counts are stored in
`_fri/<map_id>/stable_partition.lance`. Mapping identity is independent
of the FRI index UUID: updating the history rewrites its metadata, but
does not rewrite existing row-map files. The history can be opened
without loading labels; address translation reads the required blocks.

### Publication and compatibility

`AppendFragmentReuseTransitions` expresses a transition delta. Combined
atomically with a fragment rewrite, it lets the commit apply the delta
to the current history and publish destination fragments and their
mappings together. The persisted FRI details remain a snapshot of that
history.

- **Index version 0:** existing compaction format and read/write
behavior remain unchanged.
- **Index version 1:** supports legacy groups and tagged transitions in
one history.
- The first commit publishing index version 1 sets reader and writer
flag **512**. The reader flag prevents old clients from partially
interpreting the history; the writer flag prevents them from dropping
mappings during metadata maintenance. Subsequent manifests retain both
bits.

### Scope and validation

This PR contains protobuf definitions, the corresponding format
documentation, the proposed flag constant, and minimal compile adapters.
It does not enable tagged-history reads or writes. Mapping
implementations and reader integration follow in #9106 → #9064 → #9067 →
#9068 → #9107.

Replaces #9065 as the standalone spec at the bottom of native stack
#9137, based on main `31d78d170`.

`cargo fmt --all` and whitespace checks pass. Clippy and tests are
blocked by dependency resolution: main requires `object_store_opendal
0.60.1`, while the crates.io index currently offers only up to 0.60.0.

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
@LuQQiu
LuQQiu force-pushed the lu/fri-mapping-reader branch from 8810c5b to fd60a5e Compare September 15, 2026 18:46
@lance-gatekeeper lance-gatekeeper Bot removed the K-approved Latest Gatekeeper recommendation permits acceptance. label Sep 15, 2026

@lance-gatekeeper lance-gatekeeper Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Gate recommendation: approve.

The mapping seam remains appropriately scoped: ordered compaction preserves the existing bitmap/rank behavior, while the async contract lets external row-map implementations override batch reads for efficient I/O. The tests cover deletion, ordering, duplicates, invalid addresses, and single/batch equivalence.

With #9136 now merged, the intended dependency order is satisfied.

@lance-gatekeeper lance-gatekeeper Bot added K-approved Latest Gatekeeper recommendation permits acceptance. and removed K-approved Latest Gatekeeper recommendation permits acceptance. labels Sep 15, 2026

@jackye1995 jackye1995 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks good to me!

@LuQQiu
LuQQiu merged commit c7c944e into main Sep 16, 2026
43 checks passed
@LuQQiu
LuQQiu deleted the lu/fri-mapping-reader branch September 16, 2026 01:58
LuQQiu added a commit that referenced this pull request Sep 16, 2026
## What this adds

A stable-partition rewrite reads source fragments in scan order and
distributes their live rows across an ordered list of destination
fragments. Row order is preserved within each destination, but the
destination cannot be inferred from source row order as it can for
ordered compaction.

This PR implements the mapping as one immutable Lance file and exposes
it through the `MappingReader` interface introduced in #9106. It does
not integrate dataset lineage traversal or change the legacy compaction
reader/writer.

## Encoding

The row-map file contains one entry for every physical source row,
ordered by source fragment and then row offset:

```text
stable_partition.lance
├── label: nullable UInt16
│   ├── 0..N-1 → position in the ordered destination list
│   └── null   → source row was deleted
└── global buffer: cumulative counts matrix
```

For example, a label of `2` selects `destinations[2]`; it is not a
fragment ID or destination row offset.

The writer receives labels for live rows in source scan order and
inserts nulls from the source deletion vectors. This preserves one file
row per physical source row, including rows already deleted when the
rewrite ran.

The counts matrix divides the label column into 65,536-row blocks. Its
on-disk layout is:

```text
28-byte header
├── magic:                "LSPC"
├── version:              u32
├── representation:       u32 (0 = dense)
├── num_destinations:     u32
├── rows_per_block:       u32
└── total_source_rows:    u64

dense grid
└── cumulative u32 count for each (block, destination), block-major
```

The Lance schema metadata key
`lance:stable_partition:counts_buffer_index` identifies this global
buffer.

## Address translation

For source row `(fragment_id, row_offset)`, the reader first converts it
to its position in the concatenated source layout. A null label returns
a deleted row. Otherwise:

```text
destination fragment = destinations[label]
destination offset   = count(label before this block)
                     + rank(label earlier in this block)
```

Because each destination is written in source scan order, this
reconstructs its physical destination offset without storing a complete
source-to-destination address table. Reordering rows within a
destination is outside this encoding and requires a different mapping
type.

Opening the mapping reads and validates the counts metadata without
reading labels. Point and batch translations lazily read only the
touched label blocks; batch requests sweep each touched block once.

## Validation

The reader validates the label schema, counts header and dimensions,
monotonic cumulative counts, source and destination row totals, label
bounds, and per-block label/count agreement.

Validated with the stable-partition row-map and mapping-reader tests,
workspace Clippy, and `cargo fmt --all`.

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request K-approved Latest Gatekeeper recommendation permits acceptance.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants