feat(index): remap indexes through fully covered fragment-reuse transitions

[LuQQiu]

Summary

Rewrites index row addresses through FRI mappings so eligible segments can stop translating at query time. Once indexes catch up, #9161's cleanup can retire the unused history.

Builds on #9162. No protobuf or file-format changes.

What changes and why

• Full-coverage, multi-hop remapping. Supports stable-partition and ordered-compaction chains. Stable partition requires every source contribution before a segment can claim the destination fragments; partial coverage is left unchanged.
• Streaming index rewrites. Built-in indexes translate addresses in batches of at most 64K instead of building a whole-segment address HashMap. Withdrawn and excluded rows are removed so the replacement file matches its declared coverage.
• Legacy-upgrade handling. Recover source provenance when a legacy segment's bitmap already names destinations but its files still contain old addresses. This prevents row loss during subsequent remap and trim.
• Compatible API extension. Keep existing remap methods and add remap_streaming. Older plugins can use a complete, budget-checked materialized fallback; unsupported or over-budget cases skip the segment and retain its history. I/O and corruption errors still fail.

Limits and tradeoffs

No partial stable-partition remapping. Translation buffers are bounded, but builders retain their own working sets. Remapping does not itself delete mapping files.

Rebuild remains an alternative: the recorded 100M-row IVF_RQ/PQ benchmarks found remap roughly 3.2–3.9x slower than rebuild.

Tests

Covers mixed chains, coverage withdrawal, legacy upgrades, direct inspection of remapped postings, plugin fallback, and query results against scans. Windows CI verifies this head preserves #9161's stack fix.

[github-actions]

ACTION NEEDED
Lance follows the Conventional Commits specification for release automation.

The PR title and description are used as the merge commit message. Please update your PR title and description to match the specification.

For details on the error please inspect the "PR Title Check" action.

[lance-gatekeeper]

❌ Gate recommendation: request changes.

The full-coverage eligibility and coverage-restamping model is coherent, but the stable-partition implementation reintroduces the row-count-proportional remap allocation that #7237 removed after #7150 demonstrated OOM at scale. Keep this path compact or streamed, or retain the safe skip-and-rebuild fallback when a mapping exceeds a bounded memory budget; rebuilding is already the faster measured alternative for the vector index types in this PR.

[lance-gatekeeper]

This materialization holds both one address in addrs and one HashMap entry for every physical source row, then retains the map throughout the index rewrite. That restores the large-table remap OOM eliminated by #7237: #7150 measured the same per-row map at 28 GB for 500M rows and 56 GB for 1B rows, including an OOM-killed billion-row run. A full-table stable partition with a fully covering index—the workload benchmarked in this PR—takes this branch with no guard or fallback, so the maintenance worker can die before any commit.

Please keep stable-partition lookup compact or stream/batch it into index remapping. If that is not available yet, preserve the previous skip behavior and direct oversized cases to the already-faster rebuild path.

Bounded allocation reproduction run on this head

use std::{collections::HashMap, fs};

fn rss_kib() -> u64 {
    fs::read_to_string("/proc/self/status").unwrap().lines()
        .find_map(|line| line.strip_prefix("VmRSS:")?.split_whitespace().next()?.parse().ok())
        .unwrap()
}

fn main() {
    const ROWS: u64 = 5_000_000;
    let baseline = rss_kib();
    let addrs: Vec<u64> = (0..ROWS).collect();
    let after_addrs = rss_kib();
    let mut map = HashMap::with_capacity(addrs.len());
    map.extend(addrs.iter().copied().map(|addr| (addr, Some(addr + 1))));
    let after_map = rss_kib();
    println!("vec={:.1} MiB, combined={:.1} MiB",
        (after_addrs - baseline) as f64 / 1024.0,
        (after_map - baseline) as f64 / 1024.0);
    std::hint::black_box((addrs, map));
}

Run with rustc -O repro.rs -o repro && ./repro.

Expected: no row-count-proportional remap structure, matching the compact-remap contract established by #7237.

Observed: vec=39.0 MiB, combined=239.0 MiB for only 5M rows; the allocation grows linearly before index-rewrite memory.

[lance-gatekeeper]

The full-source map is gone for the other rewrite paths, but NGram still expands a compressed posting batch into an unbounded per-row vector and map. The current-head finding keeps this OOM risk actionable until that path is bounded or falls back to rebuild.

[lance-gatekeeper]

Fixed in 3e237f2: the source-row allocation was already removed by the batch translator, and this change bounds the remaining dense NGram posting path to 64K-address slices. The focused NGram regression passes on this head.

[lance-gatekeeper]

❌ Gate recommendation: request changes.

The retained-lineage admission preserves a merged segment's already-translated rows through later hops, and the new and prior withdrawal regressions pass. The NGram remap finding remains: a dense posting still expands into a row-count-sized vector and hash map before the translator's 64K batches, risking a maintenance-worker OOM before commit. Bound posting translation, or leave NGram segments unchanged for rebuild until it is bounded. #7237 removed the analogous compaction allocation after #7150 documented scale OOM.

Please mark this PR with the breaking-change label.

[lance-gatekeeper]

❌ Gate recommendation: request changes.

The six Box::pin changes affect only tests; the NGram remap finding remains. resolve still expands a dense posting into a per-row vector and hash map before the 64K translation batches, so maintenance can OOM before publishing the replacement. Translate postings in bounded chunks, or skip oversized remaps and rebuild the segment. #7237 removed the analogous allocation after #7150 documented scale OOM.

Please mark this PR with the breaking-change label.

[lance-gatekeeper]

✅ Gate recommendation: approve.

The bounded NGram posting translation closes the remaining remap-memory finding: it no longer builds a whole-posting address map before the 64K batch translator. The coverage-plan check leaves unservable segments untouched for scan fallback, while eligible full-coverage segments still remap and restamp. The focused regressions for these paths pass.

Please mark this PR with the breaking-change label.

[lance-gatekeeper]

⚠️ Gate recommendation: approve with a non-blocking risk.

Full-coverage remapping and the restored legacy API remain sound: built-in indexes stream, and the focused compatibility and tagged-remap regressions pass. The opt-in legacy fallback can exceed its stated 256 MiB materialization budget because HashMap rounds its allocation upward. On memory-limited workers this can abort maintenance instead of taking the skip; leaving that plugin on query-time translation or rebuilding avoids the path until the estimate is conservative.

[lance-gatekeeper]

The 32-byte estimate misses hash-table bucket rounding, so the legacy fallback can exceed its stated 256 MiB cap and abort instead of returning RemapUnavailable::OverBudget. For 7,350,000 fully moved addresses, this branch estimates 225.8 MiB, but HashMap::with_capacity requests 419,430,416 bytes (~400 MiB). Under a 312 MiB process limit the allocation aborted (exit 134) in a probe on this head. This is limited to legacy-only indexes that opt into stored_fragments; the built-ins stream. Accounting conservatively for rounded buckets would make the skip reliable.

Reproducer run on this head

use std::collections::HashMap;
use std::fs;
use std::hint::black_box;

fn rss_kib() -> u64 {
    fs::read_to_string("/proc/self/status").unwrap().lines()
        .find_map(|line| line.strip_prefix("VmRSS:")?.split_whitespace().next()?.parse().ok())
        .unwrap()
}

fn main() {
    let rows = 7_350_000usize;
    let estimated = rows as u64 * 32 + 65_536 * 24;
    let before = rss_kib();
    let mut map: HashMap<u64, Option<u64>> = HashMap::with_capacity(rows);
    let actual_capacity = map.capacity();
    for id in 0..rows as u64 {
        map.insert(id, Some(id + 1));
    }
    let after = rss_kib();
    println!("rows={rows} estimate_mib={:.1} actual_capacity={actual_capacity} rss_delta_mib={:.1}",
        estimated as f64 / 1024.0 / 1024.0,
        (after - before) as f64 / 1024.0);
    black_box(map);
}

rustc -O probe.rs -o probe && ./probe printed estimate_mib=225.8 actual_capacity=14680064 rss_delta_mib=400.6. Then bash -c 'ulimit -v 320000; ./probe' aborted while allocating 419,430,416 bytes. Expected: a budget overrun returns OverBudget before allocating.

[lance-gatekeeper]

Fixed in a366b9f: the fallback now accounts for rounded hash-table buckets before allocation. The 7.35M-row case is estimated at 419,430,416 bytes and returns OverBudget under the default 256 MiB budget before translation; the focused regression passes on this head.

[lance-gatekeeper]

⚠️ Gate recommendation: approve with a non-blocking risk.

The rounded-bucket fallback fix closes the previously reported budget overrun. The v0 deferred-remap path also carries source-address rows through later stable partitions and cleanup; the focused remap regressions pass.

Legacy-only plugins using the compatibility fallback may still allocate up to its 256 MiB budget in addition to their own working set; built-in indices stream. On tight-memory workers, leaving such segments on query-time translation or rebuilding them avoids that cost. The Windows Python-thread stack case was not exercised by these local tests.

[lance-gatekeeper]

⚠️ Gate recommendation: approve with a non-blocking risk.

This rebase leaves the remap behavior materially unchanged; the new commit only boxes test-fixture futures, and the focused remap regressions pass. The eager-remap stack boxing now lives in the preceding #9161; this PR retains it when adding the streaming entry point.

Legacy-only plugins using the compatibility fallback may still allocate up to its 256 MiB budget alongside their own working set; built-in indices stream. On tight-memory workers, leaving those segments on query-time translation or rebuilding them avoids that cost. The Windows Python-thread stack case was not exercised locally.

[LuQQiu]

The linux-build and linux-arm failures here are the four rest::tls tests in lance-namespace-impls (test_incomplete_initial_load_is_retried, test_malformed_ca_certificate_is_dropped_on_its_own, test_rotation_is_picked_up_once_the_interval_elapses::case_1_interval_elapsed, test_unusable_rotated_material_keeps_client), added by #8456; this stack does not touch that crate, and main fails the same four tests in the same two jobs on its own tip: https://github.com/lance-format/lance/actions/runs/36194697954 (2f5d4fb) and https://github.com/lance-format/lance/actions/runs/36170893444 (0aa3d53).

[lance-gatekeeper]

⚠️ Gate recommendation: approve with a non-blocking risk.

The squash/rebase preserves the previously reviewed remap patch. On the newer base, the focused tagged-remap, index-remap, core-remapping, and IVF_PQ remap regressions pass. Fully covered segments can stop query-time translation; ineligible segments retain their history.

The opt-in compatibility fallback for legacy-only plugins may use up to its 256 MiB mapping budget alongside the plugin's own working set; built-in indices stream. On tight-memory workers, leaving those segments on query-time translation or rebuilding them avoids that cost. The Windows Python-thread stack case was not exercised locally.