perf(index): cut fixed and per-partition overhead of IVF search

[xiaguanglei]

perf(index): cut fixed and per-partition overhead of IVF search

Summary

IVF queries pay fixed per-query and per-partition overhead — re-quantizing the
RaBitQ FastScan LUT, hashing partition cache keys, and rebuilding the partition
reconstruction — on every probe. This PR reuses per-query RaBitQ state and
memoizes partition-cache work, cutting query latency by up to −18% on SIFT-1M
with no recall change and no on-disk format change.

Why this helps

A probe-heavy IVF query touches nprobes partitions, so any per-partition or
per-query fixed cost is paid nprobes times while contributing nothing to the
actual scan. This PR removes three such costs by doing the work once and
reusing it:

• The raw-query FastScan LUT is centroid-independent, so it was being
  re-quantized once per partition. Memoizing it per query turns nprobes
  redundant quantizations into one.
• Partition cache keys were re-hashed (BLAKE3) on every probe. Precomputing
  namespace-tagged digests once at open removes the hash from the per-probe
  path.
• The whole Query (including its key array) was cloned per partition.
  Sharing it via Arc — with only the scalar dist_q_c differing per
  partition — removes nprobes array clones.

Each reuse removes an O(1) cost multiplied by nprobes, which accumulates
into the measured up-to-−18% latency reduction on SIFT-1M.

Changes

• Reuse the raw-query context across probed partitions. The raw-query
  dist_table is centroid-independent, so its FastScan LUT was being
  re-quantized once per partition. The LUTs (normal_lut / accurate_lut) are
  now memoized on RabitRawQueryContext and built lazily on first use.
  rust/lance-index/src/vector/storage.rs:314-384
• Precompute namespace-tagged BLAKE3 digests for partition cache keys,
  skipping a digest computation per probe. resolve() rejects digests from a
  foreign namespace and re-hashes, so cross-namespace reuse stays correct.
  rust/lance-core/src/cache/key.rs:136-165 ·
  rust/lance/src/index/vector/ivf/v2.rs:959-1055
• Memoize IVF partition reconstruction in its own capacity-accounted cache
  entry, keyed by object store + fragment reuse index identity, so identical
  reopens reuse the built index instead of rebuilding it. The live index
  (readers, object store, remapper, RaBitQ scratch pools) is inserted as a
  dedicated CachedReconstructedIndex entry whose weight is computed at insert
  time — rather than being mutated into the already-inserted IvfIndexState,
  which would silently grow past the index cache's memory limit without
  increasing its eviction weight.
  rust/lance/src/index/vector/ivf/v2.rs:109-150,2813-2901
• Share Query via Arc across probed partitions and add a parallel
  per-partition dist_q_c field on PreparedPartitionSearch, stopping a full
  Query clone per partition. IvfSubIndex::query_params keeps its default
  impl (no forced API break); FlatIndex / HNSW override it to avoid the
  clone.
  rust/lance/src/index/vector/ivf/v2.rs:468-479 ·
  rust/lance-index/src/vector/v3/subindex.rs:51 ·
  rust/lance-index/src/vector/flat/index.rs:82-103 ·
  rust/lance-index/src/vector/hnsw/builder.rs:1423-1446
• Harden partition_cache_key with a debug_assert! bounds guard for the
  precomputed-key array. rust/lance/src/index/vector/ivf/v2.rs:1054-1060

API note

RabitRawQueryContext is a pub type (in pub mod vector::storage, returned
by RabitQuantizationStorage::prepare_raw_query_context) that shipped in
v12.0.0-beta.18. This PR adds private normal_lut / accurate_lut memo
fields and marks it #[doc(hidden)] #[non_exhaustive] with construction going
through RabitRawQueryContext::new.

This is an internal-plumbing-type closure, not a public API change: it is a
12.0.0-beta line (no stable API contract), the six pub fields are all RaBitQ
internals no downstream crate can meaningfully construct, and closing the type
now is cheaper than after a stable release.

Performance

IVF_RQ on SIFT-1M (x86 Skylake, nlist=256, RQ5, warm, AVX-512, median-of-medians):

nprobes	latency	recall@10
8	−6.6%	0.894
32	−17.8%	0.942
128	−15.6%	0.943
256	−14.1%	0.943

Result hashes are bit-identical and recall is unchanged across the matrix. The
win concentrates at low nprobes and decays as the partition scan begins to
dominate, consistent with removing fixed per-query/per-partition costs.

The shared IVF paths (coarse select + reconstruct/cache) carry the win beyond
RQ (SIFT-1M, nlist=256, same protocol — re-tested on the same dataset):

Index	np8	np32	np128
IVF_PQ (m=16)	−11.9%	−11.6%	−18.0%
IVF_FLAT	−16.6%	−5.7%	−4.2%
IVF_SQ	−6.0%	−11.4%	−6.4%
IVF_HNSW_PQ	−18.2%	−16.1%	−8.7%
IVF_HNSW_FLAT	−16.5%	−18.9%	−5.1%

The gain holds both cold and warm, and index build is unaffected (the
wall-clock prepare delta sits within the measurement floor).

Methodology

All end-to-end measurements are interleaved A/B on the same host, driven by
identical dataset/query/ground-truth paths, so the two binaries experience the
same conditions within each cell. Everything below is reproducible from public
data and a small in-tree-toolchain harness.

• Dataset: SIFT-1M — the industry-standard
  ANN benchmark (sift.tar.gz, 168 MB). 1M × 128-d f32 base
  (sift_base.fvecs), the official 10k-query set (sift_query.fvecs), and the
  official exact top-100 ground truth (sift_groundtruth.ivecs) — no synthetic
  data, no self-computed ground truth.
• Index: IVF nlist=256, RaBitQ rq-bits=5, fast rotation, metric=l2.
• Host: x86 Skylake (AVX-512 runtime dispatch active), release-with-debug.
• Protocol: BCCBBCCB interleaving (4 baseline + 4 candidate runs per
  nprobes); each run = 6 repetitions × 200 queries after 3 warm-up rounds,
  concurrency=1. Reported latency is the median of the four run p50s per
  variant. Recall and result hashes are checked for identity against baseline on
  every cell (recall-neutral, bit-identical results).
• Base / candidate: 577091e53 vs ed799ad7a (branch
  index/perf-index-ivf-search-overhead).

The harness converts the fvecs/ivecs to Lance, builds the index, and runs the
search under the above protocol.

Behavior

• No on-disk format change. partition_cache_keys is runtime-only and excluded
  from the CacheCodec wire format; the reconstructed index is a separate
  in-memory cache entry (no codec), so it is never serialized.
• Partition selection remains byte-for-byte the upstream sort_to_indices
  path — no behavioral change.

Test plan

• cargo fmt --all
• cargo clippy --all --tests --benches -- -D warnings
• cargo test -p lance-core -p lance-index -p lance (0 failed)
• precomputed physical key + foreign-namespace ignore tests
• test_reconstructed_index_cache_key_distinguishes_inputs (reconstructed-
  index cache-key reuse rules)
• test_vector_cache_uses_current_object_store asserts the reconstructed
  index is held as a capacity-accounted entry (cache weight >= its deep size)

[lance-gatekeeper]

✅ Gate recommendation: approve.

The rebased IVF patch preserves the previously reviewed cache accounting and query behavior. Focused RaBitQ, reconstructed-index cache, and KNN ordering regressions pass on the new base.

Please mark this PR with the breaking-change label.