feat(index): support MinHash LSH scalar index - #9114
Conversation
github-actions
Bot
added
A-python
|
Important Format specification voteThis PR modifies the Lance format specification, so it requires 3 binding +1 votes from PMC members (excluding the proposer), at least one of them on the latest commit, and a minimum 72-hour voting period, weekends excluded, before it can merge. Vote by approving this PR (+1) or requesting changes (−1, a veto). See the voting process. Approvals carry over across pushes, so a rebase or a typo fix does not send everyone back to re-vote. Whoever approves the latest commit is vouching that nothing substantive has changed since the earlier approvals; if something has, ask for fresh votes. Status: ❌ Blocked — vetoed by @BubbleCal
Updated automatically by the format-spec vote gate, which re-checks every 15 minutes — just voted? Re-check now (press Run workflow; leave the input blank to re-check every open format PR). A PMC member may apply the |
![lance-gatekeeper[bot]](https://avatars.githubusercontent.com/in/4433141?s=60&v=4){kind=small}
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
lance-gatekeeper
Bot
added
K-changes
lance-gatekeeper
Bot
added
K-changes
Candidates of a MinHash LSH search are now ordered by the number of bands they share with the query. A candidate that shares m of b bands cannot be closer than (b - m) / k, so the refine step reads signatures in decreasing order of m and stops as soon as the current cutoff is no larger than the closest distance any remaining candidate could reach. The candidates that must be read regardless (enough to fill the limit, extended to the end of their shared-band group) are read first in one pass in document order; the rest follow in batches that double in size. Rows at equal distance may now come back in any order among themselves; results stay sorted by distance, then row id.
lance-gatekeeper
Bot
removed
the
K-changes
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
lance-gatekeeper
Bot
added
the
K-changes
BubbleCal
left a comment
BubbleCal
left a comment
There was a problem hiding this comment.
Both the build and query paths can materialize an entire duplicate cluster in memory. These need bounded processing before merging.
| let mut records: Vec<(u64, u32)> = | ||
| Vec::with_capacity(runs.iter().map(|run| run.rows(&partitions).len()).sum()); |
There was a problem hiding this comment.
A large duplicate cluster puts every document into the same partition within each band. Since merge_groups cannot split a partition, this allocation and the subsequent reads materialize that entire partition across all runs, regardless of LANCE_MEM_POOL_SIZE. Using the current grouping function with a target of 100 records produced a 10,000-record group. Spilling therefore does not bound peak memory for this workload, and building or merging a large duplicate cluster can OOM. The merge needs to stream records with bounded buffers, including when a single partition exceeds the budget.
There was a problem hiding this comment.
Confirmed and fixed in 429f289.
merge_groups now closes a group before it would exceed the budget, so only a single partition larger than the budget can form an oversized group, and such a group is no longer gathered: stream_merge_group reads each run's slice of the partition one chunk at a time (budget / (2 · runs) records per chunk), merges the chunk heads through a min-heap on (key, doc id), copies runs of equal keys as whole blocks, and hands batches to the bands writer through a channel that holds one batch. Peak memory of a group is therefore about half the budget plus one write batch, whatever the cluster size; groups within the budget keep the existing gather-and-sort path.
Measured with a probe of 32M identical documents (k=128, b=16): build peak RSS 14.2 GB → 1.2 GB at the same wall time, and it now follows LANCE_MEM_POOL_SIZE (256 MiB pool → 1.0 GB). A test builds a 3000-document cluster with 200-record groups and checks the streamed bands file is byte-for-byte the in-memory sort's.
| let pages = self.load_pages(&pages, metrics).await?; | ||
| let mut members: Vec<u32> = Vec::new(); | ||
| for (key, bucket_pages) in &buckets { | ||
| for page in bucket_pages.clone() { | ||
| let page = pages.get(&page).ok_or_else(|| { | ||
| Error::internal(format!("band page {page} was requested but not loaded")) | ||
| })?; | ||
| members.extend_from_slice(page.members(*key)); |
There was a problem hiding this comment.
load_pages retains an Arc for every requested page, so cache eviction cannot release those pages during candidate collection. We then copy every band's postings into members, including repeated doc ids. With 100 million identical documents and 16 bands, the page arrays alone hold about 19.2 GB and members adds at least 6.4 GB, before candidate storage and allocation overhead. Even limit=10 cannot avoid this: refinement and early stopping happen only after collection finishes. This can OOM on the duplicate-heavy workload the index targets; candidate collection needs bounded page consumption without fully expanding all postings.
There was a problem hiding this comment.
Confirmed and fixed in 429f289.
Candidate collection no longer materializes the buckets. A search now locates each bucket's exact row range from the page table and its two boundary pages, then walks the buckets together in doc id order with one cursor per band, each holding a single window of pages (one IO batch, fetched through the page cache and released as the cursor moves on). The merge yields each doc id with the number of bands it shares with the query; candidates are kept per shared-band level, at most one signature-read batch per level. The top level is refined as soon as a batch is held and the search stops once the results cannot improve, so limit=10 on a 100M-document cluster reads the first window of each bucket and one batch of signatures. A lower level that overflowed its batch is walked again from where it overflowed, against a sequential scan of the signature table when it covers much of the segment. What a search holds is one window per bucket plus one batch per level, independent of cluster size.
Measured with a probe of 32M identical documents (k=128, b=16, limit=10): 20.8 s and 8.3 GB per query → 0.55 s and 265 MB the first time (the windows enter the page cache), 9 ms afterwards. Results are unchanged: the 100M accuracy evaluation matches the previous binary line for line, and prewarmed latency on the 100M table improved for large result sets (k=10000: 17.7 → 8.9 ms; the largest cluster's root at k=10: 45.8 → 10.6 ms) while small queries are unchanged (k=10: 2.8 vs 2.7 ms).
lance-gatekeeper
Bot
removed
the
K-changes
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
lance-gatekeeper
Bot
added
the
K-changes
lance-gatekeeper
Bot
removed
the
K-changes
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
lance-gatekeeper
Bot
added
the
K-changes
…rge duplicate clusters A duplicate cluster puts all of its documents into one bucket of every band. The build merged such a partition in memory regardless of LANCE_MEM_POOL_SIZE; a search held every page of the buckets and every posting before refining. Builds now stream a merge group larger than the budget from the spill files one chunk per run, copying runs of equal keys as blocks. Searches walk the buckets together in doc id order, one window of pages per bucket through the page cache, keep one batch of candidates per shared-band level, and continue a level from where it overflowed; nothing held grows with the cluster.
lance-gatekeeper
Bot
removed
the
K-changes
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
lance-gatekeeper
Bot
added
the
K-changes
…and buckets The streaming merge gave every spill run a chunk of at least 4096 records, so its memory grew with the run count whatever LANCE_MEM_POOL_SIZE said; the chunks now split half the group budget however many runs there are, and output batches are capped at the other half. A search gave every bucket a window of one IO batch, so its memory grew with num_bands; the batch is now shared by the buckets still being walked, so a walk that outlives the others keeps reading in full batches while the total stays one batch.
lance-gatekeeper
Bot
removed
the
K-changes
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
lance-gatekeeper
Bot
added
the
K-changes
…s across levels Every spilled run kept a 65,537-entry partition map until the merge, so the maps grew with the run count, outside LANCE_MEM_POOL_SIZE (1.6 GB for 100 million identical documents at a 64 MiB budget). The maps now share a fifth of the budget: when they outgrow it, every map merges adjacent partitions, which only coarsens the merge groups. At the default budgets the spill limit is reached first, so the maps never coarsen. A search read the signatures of each shared-band level on its own, so a query needing many levels made dozens of sequential scattered reads. The candidates now go through one queue: the first read takes what the results miss, extended to the end of its level, later reads double in size, and a read is sorted as a whole before it is split into IO batches so neighbouring rows coalesce. When a bucket runs dry, every bucket running low is topped up in the same round, so walking the buckets costs one round trip per round rather than one per bucket.
lance-gatekeeper
Bot
removed
the
K-changes
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
lance-gatekeeper
Bot
added
the
K-approved
Use lance_arrow::iter_str_array (now also reading Utf8View), the memory and temporary disk settings of LanceExecutionOptions, OrderedFloat for hit ordering, and a VecDeque for the candidates waiting for their signatures.
lance-gatekeeper
Bot
removed
the
K-approved
There was a problem hiding this comment.
✅ Gate recommendation: approve.
The helper reuse leaves the accepted bounded design intact: shared execution options preserve the configured build limits, the flattened pending queue retains shared-band priority without reopening whole-cluster materialization, and the common string and float helpers preserve supported inputs and result ordering.
lance-gatekeeper
Bot
added
the
K-approved
|
Hi @BubbleCal Thanks a lot for your review. Comments are addressed. PTAL :
Thanks! |
|
Let's GO! |
@BubbleCal Thanks a lot for your help! |
2 participants
Please refer to #8820 for the full discussion
A
minhashlshscalar index returns the rows whose token shingles have the highest estimated Jaccard similarity to a query text.Text is tokenized with the full text search tokenizer, shingled, hashed and permuted into a 16-bit MinHash signature that LSH banding groups into candidates. A segment stores
signatures.lance(one signature per document) andbands.lance(fixed-width(band_key, doc_id)rows with a page table); every signature parameter lives inMinHashLshIndexDetails, whose tokenizer reusesInvertedIndexDetails. Queries read one page per band, refine the candidates by signature under the prefilter, merge segments by distance, and score rows the index does not cover on the fly unless the scan usesfast_search. Builds spill sorted runs and merge them by partition, so memory does not grow with the table; merges, updates and remaps rebuild from stored signatures.Exposed as
Scanner::minhash_searchin Rust,MinHashQueryasnearestin Python andIndexType::MinHashLshin Java.