feat(knn): expose ANNIvfBatchExec's query, width and inputs - #9262
Merged
hamersaw merged 3 commits intoSep 16, 2026
Merged
Conversation
The shared-scan batch node kept every field private, so a caller that matched it in a plan could not read the query, the number of vectors in it, or the dataset and index metadata behind it — the accessors ANNIvfSubIndexExec has had all along. Sophon's WAL union needs them to build a fresh-tier arm for a batch vector search and to rebuild the node with an inflated candidate k. Also re-export the node and QUERY_INDEX_COL from `lance::io::exec`, alongside the single-query nodes. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
hamersaw
marked this pull request as ready for review
September 16, 2026 14:56
![lance-gatekeeper[bot]](https://avatars.githubusercontent.com/in/4433141?s=60&v=4){kind=small}
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
lance-gatekeeper
Bot
added
the
K-changes
The accessors and the `io::exec` re-export had no repository coverage, so a regression in either would go unnoticed here. Plan a batch indexed search, downcast the node through the public path, and assert the five accessors hand back the scanner's inputs — with and without a prefilter, so `prefilter_source()` is exercised in both shapes. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
lance-gatekeeper
Bot
removed
the
K-changes
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
lance-gatekeeper
Bot
added
the
K-changes
Two breaks from this branch: - The plan walk used `as_any().downcast_ref()`, but DataFusion 54 puts `downcast_ref` on `dyn ExecutionPlan` itself, so the test target did not compile. Match `find_filtered_read`, which already walks plans this way. - Re-exporting `ANNIvfBatchExec` brought its doc comment into the public docs, where its intra-doc link to the private `build_dataset_prefilter` is a rustdoc error. Keep the reference, drop the link. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
lance-gatekeeper
Bot
removed
the
K-changes
lance-gatekeeper
Bot
added
the
K-approved
wkalt
reviewed
Sep 16, 2026
| } | ||
| } | ||
|
|
||
| /// Finds the batch vector-search node in a physical plan. |
wkalt
approved these changes
Sep 16, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds accessors to
ANNIvfBatchExec—query(),query_count(),dataset(),indices(),prefilter_source()— and re-exports the node andQUERY_INDEX_COLfromlance::io::exec.Why
ANNIvfSubIndexExechas hadquery(),dataset(),indices()andprefilter_source()since it was introduced.ANNIvfBatchExeclanded with every field private and onlytry_new, so a caller that matches it in a physical plan can see it exists but can't read anything out of it.That blocks plan rewriting on the batch path. A downstream rewriter that needs to add a second scan arm to a batch vector search — for example, to cover rows that aren't in the index yet — has to read the query vectors, how many there are, and the dataset and index metadata out of the base plan. The per-query union shape (HNSW, or any
refine_factor) is already reachable throughANNIvfSubIndexExec; the shared-scan fast path is not, so the two shapes can't be handled the same way.Rebuilding the node with a different candidate
kneeds the same fields, sincetry_newtakes them all and there's no way to get them back out of an existing node.Notes
query()'s doc notes thatkeyholds allquery_countvectors concatenated, which is the one non-obvious thing about reading it.🤖 Generated with Claude Code