Conversation
github-actions
Bot
added
A-index
sbrunk
force-pushed
the
combined-fields-java
branch
4 times, most recently
from
August 22, 2026 12:21
773c473 to
723c46b
Compare
sbrunk
force-pushed
the
combined-fields-java
branch
3 times, most recently
from
September 8, 2026 21:24
2a0042e to
02af44d
Compare
sbrunk
force-pushed
the
combined-fields-java
branch
3 times, most recently
from
September 25, 2026 14:18
224037e to
a7701c1
Compare
Xuanwo
pushed a commit
that referenced
this pull request
Sep 25, 2026
## TL;DR Adds a `combined_fields` full text query that scores several text columns as one virtual field. This is BM25F, the same thing Elasticsearch calls [`combined_fields`](https://www.staff.city.ac.uk/~sbrp622/papers/foundations_bm25_review.pdf) and Lucene calls `CombinedFieldQuery`. Today the only option is `MultiMatch`, which scores each column on its own and then keeps the best one. This PR covers the case where every target column is fully indexed. If a query reaches a fragment that one of the target columns has not indexed, it returns an error. Removing that restriction is the follow up PR, #9444, which adds a flat scan over those fragments. See [Stack](#stack). Language bindings are separate: #8549 (Java) and #8550 (Python). @Xuanwo this is related to some of the work you've been doing on FTS so it'd be great if you could have a look. ## Why Lance can already search several columns with `MultiMatchQuery`, but each column is scored against its own statistics, and the results are combined by taking the highest score. That is what Elasticsearch calls `best_fields`. There is no real cross field BM25. A term that is rare in `title` but common in `body` ends up with two IDF values that cannot be compared to each other, and a query like `john` in `first_name` plus `smith` in `last_name` cannot be scored as if both columns were one field. BM25F blends the statistics so the columns behave like a single field, with a weight per column. ## What is in this PR | Commit | Files | + | − | Subject | |---|---:|---:|---:|---| | `9f289302c` | 27 | 3,103 | 127 | feat(fts): add combined_fields (BM25F) cross-field search | | `30c7fc939` | 13 | 2,061 | 45 | test(fts): cover combined_fields end to end | | **Total** | **34** | **5,164** | **172** | 34 distinct files, per-commit file counts overlap | About 2,200 of the added lines are tests. The whole public API is in this PR: `CombinedFieldsQuery`, its JSON form in the FTS parser, tokenizer validation, and the row granularity rules. The follow up PR leaves all of it alone. `query.rs`, `parser.rs`, `tokenizer.rs` and `traits.rs` are unchanged there. So the API review happens here, and the follow up is only about execution. ## How it works For each query term `t` and each column `f` with weight `w_f`, following Lucene's `CombinedFieldQuery`: ``` tf'(t,d) = Σ_f w_f · tf_f(t,d) docFreq'(t) = max_f docFreq_f(t) dl'(d) = Σ_f w_f · dl_f(d) docCount' = max_f docCount_f sumTTF' = Σ_f w_f · sumTotalTermFreq_f avgdl' = sumTTF' / docCount' score(t,d) = idf'(t) · (k1+1)·tf' / (tf' + k1·(1 - b + b·dl'/avgdl')) ``` How a query runs: ``` CombinedFieldsQuery(cols, terms, weights) │ ▼ plan_combined_fields_query ── which fragments does every target column's │ index cover, and which ones have entries that │ a newer data overlay made stale? │ ├─ every target fragment covered │ │ │ ▼ │ CombinedFieldsQueryExec │ • opens every target column's FTS segments │ • looks up dl' per candidate via DocSet::doc_length_by_row_id │ instead of scanning all documents │ • scores the union of the terms' postings, bounded top-k heap │ └─ some fragment not covered → return an error naming the columns that need optimizing ``` The design has two consequences. Scores are per row, not per list element. BM25F has to join the target columns on the row address. If two columns hold lists, their elements have no correspondence, so there is no way to pair them up. `combined_fields` reports itself as row granular everywhere the granularity code asks, and rejects a target column that can only produce element documents. Corpus statistics have to use that same granularity. Releases before #7656 indexed each `List<String>` element as its own document, so those index files report `docCount` and `docFreq` per element, while this scan counts per row. Mixing the two produces wrong `idf'` and `avgdl'` values. That is what `bm25_row_stats_for_terms` is for. It counts distinct rows, and on V3 it just delegates to the normal path, because there one row is one document. ## When the query returns an error Coverage is decided per fragment rather than per row. A BM25F score needs `tf_f` and `dl_f` from every target column, so a row has to be scored either entirely from the index or not at all. A fragment counts as not covered if any target column's index is missing it, or if a newer data overlay made its entries stale. If such a fragment is in scope, a normal scan fails with: ``` Invalid user input: combined_fields requires every target column to be indexed over every scanned fragment, but 1 of 2 fragments are not fully covered. Optimize the indexes on body, title and retry. ``` `fast_search` is not affected. It only ever reads indexes, so it skips the fragments that are not covered and returns what the indexes have. The alternatives were to leave those rows out of the result, or to score them with statistics that do not describe the data. An error naming the columns to optimize seemed better than either. #9444 replaces it with a flat scan over exactly those fragments, using one shared set of corpus statistics for both sides. ## Correctness - `scalar::inverted`: 850 of 850 tests pass. That includes the row granularity statistics path, checked against real released V1 and V2 index files rather than hand built fixtures. - 13 test functions and 16 cases in `rust/lance/src/dataset/tests/dataset_fts_combined_fields.rs`. 24 tests match the `combined` filter in `lance --lib`. - Every dataset test checks exact scores against `lance_index::scalar::inverted::oracle`, a brute force BM25F reference that recomputes every statistic from the raw text and shares no code with the scan it checks. A wrong corpus size still returns almost the right rows in almost the right order, so a test that only checked which rows came back would pass anyway. - Covered: both row id schemes (row addresses and stable row ids), nulls, cross field AND, the concatenation identity (a two column query has to score the same as the same text in a single column), boost ranking, tokenizer validation, released V1 and V2 format fixtures, and the `fast_search` paths that skip uncovered fragments. The plan shape is checked as well, so a query that should return an error cannot answer from partial statistics instead. - Tie ordering is fixed by the data. The scan walks candidates in ascending row id order and sorts equal scores by score descending, then row id ascending, through `RankedDoc`. That decides both which rows survive a tie at the k-th score and the order they come back in. It has to happen inside the search, because when everything is covered the exec is returned directly with no `SortExec` above it. ## Performance There are three performance layers in this work, and only the `dl'` lookup is in this PR (`DocSet::doc_length_by_row_id`). It removes a pass over all documents just to get their lengths, which is the main cost when term frequencies are fairly uniform. MAXSCORE pruning and posting block skipping are in the branches below, so on skewed data `combined_fields` will be slower than `best_fields` until those land. ## Stack ``` combined-fields-bm25f #7905 ← this PR (part 1, indexed BM25F) └── combined-fields-bm25f-part2 #9444 (part 2, flat scan) ├── fts-builder-doc-order │ └── combined-fields-maxscore │ └── combined-fields-block-skip │ └── combined-fields-bench ├── fts-json-stream-schema ├── combined-fields-python #8550 └── combined-fields-java #8549 ``` | Branch (diff) | Base | PR | Cmts | Files | + | − | Description | |---|---|---|---:|---:|---:|---:|---| | `combined-fields-bm25f` | main | #7905 | 2 | 34 | 5,164 | 172 | BM25F over fully indexed data | | [combined-fields-bm25f-part2](sbrunk/lance@combined-fields-bm25f...combined-fields-bm25f-part2) | part 1 | #9444 | 2 | 16 | 4,262 | 112 | flat scan for fragments the indexes do not cover | | [fts-builder-doc-order](sbrunk/lance@combined-fields-bm25f-part2...fts-builder-doc-order) | part 2 | none yet | 1 | 1 | 493 | 0 | order inverted index docs by row_id at build time | | [combined-fields-maxscore](sbrunk/lance@fts-builder-doc-order...combined-fields-maxscore) | builder-doc-order | none yet | 1 | 8 | 903 | 82 | prune candidate scoring with MAXSCORE | | [combined-fields-block-skip](sbrunk/lance@combined-fields-maxscore...combined-fields-block-skip) | maxscore | none yet | 1 | 10 | 1,091 | 106 | skip posting blocks a row id seek jumps past | | [combined-fields-bench](sbrunk/lance@combined-fields-block-skip...combined-fields-bench) | block-skip | none yet | 1 | 4 | 1,191 | 1 | Lance vs Lucene BM25F validation harness | | [fts-json-stream-schema](sbrunk/lance@combined-fields-bm25f-part2...fts-json-stream-schema) | part 2 | none yet | 1 | 2 | 223 | 21 | report the real schema from `JsonTextStream` | | `combined-fields-python` | part 2 | #8550 | 2 | 5 | 227 | 1 | Python binding for `combined_fields` | | `combined-fields-java` | part 2 | #8549 | 1 | 4 | 321 | 24 | Java binding for `combined_fields` | The performance and bench branches do not have PRs yet. I wanted to agree the split first. These are all fork branches, and GitHub cannot stack PRs across forks, so each dependent PR also shows the commits of the ones below it. The compare links give the real diff for each. --------- Co-authored-by: Yang Cen <bubble-cal@outlook.com>
sbrunk
force-pushed
the
combined-fields-java
branch
from
September 25, 2026 16:57
a7701c1 to
f72d23d
Compare
…d_fields The previous commit refuses a `combined_fields` query whose target columns do not all cover every scanned fragment, so a default full-text search fails on any dataset with rows appended since the indexes were built. `MatchQuery` already unions in a flat scan for its unindexed fragments; this does the same for BM25F. Coverage is per column here, which makes it more than a copy of the single-column path. `dl'` sums each column's document length and a row absent from a column's `DocSet` contributes 0, so a fragment indexed for `title` but not `body` cannot be scored from the index at all. The indexed scan is therefore restricted to the intersection of per-column coverage and everything else goes to the flat scan, rather than splitting on the union. Both sides then score against one shared corpus. The flat side alone sees the rows no index covers, so it measures their contribution and publishes the blend; the indexed side waits for it instead of folding only its own `docCount'`/`docFreq'`/`avgdl'`. Without that, a row reached through either path would rank differently depending on which side happened to score it. Data overlays are handled by measuring rather than patching. When a target column carries an overlay-stale index entry, folding the flat row into the index statistics would double count it against the entry it replaces, and the flat scan cannot subtract what it replaced. So the corpus is measured from current data instead: every target fragment is scanned, every row folded into every column, and the index statistics left out. That costs a full scan of the target columns, so it stays confined to the stale case. `fast_search` is unchanged, being index-only by contract.
…tistics The cases that matter here are the ones where a shared corpus is easy to lose, since both scan sides must agree on `docCount'`/`docFreq'`/`avgdl'`: unindexed and partially indexed fragments, per-column index skew over both row-id schemes, a mixed indexed/flat plan, deletions followed by optimize, and overlay-stale fragments. Each asserts exact scores against the brute-force reference, because scoring the two sides against different corpora still returns the right rows in almost the right order. Also covers what only the flat path reaches: nulls and empty strings read from the scan rather than an index, list and nested columns, a column under a list, filters, and the plan shape itself, so a query that should union does not silently answer from the index alone. The external row-address prefilter is covered for the same reason, over both plan shapes: the fully covered plan ANDs the mask into the indexed scan's prefilter, while the mixed plan's flat child never reaches an index-side prefilter and is masked on its output instead. Scores are asserted there too, because the mask picks what is emitted and not the corpus it is measured against, so a surviving row must keep the score the unmasked query gave it.
sbrunk
force-pushed
the
combined-fields-java
branch
from
September 25, 2026 17:20
f72d23d to
4b24484
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Exposes the Rust
combined_fields(BM25F) query through the Java scanner APIFullTextQuery.combinedFields(queryText, columns)plus an overload taking per-column boosts and an operator.Unlike
multiMatch, which scores each column separately and keeps the best field, this treats the target columns as one virtual field and blends term statistics across them, so a single query can match terms spread over severalcolumns.
Validation (unique columns, boost count matching column count, finite boosts >= 1) stays in the Rust core and surfaces as an exception when the query runs. The Java class only rejects null or empty text and columns, and defensively copies the column and boost lists.
The core
combined_fieldsquery landed in #7905. This branch is based on #9444, which adds the flat scan for unindexed fragments, so GitHub also shows #9444's commits here. The diff for this PR alone is part 2 to java.