feat(fts): score unindexed fragments in combined_fields

[sbrunk]

TL;DR

Follow up to #7905, which added combined_fields (BM25F) but returned an error whenever a query reached a fragment that one of the target columns had not indexed. This PR replaces that error with a flat scan over those fragments, so combined_fields also works on rows added since the indexes were built.

What changes

Commit	Files	+	−	Subject
c13ec836a	15	1,899	115	feat(fts): score unindexed and partially indexed fragments in combined_fields
a10549942	4	2,374	8	test(fts): cover the combined_fields flat scan and overlay corpus statistics
Total	17	4,273	123	about 2,200 lines of it tests

The query API is untouched: query.rs, parser.rs, tokenizer.rs and traits.rs are unchanged. CombinedFieldColumn gains one field, stale_rows, which is public but internal to scoring. Most of the 123 deleted lines are in code from #7905, mainly the error path that the union plan replaces. One #7905 test changes meaning: in test_element_document_nested_lists_use_deepest_boundary, a normal scan over the appended fragment now returns all four rows instead of the coverage error. Its fast_search and post-optimize checks are unchanged.

Before, on a dataset with one appended fragment:

Invalid user input: combined_fields requires every target column to be indexed
over every scanned fragment, but 1 of 2 fragments are not fully covered.
Optimize the indexes on body, title and retry.

After, the same query returns those rows, ranked correctly against the indexed ones.

How it works

plan_combined_fields_query
        │
        └─ some fragment not covered → union of two children
             │
             ├── CombinedFieldsQueryExec  (restricted to the covered fragments)
             │
             └── FlatCombinedFieldsExec   (the remaining fragments)
                   • reads the column values and blends dl' and tf' per row
                   • publishes the corpus statistics that both children use
             │
             ▼
        UnionExec → SortExec(score DESC, row_id ASC) → fetch(k)

Both children score against the same corpus. Only the flat side can see the rows that no index covers, so it measures what they contribute and publishes the combined statistics. The indexed side waits for those rather than using only its own docCount', docFreq' and avgdl'. Getting this wrong would be hard to spot: the right rows still come back, but with scores that depend on which child happened to produce them.

A fragment counts as not covered if any target column's index is missing it, or if a newer data overlay made its entries stale. Coverage stays a per fragment decision, because a row has to be scored either entirely from the index or entirely from the scan.

A stale row needs one more step. The index holds its pre-overlay value until the index is optimized, and the flat scan folds in its current one, so counting both inflates docCount and docFreq. That is enough to reorder results after an overlay that changed no searchable value at all. InvertedIndex::bm25_row_stats_for_terms therefore takes the stale rows and subtracts what they still contribute, reading posting membership to do it. Only a column that actually has stale rows pays for that.

Correctness

• Adds 15 test functions and 23 cases to dataset_fts_combined_fields.rs, taking it from 13 functions and 16 cases to 28 and 39, plus overlay coverage in dataset_overlay_index_masking.rs. 62 tests match the combined filter in lance --lib, and scalar::inverted is at 884 of 884.
• Every dataset test checks exact scores against the brute force oracle reference, which recomputes every statistic from the raw text and shares no code with the scan. A wrong shared corpus still returns almost the right rows in almost the right order, so a test that only checked which rows came back would pass anyway.
• Covered: mixed indexed and flat plans, per column index skew over both row id schemes, partially indexed columns, an unindexed row winning top k, fragments made stale by an overlay, deletions followed by optimize, external row masks, filters, nulls and empty strings on the flat path, list and large list and nested columns, a column under a list, and the plan shape itself, so a query that should union cannot answer from the index alone.
• test_fts_combined_fields_overlay_preserves_every_score pins the stale row subtraction by comparing score bit patterns across a same-value overlay. Ids alone would not catch it, because the same rows come back either way.
• test_fts_combined_fields_tied_scores_are_deterministic covers tie ordering for both plan shapes. A fully covered dataset returns the exec with no SortExec above it, while a mixed plan gets its order from the sort's second key, so the two shapes need separate checks.

Stack

The branches that build on this one. They live on a fork, and GitHub cannot stack PRs across forks, so each dependent PR also shows this PR's commits. The compare links give the real diff for each.

main (includes #7905)
└── combined-fields-bm25f-part2           #9444  ← this PR (flat scan)
     ├── fts-builder-doc-order
     │    └── combined-fields-maxscore
     │         └── combined-fields-block-skip
     │              └── combined-fields-bench
     ├── fts-json-stream-schema
     ├── combined-fields-python           #8550
     └── combined-fields-java             #8549

Branch (diff)	Base	PR	Cmts	Files	+	−	Description
combined-fields-bm25f-part2	main	#9444	2	17	4,273	123	flat scan for fragments the indexes do not cover
fts-builder-doc-order	part 2	none yet	1	1	493	0	order inverted index docs by row_id at build time
combined-fields-maxscore	builder-doc-order	none yet	1	8	903	82	prune candidate scoring with MAXSCORE
combined-fields-block-skip	maxscore	none yet	1	10	1,091	106	skip posting blocks a row id seek jumps past
combined-fields-bench	block-skip	none yet	1	4	1,191	1	Lance vs Lucene BM25F validation harness
fts-json-stream-schema	part 2	none yet	1	1	184	1	tests for flat scans over JSON columns (the JsonTextStream schema fix itself landed upstream in #9479)
combined-fields-python	part 2	#8550	2	5	227	1	Python binding for combined_fields
combined-fields-java	part 2	#8549	1	4	321	24	Java binding for combined_fields

[lance-gatekeeper]

⚠️ Gate recommendation: approve with a non-blocking risk.

The only change since the previous reviewed head is an intra-doc link. With the indexed core in #7905 merged, this flat-scan follow-up retains its verified behavior: appended and overlay-stale fragments are scored without double-counting stale index statistics, while covered fragments stay on the indexed path.

The author accepted the remaining candidate-scoped corpus behavior: filtering or fragment selection can make ranking depend on index coverage, including after optimization with unchanged text. #9058 tracks the broader invariant; #9243 addresses single-column Match only. No change requested for this risk here.