fix(fts): preserve corpus statistics for restricted matches

[ruslan-shaydullin]

Related to #9058.

For the two reported single-column Match cases, index optimization changes the winner from alpha to beta after incorporating already appended rows, although the searchable data and query are unchanged.

This partial fix scores eligible restricted matches against committed posting statistics plus the complete unindexed corpus. Flat candidate filtering runs after corpus collection; indexed masks still constrain indexed top-k. A private owner gives each execution a fresh shared scorer. The restricted path handles an effective top-k limit of zero without constructing a zero-sized sort.

The correction covers exact, row-level scalar-string Match queries with a prefilter or explicit fragment selection on the supported append-only corpus. Indexed scoring uses V2/V3 postings. Deletion/stale-text-overlay scoring, legacy postings, fuzzy/phrase/Boolean queries, BM25F/combined_fields, and the unfiltered Hybrid approximation remain outside this correction. #9058 remains open for the broader scoring work.

Validation

Upstream CI ran on the original head and exposed outdated test_plans expectations on all four Rust platforms. Follow-up 5328449 updates the Legacy/Stable expected plans and explanatory comment for MatchCorpus, the full unindexed text scan, and separate candidate filtering; production code is unchanged. All four plan-test variants and 65 targeted FTS regressions passed locally, along with formatting, workspace clippy with warnings denied, and applicable commit hooks. Upstream validation on the updated head has now completed: all 37 current checks pass, including the Rust, Python, Java JNI, license-header, and typo workflows.

• Reproduced eight ranking failures on unchanged base a43e0500; checked scores against an independent raw-text BM25 reference.
• 65 targeted cases passed, including default query construction and offset, stable residual-candidate IDs, real BTree allow/block masks, corpus/candidate boundaries, maintenance, empty/null/deletion controls, failure/cancellation, metrics, and repeated execution.
• Compatibility suites passed: 31 FTS executor tests, 156 dataset/index tests, 46 overlay tests (one existing benchmark ignored), and 782 inverted-index tests. These groups overlap; all added cases are included in the final targeted run.
• Final workspace check (tests and benches), clippy with warnings denied, formatting, and applicable commit hooks passed. Validation used Rust 1.97 on macOS arm64 with the locked ci profile.

The global corpus requires reading/tokenizing all unindexed text, including rows outside the candidate set. Corpus construction reuses committed posting statistics without retokenizing indexed text. A scalar prefilter with an overlay-stale index can still contain a consumable OneShotExec; fresh scorer state does not make that existing input replayable. Sequential/concurrent reuse is verified on the append-only fixtures. No performance claim is made.

Thanks to @sbrunk and lance-gatekeeper[bot] for the report and reproductions during #7905; its emission-filter/shared-scorer work informed this follow-up. Developed with AI assistance and checked against the independent reference and native tests.

[ruslan-shaydullin]

Thanks @wjones127 for enabling the previous CI run. It exposed outdated expected plans in test_plans; 5328449 updates both storage-format expectations and the comment without changing production code. All four plan variants and 65 FTS regressions now pass locally, as do formatting and workspace clippy with warnings denied.

Could you approve the five code workflows on this updated head? They currently report action_required, including the Rust run. The PR description now reflects these results. Thanks!

[lance-gatekeeper]

⚠️ Gate recommendation: approve with a non-blocking risk.

The follow-up is verified: 5328449 only aligns the Legacy/Stable test_plans expectations and explanatory comment with the existing MatchCorpus plan; production code is unchanged. The restricted Match correction therefore retains the prior assessment.

The remaining non-blocking risk is operational: selective prefilters or fragment selections must read and tokenize the full unindexed text corpus, so latency and peak memory scale with the unindexed tail until the FTS index is optimized.

[ruslan-shaydullin]

Hi @sbrunk, would you be able to check the two single-column Match reproductions from #9058 against this PR (5328449)? The covered cases are exact, row-level scalar-string matches on an append-only corpus with either prefilter(true) / id < 2 or with_fragments([0]), using V2/V3 postings. With the same data and query, the selected row and BM25 score should remain stable before and after optimize_indices().

The updated head has passed Rust, Python and Java CI. This remains a partial fix: combined_fields/BM25F, deletion or stale-text overlays, legacy postings, fuzzy/phrase/Boolean queries, and unfiltered Hybrid scoring are outside its scope. A result for either original single-column reproduction would be useful; #9058 stays open for the broader work. Thanks!