feat(fts): define global BM25 statistics wire contract

[zhangstar333]

Distributed FTS executors need one query-bound, corpus-wide BM25 statistics payload so workers searching different index segments can produce comparable scores.

This is part of #8937 and supports the downstream Doris use case described in apache/doris#67435.

[zhangstar333]

@Xuanwo hi, could you please have a look at this pr when you are free, very Thanks!

[sbrunk]

Suggestion to make V1 easier to extend: use a multi-corpus layout, but restrict V1 to one corpus, document counts, Match/Phrase leaves.

 message FtsGlobalStatisticsProto {
   FtsGlobalStatisticsSchemaVersion schema_version = 1;
   uint64 dataset_version = 2;
-  string index_name = 3;
-  string column = 4;
-  repeated bytes segment_uuids = 5;
-  optional uint64 total_tokens = 6;
-  optional uint64 num_docs = 7;
-  repeated FtsTermStatisticsProto terms = 8;
-  FtsDocumentGranularity document_granularity = 9;
-  repeated FtsPreparedQueryLeafProto query_leaves = 10;
+  // Unique prepared terms referenced by any leaf, sorted by UTF-8 bytes.
+  repeated string terms = 3;
+  // V1: exactly one entry.
+  repeated FtsCorpusStatisticsProto corpora = 4;
+  repeated FtsPreparedQueryLeafProto query_leaves = 5;
 }

-message FtsTermStatisticsProto {
-  string term = 1;
-  optional uint64 document_frequency = 2;
-}
+// Statistics of one FTS index on one column, over its committed segments.
+message FtsCorpusStatisticsProto {
+  string index_name = 1;
+  string column = 2;
+  FtsDocumentGranularity document_granularity = 3;
+  // V1: only DOCUMENT.
+  FtsCorpusCountUnit count_unit = 4;
+  repeated bytes segment_uuids = 5;
+  optional uint64 num_docs = 6;
+  optional uint64 total_tokens = 7;
+  // Sorted by term_index.
+  repeated FtsCorpusTermStatisticsProto terms = 8;
+}
+
+message FtsCorpusTermStatisticsProto {
+  optional uint32 term_index = 1;           // into FtsGlobalStatisticsProto.terms
+  optional uint64 document_frequency = 2;
+}

 message FtsPreparedQueryLeafProto {
   optional uint32 leaf_ordinal = 1;
-  repeated FtsPreparedTokenProto tokens = 2;
-  optional bool has_all_query_positions = 3;
+  // V1: only MATCH or PHRASE.
+  FtsQueryLeafKind kind = 2;
+  // V1: always [0].
+  repeated uint32 corpus_indices = 3;
+  repeated FtsPreparedTokenProto tokens = 4;
+  optional bool has_all_query_positions = 5;
 }
+
+enum FtsCorpusCountUnit {
+  FTS_CORPUS_COUNT_UNIT_UNSPECIFIED = 0;
+  FTS_CORPUS_COUNT_UNIT_DOCUMENT = 1;
+}
+
+enum FtsQueryLeafKind {
+  FTS_QUERY_LEAF_KIND_UNSPECIFIED = 0;
+  FTS_QUERY_LEAF_KIND_MATCH = 1;
+  FTS_QUERY_LEAF_KIND_PHRASE = 2;
+}

FtsPreparedTokenProto is unchanged. Its term_index still points into FtsGlobalStatisticsProto.terms, which now holds only the term strings.

V1 consumers reject any payload outside these restrictions, including unknown enum values.

Why

Cross-column queries like cross-column compound (#8689) and combined_fields/BM25F (WIP in #7905), will need:

• one corpus per column
• leaves that reference one or more corpora
• for BM25F, statistics counted per row instead of per document, since legacy V1/V2 list indexes store one document per list element.

With the current flat layout, that means bumping schema_version and restructuring. Just adding fields isn't safe: V1 consumers ignore unknown fields and would score with incomplete or wrongly counted statistics, without raising an error.

With the layout above, supporting those queries later only means allowing more values: several corpora, a ROW count unit, a COMBINED_FIELDS leaf kind. There are no layout changes and no new fields. V1 consumers reject such payloads instead of misreading them, and single-column payloads stay valid across versions.
The shared term list exists because one combined_fields token is looked up in several columns, each with its own df.

Adding full cross-column support would then look like this sketch. Alternatively, we could also go with full support in V1 directly.

 message FtsGlobalStatisticsProto {
   FtsGlobalStatisticsSchemaVersion schema_version = 1;
   uint64 dataset_version = 2;
   // Unique prepared terms referenced by any leaf, sorted by UTF-8 bytes.
   repeated string terms = 3;
-  // V1: exactly one entry.
+  // Unique and sorted by (index_name, column, document_granularity, count_unit).
+  // Every corpus is referenced by at least one leaf.
   repeated FtsCorpusStatisticsProto corpora = 4;
   repeated FtsPreparedQueryLeafProto query_leaves = 5;
 }

 message FtsCorpusStatisticsProto {
   string index_name = 1;
   string column = 2;
   FtsDocumentGranularity document_granularity = 3;
-  // V1: only DOCUMENT.
   FtsCorpusCountUnit count_unit = 4;
   repeated bytes segment_uuids = 5;
   optional uint64 num_docs = 6;
   optional uint64 total_tokens = 7;
-  // Sorted by term_index.
+  // Sorted by term_index. Covers every term that a leaf bound to this
+  // corpus references.
   repeated FtsCorpusTermStatisticsProto terms = 8;
 }

 message FtsPreparedQueryLeafProto {
+  // CombinedFields emits one leaf and is not descended into.
   optional uint32 leaf_ordinal = 1;
-  // V1: only MATCH or PHRASE.
   FtsQueryLeafKind kind = 2;
-  // V1: always [0].
+  // Indices into corpora.
+  // MATCH/PHRASE: exactly one corpus, with count_unit DOCUMENT.
+  // COMBINED_FIELDS: one corpus per target column, in the query's column
+  //   order (which pairs each corpus with its weight), all with count_unit
+  //   ROW and document_granularity ROW, no duplicates.
   repeated uint32 corpus_indices = 3;
   repeated FtsPreparedTokenProto tokens = 4;
+  // Always true for COMBINED_FIELDS (no fuzziness).
   optional bool has_all_query_positions = 5;
 }

 enum FtsCorpusCountUnit {
   FTS_CORPUS_COUNT_UNIT_UNSPECIFIED = 0;
   FTS_CORPUS_COUNT_UNIT_DOCUMENT = 1;
+  // One unit per distinct row id. Required by COMBINED_FIELDS (BM25F), since
+  // legacy V1/V2 list indexes store one document per list element.
+  FTS_CORPUS_COUNT_UNIT_ROW = 2;
 }

 enum FtsQueryLeafKind {
   FTS_QUERY_LEAF_KIND_UNSPECIFIED = 0;
   FTS_QUERY_LEAF_KIND_MATCH = 1;
   FTS_QUERY_LEAF_KIND_PHRASE = 2;
+  FTS_QUERY_LEAF_KIND_COMBINED_FIELDS = 3;
 }

@BubbleCal

[lance-gatekeeper]

✅ Gate recommendation: approve.

The proposed multi-corpus layout makes future cross-column and BM25F support easier to extend. For #8937, V1 deliberately covers one corpus and binds every prepared leaf to it; query attachment remains the responsibility of the trusted planner. The current contract requires consumers to reject unsupported schema_version values.

Adding multi-corpus statistics later will require a new version and new field numbers while preserving existing tags. Older V1 consumers would then reject the new payload rather than calculate scores from partial statistics. That future migration cost does not change the recommendation for this focused V1 contract.