fix(global-index): apply global top-k to full-text search across shards

[SteNicholas]

Purpose

Linked issue: close #402

UnionGlobalIndexReader::VisitFullTextSearch ORed shard results, so a full-text search with
limit k over multiple index shards could return up to shards × k rows, and it failed when two
shards returned the same row id. OffsetGlobalIndexReader shifted global include_row_ids by the
shard offset without clipping them, so row ids before a shard became negative local ids, row ids
past it were kept, and a shard with no included row was still searched.

This PR follows Java DataEvolutionFullTextRead, which asks each split for limit rows, combines
them with ScoredGlobalIndexResult#merge and applies #topK(limit) last.

• UnionGlobalIndexReader::VisitFullTextSearch: merges shard results, where a duplicate row id
  keeps its first score, and keeps the limit highest-scoring rows. Scores are compared as Java
  Double#compare does (via FieldsComparator::CompareFloatingPoint), and ties among merged rows
  are broken by ascending row id. A null search or a non-positive limit is rejected before any
  shard is searched.
• OffsetGlobalIndexReader: takes the last row id of its shard and clips include_row_ids to
  the shard with FullTextSearch::OffsetRange(offset, to), as Java
  OffsetGlobalIndexReader(wrapped, offset, to) does. A shard whose include set becomes empty is
  not searched.
• Overlapping index ranges: as Java DataEvolutionFullTextScan#chooseIndexRanges,
  GlobalIndexScanImpl ranks a row covered by overlapping ranges of one index type only in the
  first of them in (from, to) order. Each OffsetGlobalIndexReader is given the last row ranked
  by earlier ranges (skip_to) and ranks only the rows after it, so the top limit rows of each
  shard include all of its rows that score above the global limit-th score. Passing the last
  skipped row instead of the first row to rank does not overflow when a range ends at INT64_MAX.
  A fully covered shard is not searched but still rejects a non-positive limit.

As in Java, each shard's index engine decides which of its rows tied at its cutoff score it
returns, so ascending row id order is guaranteed only among the rows the shards return.

Tests

UT:

• src/paimon/common/global_index/union_global_index_reader_test.cpp:
  TestVisitFullTextSearchUnion, TestVisitFullTextSearchKeepsTopKAcrossReaders (top-k and ties
  by row id), TestVisitFullTextSearchDuplicateRowIdKeepsFirstScore,
  TestVisitFullTextSearchOrdersSpecialScoresAsJava (NaN, infinity and signed zeros),
  TestVisitFullTextSearchAllNullptr, TestVisitFullTextSearchErrors.
• src/paimon/common/global_index/offset_global_index_reader_test.cpp:
  TestVisitFullTextSearchConvertsIncludeRowIds (clipping to the shard, and an empty include set
  that is not searched), TestVisitFullTextSearchRejectsNonPositiveLimit,
  TestVisitFullTextSearchSkipsRows, TestVisitFullTextSearchEndingAtLargestRowId. The other
  tests pass the shard end.

IT, test/inte/global_index_test.cpp:

• TestWriteCommitScanReadLuceneIndexWithPartition and
  TestWriteCommitScanReadFullTextIndexWithPartition: global top-k for every limit across two
  shards, with and without include_row_ids.
• TestLuceneSearchOverlappingIndexRanges: index ranges [0, 2], [1, 1] and [1, 3], where the
  best match is covered only by the tail of [1, 3], also with include_row_ids and a
  RowRangeIndex that prunes [0, 2].

API and Format

No signature change in include/. The documented behavior changes:

• GlobalIndexReader::VisitFullTextSearch (include/paimon/global_index/global_index_reader.h)
  returns at most limit rows across all index shards.
• FullTextSearch::limit (include/paimon/predicate/full_text_search.h) is the maximum number of
  rows returned; each index shard is asked for that many rows before the merge.

No storage format change.

Documentation

Yes. docs/source/user_guide/global_index.rst replaces the note that searches across shards return
the union of shard candidates with the global top-k behavior, tie handling, overlapping index
ranges and include_row_ids clipping.

Generative AI tooling

Generated-by: Claude Code 2.1.296 (Claude Opus 5.5)

🤖 Generated with Claude Code

[lxy-9602]

The comment for OffsetGlobalIndexReader seems to fully describe full-text search.
I think its purpose may be more than just FTS?

[lxy-9602]

It looks like the three-parameter version isn’t used in production code—can we remove it?

[lxy-9602]

I checked Java’s chooseIndexRanges—it seems more complex than the C++ side, and it passes includeRowIds rather than inside OffsetGlobalIndexReader. I wonder what’s the rationale behind this design?