feat(global-index): align FullTextSearch with Java JSON DSL query model

[SteNicholas]

Purpose

Linked issue: close #401

Java org.apache.paimon.predicate.FullTextSearch is (fieldName, query, limit) plus a nullable
includeRowIds: query is a JSON DSL string passed unchanged to the native engine, limit must
be positive and results are always scored. The C++ FullTextSearch used a different model
(SearchType, an optional limit, with_score, min_score and pre_filter), so a query written
for Java could not run in C++, and the with_score × limit matrix could return unscored or
unlimited results that Java never produces.

This PR aligns the C++ API with Java and takes option 2 of #401 for lucene-fts: remove the old
model and translate the DSL.

• FullTextSearch: takes a field name, a JSON DSL query, a positive limit and optional
  include_row_ids, which replaces pre_filter with Java includeRowIds semantics. As with
  pre_filter, these are global row ids for GlobalIndexReader and file-local row ids for
  FileIndexReader. SearchType, with_score, min_score and ReplacePreFilter are removed.
• FullTextSearch::OffsetRange(from, to): as Java FullTextSearch#offsetRange, clips
  include_row_ids to a shard range and shifts them to shard-local row ids.
  OffsetGlobalIndexReader uses it.
• GlobalIndexReader::VisitFullTextSearch returns ScoredGlobalIndexResult. UnionGlobalIndexReader
  merges scored results as for vector search, and the readers that reject full-text search (BTree,
  bitmap, lazy-filtered BTree, primary-key sorted, Lumina, file index wrapper) are updated.
• full-text reader: passes the query to the engine unchanged and drops the unscored and
  min_score paths.
• lucene-fts reader: translates match, multi_match, match_phrase (alias phrase) and
  boolean into Lucene term, boolean and phrase queries. Field names, aliases, defaults and
  validation follow core/src/query.rs and core/src/index.rs of apache/paimon-full-text:
  operators, boolean occurrences and column names are trimmed of Unicode whitespace as Rust
  str::trim does, boosts are validated after rounding to f32, duplicate known fields are
  rejected and unknown fields are ignored. Queries are parsed without recursion and may nest at
  most 127 JSON arrays and objects, not counting unknown fields, as serde_json allows in the
  engine. boost queries and fuzzy matching are rejected with NotImplemented, and prefix,
  wildcard and unlimited Lucene searches are removed.
• File index full-text reads (added in feat(file-index): support basic vector and full-text search #372): RawFileSplitRead intersects include_row_ids
  with the rows kept by deletion vectors. File index full-text search returns unscored rows, so a
  read that projects _INDEX_SCORE is rejected with NotImplemented, replacing the with_score
  check.

limit applies to each index shard. Searches across multiple shards return the union of the shard
candidates, which can exceed limit; a final global top-k is tracked in #402.

Tests

UT:

• src/paimon/common/predicate/full_text_search_test.cpp: TestOffsetRangeWithoutIncludeRowIds,
  TestOffsetRangeWithIncludeRowIds (including row ids beyond 32 bits).
• src/paimon/common/global_index/offset_global_index_reader_test.cpp:
  TestVisitFullTextSearchWithOffset, TestVisitFullTextSearchConvertsIncludeRowIds.
• src/paimon/common/global_index/union_global_index_reader_test.cpp:
  TestVisitFullTextSearchUnion. MockGlobalIndexReader returns scored results for full-text
  search.
• src/paimon/global_index/full_text/full_text_global_index_test.cpp: TestWriteAndSearch
  (multi_match on the native text field), TestIncludeRowIdsAndLimit,
  TestNestedBooleanAndBoost (nested boolean queries, and match.boost 2.0 doubling the scores
  of boost 1.0 with the same row ids), TestInvalidRead.
• src/paimon/global_index/lucene/lucene_global_index_test.cpp: TestSimple and
  TestSimpleChinese (DSL queries, boosts, Unicode whitespace, blank match and match_phrase
  text, scores and include_row_ids),
  TestNestedQuery (nested boolean queries and the nesting-depth limit, including 100000 levels),
  TestInvalidQuery (invalid JSON, unknown query types, duplicate fields, invalid values, boost
  queries and fuzzy matching).
• src/paimon/core/operation/internal_read_context_test.cpp:
  TestFileFullTextSearchScoreIsNotSupported (a read that projects _INDEX_SCORE).
• src/paimon/core/operation/raw_file_split_read_test.cpp:
  TestFullTextSearchFiltersDeletionVectorBeforeLimit (include_row_ids with deletion vectors),
  TestFullTextSearchRenamedUnprojectedFieldInOldFile.
• The file index tests from feat(file-index): support basic vector and full-text search #372 in empty_file_index_reader_test.cpp,
  file_index_reader_test.cpp, file_index_evaluator_test.cpp, read_context_test.cpp and
  table_read_test.cpp build searches with the new constructor.

IT, test/inte/global_index_test.cpp: TestLuceneWriteCommitScanReadIndexWithScore,
TestWriteCommitScanReadLuceneIndexWithPartition, TestFullTextWriteCommitScanReadIndex,
TestWriteCommitScanReadFullTextIndexWithPartition.

API and Format

Yes, public API changes in include/paimon/:

• FullTextSearch (include/paimon/predicate/full_text_search.h): the constructor is now
  FullTextSearch(field_name, query, limit, include_row_ids = std::nullopt). SearchType,
  search_type, pre_filter, with_score, min_score and ReplacePreFilter are removed, limit
  is int32_t instead of std::optional<int32_t>, and OffsetRange is added.
• GlobalIndexReader::VisitFullTextSearch (include/paimon/global_index/global_index_reader.h)
  returns Result<std::shared_ptr<ScoredGlobalIndexResult>>.

lucene-fts queries must be written in the JSON DSL. A file index full-text read through
ReadContext that projects _INDEX_SCORE fails with NotImplemented. No storage format change.

Documentation

Yes. docs/source/user_guide/global_index.rst describes the JSON DSL queries of the full-text
and lucene-fts indexes, how lucene-fts translates and validates them, and the per-shard
limit.

Generative AI tooling

Generated-by: Claude Code 2.1.296 (Claude Opus 5.5)

🤖 Generated with Claude Code

[zjw1111]

+1

[zjw1111]

+1

[lxy-9602]

+1