Skip to content

feat(file-index): support basic vector and full-text search - #372

Merged
lxy-9602 merged 9 commits into
apache:mainfrom
zjw1111:codex/file-index-vector-full-text-search
Oct 13, 2026
Merged

feat(file-index): support basic vector and full-text search#372
lxy-9602 merged 9 commits into
apache:mainfrom
zjw1111:codex/file-index-vector-full-text-search

Conversation

@zjw1111

@zjw1111 zjw1111 commented Sep 18, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

Purpose

Linked issue: #361

  • Add VectorSearch and FullTextSearch interfaces and execution paths for File Index, with a Lumina vector index backend supporting both writes and reads.
  • Return vector scores through explicitly projected _INDEX_SCORE columns, keeping scores aligned with selected physical rows across files and batches.
  • Support append-only tables and raw-convertible primary-key splits with deletion vectors. Apply deletion vectors and user pre-filters before search limits.
  • Support field renames by matching field IDs and rewriting current read options to each file's field names. Keep schema-evolution handling above File Index readers rather than passing mappings through backend interfaces.

Current scope: search limits are applied per data file. Full-text backend implementation (including Tantivy File Index), full-text score output, Predicate + Search combinations, Merge-on-Read Search, and cross-file TopK merging are deferred to follow-up work.

Tests

Focused local validation during development used the existing Debug/ASan build, not the complete test suite:

  • Unit coverage for field-ID mappings, renamed/reordered fields, same-name fields with different IDs, option rewriting, backend option propagation, unprojected search fields, deletion-vector pre-filters, and row/score alignment.
  • Real Lumina integration coverage for querying renamed fields across old/new schema versions and rebuilding vector indexes during append compaction, including NULL vectors and TopK scores. Integration tests cover Parquet, ORC, and Avro; full-text execution is covered by the shared mock backend in unit tests.
  • Build: cmake --build build --target paimon-core-test paimon-common-test paimon-write-and-read-inte-test -j 64
  • Unit tests: ./build/debug/paimon-core-test --gtest_filter='FieldMappingTest.*:FileIndexOptionsTest.*:FileIndexEvaluatorTest.*:*FileIndexSearchEvaluatorTest.*:RawFileSplitReadTest.*'
  • Format tests: ./build/debug/paimon-common-test --gtest_filter='FileIndexFormatTest.*'
  • Integration tests: ./build/debug/paimon-write-and-read-inte-test --gtest_filter='*TestLuminaVectorIndexAfterCompaction*:*TestLuminaSearchAfterRenameAcrossSchemaVersions*'
  • pre-commit on the changed files and git diff --check passed.

API and Format

  • Add ReadContextBuilder::SetVectorSearch and SetFullTextSearch.
  • Add FileIndexReader::VisitVectorSearch and VisitFullTextSearch, plus ScoredFileIndexResult for file-local row positions and aligned vector scores.
  • Expose vector scores through explicit _INDEX_SCORE projection without implicitly changing the output schema.
  • Pass backend options explicitly to FileIndexFormat::CreateReader instead of using Arrow schema metadata. Options and search requests reaching File Index readers use file field names.
  • The persisted File Index format is unchanged.

Documentation

API comments describe the search requests and score projection, with follow-up capabilities distinguished from the current implementation. No standalone user guide is included in this PR.

Generative AI tooling

Generated-by: OpenAI Codex (GPT-5).

@zjw1111
zjw1111 marked this pull request as draft September 18, 2026 05:19
@zjw1111
zjw1111 force-pushed the codex/file-index-vector-full-text-search branch from 5eefab6 to 718fa3e Compare October 8, 2026 06:09
@zjw1111
zjw1111 marked this pull request as ready for review October 8, 2026 06:10
@zjw1111
zjw1111 force-pushed the codex/file-index-vector-full-text-search branch from 338fbe6 to 0bcceba Compare October 8, 2026 08:10
lxy-9602
lxy-9602 reviewed Oct 9, 2026
View reviewed changes
Comment thread include/paimon/file_index/file_index_reader.h
lxy-9602
lxy-9602 reviewed Oct 9, 2026
View reviewed changes
Comment thread include/paimon/read_context.h
lxy-9602
lxy-9602 reviewed Oct 9, 2026
View reviewed changes
Comment thread include/paimon/read_context.h
lxy-9602
lxy-9602 reviewed Oct 9, 2026
View reviewed changes
Comment thread src/paimon/common/file_index/scored_file_index_result_test.cpp Outdated
Comment thread src/paimon/common/reader/complete_index_score_batch_reader_test.cpp Outdated
Comment thread src/paimon/core/operation/raw_file_split_read.h Outdated
Comment thread src/paimon/core/io/file_index_evaluator_test.cpp
Comment thread src/paimon/core/io/file_index_options.cpp
Comment thread src/paimon/indexer/lumina/lumina_file_index_factory.cpp
Comment thread src/paimon/format/blob/blob_format_writer.h Outdated
Comment thread src/paimon/CMakeLists.txt
Comment thread src/paimon/indexer/lumina/lumina_file_index.cpp
Comment thread src/paimon/indexer/lumina/lumina_file_index_test.cpp Outdated
lszskye
lszskye reviewed Oct 10, 2026
View reviewed changes
Comment thread src/paimon/core/io/file_index_evaluator.cpp
@zjw1111 zjw1111 changed the title feat(file-index): support file-local search with PK+DV filtering feat(file-index): support vector and full-text search Oct 12, 2026
@zjw1111 zjw1111 changed the title feat(file-index): support vector and full-text search feat(file-index): file-index support vector and full-text search Oct 12, 2026
@zjw1111 zjw1111 changed the title feat(file-index): file-index support vector and full-text search feat(file-index): support basic vector and full-text search Oct 12, 2026
lxy-9602
lxy-9602 approved these changes Oct 12, 2026
View reviewed changes

@lxy-9602 lxy-9602 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

@lszskye

lszskye commented Oct 12, 2026

Copy link
Copy Markdown
Collaborator

+1

@lxy-9602
lxy-9602 merged commit f2bb0d4 into apache:main Oct 13, 2026
16 checks passed
SteNicholas added a commit to SteNicholas/paimon-cpp that referenced this pull request Oct 13, 2026
@SteNicholas
feat(global-index): align FullTextSearch with Java JSON DSL query model
02a4ee7 
Align FullTextSearch with org.apache.paimon.predicate.FullTextSearch:

- Make FullTextSearch take a field name, a JSON DSL query, a positive
  limit and optional include_row_ids, which replaces pre_filter. Remove
  SearchType, with_score, min_score and ReplacePreFilter.
- Add FullTextSearch::OffsetRange(from, to), which clips include_row_ids
  to a shard range and shifts them to local row ids, and use it in
  OffsetGlobalIndexReader.
- Make GlobalIndexReader::VisitFullTextSearch return
  ScoredGlobalIndexResult and drop the unscored and min_score paths of
  the full-text reader.
- Translate match, multi_match, match_phrase and boolean queries into
  Lucene queries in the lucene-fts reader, validating them as the
  paimon-full-text engine does. Reject boost queries and fuzzy
  matching, and remove prefix, wildcard and unlimited Lucene searches.
  Parse queries without recursion and reject queries that nest more
  than 127 JSON arrays and objects, as serde_json does in the engine.
- Move file index full-text reads from apache#372 to include_row_ids, and
  reject a file full-text read that projects _INDEX_SCORE in place of
  its with_score check.
- Update tests and global index docs, and document that limit applies
  to each index shard until apache#402 adds a global top-k.

Closes apache#401.
SteNicholas added a commit to SteNicholas/paimon-cpp that referenced this pull request Oct 13, 2026
@SteNicholas
feat(global-index): align FullTextSearch with Java JSON DSL query model
2f31b8a 
Align FullTextSearch with org.apache.paimon.predicate.FullTextSearch:

- Make FullTextSearch take a field name, a JSON DSL query, a positive
  limit and optional include_row_ids, which replaces pre_filter. Remove
  SearchType, with_score, min_score and ReplacePreFilter.
- Add FullTextSearch::OffsetRange(from, to), which clips include_row_ids
  to a shard range and shifts them to local row ids, and use it in
  OffsetGlobalIndexReader.
- Make GlobalIndexReader::VisitFullTextSearch return
  ScoredGlobalIndexResult and drop the unscored and min_score paths of
  the full-text reader.
- Translate match, multi_match, match_phrase and boolean queries into
  Lucene queries in the lucene-fts reader, validating them as the
  paimon-full-text engine does. Reject boost queries and fuzzy
  matching, and remove prefix, wildcard and unlimited Lucene searches.
  Parse queries without recursion and reject queries that nest more
  than 127 JSON arrays and objects, as serde_json does in the engine.
- Use include_row_ids in file index full-text reads from apache#372, and
  replace their with_score check by rejecting reads that project
  _INDEX_SCORE.
@SteNicholas SteNicholas mentioned this pull request Oct 13, 2026
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@lxy-9602 lxy-9602 lxy-9602 approved these changes
+1 more reviewer
@lszskye lszskye lszskye left review comments
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@zjw1111 @lszskye @lxy-9602