Skip to content

[core] Expose phased hybrid search and pure score fusion - #1206

Merged
JingsongLi merged 3 commits into
apache:mainfrom
JingsongLi:codex/native-hybrid-route-execution
Oct 23, 2026
Merged

JingsongLi merged 3 commits into
apache:mainfrom
JingsongLi:codex/native-hybrid-route-execution

Conversation

@JingsongLi

@JingsongLi JingsongLi commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • Add Java-style route_builders, to_route_result, and rank to hybrid search, with owned typed route/result handles. Reuse Core fusion for complete queries and independently executed routes.
  • Unify vector, full-text, hybrid and route execute() results as GlobalIndexResult; remove execute_scored() and migrate Core, DataFusion and SDK callers.
  • Retain the actual mutable vector/full-text builder in each route. Core handles share configuration; repeated SDK search_builder() calls return the same object. Changes affect subsequent route execution. Capture one configuration before I/O so edits do not alter an active request.
  • Capture one authorized snapshot for all routes, including empty and retained views. Reconfiguration preserves that captured view.
  • Expose pure HybridSearchRanker and HybridSearchWeightedResult for external scored results without table access or query validation.
  • Preserve compatible captured read contexts, and return contextless scores for external or conflicting views.
  • Keep the Python binding thin by delegating single vector/full-text builder state to Core. Update stubs, docs and regressions. No dependencies are added.

This covers existing PyPaimon phased hybrid operations. The physical primary-key hybrid API keeps its existing materialized-read contract.

Review follow-up

  • Require shared FileIO backend provenance before attaching a fused result's read context. Matching table metadata alone cannot prove identical credential scopes. A regression uses one immutable snapshot with disjoint partition credentials: each input query/read succeeds, fusion keeps all scores, and conflicting contexts are dropped. Cloned compatible contexts remain usable.
  • Review the complete Core, binding and Python diff again after repairs; no additional actionable findings remain.

Verification

  • 112 Core search-builder tests passed, covering hybrid, vector, batch vector and full-text behavior, retained route mutation, active-request isolation and snapshot ownership.
  • Eight embedded SDK hybrid and search-plan/API tests passed, including builder identity, configuration mutation and removal of execute_scored.
  • 90 real PyPaimon Native joint cases passed (62 Native plans), including mutable SDK routes and both public execution backends. The actual SDK was registered inside a Cargo test executable; no Native extension was installed or rebuilt into a Python environment. The temporary runner is not committed.
  • 170 focused companion Python tests passed; Arrow 16 focused checks from the initial PR: 23 passed.
  • Rust 1.98.0 all-targets Clippy for Core, DataFusion and bindings with warnings denied; default-feature Core check; rustfmt; stub parsing; scoped Python lint and diff checks passed.
  • Independent API, lifetime, snapshot and Java-semantics review found no remaining actionable issue.

Companion Python PR: apache/paimon#10769 (draft until this API reaches Rust main). Its Native CI continues to build Rust main.

@leaves12138 leaves12138 left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@JingsongLi
JingsongLi merged commit d8c92f4 into apache:main Oct 23, 2026
15 of 16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants