Skip to content

[Python] Dispatch hybrid search phases to native execution - #10769

Merged
JingsongLi merged 3 commits into
apache:masterfrom
JingsongLi:codex/native-hybrid-route-execution
Oct 23, 2026
Merged

JingsongLi merged 3 commits into
apache:masterfrom
JingsongLi:codex/native-hybrid-route-execution

Conversation

@JingsongLi

@JingsongLi JingsongLi commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • Dispatch the existing public hybrid route_builders() and rank() operations to Native execution. Rust captures one authorized snapshot for the routes and performs score fusion.
  • Keep ordinary, configurable public vector/full-text builders in route.search_builder, including their scan/read methods, metadata transport, and empty or retained read views.
  • Specify and verify that editing a route's retained search builder changes subsequent route execution while retaining the captured snapshot. Cover query, limit and filter changes through the SDK and both public backends.
  • Keep to_route_result() as shared public result validation, preserving scored result identity and normalizing empty unscored selections.
  • Preserve pure in-memory ranking without REST/table access, authorization, or validation of unused queries. Supplied results can be empty, reordered, repeated, or externally evaluated; use the current final limit/ranker.
  • Align Python scores, weights, intermediate arithmetic and accumulation with current Java float32 semantics. Add finite weight boundary and exact Java score fixtures.
  • Preserve custom abstract builder implementations and backend isolation. Native phase failures propagate without Python replay.
  • Update hybrid search documentation. No public PyPaimon API or dependency changes.

Review follow-up

  • Preserve public route/conversion/rank hooks for custom abstract hybrid builders that store a native-enabled table. Only the default implementation enters the Native configuration protocol. The added cases reproduce the original AttributeError before the fix.
  • Independently verify custom scheduled execution with concurrency 1 and 2, both with and without stored table state. A second complete review found no additional actionable issue.

Verification

  • 170 focused Python phase, ranker, execution, search filter, authorization and Ray coordinator tests passed.
  • Arrow 16 focused checks from the initial PR: 23 passed.
  • 90 real Native joint cases passed against the companion Rust implementation, exercising 62 Native plans. The actual SDK was registered inside a Cargo test executable; no Native extension was installed or rebuilt into a Python environment, and the temporary runner is not committed.
  • Companion Rust: 112 Core search-builder tests and eight embedded SDK tests passed. Rust 1.98.0 all-targets Clippy for Core, DataFusion and bindings with warnings denied passed.
  • Scoped flake8 and diff checks passed. Independent review found no remaining actionable issue.

Dependency

Depends on apache/paimon-rust#1206, which unifies execute() -> GlobalIndexResult, removes execute_scored and retains live route builder state. Keep this PR draft until the new APIs reach Rust main. Native CI continues to build Rust main; no SDK version probes, old-API compatibility or CI pins are introduced.

@JingsongLi
JingsongLi marked this pull request as ready for review October 23, 2026 07:11

@leaves12138 leaves12138 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@JingsongLi
JingsongLi merged commit 7befc15 into apache:master Oct 23, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants