feat: add ASOF join logical semantics by Xuanwo · Pull Request #23829 · apache/datafusion

Xuanwo · 2026-07-23T05:28:25Z

Which issue does this PR close?

Part of ASOF join support / Specialize Range Joins #318.
Umbrella PR: feat: support ASOF joins #23738.
Depends on feat: add ASOF join physical operator #23828 (merged).

Rationale for this change

This is the logical-planning layer of the ASOF JOIN stack. It defines the
logical contract and planner behavior separately from the SQL frontend and
serialization formats.

#23828 is merged, so this PR's diff against main is the isolated logical
layer. It no longer depends on the optional floating-point follow-up #24375.

What changes are included in this PR?

Add LogicalPlan::AsOfJoin, AsOfJoin, and AsOfMatch.
Validate deterministic expressions, input ownership, supported match
operators, equality-key types, and USING constraints.
Add LogicalPlanBuilder entry points and schema construction that preserves
both qualified USING keys while exposing one unqualified wildcard key.
Integrate ASOF joins with tree transforms, display, type coercion, projection
pruning, row bounds, and physical planning.
Plan logical ASOF joins to the broadcast-based AsOfJoinExec from feat: add ASOF join physical operator #23828.
Fail closed at proto, SQL unparser, and Substrait boundaries until their
owning stack layers add explicit support.
Defer ASOF-specific functional-dependency refinement to perf: preserve left functional dependencies for ASOF joins #24799 and filter
pushdown to perf: push left filters through ASOF joins #24801 so each optimization can be reviewed independently.

Are these changes tested?

Yes:

cargo fmt --all
cargo clippy --all-targets --all-features -- -D warnings
cargo test -p datafusion-expr min_rows_of_joins --all-features
cargo test -p datafusion-substrait asof_join_fails_closed_until_substrait_has_an_extension --all-features
The extended workspace test command from the contributor guide

Are there any user-facing changes?

This adds logical-plan and builder APIs for ASOF joins. SQL syntax, DataFrame
APIs, and plan serialization are intentionally left to dependent stack PRs.
Floating equality keys remain rejected by the merged physical operator unless
the independent follow-up #24375 is also included.

As with any new public LogicalPlan variant, downstream exhaustive matches must
add an arm. The variant is appended so existing variants retain their
PartialOrd ordering; maintainers should still treat the enum addition as a
Rust source-compatibility break.

This PR can be reviewed independently now that #23828 has merged. The
optimization follow-ups #24799 and #24801 are not required by the core ASOF
stack.

github-actions · 2026-07-23T05:47:00Z

Thank you for opening this pull request!

Reviewer note: cargo-semver-checks reported the current version number is not SemVer-compatible with the changes in this pull request (compared against the base branch).

Details

     Cloning apache/main
    Building datafusion v55.0.0 (current)
       Built [  63.630s] (current)
     Parsing datafusion v55.0.0 (current)
      Parsed [   0.038s] (current)
    Building datafusion v55.0.0 (baseline)
       Built [  62.116s] (baseline)
     Parsing datafusion v55.0.0 (baseline)
      Parsed [   0.039s] (baseline)
    Checking datafusion v55.0.0 -> v55.0.0 (no change; assume patch)
     Checked [   0.928s] 223 checks: 223 pass, 31 skip
     Summary no semver update required
    Finished [ 129.033s] datafusion
    Building datafusion-expr v55.0.0 (current)
       Built [  31.273s] (current)
     Parsing datafusion-expr v55.0.0 (current)
      Parsed [   0.082s] (current)
    Building datafusion-expr v55.0.0 (baseline)
       Built [  30.907s] (baseline)
     Parsing datafusion-expr v55.0.0 (baseline)
      Parsed [   0.083s] (baseline)
    Checking datafusion-expr v55.0.0 -> v55.0.0 (no change; assume patch)
     Checked [   1.853s] 223 checks: 222 pass, 1 fail, 0 warn, 31 skip

--- failure enum_variant_added: enum variant added on exhaustive enum ---

Description:
A publicly-visible enum without #[non_exhaustive] has a new variant.
        ref: https://doc.rust-lang.org/cargo/reference/semver.html#enum-variant-new
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.50.0/src/lints/enum_variant_added.ron

Failed in:
  variant LogicalPlan:AsOfJoin in /home/runner/work/datafusion/datafusion/datafusion/expr/src/logical_plan/plan.rs:301
  variant LogicalPlan:AsOfJoin in /home/runner/work/datafusion/datafusion/datafusion/expr/src/logical_plan/plan.rs:301

     Summary semver requires new major version: 1 major and 0 minor checks failed
    Finished [  65.491s] datafusion-expr
    Building datafusion-optimizer v55.0.0 (current)
       Built [  28.977s] (current)
     Parsing datafusion-optimizer v55.0.0 (current)
      Parsed [   0.033s] (current)
    Building datafusion-optimizer v55.0.0 (baseline)
       Built [  28.583s] (baseline)
     Parsing datafusion-optimizer v55.0.0 (baseline)
      Parsed [   0.034s] (baseline)
    Checking datafusion-optimizer v55.0.0 -> v55.0.0 (no change; assume patch)
     Checked [   0.226s] 223 checks: 223 pass, 31 skip
     Summary no semver update required
    Finished [  58.910s] datafusion-optimizer
    Building datafusion-physical-plan v55.0.0 (current)
       Built [  40.299s] (current)
     Parsing datafusion-physical-plan v55.0.0 (current)
      Parsed [   0.164s] (current)
    Building datafusion-physical-plan v55.0.0 (baseline)
       Built [  40.592s] (baseline)
     Parsing datafusion-physical-plan v55.0.0 (baseline)
      Parsed [   0.173s] (baseline)
    Checking datafusion-physical-plan v55.0.0 -> v55.0.0 (no change; assume patch)
     Checked [   1.069s] 223 checks: 223 pass, 31 skip
     Summary no semver update required
    Finished [  83.760s] datafusion-physical-plan
    Building datafusion-proto v55.0.0 (current)
       Built [  57.857s] (current)
     Parsing datafusion-proto v55.0.0 (current)
      Parsed [   0.019s] (current)
    Building datafusion-proto v55.0.0 (baseline)
       Built [  58.197s] (baseline)
     Parsing datafusion-proto v55.0.0 (baseline)
      Parsed [   0.020s] (baseline)
    Checking datafusion-proto v55.0.0 -> v55.0.0 (no change; assume patch)
     Checked [   0.172s] 223 checks: 223 pass, 31 skip
     Summary no semver update required
    Finished [ 117.727s] datafusion-proto
    Building datafusion-sql v55.0.0 (current)
       Built [  46.275s] (current)
     Parsing datafusion-sql v55.0.0 (current)
      Parsed [   0.035s] (current)
    Building datafusion-sql v55.0.0 (baseline)
       Built [  46.246s] (baseline)
     Parsing datafusion-sql v55.0.0 (baseline)
      Parsed [   0.037s] (baseline)
    Checking datafusion-sql v55.0.0 -> v55.0.0 (no change; assume patch)
     Checked [   0.329s] 223 checks: 223 pass, 31 skip
     Summary no semver update required
    Finished [  94.340s] datafusion-sql
    Building datafusion-substrait v55.0.0 (current)
       Built [ 314.635s] (current)
     Parsing datafusion-substrait v55.0.0 (current)
      Parsed [   0.019s] (current)
    Building datafusion-substrait v55.0.0 (baseline)
       Built [ 319.294s] (baseline)
     Parsing datafusion-substrait v55.0.0 (baseline)
      Parsed [   0.019s] (baseline)
    Checking datafusion-substrait v55.0.0 -> v55.0.0 (no change; assume patch)
     Checked [   0.328s] 223 checks: 223 pass, 31 skip
     Summary no semver update required
    Finished [ 636.849s] datafusion-substrait

codecov-commenter · 2026-07-23T05:56:54Z

Codecov Report

❌ Patch coverage is 30.34301% with 264 lines in your changes missing coverage. Please review.
✅ Project coverage is 81.48%. Comparing base (c4910e0) to head (65752d9).
⚠️ Report is 20 commits behind head on main.

Files with missing lines	Patch %	Lines
datafusion/expr/src/logical_plan/plan.rs	40.51%	101 Missing and 15 partials ⚠️
datafusion/expr/src/logical_plan/tree_node.rs	0.00%	43 Missing ⚠️
datafusion/core/src/physical_planner.rs	0.00%	41 Missing ⚠️
datafusion/optimizer/src/analyzer/type_coercion.rs	0.00%	28 Missing ⚠️
datafusion/expr/src/logical_plan/builder.rs	60.71%	17 Missing and 5 partials ⚠️
datafusion/expr/src/logical_plan/display.rs	0.00%	10 Missing ⚠️
datafusion/optimizer/src/optimizer.rs	0.00%	4 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff            @@
##             main   #23829     +/-   ##
=========================================
  Coverage   81.47%   81.48%             
=========================================
  Files        1122     1123      +1     
  Lines      403629   406478   +2849     
  Branches   403629   406478   +2849     
=========================================
+ Hits       328866   331212   +2346     
- Misses      55510    55880    +370     
- Partials    19253    19386    +133

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Xuanwo · 2026-08-28T05:53:56Z

cc @2010YOUY01 & @jayzhan211, we can work on this PR now!

jayzhan211

Thanks @Xuanwo , LGTM

Xuanwo · 2026-08-30T05:00:38Z

Thanks @Xuanwo , LGTM

Thank you @jayzhan211 for the review 🥰

2010YOUY01

Thank you, this is great. Could you wait a few days? I need more time to review.

After a quick look, I'm wondering whether it's possible to remove the two pure optimizations marked below and add them in follow-up PRs.

They seem quite tricky, and we've had many related bugs before, so we should add more test coverage targeting those specific optimizations, and potentially better explain the mechanism and hidden assumptions in the separate PRs.

2010YOUY01 · 2026-08-30T09:39:09Z

+        left: Arc<LogicalPlan>,
+        right: Arc<LogicalPlan>,
+        on: Vec<(Expr, Expr)>,
+        match_condition: AsOfMatch,


nit and optional: would it be better to use Expr instead of AsOfMatch?

The reason is that the typed AsOfMatch can only do part of the validation. It guarantees the expr has the form expr1 comparator expr2, but later we still need to validate expr1 and expr2 further. I think keeping them in one place might be better — the "scattered implementation" pattern is bad in most cases.

This might just be personal taste though. I don't see it as a major issue, happy to be convinced otherwise.

I'd prefer to keep AsOfMatch. It gives the planner and downstream APIs the left/op/right structure they all need; using a generic Expr would make each consumer unpack the binary form again. Schema-dependent checks still belong in AsOfJoin::try_new.

Xuanwo · 2026-08-30T17:46:53Z

Thank you, this is great. Could you wait a few days? I need more time to review.

Sure, take your time! I have moved some independent optimizations in to dedicated PRs as follow-ups.

github-actions Bot added sql SQL Planner logical-expr Logical plan and expressions optimizer Optimizer rules core Core DataFusion crate substrait Changes to the substrait crate proto Related to proto crate physical-plan Changes to the physical-plan crate labels Jul 23, 2026

Xuanwo mentioned this pull request Jul 23, 2026

feat: support ASOF JOIN SQL #23830

Draft

github-actions Bot added the auto detected api change Auto detected API change label Jul 23, 2026

This was referenced Jul 23, 2026

feat: add ASOF joins to DataFrame API #23831

Draft

feat: serialize ASOF join plans #23832

Draft

bench: add ASOF join benchmarks #23833

Draft

feat: support ASOF joins #23738

Open

github-actions Bot added the physical-expr Changes to the physical-expr crates label Aug 14, 2026

Xuanwo force-pushed the xuanwo/asof-logical branch from 6429c60 to 1a925ce Compare August 20, 2026 17:56

Xuanwo added 2 commits August 28, 2026 13:07

feat: add ASOF join logical semantics

c665165

fix: preserve ASOF minimum row bound

eace163

Xuanwo force-pushed the xuanwo/asof-logical branch from 1a925ce to eace163 Compare August 28, 2026 05:40

github-actions Bot removed physical-expr Changes to the physical-expr crates physical-plan Changes to the physical-plan crate labels Aug 28, 2026

Xuanwo marked this pull request as ready for review August 28, 2026 05:53

jayzhan211 mentioned this pull request Aug 29, 2026

refactor(asof_join): enhance match value handling and comparator caching #24519

Merged

jayzhan211 approved these changes Aug 30, 2026

View reviewed changes

2010YOUY01 reviewed Aug 30, 2026

View reviewed changes

refactor: defer ASOF logical optimizations

65752d9

github-actions Bot added the physical-plan Changes to the physical-plan crate label Aug 30, 2026

This was referenced Aug 30, 2026

perf: preserve left functional dependencies for ASOF joins #24799

Draft

perf: push left filters through ASOF joins #24801

Draft

Navigation Menu

Uh oh!

feat: add ASOF join logical semantics - #23829

feat: add ASOF join logical semantics#23829
Xuanwo wants to merge 3 commits into
apache:mainfrom
Xuanwo:xuanwo/asof-logical

Xuanwo commented Jul 23, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Jul 23, 2026 •

edited

Loading

Uh oh!

codecov-commenter commented Jul 23, 2026 •

edited

Loading

Uh oh!

Xuanwo commented Aug 28, 2026

Uh oh!

jayzhan211 left a comment

Uh oh!

Xuanwo commented Aug 30, 2026

Uh oh!

2010YOUY01 left a comment •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

2010YOUY01 Aug 30, 2026

Uh oh!

Xuanwo Aug 30, 2026

Uh oh!

Xuanwo commented Aug 30, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

Xuanwo commented Jul 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

github-actions Bot commented Jul 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov-commenter commented Jul 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Xuanwo commented Aug 28, 2026

Uh oh!

jayzhan211 left a comment

Choose a reason for hiding this comment

Uh oh!

Xuanwo commented Aug 30, 2026

Uh oh!

2010YOUY01 left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

2010YOUY01 Aug 30, 2026

Choose a reason for hiding this comment

Uh oh!

Xuanwo Aug 30, 2026

Choose a reason for hiding this comment

Uh oh!

Xuanwo commented Aug 30, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Xuanwo commented Jul 23, 2026 •

edited

Loading

github-actions Bot commented Jul 23, 2026 •

edited

Loading

codecov-commenter commented Jul 23, 2026 •

edited

Loading

2010YOUY01 left a comment •

edited

Loading