Skip to content

feat: support ASOF JOIN SQL - #23830

Draft
Xuanwo wants to merge 5 commits into
apache:mainfrom
Xuanwo:xuanwo/asof-sql
Draft

feat: support ASOF JOIN SQL#23830
Xuanwo wants to merge 5 commits into
apache:mainfrom
Xuanwo:xuanwo/asof-sql

Conversation

@Xuanwo

@Xuanwo Xuanwo commented Jul 23, 2026

Copy link
Copy Markdown
Member

Which issue does this PR close?

Rationale for this change

This is the SQL frontend layer of the ASOF JOIN stack. It adds syntax and
unparsing only after the physical and logical contracts have been separated
into their own reviewable PRs.

GitHub shows the cumulative stack because fork branches cannot be selected as
upstream PR bases. The isolated diff for this layer is:
Xuanwo/datafusion@xuanwo/asof-logical...xuanwo/asof-sql

What changes are included in this PR?

  • Plan ASOF JOIN ... MATCH_CONDITION (...) with optional ON or USING
    equality keys.
  • Reject unsupported match shapes and non-equality ON predicates.
  • Unparse ASOF joins while preserving right-side candidate preselection and
    nested join scope.
  • Document the supported SQL syntax and semantics, including qualified
    USING keys, left-partitioned broadcast execution, the full-right memory
    requirement, repeated scans, and the absence of spill/repartitioned ASOF.
  • Add SQL integration and sqllogictest coverage for all four match directions,
    coercion, equality-free joins, USING, invalid contracts, EXPLAIN, boundedness,
    and optimized-plan round trips.
  • Verify the broadcast topology with a multi-partition left input: left
    partitioning is preserved while the right input is single-partitioned.

Are these changes tested?

Yes:

  • cargo fmt --all
  • ./ci/scripts/doc_prettier_check.sh --write --allow-dirty
  • cargo clippy --all-targets --all-features -- -D warnings
  • cargo test -p datafusion --test core_integration asof --all-features
  • cargo test -p datafusion-sqllogictest --test sqllogictests --all-features -- asof_join.slt
  • The extended workspace test command from the contributor guide

Are there any user-facing changes?

Users can express Snowflake-style ASOF joins in SQL with
MATCH_CONDITION, optional equality keys, and <, <=, >, or >= match
directions. With USING, wildcard output exposes one unqualified key while
both qualified input keys remain addressable. The user guide also documents
the initial broadcast strategy and its memory/no-spill limitations.

This PR should be reviewed and merged after #23829. It does not depend on the
optional floating-point follow-up #24375.

@github-actions github-actions Bot added documentation Improvements or additions to documentation sql SQL Planner logical-expr Logical plan and expressions optimizer Optimizer rules core Core DataFusion crate sqllogictest SQL Logic Tests (.slt) substrait Changes to the substrait crate proto Related to proto crate physical-plan Changes to the physical-plan crate labels Jul 23, 2026
@codecov-commenter

codecov-commenter commented Jul 23, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 67.23869% with 210 lines in your changes missing coverage. Please review.
✅ Project coverage is 81.52%. Comparing base (c4910e0) to head (15ddfb8).
⚠️ Report is 20 commits behind head on main.

Files with missing lines Patch % Lines
datafusion/sql/src/unparser/plan.rs 51.16% 77 Missing and 7 partials ⚠️
datafusion/expr/src/logical_plan/plan.rs 59.48% 63 Missing and 16 partials ⚠️
datafusion/sql/src/relation/join.rs 83.33% 7 Missing and 8 partials ⚠️
datafusion/expr/src/logical_plan/display.rs 0.00% 10 Missing ⚠️
datafusion/expr/src/logical_plan/builder.rs 83.92% 1 Missing and 8 partials ⚠️
datafusion/core/src/physical_planner.rs 85.36% 4 Missing and 2 partials ⚠️
datafusion/expr/src/logical_plan/tree_node.rs 93.02% 0 Missing and 3 partials ⚠️
datafusion/optimizer/src/analyzer/type_coercion.rs 92.85% 1 Missing and 1 partial ⚠️
datafusion/optimizer/src/optimizer.rs 50.00% 0 Missing and 2 partials ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main   #23830      +/-   ##
==========================================
+ Coverage   81.47%   81.52%   +0.05%     
==========================================
  Files        1122     1123       +1     
  Lines      403629   406730    +3101     
  Branches   403629   406730    +3101     
==========================================
+ Hits       328866   331597    +2731     
- Misses      55510    55721     +211     
- Partials    19253    19412     +159     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@github-actions

github-actions Bot commented Jul 23, 2026

Copy link
Copy Markdown

Thank you for opening this pull request!

Reviewer note: cargo-semver-checks reported the current version number is not SemVer-compatible with the changes in this pull request (compared against the base branch).

Details
     Cloning apache/main
    Building datafusion v55.0.0 (current)
       Built [  60.809s] (current)
     Parsing datafusion v55.0.0 (current)
      Parsed [   0.034s] (current)
    Building datafusion v55.0.0 (baseline)
       Built [  58.678s] (baseline)
     Parsing datafusion v55.0.0 (baseline)
      Parsed [   0.035s] (baseline)
    Checking datafusion v55.0.0 -> v55.0.0 (no change; assume patch)
     Checked [   0.575s] 223 checks: 223 pass, 31 skip
     Summary no semver update required
    Finished [ 122.217s] datafusion
    Building datafusion-expr v55.0.0 (current)
       Built [  28.713s] (current)
     Parsing datafusion-expr v55.0.0 (current)
      Parsed [   0.075s] (current)
    Building datafusion-expr v55.0.0 (baseline)
       Built [  29.014s] (baseline)
     Parsing datafusion-expr v55.0.0 (baseline)
      Parsed [   0.075s] (baseline)
    Checking datafusion-expr v55.0.0 -> v55.0.0 (no change; assume patch)
     Checked [   1.156s] 223 checks: 222 pass, 1 fail, 0 warn, 31 skip

--- failure enum_variant_added: enum variant added on exhaustive enum ---

Description:
A publicly-visible enum without #[non_exhaustive] has a new variant.
        ref: https://doc.rust-lang.org/cargo/reference/semver.html#enum-variant-new
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.50.0/src/lints/enum_variant_added.ron

Failed in:
  variant LogicalPlan:AsOfJoin in /home/runner/work/datafusion/datafusion/datafusion/expr/src/logical_plan/plan.rs:301
  variant LogicalPlan:AsOfJoin in /home/runner/work/datafusion/datafusion/datafusion/expr/src/logical_plan/plan.rs:301

     Summary semver requires new major version: 1 major and 0 minor checks failed
    Finished [  60.245s] datafusion-expr
    Building datafusion-optimizer v55.0.0 (current)
       Built [  26.803s] (current)
     Parsing datafusion-optimizer v55.0.0 (current)
      Parsed [   0.030s] (current)
    Building datafusion-optimizer v55.0.0 (baseline)
       Built [  27.686s] (baseline)
     Parsing datafusion-optimizer v55.0.0 (baseline)
      Parsed [   0.031s] (baseline)
    Checking datafusion-optimizer v55.0.0 -> v55.0.0 (no change; assume patch)
     Checked [   0.157s] 223 checks: 223 pass, 31 skip
     Summary no semver update required
    Finished [  55.545s] datafusion-optimizer
    Building datafusion-physical-plan v55.0.0 (current)
       Built [  38.728s] (current)
     Parsing datafusion-physical-plan v55.0.0 (current)
      Parsed [   0.149s] (current)
    Building datafusion-physical-plan v55.0.0 (baseline)
       Built [  38.525s] (baseline)
     Parsing datafusion-physical-plan v55.0.0 (baseline)
      Parsed [   0.148s] (baseline)
    Checking datafusion-physical-plan v55.0.0 -> v55.0.0 (no change; assume patch)
     Checked [   0.654s] 223 checks: 223 pass, 31 skip
     Summary no semver update required
    Finished [  79.345s] datafusion-physical-plan
    Building datafusion-proto v55.0.0 (current)
       Built [  54.351s] (current)
     Parsing datafusion-proto v55.0.0 (current)
      Parsed [   0.017s] (current)
    Building datafusion-proto v55.0.0 (baseline)
       Built [  54.815s] (baseline)
     Parsing datafusion-proto v55.0.0 (baseline)
      Parsed [   0.018s] (baseline)
    Checking datafusion-proto v55.0.0 -> v55.0.0 (no change; assume patch)
     Checked [   0.115s] 223 checks: 223 pass, 31 skip
     Summary no semver update required
    Finished [ 110.498s] datafusion-proto
    Building datafusion-sql v55.0.0 (current)
       Built [  42.792s] (current)
     Parsing datafusion-sql v55.0.0 (current)
      Parsed [   0.031s] (current)
    Building datafusion-sql v55.0.0 (baseline)
       Built [  42.874s] (baseline)
     Parsing datafusion-sql v55.0.0 (baseline)
      Parsed [   0.033s] (baseline)
    Checking datafusion-sql v55.0.0 -> v55.0.0 (no change; assume patch)
     Checked [   0.222s] 223 checks: 223 pass, 31 skip
     Summary no semver update required
    Finished [  87.098s] datafusion-sql
    Building datafusion-sqllogictest v55.0.0 (current)
       Built [ 102.265s] (current)
     Parsing datafusion-sqllogictest v55.0.0 (current)
      Parsed [   0.021s] (current)
    Building datafusion-sqllogictest v55.0.0 (baseline)
       Built [ 102.737s] (baseline)
     Parsing datafusion-sqllogictest v55.0.0 (baseline)
      Parsed [   0.023s] (baseline)
    Checking datafusion-sqllogictest v55.0.0 -> v55.0.0 (no change; assume patch)
     Checked [   0.090s] 223 checks: 223 pass, 31 skip
     Summary no semver update required
    Finished [ 208.177s] datafusion-sqllogictest
    Building datafusion-substrait v55.0.0 (current)
       Built [ 313.000s] (current)
     Parsing datafusion-substrait v55.0.0 (current)
      Parsed [   0.018s] (current)
    Building datafusion-substrait v55.0.0 (baseline)
       Built [ 318.409s] (baseline)
     Parsing datafusion-substrait v55.0.0 (baseline)
      Parsed [   0.017s] (baseline)
    Checking datafusion-substrait v55.0.0 -> v55.0.0 (no change; assume patch)
     Checked [   0.210s] 223 checks: 223 pass, 31 skip
     Summary no semver update required
    Finished [ 633.692s] datafusion-substrait

@github-actions github-actions Bot added the auto detected api change Auto detected API change label Jul 23, 2026
@github-actions github-actions Bot added the physical-expr Changes to the physical-expr crates label Aug 14, 2026
@github-actions github-actions Bot removed physical-expr Changes to the physical-expr crates physical-plan Changes to the physical-plan crate labels Aug 28, 2026
@github-actions github-actions Bot added the physical-plan Changes to the physical-plan crate label Aug 30, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

auto detected api change Auto detected API change core Core DataFusion crate documentation Improvements or additions to documentation logical-expr Logical plan and expressions optimizer Optimizer rules physical-plan Changes to the physical-plan crate proto Related to proto crate sql SQL Planner sqllogictest SQL Logic Tests (.slt) substrait Changes to the substrait crate

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants