feat: support floating-point ASOF equality keys by Xuanwo · Pull Request #24375 · apache/datafusion

Xuanwo · 2026-08-14T16:52:20Z

Which issue does this PR close?

Part of ASOF join support / Specialize Range Joins #318.
Umbrella PR: feat: support ASOF joins #23738.
Independent follow-up to feat: add ASOF join physical operator #23828 (merged).

Rationale for this change

The physical ASOF operator in #23828 rejects floating-point equality keys because
Arrow's total ordering distinguishes -0.0 from +0.0, while join equality
treats them as equal. That mismatch can split one equality group during the
required input sort and make the forward-only ASOF scan choose the wrong row.

This PR adds complete Float16, Float32, and Float64 equality-key support as an
independent follow-up. The logical, SQL, DataFrame, serialization, and benchmark
PRs in #23738 do not depend on it and can merge first while floating equality
keys remain rejected during physical planning.

#23828 is merged, so this PR's diff against main is the isolated
floating-point equality layer.

What changes are included in this PR?

Add an order-preserving NormalizeFloatZeroExpr that maps -0.0 to
+0.0 without changing other values.
Use the expression only for floating-point ASOF equality-key ordering, so the
required sort order agrees with the existing join-key comparator.
Add protobuf encode/decode support so plans containing the required sort
expression round-trip correctly.
Add a regression covering Float16, Float32, and Float64 where
(-0.0, ts=10) followed by (+0.0, ts=1) must remain one group and select
ts=1 for a left (+0.0, ts=5) row.

Are these changes tested?

Yes:

cargo fmt --all
cargo clippy --all-targets --all-features -- -D warnings
ASOF physical operator tests, including all three floating-point types
Logical and physical protobuf round-trip tests
Extended workspace tests from the contributor guide
FFI integration tests

Are there any user-facing changes?

ASOF equality keys can use Float16, Float32, and Float64 while treating signed
zero consistently with join equality. This layer also adds a public physical
expression and an append-only protobuf oneof variant; existing wire tags are not
reused.

github-actions · 2026-08-14T16:58:24Z

Thank you for opening this pull request!

Reviewer note: cargo-semver-checks reported the current version number is not SemVer-compatible with the changes in this pull request (compared against the base branch).

Details

     Cloning apache/main
    Building datafusion-physical-expr v55.0.0 (current)
       Built [  48.779s] (current)
     Parsing datafusion-physical-expr v55.0.0 (current)
      Parsed [   0.039s] (current)
    Building datafusion-physical-expr v55.0.0 (baseline)
       Built [  23.618s] (baseline)
     Parsing datafusion-physical-expr v55.0.0 (baseline)
      Parsed [   0.040s] (baseline)
    Checking datafusion-physical-expr v55.0.0 -> v55.0.0 (no change; assume patch)
     Checked [   0.463s] 223 checks: 223 pass, 31 skip
     Summary no semver update required
    Finished [  74.438s] datafusion-physical-expr
    Building datafusion-physical-plan v55.0.0 (current)
       Built [  30.910s] (current)
     Parsing datafusion-physical-plan v55.0.0 (current)
      Parsed [   0.124s] (current)
    Building datafusion-physical-plan v55.0.0 (baseline)
       Built [  32.382s] (baseline)
     Parsing datafusion-physical-plan v55.0.0 (baseline)
      Parsed [   0.123s] (baseline)
    Checking datafusion-physical-plan v55.0.0 -> v55.0.0 (no change; assume patch)
     Checked [   0.933s] 223 checks: 223 pass, 31 skip
     Summary no semver update required
    Finished [  65.948s] datafusion-physical-plan
    Building datafusion-proto v55.0.0 (current)
       Built [  44.854s] (current)
     Parsing datafusion-proto v55.0.0 (current)
      Parsed [   0.015s] (current)
    Building datafusion-proto v55.0.0 (baseline)
       Built [  44.076s] (baseline)
     Parsing datafusion-proto v55.0.0 (baseline)
      Parsed [   0.015s] (baseline)
    Checking datafusion-proto v55.0.0 -> v55.0.0 (no change; assume patch)
     Checked [   0.146s] 223 checks: 223 pass, 31 skip
     Summary no semver update required
    Finished [  90.879s] datafusion-proto
    Building datafusion-proto-models v55.0.0 (current)
       Built [  20.811s] (current)
     Parsing datafusion-proto-models v55.0.0 (current)
      Parsed [   0.104s] (current)
    Building datafusion-proto-models v55.0.0 (baseline)
       Built [  20.277s] (baseline)
     Parsing datafusion-proto-models v55.0.0 (baseline)
      Parsed [   0.105s] (baseline)
    Checking datafusion-proto-models v55.0.0 -> v55.0.0 (no change; assume patch)
     Checked [   2.286s] 223 checks: 222 pass, 1 fail, 0 warn, 31 skip

--- failure enum_variant_added: enum variant added on exhaustive enum ---

Description:
A publicly-visible enum without #[non_exhaustive] has a new variant.
        ref: https://doc.rust-lang.org/cargo/reference/semver.html#enum-variant-new
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.50.0/src/lints/enum_variant_added.ron

Failed in:
  variant ExprType:NormalizeFloatZero in /home/runner/work/datafusion/datafusion/datafusion/proto-models/src/generated/prost.rs:1630
  variant ExprType:NormalizeFloatZero in /home/runner/work/datafusion/datafusion/datafusion/proto-models/src/generated/prost.rs:1630

     Summary semver requires new major version: 1 major and 0 minor checks failed
    Finished [  44.979s] datafusion-proto-models

codecov-commenter · 2026-08-14T17:16:03Z

Codecov Report

❌ Patch coverage is 63.32180% with 106 lines in your changes missing coverage. Please review.
✅ Project coverage is 81.36%. Comparing base (0eecdfc) to head (ee80125).
⚠️ Report is 44 commits behind head on main.

Files with missing lines	Patch %	Lines
datafusion/proto-models/src/generated/pbjson.rs	0.00%	57 Missing ⚠️
...sical-expr/src/expressions/normalize_float_zero.rs	71.73%	27 Missing and 12 partials ⚠️
datafusion/physical-plan/src/joins/asof_join.rs	90.32%	2 Missing and 7 partials ⚠️
datafusion/proto/src/physical_plan/from_proto.rs	0.00%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #24375      +/-   ##
==========================================
+ Coverage   81.31%   81.36%   +0.05%     
==========================================
  Files        1117     1119       +2     
  Lines      395987   398916    +2929     
  Branches   395987   398916    +2929     
==========================================
+ Hits       321993   324593    +2600     
- Misses      55177    55332     +155     
- Partials    18817    18991     +174

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

## Which issue does this PR close? - Part of #318. - Umbrella PR: apache#23738. - Follow-up for floating-point equality keys: apache#24375. ## Rationale for this change This is the first layer of the ASOF JOIN stack. It establishes a broadcast-based physical execution contract independently so later floating-point equality, logical-plan, SQL, DataFrame, and serialization changes can be reviewed as smaller follow-up PRs. The initial implementation deliberately favors the simpler broadcast design: the right input must fit in memory and each left partition scans the shared right-side batches. A repartitioned implementation can be evaluated separately without changing the ASOF semantics introduced here. Floating-point equality keys are rejected in this base layer because Arrow's required sort order distinguishes `-0.0` from `+0.0` while join equality does not. apache#24375 adds the required ordering normalization as an independently reviewable layer. ## What changes are included in this PR? - Add `AsOfJoinExec` for left-preserving, Snowflake-style ASOF semantics. - Coalesce and collect the ordered right input once, then share it across all left partitions. - Keep the left input partitioned so each partition can scan independently and preserve the left-side output partitioning. - Preserve merge state across input and output batch boundaries. - Reserve each retained Arrow buffer exactly once, including when right-side batches are zero-copy slices, and expose build, match, and output metrics. - Define output properties and statistics for the broadcast execution model. - Reject floating-point equality keys until apache#24375 supplies a sort/equality contract that handles signed zero correctly. - Add physical operator tests covering match directions, equality groups, batch boundaries, unmatched rows, invalid contracts, shared-buffer memory accounting, multi-partition broadcast execution, and float-key rejection. ## Are these changes tested? Yes: - `cargo fmt --all` - `cargo clippy --all-targets --all-features -- -D warnings` - `cargo test -p datafusion-physical-plan joins::asof_join --all-features` - Extended workspace tests from the contributor guide - FFI integration tests ## Are there any user-facing changes? This adds a new physical operator API. The base operator deliberately rejects floating-point equality keys; apache#24375 adds full Float16, Float32, and Float64 support. SQL and DataFrame APIs are left to later dependent PRs. --------- Co-authored-by: Yongting You <2010youy01@gmail.com>

Xuanwo · 2026-08-24T08:17:54Z

cc @2010YOUY01, this PR is now ready for review!

jayzhan211 · 2026-08-26T12:18:55Z

+/// order-preserving but not strictly order-preserving because it collapses the
+/// two signed-zero representations.
+#[derive(Debug, Eq)]
+pub struct NormalizeFloatZeroExpr {


I still have a question about why we introduce another PhysicalExpr to handle this type issue instead of just calling normalize_float_zero.

Good question. normalize_float_zero works on an already evaluated array, but here normalization must be part of the required ordering.

For example, raw [key, ts] may be (-0, 10), (+0, 1). Normalizing only during comparison makes them one equality group but leaves ts as [10, 1], breaking the forward scan. We need the input sorted by [normalize_float_zero(key), ts].

The new PhysicalExpr is the plan-level wrapper around the existing helper, allowing SortExec, ordering enforcement, and protobuf to represent this without changing float sorting globally.

Let me know if this addressed your concerns. I'm open to better ideas!

I'll take a look at this. My read is that it's handling the edge cases for asof join.

One thing I'm worried about: we'd end up with NormalizeFloatZeroExpr and normalize_float_zero spread across a lot of call sites, which seems likely to cause confusion down the line.

Would it make sense to land a minimal PR first — just enough to get a simple asof join working — and then iterate on the other cases after that? What do you think?

Yep, only ASOF join use this part of logic. I'm open to leave it as a follow-up. I'll adjust the stack and get it a out so that we can work on ASOF join first.

github-actions Bot added physical-expr Changes to the physical-expr crates proto Related to proto crate physical-plan Changes to the physical-plan crate labels Aug 14, 2026

github-actions Bot added the auto detected api change Auto detected API change label Aug 14, 2026

fix: support floating ASOF equality keys

8f501fd

Xuanwo force-pushed the xuanwo/asof-float-equality branch from 6b0bd0e to 8f501fd Compare August 20, 2026 17:56

2010YOUY01 mentioned this pull request Aug 23, 2026

refactor(asof_join): enhance match value handling and comparator caching #24519

Merged

Xuanwo marked this pull request as ready for review August 24, 2026 08:17

fix: normalize signed-zero expression bounds

ee80125

github-actions Bot added the documentation Improvements or additions to documentation label Aug 24, 2026

jayzhan211 reviewed Aug 26, 2026

View reviewed changes

Navigation Menu

Uh oh!

feat: support floating-point ASOF equality keys - #24375

feat: support floating-point ASOF equality keys#24375
Xuanwo wants to merge 2 commits into
apache:mainfrom
Xuanwo:xuanwo/asof-float-equality

Xuanwo commented Aug 14, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Aug 14, 2026 •

edited

Loading

Uh oh!

codecov-commenter commented Aug 14, 2026 •

edited

Loading

Uh oh!

Xuanwo commented Aug 24, 2026

Uh oh!

jayzhan211 Aug 26, 2026

Uh oh!

Xuanwo Aug 26, 2026

Uh oh!

jayzhan211 Aug 27, 2026 •

edited

Loading

Uh oh!

Xuanwo Aug 28, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

Xuanwo commented Aug 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

github-actions Bot commented Aug 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov-commenter commented Aug 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Xuanwo commented Aug 24, 2026

Uh oh!

jayzhan211 Aug 26, 2026

Choose a reason for hiding this comment

Uh oh!

Xuanwo Aug 26, 2026

Choose a reason for hiding this comment

Uh oh!

jayzhan211 Aug 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Xuanwo Aug 28, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Xuanwo commented Aug 14, 2026 •

edited

Loading

github-actions Bot commented Aug 14, 2026 •

edited

Loading

codecov-commenter commented Aug 14, 2026 •

edited

Loading

jayzhan211 Aug 27, 2026 •

edited

Loading