Skip to content

feat(spark): add equal_null - #24827

Open
advitrocks9 wants to merge 1 commit into
apache:mainfrom
advitrocks9:b8-spark-equal-null
Open

feat(spark): add equal_null#24827
advitrocks9 wants to merge 1 commit into
apache:mainfrom
advitrocks9:b8-spark-equal-null

Conversation

@advitrocks9

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

No issue filed. Part of #15914, which tracks the Spark function library.

Rationale for this change

equal_null is missing, and the misc module it belongs in had no functions in it at all. In
Spark it is an alias: EqualNull(l, r) is rewritten to EqualNullSafe, the <=> operator, so
it is null-safe equality. Both NULL is true, and the result is never NULL:

SELECT equal_null(NULL, NULL);              -- true
SELECT equal_null(NULL::int, 1::int);       -- false

DataFusion already has that operator as IS NOT DISTINCT FROM, so this mirrors Spark's own
structure instead of writing a second comparison kernel.

What changes are included in this PR?

misc/equal_null.rs and its registration. simplify() rewrites to the operator, which is
Spark's replacement field, and invoke_with_args calls apply_cmp with the same operator so
the function also works with the logical optimizer disabled, as the crate README requires for
Comet.

The .slt stub is filled in with 27 assertions. Two of its commented-out queries were malformed:
the porting script wrote one cast per distinct typeof() key, so a call with two identical
literals lost an argument. The same artifact affects 10 more pairs in 6 other spark files, left
alone here.

Two divergences from Spark are left alone because they belong to the operator, not to this
function. Spark treats -NaN and NaN as equal and DataFusion does not, which is true of
IS NOT DISTINCT FROM generally. Spark also rejects maps at analysis since MapType is not
orderable, while comparison_coercion here accepts them.

Are these changes tested?

Yes, spark/misc/equal_null.slt goes from a skipped stub to 27 assertions covering the truth
table, float ordering, columns, arrays, structs, decimals and the arity errors.

Reverting simplify() to plain Eq fails 9 of them, so the file is not passing on constant
folding. Putting invoke_with_args back to a stub fails the two queries that run with the
optimizer off, and nothing else.

Are there any user-facing changes?

Yes, equal_null is a new function in datafusion-spark. Nothing existing changes.

Rewrites to IS NOT DISTINCT FROM in simplify(), mirroring Spark's own
RuntimeReplaceable alias of EqualNullSafe. Fills the porting-script stub,
restoring the two queries that lost an argument to the typeof() dedup.
@github-actions github-actions Bot added sqllogictest SQL Logic Tests (.slt) spark labels Aug 31, 2026
@codecov-commenter

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 85.71429% with 6 lines in your changes missing coverage. Please review.
✅ Project coverage is 81.57%. Comparing base (408fc6f) to head (c38bb51).
⚠️ Report is 39 commits behind head on main.

Files with missing lines Patch % Lines
datafusion/spark/src/function/misc/equal_null.rs 85.36% 4 Missing and 2 partials ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main   #24827      +/-   ##
==========================================
+ Coverage   81.42%   81.57%   +0.15%     
==========================================
  Files        1120     1124       +4     
  Lines      402021   406651    +4630     
  Branches   402021   406651    +4630     
==========================================
+ Hits       327357   331741    +4384     
+ Misses      55484    55467      -17     
- Partials    19180    19443     +263     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

spark sqllogictest SQL Logic Tests (.slt)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants