feat(spark): add equal_null - #24827
Open
advitrocks9 wants to merge 1 commit into
Open
Conversation
Rewrites to IS NOT DISTINCT FROM in simplify(), mirroring Spark's own RuntimeReplaceable alias of EqualNullSafe. Fills the porting-script stub, restoring the two queries that lost an argument to the typeof() dedup.
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #24827 +/- ##
==========================================
+ Coverage 81.42% 81.57% +0.15%
==========================================
Files 1120 1124 +4
Lines 402021 406651 +4630
Branches 402021 406651 +4630
==========================================
+ Hits 327357 331741 +4384
+ Misses 55484 55467 -17
- Partials 19180 19443 +263 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
No issue filed. Part of #15914, which tracks the Spark function library.
Rationale for this change
equal_nullis missing, and themiscmodule it belongs in had no functions in it at all. InSpark it is an alias:
EqualNull(l, r)is rewritten toEqualNullSafe, the<=>operator, soit is null-safe equality. Both NULL is true, and the result is never NULL:
DataFusion already has that operator as
IS NOT DISTINCT FROM, so this mirrors Spark's ownstructure instead of writing a second comparison kernel.
What changes are included in this PR?
misc/equal_null.rsand its registration.simplify()rewrites to the operator, which isSpark's
replacementfield, andinvoke_with_argscallsapply_cmpwith the same operator sothe function also works with the logical optimizer disabled, as the crate README requires for
Comet.
The
.sltstub is filled in with 27 assertions. Two of its commented-out queries were malformed:the porting script wrote one cast per distinct
typeof()key, so a call with two identicalliterals lost an argument. The same artifact affects 10 more pairs in 6 other spark files, left
alone here.
Two divergences from Spark are left alone because they belong to the operator, not to this
function. Spark treats
-NaNandNaNas equal and DataFusion does not, which is true ofIS NOT DISTINCT FROMgenerally. Spark also rejects maps at analysis sinceMapTypeis notorderable, while
comparison_coercionhere accepts them.Are these changes tested?
Yes,
spark/misc/equal_null.sltgoes from a skipped stub to 27 assertions covering the truthtable, float ordering, columns, arrays, structs, decimals and the arity errors.
Reverting
simplify()to plainEqfails 9 of them, so the file is not passing on constantfolding. Putting
invoke_with_argsback to a stub fails the two queries that run with theoptimizer off, and nothing else.
Are there any user-facing changes?
Yes,
equal_nullis a new function indatafusion-spark. Nothing existing changes.