Repository navigation
fix: compare Duration and Interval as exact elapsed time in comparisons - #26742
Open
nikolas-sapa wants to merge 1 commit into
Open
nikolas-sapa wants to merge 1 commit into
nikolas-sapa wants to merge 1 commit into
Conversation
Comparing a Duration (e.g. ts1 - ts2) with an INTERVAL that has a days or months component was decided by the wrong field: the mixed comparison coerced both sides to Interval(MonthDayNano), whose derived ordering compares months, then days, then nanoseconds, so a nanoseconds-only duration always compared incorrectly against any interval with non-zero days (e.g. a 10-day duration satisfied <= INTERVAL '3 days'). Route mixed Duration/Interval comparisons through a dedicated coercion that compares exact elapsed time by coercing both sides to Duration(Nanosecond). Intervals whose months and days are zero compare exactly; any other interval fails loudly when cast (during constant folding or at execution) instead of silently comparing on the wrong field. Interval(YearMonth) and Interval(DayTime) have no exact conversion and fail at planning, as before. Duration-vs-Duration and Interval-vs-Interval comparison coercion is unchanged. Closes apache#26671
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Durationto anIntervalwith a days field returns wrong results (e.g.ts - ts <= INTERVAL '3 days') #26671Rationale for this change
Comparing a
Duration(e.g.ts1 - ts2) with anINTERVALthat has a days or monthscomponent silently returns wrong results. For example, both of these returned
truefor a 10-day duration:
so range filters built on such comparisons silently filter the wrong rows.
Root cause: the mixed comparison coerced both sides to
Interval(MonthDayNano).arrow-cast represents the duration side as
IntervalMonthDayNano(0, 0, nanos), andIntervalMonthDayNano's derived ordering compares months, then days, then nanoseconds —so any interval with non-zero days or months decides the comparison on a field the
duration side can never influence.
What changes are included in this PR?
Duration/Intervalcomparisons now compare exact elapsed time: both sidescoerce to
Duration(Nanosecond)via a newduration_interval_comparison_coercion,routed from
comparison_coercion_innerso it covers comparison operators,INlists,
CASE/WHENandBETWEEN.INTERVAL '240 hours') compareexactly and correctly.
constant folding or at execution — with Arrow's "Cannot convert interval containing
non-zero months or days to duration", instead of silently returning wrong results.
Interval(YearMonth)andInterval(DayTime)have no exact duration conversion andfail at planning ("cannot infer common argument type"), as before.
Duration-vs-DurationandInterval-vs-Intervalcomparison coercion is unchanged:both still compare via
Interval(MonthDayNano)'s ordering, consistent with Postgres(so
interval '3 days' > interval '240 hours'remains true). Only mixed pairs change.Design note for reviewers: an alternative semantics would treat a day as exactly
86,400 seconds and only reject months-bearing intervals. This PR deliberately follows
Arrow's cast semantics instead: a "day" added to a timezone-aware timestamp is not
always 86,400 seconds (DST transitions), so treating days as fixed in comparisons
would silently disagree with
ts + INTERVAL '1 day'arithmetic — erroring keeps thetwo consistent, and mirrors Arrow's own refusal to convert. Happy to switch to
days == 0-only-erroring if maintainers prefer.What is the testing strategy for this PR?
test_duration_interval_comparison_coercionindatafusion/expr-common/src/type_coercion/binary/tests/comparison.rscovers all sixcomparison operators plus
IS DISTINCT FROM/IS NOT DISTINCT FROM, all fourduration units, both operand orders, the
YearMonth/DayTimerejection, and thatDuration-vs-Duration / Interval-vs-Interval coercions are unchanged.
datafusion/sqllogictest/test_files/datetime/interval.slt:the constant-folded path (scalar duration vs scalar interval), the runtime path
(column values, both operand orders), an
IN-list with the mixed pair,pure-nanosecond intervals comparing exactly, and the exact error messages for
months/days-bearing intervals and for
Interval(YearMonth)/Interval(DayTime).cargo test -p datafusion-expr-common --lib(222 passed),the
datetime/intervalsqllogictest files,cargo fmt --all -- --check,cargo clippy --all-targets --workspace --all-features -- -D warnings, and the fullworkspace suite via
RUST_BACKTRACE=1 cargo xtask ci step test workspace(13,043 passed / 0 failed / 15 ignored on the pre-amend tree; the amend adds only
test assertions, re-verified by the targeted runs above).
Are there any user-facing changes?
Yes — a behavior change from silently wrong results to explicit errors for
Duration-vs-Interval comparisons where the interval has non-zero months or days;
pure-time intervals now compare correctly. No public API changes.