Skip to content

fix: disable aggregate dynamic filter when unsupported expressions are present (#24816) - #24818

Closed
AboEl3iz wants to merge 2 commits into
apache:mainfrom
AboEl3iz:fix-aggregate-dynamic-filter
Closed

fix: disable aggregate dynamic filter when unsupported expressions are present (#24816)#24818
AboEl3iz wants to merge 2 commits into
apache:mainfrom
AboEl3iz:fix-aggregate-dynamic-filter

Conversation

@AboEl3iz

Copy link
Copy Markdown

Which issue does this PR close?

Rationale for this change

When an aggregate query contains both supported expressions (such as MIN(a)) and unsupported expressions (such as MIN(c + 1)), DataFusion previously built a dynamic filter using only the supported expressions.

Because all aggregate expressions in an AggregateExec share the same input scan stream, pushing a dynamic filter constructed from a subset of aggregate expressions prunes rows required by the unsupported expressions, producing incorrect aggregate results.

Aggregate dynamic filter pushdown must be an all-or-nothing optimization: if any aggregate expression in the AggregateExec cannot produce a valid dynamic filter predicate, dynamic filtering must be disabled (dynamic_filter = None) for that node.

What changes are included in this PR?

  • Updated AggregateExec::init_dynamic_filter in datafusion/physical-plan/src/aggregates/mod.rs to return early when an aggregate argument is not a single Column reference.
  • Added unit test test_dynamic_filter_disabled_when_unsupported_expr_present in datafusion/physical-plan/src/aggregates/mod.rs.
  • Added integration test dynamic_rg_pruning_disabled_when_unsupported_aggregate_present in datafusion/core/tests/parquet/dynamic_row_group_pruning.rs reproducing issue Aggregate dynamic filter can prune rows required by unsupported expressions #24816.

What is the testing strategy for this PR?

  • Unit test: test_dynamic_filter_disabled_when_unsupported_expr_present in datafusion-physical-plan checks that dynamic_filter remains None when an unsupported expression like MIN(a + 1) is present.
  • Integration test: dynamic_rg_pruning_disabled_when_unsupported_aggregate_present in datafusion/core/tests/parquet/dynamic_row_group_pruning.rs verifies that queries with MIN(a), MAX(a), MAX(b), MIN(c + 1) evaluate correctly (71) without row groups being pruned.

Are there any user-facing changes?

No breaking API or user-facing changes. Correctness fix for aggregate queries with dynamic filter pushdown.

@github-actions github-actions Bot added core Core DataFusion crate physical-plan Changes to the physical-plan crate labels Aug 31, 2026
@2010YOUY01

Copy link
Copy Markdown
Contributor

It's a duplicate of an earlier PR #24817 without coordination, so close it for now.

Please reopen if you can communicate with the author/reviewers and agree on proceeding with this PR instead.

@2010YOUY01 2010YOUY01 closed this Aug 31, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

core Core DataFusion crate physical-plan Changes to the physical-plan crate

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Aggregate dynamic filter can prune rows required by unsupported expressions

2 participants