Skip to content

refactor(hash-aggr): Let partial-reduce mode aggregation early emit when OOM - #24785

Open
2010YOUY01 wants to merge 2 commits into
apache:mainfrom
2010YOUY01:split-aggr-partial-reduce-early-emit
Open

refactor(hash-aggr): Let partial-reduce mode aggregation early emit when OOM#24785
2010YOUY01 wants to merge 2 commits into
apache:mainfrom
2010YOUY01:split-aggr-partial-reduce-early-emit

Conversation

@2010YOUY01

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Part of #22710

Rationale for this change

See #24486 for the rationale. That discussion covers the issue and three potential solutions: error, early-emit, and spill.

This PR implements the early-emit behavior agreed on there.

What changes are included in this PR?

Key changes:
In datafusion/physical-plan/src/aggregates/partial_reduce_stream.rs

  • top comment explains the high-level ideas
  • To understand the implementation, start from poll_next() and follow along

Are these changes tested?

UTs (note partial-reduce aggregation can't be planned from SQL, so we can't do sqllogictests here)

Are there any user-facing changes?

No

@github-actions github-actions Bot added the physical-plan Changes to the physical-plan crate label Aug 30, 2026
@codecov-commenter

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 69.50355% with 43 lines in your changes missing coverage. Please review.
✅ Project coverage is 81.52%. Comparing base (c429919) to head (969bfc9).
⚠️ Report is 142 commits behind head on main.

Files with missing lines Patch % Lines
...sical-plan/src/aggregates/partial_reduce_stream.rs 64.35% 34 Missing and 2 partials ⚠️
datafusion/physical-plan/src/aggregates/mod.rs 82.50% 0 Missing and 7 partials ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main   #24785      +/-   ##
==========================================
+ Coverage   81.24%   81.52%   +0.28%     
==========================================
  Files        1113     1123      +10     
  Lines      392744   406130   +13386     
  Branches   392744   406130   +13386     
==========================================
+ Hits       319090   331115   +12025     
- Misses      54900    55649     +749     
- Partials    18754    19366     +612     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

physical-plan Changes to the physical-plan crate

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants