Skip to content

fix: use HashTable::allocation_size() in GroupValuesColumn::size() - #26093

Draft
mohitgurav20 wants to merge 1 commit into
apache:mainfrom
mohitgurav20:fix/exact-hashtable-accounting-group-values-column
Draft

mohitgurav20 wants to merge 1 commit into
apache:mainfrom
mohitgurav20:fix/exact-hashtable-accounting-group-values-column

Conversation

@mohitgurav20

@mohitgurav20 mohitgurav20 commented Oct 6, 2026 •

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Rationale for this change

GroupValuesColumn stores its hash map as a HashTable<(u64, GroupIndexView)>,
but size() was computing the map's contribution using a hand-maintained
map_size field — incremented via insert_accounted on every insert.
That only accounts for the raw entry storage, not hashbrown's internal
control bytes or trailing layout padding.

The result: GroupValues::size() quietly underreports real memory,
especially right after a table growth. This weakens memory-pool
enforcement and spill decisions.

ArrowBytesMap already solves this correctly with HashTable::allocation_size().
This PR brings GroupValuesColumn in line with that (same approach as merged #25750 for GroupValuesRows and #25825 for GroupValuesPrimitive).

What changes are included in this PR?

Only datafusion/physical-plan/src/aggregates/group_values/multi_group_by/mod.rs is touched.

  • Removed the map_size: usize field and every insert_accounted call that maintained it.
  • Replaced insert_accounted with insert_unique (hashbrown's plain insert) — no behavioural change, just stops threading an accumulator through.
  • GroupValuesColumn::size() now calls self.map.allocation_size() directly, matching ArrowBytesMap::size().
  • Retains existing accounting for group_index_lists, emit_group_index_list_buffer, vectorized_operation_buffers, hashes_buffer, and size_of::<Self>().
  • Cleaned up now-unused HashTableAllocExt imports.

What is the testing strategy for this PR?

Four focused unit tests added, one per acceptance criterion from the issue:

Test What it verifies
map_allocation_size_included_in_size_after_growth After 128 inserts (forces a resize), size() covers the full allocation_size()
collision_chain_size_is_separate_from_map_allocation Forced collision: map allocation and group_index_lists are tracked independently
size_reflects_retained_capacity_after_partial_emit EmitTo::First doesn't shrink the map allocation — retained capacity stays charged
reuse_after_full_emit_produces_correct_groups After a full emit + reuse, group IDs restart correctly and size() stays accurate

All 122 tests in aggregates::group_values::multi_group_by pass locally.

Are there any user-facing changes?

No. This is a memory-accounting fix only. Group values, group IDs, and query
results are completely unchanged.

@github-actions github-actions Bot added the physical-plan Changes to the physical-plan crate label Oct 6, 2026
@mohitgurav20 mohitgurav20 changed the title Fix/exact hashtable accounting group values column fix: use HashTable::allocation_size() in GroupValuesColumn::size() Oct 6, 2026
@codecov-commenter

codecov-commenter commented Oct 6, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 82.82%. Comparing base (454ca08) to head (e27ed80).

Additional details and impacted files
@@            Coverage Diff             @@
##             main   #26093      +/-   ##
==========================================
+ Coverage   82.80%   82.82%   +0.01%     
==========================================
  Files        1149     1149              
  Lines      452213   452785     +572     
  Branches   452213   452785     +572     
==========================================
+ Hits       374473   375029     +556     
- Misses      54945    54957      +12     
- Partials    22795    22799       +4     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@mohitgurav20
mohitgurav20 force-pushed the fix/exact-hashtable-accounting-group-values-column branch from e27ed80 to 4fea79e Compare October 25, 2026 09:54
Fixes apache#25736

The previous implementation tracked map memory via a map_size: usize
field that was updated incrementally using insert_accounted. That
approach charges only the entry-capacity portion of the hashbrown
allocation, silently omitting control bytes and trailing layout. The
resulting GroupValues::size() underreports the true retained memory,
particularly for small or recently grown tables.

This commit replaces the approximate accounting with a direct call to
HashTable::allocation_size(), which reports the complete allocation as
observed by the allocator. This is the same approach already used by
ArrowBytesMap::size() in physical-expr-common.

Changes:
- Remove the map_size: usize field and all insert_accounted call
  sites that maintained it; use insert_unique (the plain hashbrown
  insert) instead.
- Remove the now-unused HashTableAllocExt import from the production
  path (the test module retains what it still needs).
- Remove the size_of import that was only used in the deleted
  clear_shrink map_size update.
- Update GroupValuesColumn::size() to use self.map.allocation_size().
- Add four focused unit tests covering the acceptance criteria from the
  issue: map growth accounting, forced-collision chain independence,
  retained capacity after partial emit, and correct reuse after full emit.
@mohitgurav20
mohitgurav20 force-pushed the fix/exact-hashtable-accounting-group-values-column branch from 4fea79e to 8d97aa1 Compare October 25, 2026 10:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

physical-plan Changes to the physical-plan crate

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Use Exact Hash-Table Allocation Accounting in GroupValuesColumn

2 participants