Skip to content

Numba-enhanced Data Averaging (and more!) - #1694

Merged
bhazelton merged 8 commits into
mainfrom
must_average_faster
Aug 19, 2026
Merged

Numba-enhanced Data Averaging (and more!)#1694
bhazelton merged 8 commits into
mainfrom
must_average_faster

Conversation

@kartographer

Copy link
Copy Markdown
Contributor

Description

A new numba kernel for averaging data (with arbitrary grouping along one axis) has been added and integrated into UVData.frequency_average and UVData.downsample_in_time, which should substantially increase its speed (via multithreaded processing) and significantly reduce the memory footprint of the averaging step.

Motivation and Context

The primary motivation for this was addressing the speed and memory issue with frequency averaging, given the prior report about MWA data suffering a ~6x increase in memory footprint, although because the new kernel is somewhat flexibly designed to handle irregularly grouped arrays, I took the opportunity to add a few additional features to frequency_average (including an option to sum across nsample_array, which is appropriate for when it's being used to carry weights, as is the case in a few different filetypes).

Closes #730
Closes #1311
Closes #1624

Types of changes

  • New feature (non-breaking change which adds functionality)

Checklist:

New feature checklist:

  • I have added or updated the docstrings associated with my feature using the numpy docstring format.
  • I have updated the tutorial to highlight my new feature (if appropriate).
  • I have added/updated tests to cover my new feature.
  • I have updated the CHANGELOG.

@codecov

codecov Bot commented Aug 15, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 99.94%. Comparing base (a12cae9) to head (d934328).

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1694      +/-   ##
==========================================
+ Coverage   99.84%   99.94%   +0.09%     
==========================================
  Files          68       70       +2     
  Lines       23506    23562      +56     
==========================================
+ Hits        23470    23548      +78     
+ Misses         36       14      -22     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@bhazelton bhazelton added performance A performance improvement. enhancement UVData and removed enhancement labels Aug 18, 2026

@bhazelton bhazelton left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks fantastic, thank you! I just have a couple comments on docstrings. My student is going to test this soon on a full MWA dataset.

Comment thread src/pyuvdata/utils/averaging.py Outdated
Comment thread CHANGELOG.md

@bhazelton bhazelton left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ldress reports that it works excellently for her. Thank you so much @kartographer !

@bhazelton
bhazelton merged commit 1cc591f into main Aug 19, 2026
64 of 66 checks passed
@bhazelton
bhazelton deleted the must_average_faster branch August 19, 2026 21:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement performance A performance improvement. UVData

Projects

None yet

Development

Successfully merging this pull request may close these issues.

High memory usage when frequency averaging MWA correlator fits files Make UVData.frequency_average more flexible Frequency averaging

2 participants