Added Pandas 3 Support

[leowerneck]

Change Summary

Closes #3380.

Overview

Add support for both pandas 2.x and pandas 3.x. The changes account for pandas 3's copy-on-write behavior and default string extension arrays while preserving the behavior of existing processing code under pandas 2.2.3.

The compatibility updates include:

• Requesting writable NumPy copies when arrays obtained from pandas are mutated.
• Converting pandas extension arrays to NumPy-backed arrays at the CDF serialization boundary, without modifying the input dataset.
• Avoiding mixed-type assignment into pandas Series created from Excel rows.
• Preserving None values in IDEX trigger-mode object arrays.
• Using direct xarray indexing when the intended operation is to select and drop complete records from mixed-type datasets.
• Returning a copied, vectorized NumPy result from the Ultra energy lookup.
• Making test dimensions and writable-copy requirements explicit where newer pandas/xarray behavior exposes index-backed or read-only arrays.

These changes do not use pandas-version checks or alter global pandas options.

File changes

• imap_processing/ccsds/excel_to_xtce.py

  • Convert an Excel state row to a plain mapping before replacing a hexadecimal string with an integer, avoiding assignment of an integer into a string-typed pandas Series.

• imap_processing/cdf/utils.py

  • Convert pandas extension-array variables to NumPy-backed variables immediately before calling cdflib.
  • Preserve the original dataset, variable attributes, and encodings during the conversion.

• imap_processing/ialirt/l0/process_swe.py

  • Use boolean isel() operations when filtering complete records from a mixed-type Dataset. This avoids asking xarray to create fill values for pandas string extension arrays when the rejected records are immediately dropped.

• imap_processing/ialirt/utils/grouping.py

  • Apply the same direct-indexing approach to shared I-ALiRT group filtering.

• imap_processing/idex/idex_l1b.py

  • Explicitly construct trigger-mode object arrays so no-trigger values remain None rather than being inferred as string missing values/NaN.

• imap_processing/ultra/l1b/lookup_utils.py

  • Convert lookup indices to an integer NumPy dtype and return values from a copied NumPy array rather than a pandas Series.

• imap_processing/tests/ialirt/unit/test_process_swapi.py

  • Use to_numpy(copy=True) in five places where the returned arrays are subsequently mutated.

• imap_processing/tests/cdf/test_utils.py

  • Add regression coverage for passing NumPy-backed string data to cdflib without mutating the source Dataset.

• imap_processing/tests/mag/test_mag_l2.py

  • Copy epoch data before mutating the expected result.

• imap_processing/tests/swapi/test_swapi_l1.py

  • Declare the shared epoch dimension explicitly for mutable test variables.

• imap_processing/tests/ultra/unit/test_ultra_l1b_culling.py

  • Declare shared spin, energy_bin, and energy_edge dimensions explicitly so mutable test variables are not implicitly constructed as index coordinates.

Testing

Two clean Poetry environments were exercised with the test and development extras. The dependency set was held constant apart from the pandas version (Python 3.12, NumPy 2.2.6, and xarray 2025.4.0).

Baseline results before the compatibility fixes:

• pandas 2.2.3: 1,825 passed, 4 skipped, 4 xfailed.
• pandas 3.0.5: 25 failed, 1,800 passed, 4 skipped, 4 xfailed.

Post-fix full-suite results from:

poetry run pytest -n auto

• pandas 2.2.3: 1,826 passed, 4 skipped, 4 xfailed.
• pandas 3.0.5: 1,826 passed, 4 skipped, 4 xfailed.

[tmplummer]

It is concerning to me that upgrading to pandas 3 breaks xarray DataArray.where functionality. This fix seems like a hack to avoid some underlying issue.

[leowerneck]

The issue isn't DataArray.where() specifically, but DataArray.where(..., drop=True). Numeric DataArray.where() calls still work and were preserved, but when the intent is "keep records whose epoch satifies this condition", then:

dataset.where(condition, drop=True)

was replaced with:

dataset.isel(epoch=condition.values)

Using isel here avoids e.g., unnecessary fill-value computation.

[tmplummer]

Using isel here avoids e.g., unnecessary fill-value computation.

Very good point.

[tmplummer]

@bryan-harter is the author of cdflib. It feels to me like this should be a ticket for adding pandas 3 support to cdflib.

[Copilot]

Pull request overview

Broadens pandas 3 compatibility across data processing, serialization, and tests.

Changes:

• Copies arrays before mutation and uses NumPy-backed lookups.
• Adds explicit xarray dimensions and positional filtering.
• Normalizes pandas extension arrays and inferred string/state values.

Reviewed changes

Copilot reviewed 11 out of 11 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
imap_processing/ultra/l1b/lookup_utils.py	Uses integer NumPy indexing for energy lookup.
imap_processing/tests/ultra/unit/test_ultra_l1b_culling.py	Adds explicit test dimensions.
imap_processing/tests/swapi/test_swapi_l1.py	Adds explicit epoch dimensions.
imap_processing/tests/mag/test_mag_l2.py	Copies mutable expected epoch data.
imap_processing/tests/ialirt/unit/test_process_swapi.py	Copies arrays before mutation.
imap_processing/tests/cdf/test_utils.py	Tests extension-array conversion.
imap_processing/idex/idex_l1b.py	Preserves object-backed trigger modes.
imap_processing/ialirt/utils/grouping.py	Uses positional boolean filtering.
imap_processing/ialirt/l0/process_swe.py	Uses positional filtering for SWE data.
imap_processing/cdf/utils.py	Converts extension arrays before CDF serialization.
imap_processing/ccsds/excel_to_xtce.py	Converts state rows to plain mappings.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.