Skip to content

remove redundant attrs leaked from raw data - #688

Merged
rhaegar325 merged 1 commit into
mainfrom
dimension_leak_to_cmorised_data
Sep 3, 2026
Merged

remove redundant attrs leaked from raw data#688
rhaegar325 merged 1 commit into
mainfrom
dimension_leak_to_cmorised_data

Conversation

@rhaegar325

Copy link
Copy Markdown
Collaborator

Fix

Branch fix_outname_issue.

src/access_moppy/base.py

An allow-list of the table fields that describe the variable, and a shared helper applying it:

_CMOR_VARIABLE_ATTRIBUTES = frozenset({
    "standard_name", "long_name", "units", "cell_methods", "cell_measures",
    "comment", "positive", "valid_min", "valid_max",
    "flag_values", "flag_meanings", "_FillValue", "missing_value",
})

def _apply_cmor_variable_attributes(self, cmor_attrs):
    self.ds[self.cmor_name].attrs.update({
        k: v for k, v in cmor_attrs.items()
        if k in self._CMOR_VARIABLE_ATTRIBUTES and v not in (None, "")
    })

Two decisions carry the safety of this change.

The filter reads the table entry, not the variable's final attributes. Its reach is therefore one assignment: coordinates, units_metadata and anything inherited from the source file are set elsewhere and cannot be touched, so a name missing from the allow-list cannot silently delete them. A filter over the final attributes would have to enumerate them all, and a gap there deletes real metadata.

_FillValue and missing_value must be in the list. They are not table fields at all — _get_variable_entry injects them (defaulting to 1e20, since no CMIP6 or CMIP7 entry carries either), and base.py:2388 reads _FillValue back from the variable attributes to create the netCDF variable. wcrp_cmip7 requires both at severity HIGH with a constant of 1e20. Dropping them would write files with no fill value and two weight-3 compliance failures — worse than the leak.

The allow-list is a complete partition, not a guess: the CMIP6 and CMIP7 tables carry 16 distinct fields between them, plus the 2 injected ones; 13 are written and 5 are directives.

src/access_moppy/{atmosphere,ocean,sea_ice}.py

Each of the three attrs.update(...) blocks becomes self._apply_cmor_variable_attributes(cmor_attrs). _drop_stale_range_attributes still runs immediately afterwards and is unchanged.

@codecov

codecov Bot commented Sep 3, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 78.6%. Comparing base (43f4d94) to head (5f72cf3).

Additional details and impacted files
@@          Coverage Diff          @@
##            main    #688   +/-   ##
=====================================
  Coverage   78.6%   78.6%           
=====================================
  Files         41      41           
  Lines       9018    9021    +3     
  Branches    1687    1687           
=====================================
+ Hits        7091    7094    +3     
  Misses      1592    1592           
  Partials     335     335           
Flag Coverage Δ
unit 78.6% <100.0%> (+<0.1%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@rhaegar325
rhaegar325 merged commit 166e8ca into main Sep 3, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant