fix_cmip7_repack_compression

[rhaegar325]

Fix cmip7repack silently leaving every 3-D CMIP7 field uncompressed

Problem

createVariable passes no chunksizes, so netCDF4's default applies.
cmip7repack then refuses to rechunk in two situations, and MOPPy's CMIP7
output hits both:

(a) A second timestep does not fit. The default chunk is one timestep, and
two of them exceed the target.

(b) The default chunk already exceeds the target. For the largest fields
netCDF4's default chunk is multi-dimensional rather than one timestep.

variable class	shape	default chunk	chunk bytes	branch
p19 atmos	(19,144,192) f4	[1,19,144,192]	2,101,248	(a), over 4 MiB by 0.195 %
cl/cli/clw	(38,145,192) f4	[1,38,145,192]	4,231,680	(b)
ocean 3-D	(50,300,360) f4	[1,25,150,180]	4,500,000	(b)
surface atmos	(145,192) f4	[1,145,192]	111,360	— fine, [37,145,192]
ocean 2-D	(300,360) f4	[1,300,360]	432,000	— fine, [9,300,360]

For p19 the miss is 8,192 bytes:

one timestep = 19 × 144 × 192 × 4 =  2,101,248 B
two timesteps                     =  4,202,496 B
cmip7repack default -d            =  4,194,304 B  (4 MiB)

In a one-year daily run this is the seven p19 variables ta ua va hus hur wap zg — about 100 GiB of the 111 GiB run. Monthly runs add cl/cli/clw and
the 3-D ocean fields.

The failure was invisible because tests/unit/test_qc_gates.py:189 mocked
subprocess and asserted pass on exit 0 — pinning the blind spot in place
with a passing test. This is the second defect to reach production through this
gate; see ISSUE-cmip7-out-name-cmip6-fallback.md.

Fix

1. Size -d per variable — base.py:2710

_cmip7repack_chunk_target reopens the written file, resolves the data variable
the way cmip7repack does (the variable_id global attribute), and asks for two
timesteps' worth:

return max(default, 2 * slice_bytes)

-d must be at least the 4 MiB default, so surface fields keep it unchanged.
Branch (b) is why the target is sized from the slice rather than merely nudged
past 4 MiB: for ocean 3-D it has to clear the existing 4,500,000 B chunk before
cmip7repack will touch the variable at all.

Preferred over dropping the mip_era != "CMIP7" guard in use_compression:
that would fight the design intent (repack owns CMIP7 packing) and would not fix
the chunk layout.

2. Verify the repack happened — base.py:2748

_verify_repack_compression reopens the file after the subprocess returns and
reads filters()['zlib'] off the data variable. The gate now records warn with
a reason instead of pass when it is false:

reason = self._verify_repack_compression(path)
if reason:
    logger.warning("cmip7repack did not compress %s: %s", path, reason)
    self._record_gate("repack", "warn", tool="cmip7repack", message=reason)
else:
    self._record_gate("repack", "pass", tool="cmip7repack")

Worth having independently of (1). The man page's "or the data in the input file
only has one chunk" branch means a variable can still come back uncompressed
however -d is sized — and check_cmip7_packing does not catch this defect
either (see Verification), so this read-back is the only check that does.

Cost

+49 s CPU per 769 MB file — about +15 min CPU per p19 variable, +1.8 h across
the seven, taking them from ~70 min to ~85 min CPU (+21 %). It saves about
47 GiB on the daily run. Both figures are measured.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 78.7%. Comparing base (166e8ca) to head (6252d94).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##            main    #692     +/-   ##
=======================================
+ Coverage   78.6%   78.7%   +0.1%     
=======================================
  Files         41      41             
  Lines       9021    9061     +40     
  Branches    1687    1693      +6     
=======================================
+ Hits        7094    7134     +40     
  Misses      1592    1592             
  Partials     335     335             

Flag	Coverage Δ
unit	78.7% <100.0%> (+0.1%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.