HIT L2: added data transformation into 10-minute chunks to L2 - #3377
HIT L2: added data transformation into 10-minute chunks to L2#3377leowerneck wants to merge 4 commits into
Conversation
tmplummer
left a comment
tmplummer
left a comment
There was a problem hiding this comment.
A few things to consider.
| dataset["epoch"].attrs["DELTA_MINUS_VAR"] = "epoch_delta" | ||
| dataset["epoch"].attrs["DELTA_PLUS_VAR"] = "epoch_delta" |
There was a problem hiding this comment.
It would be nice if these were defined in the imap_hit_l2_variable_attrs.yaml file so that everything is in that single place, but that would still require custom code here to get those values and set them only to the macropixel product. Adding something like this to the above for dim in dataset.dims: loop:
if dim != "epoch":
...
elif "macropixel" in logical_source:
dataset["epoch"].attrs.update(attr_mgr.get_variable_attributes("epoch_macropixel", check_schema=False))
Thoughts?
| for species, num_energy_levels in species_energy: | ||
| energy_dim = f"{species}_energy_mean" | ||
| species_variables = [ | ||
| var | ||
| for var in macropixel_dataset.data_vars | ||
| if macropixel_dataset[var].dims[:2] == ("epoch", energy_dim) | ||
| ] | ||
|
|
||
| for energy_i in range(num_energy_levels): | ||
| for var in species_variables: | ||
| _data = macropixel_dataset[var].values[species_i::10, energy_i] | ||
| transformed_dataset[var].values[:, energy_i] = _data | ||
| species_i += 1 |
There was a problem hiding this comment.
Can you add comments in here? It looks like it is doing some sort of diagonal gather but it is a bit hard to follow.
There was a problem hiding this comment.
Thanks for pointing that out. I added comments and changed variables name to hopefully make things a bit clearer. Let me know if it's still not so.
| RECORD_VARYING: RV | ||
| SCALETYP: linear | ||
| UNITS: ns | ||
| VALIDMAX: 86000000000000 |
There was a problem hiding this comment.
Why such a specific value rather than double integer max?
There was a problem hiding this comment.
I took these straight out of L3, but I'll admit I'm not completely sure why the values were chosen.
There was a problem hiding this comment.
It's roughly a day (86000 seconds) which strikes me as Grant throwing up his hands and saying "I don't know, a day?" Maybe 1e14 makes more sense. Anything between "about an hour" and "about a day" is probably reasonable (we all know it's going to be awfully close to 300 seconds the vast majority of the time).
| FORMAT: I19 | ||
| LABLAXIS: Epoch delta | ||
| NAME: epoch_delta | ||
| RECORD_VARYING: RV |
There was a problem hiding this comment.
Where does this come from? It's the first time I have seen it and I can't find it in SPDF's list of variable attributes. https://spdf.gsfc.nasa.gov/istp_guide/vattributes.html
There was a problem hiding this comment.
L3 puts certain aspects of the variable definition in the SAMMI YAML; this is a difference between L2 and L3 that should be addressed in the move.
Co-authored-by: Tim Plummer <timothy.plummer@lasp.colorado.edu>
…e names around transformation algorithm
tmplummer
left a comment
tmplummer
left a comment
There was a problem hiding this comment.
This looks good to me.
3 participants
Change Summary
Closes #3327.
Overview
This PR moves the HIT macropixel 10-minute chunk transformation from L3 to L2. It coordinates with PR 161 in
imap_L3_processingthat closes issue 143 of that repository.The L2 macropixel product now consolidates the ten one-minute packetized records into a single record for each 10-minute
integration period. It also assigns the integration midpoint as the epoch and provides five-minute
DELTA_MINUSandDELTA_PLUSvalues throughepoch_delta.File changes
The following changes were made:
imap_processing/hit/l2/hit_l2.pytransform_to_10_minute_chunks.combinations.
epoch_deltafor each record.process_macropixel_intensityto return the transformed dataset.epoch_deltafor bothDELTA_MINUS_VARandDELTA_PLUS_VAR.imap_processing/cdf/config/imap_constant_attrs.yamlepoch_delta.Testing
Ran the focused HIT L2 test suite:
All 17 tests passed.