Adding MAG L1C day boundary tests - #3318
Closed
maxineofficial wants to merge 1 commit into
Closed
Conversation
sapols
added a commit
to sapols/imap_processing
that referenced
this pull request
Jul 6, 2026
…AP-Science-Operations-Center#2925) When the processing day opens with a gap, generate the missing normal-mode timestamps on the previous day's grid - inheriting its ending cadence and phase from an optional previous-day norm L1B dataset - so the L1C timeline stays regular across the day boundary. With no (usable) previous-day file, behavior is unchanged and single-day processing still succeeds. - mag_l1c()/process_mag_l1c() gain an optional previous_day_dataset. The inherited grid anchors on the last sample within the previous 24-hour day and snaps its cadence against the known MAG rates via _is_expected_rate. With no normal-mode data at all, the whole window continues that grid. - Mag CLI splits L1C dependencies by start date with get_valid_inputs_for_start_date; the previous day's norm L1B (delivered via a date_range entry in imap_mag_dependencies.yaml) routes to previous_day_dataset. Mode/sensor validation lives in mag_l1c. - generate_missing_timestamps no longer routes integer nanosecond gap bounds through float64 (np.rint), which shifted generated timestamps by up to 64 ns at TTJ2000 scales. - Un-xfail the day-boundary tests from IMAP-Science-Operations-Center#3318; their phase offset moves to a multiple of 128 ns because the float64 epoch columns cannot represent other offsets exactly at 2025-era epochs. Shared test helper _ttj2000_day_bounds is replaced by the production _expected_day_ns.
This was referenced Jul 6, 2026
Closed
Collaborator
|
Think this can now be closed in favour of #3323 now? |
sapols
added a commit
that referenced
this pull request
Aug 13, 2026
) When a MAG L1C processing day opens with a gap, the timestamps generated for that gap now continue the previous day's L1C timeline instead of counting from the window boundary, so the timeline stays regular across the day boundary (SDC Data Validation 4.3.3 cases T017 and T018). Without a previous-day file, single-day processing is unchanged apart from the precision fix below. - mag_l1c() and process_mag_l1c() take an optional previous_day_dataset. The continued timeline anchors on that file's last sample inside its own 24-hour day (trailing buffer samples excluded) and steps at its ending cadence, snapped against the known MAG rates. With no normal-mode data at all, the whole window continues that timeline. Only the clock is inherited; previous-day vectors never enter the output. - The Mag CLI splits L1C dependencies by start date with get_valid_inputs_for_start_date: current-day L1B files are the one-or-two inputs as before, and the previous day's L1C, delivered by orchestration in IMAP-Science-Operations-Center/sds-data-manager#1471, routes to previous_day_dataset. - A previous-day file that is not L1C for this sensor, or that has no epochs, raises ValueError: either can only mean the wrong file was delivered or produced upstream. Genuine data conditions (fewer than two samples before midnight, a cadence matching no known MAG rate) log a warning and fall back to single-day behavior. - generate_missing_timestamps now requires integer nanosecond gap bounds and raises on floats. Dropping the previous float64/np.rint path moves gap-fill timestamps at 8+ vectors per second by up to 128 ns relative to previously generated products, even on days with no previous-day file. - Un-xfails the day-boundary tests added in #3318. Closes #2925. Co-authored-by: Maxine Hartnett <maxine.hartnett@lasp.colorado.edu> Co-authored-by: Tim Plummer <timothy.plummer@lasp.colorado.edu>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This change adds unit tests for the most basic form of MAG L1C boundaries - ie, a single L1B file from the previous day is used to fill gaps at the beginning of a current L1B file.