Description of the issue
Follow-up to #2925. The PR addressing that issue (#3323) implements the simplest useful form of cross-day timeline inheritance: when the processing day opens with a gap (including days with no normal-mode data at all), the generated timestamps continue the previous day's timeline, taken from an optional previous-day L1C dependency. Several related items were deliberately left out to keep that change small and un-opinionated about gap management; this issue tracks them.
Remaining work
1. T018 (continue the previous day's L1C timeline): IMPLEMENTED, pending prod deploy and verification
MAG's top priority, per @alastairtree below. Both halves are merged: the processing side in #3323 (742d7a81) and the delivery side in IMAP-Science-Operations-Center/sds-data-manager#1471 (d83ce32), which delivers the previous day's L1C via a custom_behavior/mag.py. To generate day N's L1C for one sensor, the inputs are: day N L1B burst, day N L1B norm, and day N-1 L1C if it exists. Because the previous day's L1C carries real NM vectors where they existed and interpolated ones otherwise, this single source covers both T017 and T018 in section 4.3.3 of the SDC Data Validation Document.
Remaining: the sds-data-manager prod deploy (on us), and then verification: regenerate a few known day-boundary gap days, confirm the continued timeline in the real L1C products, and report back on #3323 (promised there).
2. T019 vs T024 specification conflict
T019 (no NM data at all; NM timeline created from BM) says to anchor the synthetic NM timeline at the first BM timestamp plus half the NM sampling period. That conflicts with the day-aligned grid already shipped and validated against T024 (fix #2274): for T024's first BM timestamp 12:38:02.000442, the validated first NM timestamp is the day-aligned 12:38:02.500, whereas the T019 rule gives 12:38:02.250442. Both cannot hold for the same burst-only scenario. Needs a decision from the MAG team (a live discussion is probably more effective than email). Note that #3323 adds a third behavior for this scenario when a previous-day file exists (inherit the previous day's grid); the T019-vs-T024 question is specifically about burst-only days with no usable neighbor.
3. MAG-approved validation data for T017-T019
The SDC Data Validation Document lists the input/output for T017-T019 as "No data provided". The unit tests merged with #3323 (from #3318) lock in the intended leading-gap behavior with synthetic scenarios, but MAG-approved validation data should replace or confirm them when it arrives. (Requested from the MAG team by email; still pending.)
4. Gap management beyond gaps at the beginning of the day
The #3323 change is deliberately un-opinionated about everything except leading gaps:
- Mid-day and trailing gaps still generate timestamps anchored on the current day's own samples.
- Whether following-day data is ever needed (current position: no - the previous day is enough, and any cross-day misalignment lands at the day boundary, which is as good a place as any).
- How inherited cadence should interact with rate changes near the boundary.
These need MAG team input on their exact expectations before further behavior is committed.
Accepted properties (no work planned)
5. Timestamp precision
The L1C pipeline carries the timeline in float64 array columns. At 2025-era TTJ2000 magnitudes float64 resolves only multiples of 128 ns, so stored L1C epochs can differ from their exact integer values by up to 64 ns (growing to 128 ns after mid-2036 and 256 ns after 2073 as epochs cross float64 binade boundaries). Per MAG expectations relayed in this review comment, nanosecond precision is not expected at the instrument level and the ~1 us atol used across the MAG validation tests reflects the expected precision - so float64 storage stays well within expectations for the mission lifetime, an int64 end-to-end timeline refactor is not needed, and the #3323 grid-continuity tests assert at that same ~1 us precision. Recorded here so the storage quantization is a known, accepted property rather than a future surprise.
Related, and already landed in #3323: generate_missing_timestamps used to route integer gap bounds through np.rint (float64), quantizing every generated timestamp onto that same 128 ns grid. With the fix, regenerated L1C products can show gap-fill timestamps at rates of 8+ vec/s (whose periods are not multiples of 128 ns) shifted by up to 128 ns - and occasionally one row more or fewer at a gap edge - relative to previously generated products. The new values are the exact intended grid; fills at 1, 2, and 4 vec/s are byte-identical to before. Diffs against archived products in those cases are the bug fix, not a regression.
Production wiring reminder
The previous-day L1C delivery is merged at IMAP-Science-Operations-Center/sds-data-manager#1471 (custom_behavior; see item 1). Until sds-data-manager is deployed to prod, the timeline-continuation path is dormant.
Description of the issue
Follow-up to #2925. The PR addressing that issue (#3323) implements the simplest useful form of cross-day timeline inheritance: when the processing day opens with a gap (including days with no normal-mode data at all), the generated timestamps continue the previous day's timeline, taken from an optional previous-day L1C dependency. Several related items were deliberately left out to keep that change small and un-opinionated about gap management; this issue tracks them.
Remaining work
1. T018 (continue the previous day's L1C timeline): IMPLEMENTED, pending prod deploy and verification
MAG's top priority, per @alastairtree below. Both halves are merged: the processing side in #3323 (
742d7a81) and the delivery side in IMAP-Science-Operations-Center/sds-data-manager#1471 (d83ce32), which delivers the previous day's L1C via acustom_behavior/mag.py. To generate day N's L1C for one sensor, the inputs are: day N L1B burst, day N L1B norm, and day N-1 L1C if it exists. Because the previous day's L1C carries real NM vectors where they existed and interpolated ones otherwise, this single source covers both T017 and T018 in section 4.3.3 of the SDC Data Validation Document.Remaining: the sds-data-manager prod deploy (on us), and then verification: regenerate a few known day-boundary gap days, confirm the continued timeline in the real L1C products, and report back on #3323 (promised there).
2. T019 vs T024 specification conflict
T019 (no NM data at all; NM timeline created from BM) says to anchor the synthetic NM timeline at the first BM timestamp plus half the NM sampling period. That conflicts with the day-aligned grid already shipped and validated against T024 (fix #2274): for T024's first BM timestamp 12:38:02.000442, the validated first NM timestamp is the day-aligned 12:38:02.500, whereas the T019 rule gives 12:38:02.250442. Both cannot hold for the same burst-only scenario. Needs a decision from the MAG team (a live discussion is probably more effective than email). Note that #3323 adds a third behavior for this scenario when a previous-day file exists (inherit the previous day's grid); the T019-vs-T024 question is specifically about burst-only days with no usable neighbor.
3. MAG-approved validation data for T017-T019
The SDC Data Validation Document lists the input/output for T017-T019 as "No data provided". The unit tests merged with #3323 (from #3318) lock in the intended leading-gap behavior with synthetic scenarios, but MAG-approved validation data should replace or confirm them when it arrives. (Requested from the MAG team by email; still pending.)
4. Gap management beyond gaps at the beginning of the day
The #3323 change is deliberately un-opinionated about everything except leading gaps:
These need MAG team input on their exact expectations before further behavior is committed.
Accepted properties (no work planned)
5. Timestamp precision
The L1C pipeline carries the timeline in float64 array columns. At 2025-era TTJ2000 magnitudes float64 resolves only multiples of 128 ns, so stored L1C epochs can differ from their exact integer values by up to 64 ns (growing to 128 ns after mid-2036 and 256 ns after 2073 as epochs cross float64 binade boundaries). Per MAG expectations relayed in this review comment, nanosecond precision is not expected at the instrument level and the ~1 us
atolused across the MAG validation tests reflects the expected precision - so float64 storage stays well within expectations for the mission lifetime, an int64 end-to-end timeline refactor is not needed, and the #3323 grid-continuity tests assert at that same ~1 us precision. Recorded here so the storage quantization is a known, accepted property rather than a future surprise.Related, and already landed in #3323:
generate_missing_timestampsused to route integer gap bounds throughnp.rint(float64), quantizing every generated timestamp onto that same 128 ns grid. With the fix, regenerated L1C products can show gap-fill timestamps at rates of 8+ vec/s (whose periods are not multiples of 128 ns) shifted by up to 128 ns - and occasionally one row more or fewer at a gap edge - relative to previously generated products. The new values are the exact intended grid; fills at 1, 2, and 4 vec/s are byte-identical to before. Diffs against archived products in those cases are the bug fix, not a regression.Production wiring reminder
The previous-day L1C delivery is merged at IMAP-Science-Operations-Center/sds-data-manager#1471 (custom_behavior; see item 1). Until sds-data-manager is deployed to prod, the timeline-continuation path is dormant.