Repository navigation
BlackrockRawIO: extend gap_tolerance_ms to the standard nsx formats - #1880
h-mayorquin wants to merge 23 commits into
Conversation
Take master's blackrockrawio.py wholesale. The branch's own buffer API migration is superseded by NeuralEnsemble#1820, which merged after review. The gap feature is re-applied on top of master's version in the following commit.
Master detects gaps only for PTP files, where every sample carries its own timestamp. Standard-format files were segmented by making each data block a segment, with no gap detection and no way to merge blocks that a pause never really separated. Split the segmentation into a dispatcher plus one segmenter per format, and give the standard path block-level gap detection through the same _classify_gaps step the PTP path uses. Both hand it the interval between consecutive samples, so the two paths share one threshold and one report. Merging blocks means a segment can now span several of them, at different file offsets, so segments carry a list of memmap specs rather than a single one and _read_multi_block_chunk stitches reads that straddle a boundary. Backward jumps now split as well. A clock reset makes the timestamps run backwards rather than leaving a gap, so there is no duration to compare against the tolerance and the blocks either side must never merge.
_format_gap_report derived a gap's position from a per-sample timestamp array, asking that array to be both the time of the sample before each gap and, via its first element, the start of the recording. Per-sample timestamps satisfy both, so the PTP path it was written for was correct. The standard path has no such array: its file stores one timestamp per block. It passed block timestamps, which got the origin right but not the position, so a gap was reported at the start of the block preceding it rather than at the sample it follows. In pause_correct that put a gap 4 s into the recording at 0.000000 s. Pass the resolved quantities instead, so the report renders what it is given and assumes nothing about how a format stores its timing. The standard path works out the sample before each gap from its block scalars, which is exact because within a block the timing is uniform by definition, and avoids synthesizing per-sample timestamps that would run to gigabytes on a long recording. Both paths now report in the same sample-level terms, which keeps the report reusable as the gap API reaches other readers (issue NeuralEnsemble#1773). Also only build the report when about to raise, rather than on every load.
An empty range touched no block, leaving nothing to concatenate and raising ValueError, where an unmerged segment returns an empty array. Filling a preallocated output gives the empty range a correctly shaped result and keeps merged and unmerged segments behaving alike. The blocks are also no longer all held at once. With a channel list each block is a copy rather than a view, so concatenating held every one of them; now each is written into the output and released. A range inside a single block is still returned as a view, as before.
_get_blackrock_timestamps reconstructed non-PTP times as t_start + index/rate. That was correct only because segmentation gave every data block its own segment, and within a block the sampling is uniform by definition. Nothing stated that dependency. Merging blocks under gap_tolerance_ms removes it. A merged segment has one t_start and a pause somewhere inside it, so the arithmetic walks straight through the pause and reports every later sample early by its length. The accessor returned 4.0 s for a sample pause_correct timestamps at 31.0087 s, without a line of it changing. Keep each contributing block's timestamp on the segment and interpolate per block, which is where uniform spacing is a fact of the format rather than an artifact of the merge. gap_tolerance_ms decides how samples are grouped into segments; it does not decide what time a sample was recorded at. The segment's own t_start and t_stop were always right, being the first and last sample, and stay so. What a merged segment cannot express is the pause in between, which is exactly why these timestamps are worth reporting separately.
…ap report Raising by default earns its place when the reader cannot decide for the caller. A forward gap qualifies: merging across one distorts the segment's uniform time base, splitting gives more segments, and only the caller knows which is acceptable. A backward jump does not. It has no duration to compare against gap_tolerance_ms, so it always becomes a segment boundary and there is nothing to choose. Raising asked a question with one possible answer, and pointed the caller at a parameter that has no effect on it: whatever value they supplied, the answer was the same. Warn instead, naming what was found and what was done about it, and leave backward jumps out of the gap report, which is about gaps. The error now reports only what gap_tolerance_ms can act on, so its advice is true of everything in it. This narrows the breaking change to files with real forward gaps. A file whose only discontinuity is a clock reset loads as it did before, into the same segments, and test_segment_detection_reset goes back to master unchanged. The nev path already warns about resets on these files, so the signal path now says the same thing.
…_logic_to_blocks # Conflicts: # neo/rawio/blackrockrawio.py
cboulay
left a comment
There was a problem hiding this comment.
Thanks Heberto, I like where this lands: gap_tolerance_ms means the same thing for every Blackrock variant, and _segment_nsx is split into per-format segmenters. It also fixes three_data_blocks / ten_data_blocks_with_millisecond_gaps at a large tolerance. I ran it on all 53 gin Blackrock sets at None, 0 and 1e6 against master, and on a couple of our spec 3.0 PTP recordings. Two bugs and one memory concern before I'd approve:
_read_multi_block_chunkdoesn't clampi_stop. Ontwo_data_blocks_after_pausemerged into one 200-sample segment,get_analogsignal_chunk(0, 0, 50, 250)returns 200 rows, and the last 50 come fromnp.empty. A range inside one block returns a shorter slice, as before.- An nsX with only empty blocks raises
IndexErroratblock_infos[0]. I made one from a basic header plus a single 0-point data header. On master it loads with 0 segments. - In the PTP path,
last_sample_indicesandlast_sample_positionsare built for every sample just to describe the few gaps. Peak memory inparse_headergoes from about 17 to 34 bytes per sample (tracemalloc on a 1.1 M-sample file), which matters for large PTP files on small cloud runners. Could_classify_gapsreturn the forward gap indices and let each caller build the report only for those? That would also let a segmenter drop gaps it knows are harmless before deciding whether to raise. That's exactly what packets written out of order need (#1937).
Two smaller points:
gap_tolerance_ms=0isn't quite "every block its own segment": blocks ≤ 2 periods apart still merge. Soone_sample_blocksgives 101-sample segments starting one tick earlier, where master gave 100. I think that's fine, but the description and docstring should say so.- On the default: every paused 2.2/2.3 file in gin now raises out of the box, which will reach SpikeInterface users quickly. Would you consider a release where
Nonestill splits at every block and emits a FutureWarning naminggap_tolerance_ms, before switching to the error?
On our spec 3.0 test recording (signal generator, linked in #1924) the PTP behaviour matches master, as the description says. With the reordered copy, the default raises on the out-of-order packets, and gap_tolerance_ms=0 produces 52- and 3-sample segments around them. That's the #1937 case, not a regression here.
Line notes
neo/rawio/blackrockrawio.py:794:i_stoppast the end of the segment leaves rows ofoutuninitialized when the range spans more than one block, while a single-block range returns fewer rows instead. Suggesti_stop = total_samples if i_stop is None else min(i_stop, total_samples). A regression test would be cheap:two_data_blocks_after_pausewith a large tolerance andi_stop = size + 50.neo/rawio/blackrockrawio.py:1634: If every block is empty this list is empty, andblock_infos[0]at 1681 raisesIndexError. Master returned 0 segments here; an earlyreturn []would restore that.neo/rawio/blackrockrawio.py:1722: These two N-length arrays, plus the float64 copy and diff above, double peak memory for large PTP files, but they're only needed at the gap indices. Could_classify_gapsreturn forward gap indices and the caller build the report fromtimestamps[idx]? Doing the diff on the integer ticks rather than float seconds would also save a copy.neo/rawio/blackrockrawio.py:1582: Raising here means a caller can't remove gaps it knows are benign before the error. Splitting "find gaps" from "raise / apply tolerance" would make that possible; the out-of-order packets in #1937 are the case I have in mind.neo/rawio/blackrockrawio.py:129: This note says pauses in 2.2/2.3 are split into segments automatically, which is no longer true by default. Could it point togap_tolerance_ms?neo/rawio/blackrockrawio.py:125: Worth documenting here three things: a non-zero tolerance places samples after a merged gap early undert_start + i / sampling_rate;_get_blackrock_timestampskeeps the recorded times; and 0 still merges blocks closer than 2 sampling periods.
…alve parse memory
…_logic_to_blocks # Conflicts: # neo/test/rawiotest/test_blackrockrawio.py
…and report the gaps in the FutureWarning
…, as those keep splitting
|
Thanks for the review,
On your two smaller points: |
Right now a standard-format Blackrock file (spec 2.2, 2.3 and 3.0) gets one segment per data block and
gap_tolerance_msis accepted but ignored. Files can arrive split into hundreds of segments this way (the recording in SpikeInterface #2306 loads as 280 blocks in its ns2 and 387 in its ns5). This PR extendsgap_tolerance_ms, which already works for PTP (Precision Time Protocol) files since #1789, to the standard formats. The gaps are detected between the block timestamps and by default the reader throws an error with a gap report.gap_tolerance_ms=0keeps every block as its own segment, as before, and a larger value merges the blocks whose gap is below it.Some choices to discuss:
t_start + index / sampling_rateplaces the samples earlier than they were recorded, by the length of the gap._get_blackrock_timestampskeeps returning the recorded times whatever the tolerance.gap_tolerance_ms=Nonemeans the same for every Blackrock variant, as it already does for PTP files.I also tested this against the new example datasets on gin (https://gin.g-node.org/NeuralEnsemble/ephy_testing_data/pulls/192) and fixed two problems they showed. Empty blocks (0 samples, some with a placeholder timestamp) are now left out of the gap detection, and matching a spec 2.3 nev to a merged segment now uses the end of its last block. Files whose nev and nsX disagree on their segments (for example
three_data_blockswithgap_tolerance_ms=0) still fail as on master, as that comes from_match_nsx_and_nev_segment_idsand is a separate problem.