Skip to content

BlackrockRawIO: extend gap_tolerance_ms to the standard nsx formats - #1880

Open
h-mayorquin wants to merge 23 commits into
NeuralEnsemble:masterfrom
h-mayorquin:blackrock_add_gap_logic_to_blocks
Open

h-mayorquin wants to merge 23 commits into
NeuralEnsemble:masterfrom
h-mayorquin:blackrock_add_gap_logic_to_blocks

Conversation

@h-mayorquin

@h-mayorquin h-mayorquin commented Jul 17, 2026 •

Copy link
Copy Markdown
Contributor

Right now a standard-format Blackrock file (spec 2.2, 2.3 and 3.0) gets one segment per data block and gap_tolerance_ms is accepted but ignored. Files can arrive split into hundreds of segments this way (the recording in SpikeInterface #2306 loads as 280 blocks in its ns2 and 387 in its ns5). This PR extends gap_tolerance_ms, which already works for PTP (Precision Time Protocol) files since #1789, to the standard formats. The gaps are detected between the block timestamps and by default the reader throws an error with a gap report. gap_tolerance_ms=0 keeps every block as its own segment, as before, and a larger value merges the blocks whose gap is below it.

Some choices to discuss:

  • In the standard formats, a block that starts before the previous one ends (a clock reset) has no duration to compare with a tolerance, so it always starts a new segment with a warning, as every block already did. Whether a reset should start a new block instead is the question in Are the segments of a block ordered in time? #1937 and this PR does not change it. Backward steps between PTP samples stay as they are on master: they come from packets written out of order, and how to handle them is also being discussed in Are the segments of a block ordered in time? #1937.
  • After a merged gap, t_start + index / sampling_rate places the samples earlier than they were recorded, by the length of the gap. _get_blackrock_timestamps keeps returning the recorded times whatever the tolerance.
  • This is a breaking change: a multi-block standard file that used to load now throws an error unless a tolerance is given. I did this on purpose so gap_tolerance_ms=None means the same for every Blackrock variant, as it already does for PTP files.

I also tested this against the new example datasets on gin (https://gin.g-node.org/NeuralEnsemble/ephy_testing_data/pulls/192) and fixed two problems they showed. Empty blocks (0 samples, some with a placeholder timestamp) are now left out of the gap detection, and matching a spec 2.3 nev to a merged segment now uses the end of its last block. Files whose nev and nsX disagree on their segments (for example three_data_blocks with gap_tolerance_ms=0) still fail as on master, as that comes from _match_nsx_and_nev_segment_ids and is a separate problem.

Take master's blackrockrawio.py wholesale. The branch's own buffer API
migration is superseded by NeuralEnsemble#1820, which merged after review. The gap
feature is re-applied on top of master's version in the following commit.
Master detects gaps only for PTP files, where every sample carries its own
timestamp. Standard-format files were segmented by making each data block a
segment, with no gap detection and no way to merge blocks that a pause never
really separated.

Split the segmentation into a dispatcher plus one segmenter per format, and give
the standard path block-level gap detection through the same _classify_gaps step
the PTP path uses. Both hand it the interval between consecutive samples, so the
two paths share one threshold and one report.

Merging blocks means a segment can now span several of them, at different file
offsets, so segments carry a list of memmap specs rather than a single one and
_read_multi_block_chunk stitches reads that straddle a boundary.

Backward jumps now split as well. A clock reset makes the timestamps run
backwards rather than leaving a gap, so there is no duration to compare against
the tolerance and the blocks either side must never merge.
_format_gap_report derived a gap's position from a per-sample timestamp array,
asking that array to be both the time of the sample before each gap and, via its
first element, the start of the recording. Per-sample timestamps satisfy both, so
the PTP path it was written for was correct.

The standard path has no such array: its file stores one timestamp per block. It
passed block timestamps, which got the origin right but not the position, so a
gap was reported at the start of the block preceding it rather than at the sample
it follows. In pause_correct that put a gap 4 s into the recording at 0.000000 s.

Pass the resolved quantities instead, so the report renders what it is given and
assumes nothing about how a format stores its timing. The standard path works out
the sample before each gap from its block scalars, which is exact because within a
block the timing is uniform by definition, and avoids synthesizing per-sample
timestamps that would run to gigabytes on a long recording. Both paths now report
in the same sample-level terms, which keeps the report reusable as the gap API
reaches other readers (issue NeuralEnsemble#1773).

Also only build the report when about to raise, rather than on every load.
An empty range touched no block, leaving nothing to concatenate and raising
ValueError, where an unmerged segment returns an empty array. Filling a
preallocated output gives the empty range a correctly shaped result and keeps
merged and unmerged segments behaving alike.

The blocks are also no longer all held at once. With a channel list each block is
a copy rather than a view, so concatenating held every one of them; now each is
written into the output and released. A range inside a single block is still
returned as a view, as before.
_get_blackrock_timestamps reconstructed non-PTP times as t_start + index/rate.
That was correct only because segmentation gave every data block its own segment,
and within a block the sampling is uniform by definition. Nothing stated that
dependency.

Merging blocks under gap_tolerance_ms removes it. A merged segment has one t_start
and a pause somewhere inside it, so the arithmetic walks straight through the pause
and reports every later sample early by its length. The accessor returned 4.0 s for
a sample pause_correct timestamps at 31.0087 s, without a line of it changing.

Keep each contributing block's timestamp on the segment and interpolate per block,
which is where uniform spacing is a fact of the format rather than an artifact of
the merge. gap_tolerance_ms decides how samples are grouped into segments; it does
not decide what time a sample was recorded at.

The segment's own t_start and t_stop were always right, being the first and last
sample, and stay so. What a merged segment cannot express is the pause in between,
which is exactly why these timestamps are worth reporting separately.
…ap report

Raising by default earns its place when the reader cannot decide for the caller.
A forward gap qualifies: merging across one distorts the segment's uniform time
base, splitting gives more segments, and only the caller knows which is
acceptable.

A backward jump does not. It has no duration to compare against gap_tolerance_ms,
so it always becomes a segment boundary and there is nothing to choose. Raising
asked a question with one possible answer, and pointed the caller at a parameter
that has no effect on it: whatever value they supplied, the answer was the same.

Warn instead, naming what was found and what was done about it, and leave backward
jumps out of the gap report, which is about gaps. The error now reports only what
gap_tolerance_ms can act on, so its advice is true of everything in it.

This narrows the breaking change to files with real forward gaps. A file whose only
discontinuity is a clock reset loads as it did before, into the same segments, and
test_segment_detection_reset goes back to master unchanged. The nev path already
warns about resets on these files, so the signal path now says the same thing.
@h-mayorquin h-mayorquin self-assigned this Jul 17, 2026

@cboulay cboulay left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks Heberto, I like where this lands: gap_tolerance_ms means the same thing for every Blackrock variant, and _segment_nsx is split into per-format segmenters. It also fixes three_data_blocks / ten_data_blocks_with_millisecond_gaps at a large tolerance. I ran it on all 53 gin Blackrock sets at None, 0 and 1e6 against master, and on a couple of our spec 3.0 PTP recordings. Two bugs and one memory concern before I'd approve:

  1. _read_multi_block_chunk doesn't clamp i_stop. On two_data_blocks_after_pause merged into one 200-sample segment, get_analogsignal_chunk(0, 0, 50, 250) returns 200 rows, and the last 50 come from np.empty. A range inside one block returns a shorter slice, as before.
  2. An nsX with only empty blocks raises IndexError at block_infos[0]. I made one from a basic header plus a single 0-point data header. On master it loads with 0 segments.
  3. In the PTP path, last_sample_indices and last_sample_positions are built for every sample just to describe the few gaps. Peak memory in parse_header goes from about 17 to 34 bytes per sample (tracemalloc on a 1.1 M-sample file), which matters for large PTP files on small cloud runners. Could _classify_gaps return the forward gap indices and let each caller build the report only for those? That would also let a segmenter drop gaps it knows are harmless before deciding whether to raise. That's exactly what packets written out of order need (#1937).

Two smaller points:

  • gap_tolerance_ms=0 isn't quite "every block its own segment": blocks ≤ 2 periods apart still merge. So one_sample_blocks gives 101-sample segments starting one tick earlier, where master gave 100. I think that's fine, but the description and docstring should say so.
  • On the default: every paused 2.2/2.3 file in gin now raises out of the box, which will reach SpikeInterface users quickly. Would you consider a release where None still splits at every block and emits a FutureWarning naming gap_tolerance_ms, before switching to the error?

On our spec 3.0 test recording (signal generator, linked in #1924) the PTP behaviour matches master, as the description says. With the reordered copy, the default raises on the out-of-order packets, and gap_tolerance_ms=0 produces 52- and 3-sample segments around them. That's the #1937 case, not a regression here.

Line notes

  • neo/rawio/blackrockrawio.py:794: i_stop past the end of the segment leaves rows of out uninitialized when the range spans more than one block, while a single-block range returns fewer rows instead. Suggest i_stop = total_samples if i_stop is None else min(i_stop, total_samples). A regression test would be cheap: two_data_blocks_after_pause with a large tolerance and i_stop = size + 50.
  • neo/rawio/blackrockrawio.py:1634: If every block is empty this list is empty, and block_infos[0] at 1681 raises IndexError. Master returned 0 segments here; an early return [] would restore that.
  • neo/rawio/blackrockrawio.py:1722: These two N-length arrays, plus the float64 copy and diff above, double peak memory for large PTP files, but they're only needed at the gap indices. Could _classify_gaps return forward gap indices and the caller build the report from timestamps[idx]? Doing the diff on the integer ticks rather than float seconds would also save a copy.
  • neo/rawio/blackrockrawio.py:1582: Raising here means a caller can't remove gaps it knows are benign before the error. Splitting "find gaps" from "raise / apply tolerance" would make that possible; the out-of-order packets in #1937 are the case I have in mind.
  • neo/rawio/blackrockrawio.py:129: This note says pauses in 2.2/2.3 are split into segments automatically, which is no longer true by default. Could it point to gap_tolerance_ms?
  • neo/rawio/blackrockrawio.py:125: Worth documenting here three things: a non-zero tolerance places samples after a merged gap early under t_start + i / sampling_rate; _get_blackrock_timestamps keeps the recorded times; and 0 still merges blocks closer than 2 sampling periods.

@h-mayorquin

Copy link
Copy Markdown
Contributor Author

Thanks for the review,

  1. I fixed the bug that extracted values outside of the segment. There is a test for this now.
  2. I am not sure this is the right behavior but I changed the code so this keeps doing what master does.
  3. Yes, I refactor this part of the code so only what needs reported gets loaded. Thanks for taking a look at this.

On your two smaller points:
The docstrings are fixed and I have added a deprecation period for the change of behavior. The deprecation will be a future warning for 0.16 which carries the report already so useres will get information about the temporal structure of their samples as soon as possible but not a breaking change.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants