Repository navigation
Blackrock: stop dropping NEV spikes and events at segment edges (1 ns timestamps) - #1925
Merged
h-mayorquin merged 3 commits intoOct 28, 2026
Conversation
…nts by id At the 1 ns timestamp resolution of PTP files, spikes and events at the edge of a segment were kept or dropped by rounding: the segment bounds went through float seconds back to ticks, and np.searchsorted compared uint64 timestamps with a Python int as float64. spike_count and event_count used the same window and could disagree with the timestamps returned. Without t_start/t_stop, items are selected by segment id alone, as the getters already did, and the counts select the same way. Explicit bounds are compared with the times neo reports (timestamps as float64 seconds), so a bound taken from neo, such as segment_t_start, selects the item it came from. t_stop stays exclusive; documented, with the advice to pass t_stop + 1e-5 to include an item at t_stop.
h-mayorquin
requested changes
Oct 28, 2026
h-mayorquin
left a comment
Contributor
There was a problem hiding this comment.
This looks good. Using the segment ids that were already computed instead of recalculating the membership from the time windows is better as it states the identity directly. This is what I did in #1880.
The fixes seem OK to me but my only qualms are about testing. My requests:
- Add
blackrock/segment/ResetCorrect/resettoentities_to_test. It has a real clock reset, so the split stays whatever we decide in #1937 (two segments now, two blocks if a reset starts a new block). The shared checks already comparespike_countwith the timestamps returned for each segment (read_spike_timesinrawio_compliance.py), and on this file they fail on master (nb_spike 40 != 188) and pass here. So the count fix needs no new test code, and the other shared checks run on a multi-segment Blackrock file for the first time. - For the spikes lost at the segment edges a specific test is still needed because it takes the PTP file read without its
.nsx. I think it could be simpler and use only the public API: check that the spikes returned over every block, segment and unit add up to the 999 in the file (master returns 995). Something like:
def test_nev_spikes_at_segment_edges_ptp(self):
reader = BlackrockRawIO(
filename=self.get_local_path("blackrock/blackrock_ptp_with_missing_samples/Hub1-NWBtestfile_neural_wspikes"),
nsx_to_load=[],
)
reader.parse_header()
returned = 0
for block_index in range(reader.block_count()):
for seg_index in range(reader.segment_count(block_index)):
for unit_index in range(reader.spike_channels_count()):
returned += reader.get_spike_timestamps(block_index, seg_index, unit_index, None, None).size
self.assertEqual(returned, 999)I think this way of testing is better because:
- Adding the file to
entities_to_testreuses the checks that every reader already runs, so the multi-segment case gets all of them and not only the count. - The specific test uses only the public API, so it does not break if
nev_dataorinternal_unit_idschange in a refactor. - It checks what the shared checks cannot, that every spike in the file comes back somewhere. That has to hold for any segmentation, so it keeps working whatever we decide in #1937.
- The expected value comes from the file itself, the number of spikes it holds, so it does not depend on how the reader assigns the segments.
The current test failures come from the gin 403 and not from this PR. #1938 should fix them.
…t PTP edge spikes through the public API
h-mayorquin
approved these changes
Oct 28, 2026
h-mayorquin
left a comment
Contributor
There was a problem hiding this comment.
This is ready, thanks for the changes on the testing.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
For a multi-segment
.nev,BlackrockRawIOclipped spikes and events to the segment's time window: the bounds were float seconds converted back to ticks, then compared withnp.searchsorted(which comparesuint64with a Pythonintas float64). Float64 seconds at 1.75e9 (PTP files, 1 ns resolution) are only precise to about 240 ns, so items at a segment's edge were kept or dropped by rounding.spike_count/event_countused the same window and could disagree withget_spike_timestamps. Non-PTP files (30 kHz ticks) are not affected.On the NEV of
blackrock_ptp_with_missing_samplesopened alone (4 segments), 4 of 999 spikes were lost, with and without explicit bounds. Converting explicit bounds to exact integer ticks does not fix it (5 lost), because the rounding is in the float seconds the caller passes.Changes:
t_start/t_stop, items are selected by segment id alone (the getters already did), andspike_count/event_countdo the same.segment_t_start) selects the item it came from.t_stopstays exclusive. This is now documented, with the advice to passt_stop + 1e-5to include an item att_stop(the old+1 tickin the default path was a no-op at 1 ns).The new test checks every unit of every segment of that file against the segment ids, with and without bounds, and fails on the old code.
Found while working on Blackrock reading with Colin McNabb at Blackrock Neurotech.