Repository navigation
Reading Blackrock recordings from fsspec URLs: is remote reading in scope for rawio? #1931
Description
We convert Blackrock recordings (NSx, NEV, PTP) that live in cloud blob storage. Today that means downloading each file first. We have a working implementation of BlackrockRawIO that reads straight from an fsspec URL (abfs://, s3://, ...) with byte-range reads, and we would like to upstream what you are willing to take. Before opening PRs we would like to know whether remote reading is something rawio should support at all, because neo is built around local files and memory maps and nothing in rawio reads remote files today (the only fsspec use I found is storage_options for kerchunk in xarray_utils.py).
What we have
BlackrockRawIO(filename="abfs://container/rec", storage_options={...}). A name containing://goes throughfsspec.core.url_to_fs; anything else behaves exactly as now.fsspecstays an optional import (aneo[remote]extra).- Local files keep the
np.memmappath unchanged. The remote path copies byte ranges instead, because a remote file cannot be memory-mapped. - Headers, signal chunks (standard and PTP) and
.nevpackets are read as byte ranges. A.nevis scanned in pieces instead of being read whole. - On every Blackrock test file in
ephy_testing_data, remote and local reads agree: headers, all signal chunks, timestamps, spikes, waveforms and events, including.nevscans in pieces as small as one packet. (Tests use an fsspec filesystem over the same local files, so CI needs no credentials.) - Memory: opening and streaming a 2.43 GB
.nevthrough fsspec peaked at 0.32 GB, against 4.63 GB for the existing local code path.
Proposed split, smallest first
- Generic file access (small, reusable by other rawios): helpers in
rawio/utils.pyto read records and strided values from an open file, plus an optional-fsspec opener. No change toBaseRawIO. - Blackrock streaming: chunked reads without mmap, PTP over a URL,
.nevstreaming. Blackrock-specific, most of the code. iter_spikes(): new public API that streams every unit's spikes in one pass. The per-unit methods each have to read the whole.nev, so a remote file would otherwise be read once per unit. This is the part we expect to need the most discussion.
Independent of this, we have sent or are sending small bug fixes as separate PRs (a .nev over 4 GiB failing under NumPy 2: #1929; the has_buffer_description_api() check: #1930).
Questions
- Is remote reading in scope for rawio, or do you see it as out of scope for neo?
- If it is: should it be opt-in per rawio (what we did for Blackrock), or a
BaseRawIOfeature (astorage_optionsargument, with file access going through the base class)? Per-rawio is much smaller, since 21 rawio modules usenp.memmap,np.fromfileormmapdirectly, but it leaves other formats to adopt it one by one. - Would you accept
iter_spikes()as new rawio API? If not, would you prefer we keep it out of neo? - Is the extra maintenance of a remote code path inside
blackrockrawio.py(about 700 added lines, tested against local files) acceptable? If not, we will maintain it in a fork and only send the generic pieces.
Happy to split or drop any part.
Activity
zm711 commentedon Oct 24, 2026
@cboulay thanks for bringing this up. I think traditionally we've been doing most remote reading through spikeinterface rather than directly in Neo. What do you think @h-mayorquin and @alejoe91 ?
h-mayorquin commentedon Oct 28, 2026
Thanks for writing this up so carefully and for asking before opening the PRs (and for measuring the memory!). I thought a lot about this before answering. These are my answers, @alejoe91 or @samuelgarcia might see some of them differently: 1) I would not add remote reading to rawio yet, I would like to get the reader closer first. 2) If we do it, I would prefer something general for all the readers instead of an opt-in per reader, but the interface is an open question for the other maintainers. 3) I would not add
iter_spikes()as Blackrock-only API (see the lazy waveforms below for the per-unit reads). 4) For now I would prefer not to maintain a second code path inblackrockrawio.py, so your second option of keeping the remote path in your fork and sending the generic pieces sounds good to me.The reason is that I tried to move the Blackrock reader in this direction in #1820 (describing the buffers instead of keeping memmaps open) but I found the following challenges:
gap_tolerance_msto the standard nsx formats #1880 merges blocks into one segment and then a segment is no longer one region..nevhas no index. To know which units each electrode has and where the clock resets are, every packet has to be read at parse.Did you manage to overcome them?
What I would like to do is to move forward in that direction slowly by doing small improvements such as:
seekandread) instead of callingnp.fromfileandnp.memmapon the path. I opened Blackrock: read headers through an open file instead ofnp.fromfileandnp.memmap#1940 for this..nevspike waveforms lazily instead of copying every spike packet into memory at parse. I opened BlackrockRawIO: read.nevspike waveforms lazily instead of copying them at parse #1941 with a prototype and some profiling. With this, the per-unit reads should not need to scan the file again.I think this will make your own solution simpler. It should also get the reader to a point where the changes needed for remote reading are minimal and then we can focus the discussion on a robust design for it.
If you have time, it would be great if you could review #1940 and #1880. In #1880 the PTP part touches the same code as the reordering you mentioned in #1937.
cboulay commentedon Oct 28, 2026
Thanks Heberto, that plan works for me: remote reading stays in our fork for now, and I'll send the generic pieces in small PRs. On your three challenges, here's where our fork landed. We worked around them rather than solving them in general:
gap_tolerance_msto the standard nsx formats #1880) can't be one region, so for those we'd return no buffer description and fall back to chunked reads, rather than invent a multi-region format..nev: one streaming pass at parse counts spikes per unit and finds resets, anditer_spikes()streams them afterwards. Your table in BlackrockRawIO: read.nevspike waveforms lazily instead of copying them at parse #1941 is a cleaner version of the same idea behind the normal public API. I'd rather adopt it and dropiter_spikes()than add Blackrock-only API.I've reviewed #1940 and #1880.