Repository navigation
Conversation
- EvioFile: keep scanning past a record with the last-record bit while another record header (magic word) follows. - Record.scan_events: when a record has no event index array, walk events by the length word of each event bank. - EvioFile.__del__: ignore BufferError when numpy views of the mmap are still alive. Before, these files showed 1 record and 0 events. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A local web page that decodes EVIO v6 files in the browser: file checks, occupancy, rate, charge and timing plots (file, ROC and channel level), and a structure tree down to single hit words. `pyevio web [DIR]` serves it with a folder browser on localhost. Also adds examples/fadc250_stream_to_parquet.py and docs/web_viewer.md. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Brings cissieAB:web-viewer into the gui-fix branch as the base for the web viewer fix program (space/gui-plan.md). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Flat-layout auto-discovery picked up space/ as a second top-level package and refused to build, so uv run / pip install failed. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
`uv run pytest` now works without --extra test, in worktrees too. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
F10: test_complex_event_structure and test_real_world_event_structure failed on main and gui-fix because their fixtures were wrong. A length word counts the words that follow it, so the length-11 bank at 0x28 of the real-world stream spans words 10..21 and the 0x000F header at 0x50 is nested inside it: the event has two top-level banks. The complex fixture declared lengths that disagreed with its byte sizes under any convention. Both tests now build EVIO-consistent data, pin the nested structure, and pass. Status: RESOLVED. The investigation exposed that Bank treats the length word as counting itself (size, end_offset and data_length one word short; to_numpy drops the last payload word of every leaf bank). The new strict XFAIL test_bank_length_counts_words_that_follow pins that defect as F38. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
F35: tests/golden/test_golden.py runs one test per row of the evio-optim golden manifest (44 rows, sha256 verified): header event-count sum, events found by scan_events, no event past EOF, and the I9 error contract for v4 and v1-3 files. The .dump rows skip (no file header). Today's result: 4 pass (writer_noidx_le, zeroev_le for cpp and java), 36 strict XFAIL with the finding number: F3 trailer ghosts, F4 compressed records, F9 generic version error, F39 HIPO file type id rejected. No golden row exercises F8. tests/golden/test_corpus.py checks record and event counts of the eight converted SRO corpus files; marked slow, runs only with PYEVIO_CORPUS=1. All eight are strict XFAIL on F3 (trailer counted as a record, trailer index read as events); the LZ4 files also on F4. pyproject: pyyaml in the dev group, the slow marker registered. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
F16: one copy of the hit-word layout (bits 0-12 charge, 13-16 channel, 17-30 time, bit 31 flag; t_ns = time * 4) and of the frame walk (event 0xFF60 -> SIB 0xFF31 with TSS 0x32 -> ROC banks -> stream-info 0xFF30 -> payload banks) in pyevio/decoders/fadc250_stream.py, as pure functions over uint32 event words. F13: no caps on ROC or port counts; both streams of a crate merge by ROC id with the stream identity (ROC bank num) kept per hit. F14: empty payload banks (len 1, num 1) and data type 0x00 are valid; empties are counted per port. F19: validation raises ValueError on the raw field, never assert. Bank extents use length + 1 words throughout (F38-safe). Tests prove hit columns and counts on synthetic frames in both byte orders; a bank length past the event end stops the walk with errors counted and no exception; the batch form concatenates with correct first_hit_index per frame; on the corpus (PYEVIO_CORPUS=1) record 0 decodes to the sum of (payload length - 1) per ROC id from an independent struct walk: 499,660 hits, 264 ROC banks, 45 empty banks. Throughput: 592 MB/s on one record, 321 MB/s over 200 records. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
F16: the example and cli/debug.py call pyevio.decoders.fadc250_stream instead of private copies of the walk and the hit layout. F17: the example survives a last ROC bank without children (the crash shape: IndexError past the event end) — pinned by a test with that frame shape and with an empty payload bank in the last ROC bank. F20: hits sorted by (frame, event index, t_ns), frames by (frame, event index), so hit ranges stay valid with duplicate frame numbers. F18 (partial): the example uses get_event_offsets and keeps no Event objects; EvioFile's Record cache remains core's concern. Parquet schema: new `stream` column, `port` widened to uint16, DECODER_VERSION "2". New finding fixed on the way: cli/debug.py called Record.get_record_and_event, which does not exist (it belongs to EvioFile), so `pyevio debug` printed an error for every event; it now calls Record.get_event. The `length <= 2` gate that skipped decoding of any payload bank with two or more hits is gone; a stream-frame summary table prints for 0xFF60 events. Pinned by test_debug_command_decodes_stream_frame. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
F38: Bank sized itself as length*4, dropping the last payload word of every leaf; child walks only succeeded through skip-a-word heuristics. Bank.size, data_length and end_offset now use (length + 1) words, get_children advances by that and stops at the first child whose length is 0, has no room for a header or overruns the parent, recording why in Bank.errors. RocTimeSliceBank and StreamInfoBank get the same convention (segment TSS end, payload bank stride, bank end). Tests: test_bank_length_counts_words_that_follow flips XFAIL -> PASS; the six create_mock_bank fixtures migrate to evio_bank(); the three heuristic tests (minimal length, zero length, overrun) now pin stop-and-record. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
F3: RecordHeader gains header_type (bits 28-31 of word 5) and is_trailer; Record.scan_events returns nothing for a trailer, uses min(event_count, index entries), stops at data_end and records an overrun in Record.errors. Per D12, EvioFile.record_count / get_record / iter_records cover data records only; the trailer is EvioFile.trailer (or None). F9: EvioFile._detect_layout reads the byte order from the magic at word 7, takes a file type id in word 0 as a v6 file header and otherwise raises one ValueError naming "version N" and the evio tools (evio2xml, eviocopy). F39: the HIPO id 0x4849504F is accepted, in either byte order, because HIPO writers store the ASCII bytes in order regardless of the file byte order. Also: EvioFile.get_event unpacked a 2-tuple into four names; BufferReader and RecordHeader no longer print hex dumps before raising. Tests: 18 golden rows (F3, F9, F39) and the four uncompressed corpus rows flip XFAIL -> PASS; tests/test_record_scanning.py adds trailer-with/without-index, trailer-only, count-bounding, bare-v4-block and HIPO-id cases. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…(F38, F3, F9, F39) Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
F42 / D14 / F13: the frame walk reads each ROC bank's stream-info 0x41
payload-port map (uint16 entries module<<8 | port, two per word) and
keeps module_id per ROC bank, per empty bank and per non-fADC bank. Hits
come only from module 0 (fADC250) ports; other modules' payload words
(DCRB channel masks on 6 of the 12 crates in run 791) go to
non_fadc_words, per ROC bank and per (event, roc, stream, port). A port
with no map entry decodes as fADC250 and is counted in
n_unmapped_payload_banks. validate() checks hits + non-fADC words =
payload words.
Tests: a two-module synthetic frame in both byte orders (DCRB words never
reach the hit columns), the unmapped-port rule, a malformed 0x41 segment
stops at the bank end with an error and the walk continues; on corpus
record 0 the map names fADC250 for exactly {7,9,25,27,29,84} and DCRB for
{44,45,46,53,54,55}, hits per rocid equal the sum of (payload length - 1)
over fADC ports from an independent struct walk: 99,114 hits, 400,546
DCRB words, 499,660 payload words. pyevio debug shows module, hits and
other words per ROC bank; the Parquet example records non_fadc_words in
its metadata.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…gate Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
F11: frame stride = the most common frame delta; missing frames counted against it; duplicates, out-of-order frames and timestamp-delta outliers reported. F12: frame period = median of timestamp delta per frame delta; the tick length is the knob tick_ns (default 1 ns, D2). F13: histogram axes are sized from the ROC and port sets found in the data; worker partials grow and add; the stream identity comes from the decoder. F28 (binning): the rate series is binned by time from the first to the last timestamp and normalized by the frames each bin covers (hits per second of live time). A serial header scan (frame and timestamp per event, 0.3 s for 86,794 events) fixes the time axis and the frame checks; a forkserver pool decodes record ranges, each worker opening the file and returning numpy partials that add. Trailer records are skipped; events past the record end are counted, never read. StreamStats round-trips through .npz with a version field. Tests: a synthetic striped file (stride 4, a duplicate, a gap, a non-stream event, a trailer, one fADC250 and one DCRB crate) proves totals, per-port module ids, occupancy, charge/time bin placement, rate normalization, 1-worker == 2-worker results and the npz round trip; the corpus smoke (200 records) matches decode_frames, stride 4, 0 missing, period 65,536 ticks. Full corpus file sro_000791.evio.00000.v6, 22 workers: 15.55 GB in 30.5 s (509 MB/s); 764,009,774 fADC250 hits = the PHASE0 reference 763,992,122 plus the 17,652 hits of the two salvaged tail events. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
pyevio stats FILE [--out FILE.npz] [--workers N] [--rate-bins N] [--tick-ns X] [--records N] runs stream_stats and prints totals, payload words per module, frame checks, span vs live time, hit rate, wall time with MB/s over the bytes processed, and a per-ROC table. Test: test_stats_command_prints_summary. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…F11, F12, F13, F28) Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
After PHASE1 batch 1 (D12) EvioFile.record_count and get_record cover data records only, so the frame scan never visits the trailer. n_records now means data records and the trailer counts once the scan covers the whole file. Test expectation updated: 3 data records, 1 trailer. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
F45: a bank of segments (0xd/0x20) or tagsegments (0xc) holds 1-word-header children; Bank.get_children read them as 2-word banks, taking a segment's first payload word as a header. Segment and TagSegment (Bank subclasses) parse their own header word; Bank.child_class() picks the class from the parent's data type and the walk reads each child's length through it. F5: Bank takes a limit (parent end, event end or buffer end) and a length past it is recorded in Bank.errors and clamped, so no read runs past it; Event.get_bank passes the event end; a buffer too short for the header is refused with ValueError before any struct read. Tests: test_sro_event_segments_are_one_word_structures (the SIB and stream-info banks of a synthetic SRO frame), test_tagsegment_container_children, test_segment_walk_stops_at_overrun, test_bank_clamped_to_limit. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…e cleanly
F4: Record.buffer is the mmap for a plain record and, for LZ4 / LZ4_BEST /
gzip (word 10 bits 28-31), the decompressed bytes built on first access:
LZ4 block format fed compressed_length_words*4 - pad3 bytes with the output
size index + padded user header + word 9 (word 9 counts the events only);
gzip through zlib (wbits 47). Index, user header and events are parsed from
that buffer, so event offsets of a compressed record are buffer offsets.
EvioFile caches one Record (the last accessed) so a compressed file never
sits in memory whole; Record.release() drops the buffer.
F8: user header lengths are padded to whole words in first_record_offset
and Record.data_start.
F5: a final record whose declared length passes EOF is kept, its complete
events are walked and ("truncated_tail", declared, present, salvaged) goes
to EvioFile.checks; get_total_event_count counts the salvaged events for
such a record; bytes that are no record header, or a header the scan cannot
step over, end the scan with ("bad_record_header", offset, reason).
RecordHeader checks the buffer holds its 8 or 14 words and exposes
pad1/pad2/pad3.
F7: EvioFile.close() closes the mmap and the file in separate try/finally;
a BufferError from a live numpy view is re-raised naming the cause after
the file is closed; __exit__ propagates it, __del__ stays silent.
Also: pyevio event -r looked the event up with a global index and searched
get_records() by identity; it now uses record.get_event.
Tests: all 5 F4 golden rows and 4 LZ4 corpus rows flip XFAIL -> PASS; the
golden rows also assert record.errors == [] and checks == [];
tests/test_record_scanning.py adds synthetic lz4/lz4best/gzip records with
pad3, odd user headers at file and record level, a truncated tail with
salvage, garbage after a record, a zero-length header, a root bank past its
event, and the two close() cases.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…ildren (F4, F5, F7, F8, F45) Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
After PHASE1 batch 2, event offsets of a compressed record index Record.buffer (the decompressed bytes), and EvioFile.close() raises BufferError while a numpy view of the mmap is alive (F7). stream_stats, the Parquet example, pyevio debug and the corpus test read event and bank words from the owning object's buffer (Record.buffer, Bank.mm, Event.mm) instead of the file mmap, bound against that buffer, release each record after use, and delete their last views before the `with` block ends. stream_stats no longer reaches into the removed EvioFile._records cache. F48: the example kept mmap views past the `with` block — the leak F7 is meant to expose. F4 follow-up: callers that read from evio_file.mm would decode garbage on LZ4/gzip files. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…tail
F2: a file with no file type id is read as the JLab SRO stream when every
8-word block header in the chain has version word exactly 4 (no bit-info
bits) and the magic at word 7 (D17: the first block of an evio-library v4
file is identical, so the whole chain is walked and any bit-info bit —
last-block 0x200, dictionary 0x100 — or another version raises the F9
error; a 14-word v6 record stream without a file header gets its own
message). EvioFile.header_layout and FileHeader.layout are "sro-8word",
EvioFile.header is FileHeader.for_sro_stream(endian) (version 6 content
in a version-4 container, record_count filled from the scan), the record
headers carry header_length 8 and RecordHeader.parse accepts that shape,
trailer is None, first_record_offset is 0.
F6: EvioFile._record_header_problem validates a header before a Record is
built: 32 bytes present, no file type id in word 0 (a concatenated file),
magic at word 7 (a byte-swapped header is named), header length 8 or 14
with its words present, record length at least the header. A problem ends
the scan with ("bad_record_header", offset, reason) in checks; _is_record_at
also requires the record to fit the file.
D4: a final record whose declared length passes the end of the file stays
the last record (is_truncated), its complete events are walked and
("truncated_tail", declared, present, salvaged) goes to checks.
Tests: tests/test_sro_layout.py builds SRO-shape files (two blocks of
nested stream frames, a third cut mid-event: record count, event count,
salvage, checks, bank tree through Segment children), refuses the v4
chains with the last-block or dictionary bit, v3 and v2 chains and a v6
buffer dump, names byte-swapped and garbage blocks and a second file
header, and reads the raw files sro_000791.evio.00000/00001 when present:
7892 + 1 records, 86,794 / 86,805 events, the reference tail byte counts,
open in under a second, and stream_stats over 20 records (220 frames,
stride 4). pyevio stats on the raw file 00000: 764,009,774 fADC250 hits =
the PHASE0 reference plus the two salvaged tail events; 25.6 s, 20 workers.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…ail salvage (F2, F6, D4) Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…ks in info A bare stream of 14-word version-6 records with no file header (an evio writer buffer, the golden .dump rows) opens as layout "v6-records" with first_record_offset 0 and the stand-in FileHeader.for_record_stream(endian) (record_count filled from the scan); the dictionary in the record user header is skipped by the padded user header length, as for any record. EvioFile.record_header_problem(offset) is public, for a record table's "header quirks" column: the same wording the scan stores in checks. pyevio info prints the layout, the content and container versions when they differ, and a File Checks section (truncated tail with byte counts and events kept, bad record header with offset and reason) via describe_check(). Tests: the four .dump golden rows flip from skip to pass with their manifest event counts and assert layout "v6-records"; test_sro_layout.py adds a synthetic two-record buffer dump and a CliRunner test of pyevio info on the truncated synthetic SRO file. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
What pyevio reads (v6 and HIPO-id files, LZ4/LZ4_BEST/gzip records, trailers, SRO hardware streams with cut tails, the version error for v4/v1-3 and for bare v6 record streams), the API in short (EvioFile -> Record -> Event -> Bank/Segment/TagSegment; checks, errors, close() and the BufferError rule, Record.buffer for compressed records, truncated-tail semantics, the trailer excluded from record_count), the fADC250 streaming decoder and stream_stats with short examples, every CLI command including pyevio stats, installation with the pyevio[web] extra named, and the development setup (uv sync, uv run pytest, GOLDEN_MANIFEST, PYEVIO_CORPUS=1). Removed as false against the code: the dump options --depth/--preview/ --hexdump, the hex offset 0x100 (click INT), the RocTimeSliceBank API example (F41/F46), the 0xFF30/0xFF31 bank roles, "info is the default command", the dependency list without textual. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Frame layout (0xFF60 -> SIB 0xFF31 with TSS 0x32 and AIS 0x42 -> ROC banks tag rocid, num stream mask 0x21/0x22 -> stream-info 0xFF30 with TSS 0x31 and payload map 0x41 -> payload banks tag port, type 0x00, len 1/num 1 empty), the hit-word table with the evidence note (independent parser agrees per rocid x port; per-channel split unconfirmed by the DAQ group), module types from the map (fADC250 decoded, DCRB counted), the two tick lengths (4 ns hit ticks, tick_ns knob for timestamps, 65,536 ticks per frame), striping and the stride check, the stream_stats fields, .npz layout and version field, and the file quirks with the check each one produces. Numbers from reports/phase0_sro_reference.md, phase1_findings.md and phase2_findings.md. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Keep-a-Changelog format, one Unreleased section: Added (SRO streams, compression, tail salvage, checks/errors, Segment/TagSegment, the shared decoder, stream_stats, pyevio stats, the web viewer and Parquet example, the golden harness), Changed (trailer out of record_count, bank length semantics, buffer offsets for compressed records, Parquet schema, packaging), Fixed (one line per finding with its symptom in user terms: F38, F3, F4, F5, F6, F7, F8, F9, F39, F43, F45, F49, F2, F10, F37, F40, F44, F16, F17, F19, F20, F42, F13, F14, F11, F12, F28, F48, plus the pre-ledger last-record-bit fix 88b188d without a number). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…ayout
Fixed: the last-record bit is bit 9 (the prose said 10, which is the file
header's trailer-index flag); word 8 counts the event bytes alone, index
array and user header excluded (verified on the C++ and Java writers,
phase0 reference); pad 3 counts filler bytes of the compressed data; the
event index is written by the evio library but absent in some streams and
pyevio walks by event length words then; 0xcoda0100 -> 0xc0da0100; header
length 8 exists in hardware output. Added how pyevio scans (header
validation, bad_record_header and truncated_tail checks, last-record bit
tolerance), the trailer's 0x30000206 word and its exposure as
EvioFile.trailer, and an "SRO hardware layout" section (no file header,
8-word block headers {len, block#, 8, evcount, 0, 4, 0, magic}, no index,
no trailer, cut tail, the whole-chain detection rule).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
`pyevio debug` printed a frame's error messages and skipped the new warning messages. The changelog header ran to F54, and the cache-key change (F60) and the new `pyevio stats` rows (F61) had no entry. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XDaEfG6VfRLmUbbHW3Uit5
The console viewer is the optional extra `pyevio[gui]` (textual, textual-plotext), symmetric with `pyevio[web]`; `textual` leaves the core dependencies, so the library installs with click, lz4, rich, numpy and matplotlib only. `pyevio gui [PATH]` opens a file or a folder browser and prints an install hint with exit code 1 without the extra. The `pyevio ui` command and `pyevio/ui/` (a hard-coded sample tree) are removed; nothing imported them. The EVIO file-name filter moves from `pyevio.web.routes` (which imports FastAPI) to `pyevio.web`, so the console folder browser can use it without the web extra. pytest-asyncio joins the dev group for Textual's headless pilot tests (asyncio_mode auto). Q33, Q34, Q39 (D20—D22). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XDaEfG6VfRLmUbbHW3Uit5
`pyevio.gui.session.Session` (no Textual) owns the open EvioFile, loads the shared statistics cache or starts the same AnalysisJob thread the web server uses, and serves summary, histograms, record rows, structure nodes and raw words through the `pyevio.web` model functions, so both viewers show the same numbers. Opening an unreadable file raises before the current file is touched; switching files cancels the running analysis. `PyEvioGui`: header, status strip with progress and the last error, four tabs (Overview, Plots, Records, Structure, keys 1—4) and the open screen (`o`): a directory tree filtered to EVIO names (Ctrl+A for every file), a path input, Backspace for the parent folder, Escape to cancel. The app polls the job on a timer and drives the panes through three hooks (file_closed, file_opened, stats_ready); panes hold model dicts only, so EvioFile.close() never meets a live view. The four panes are placeholders to be filled per tab. Q36, Q37, Q38 (D23—D25). Tests: tests/gui/test_session.py (open, cache hit, refusal, cancel on switch) and tests/gui/test_app.py (start with a file, tabs, open screen, unreadable file keeps the current one, start without a file). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XDaEfG6VfRLmUbbHW3Uit5
The Overview pane renders the build_summary dict the app hands it: the totals (records and trailer, events by kind, fADC250 hits, hit rate, hits per frame, duration and live time, modules, decode errors and warnings), "what stands out" with the error and warn checks and the first decode messages (the web viewer's rule; info checks stay in the checks table), the full checks table with the status coloured, the file facts including the scan checks in words, and one row per ROC. It says the analysis is running until stats_ready and that no file is open after file_closed, so a failed open of another file keeps the current overview. The Records pane is a row-cursor DataTable over one page of 500 record rows at a time, loaded in file_opened before the analysis is done. The ] and [ keys replace the page, and moving the cursor past the last or first row of the page loads the next or previous one, so a file of a million records costs one page of strings. The row key is the record's node id, and Enter opens that record in the Structure tab. Both panes keep only the dicts the session returns, never a Record or a view of the file, so switching files cannot trip EvioFile.close(). Pilot tests cover the synthetic file's checks and ROC rows, the hook states, the trailer row, Enter, and paging on a 600-record file. I38, I39. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XDaEfG6VfRLmUbbHW3Uit5
The console viewer's Structure tab now matches the browser's. The left side is a Textual Tree over pyevio.web.structure.children_of: the file, its records and trailer, events, banks and segments labeled by role inside stream frames (stream info, ROC bank with ROC id and stream, ROC stream info, payload-port map, payload banks with port, module and hit count) down to decoded hit words. Children load one page of 500 when a node is expanded; a page short of the total ends with a "more" row that loads the next page in place, so a file with thousands of records or a record with thousands of events never loads whole. The right side shows the highlighted node: every field of its dict, its walk errors in red, the decoded payload-port map and a hex dump of its raw words read through words_at, from the file for record headers and from the record buffer for events and banks, with a note when that buffer is decompressed; n and p move the 64-word window. show_node expands the path to an id (paging as needed) and selects it; the Records tab and the app's show_structure_node use it. Panes keep only the session's dicts, never a Record or Bank, so file switching stays safe. The detail column is not focusable on purpose: a focused widget inside a hidden TabPane makes Textual re-activate that tab, which would trap the number keys. Tests drive the pilot through the synthetic stream file, a 600-record file and the LZ4 golden file. I40. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XDaEfG6VfRLmUbbHW3Uit5
The Plots tab of pyevio gui now draws what the browser's Plots tab
draws, with textual-plotext: hits per frame, hit rate over time and the
ROC-by-port occupancy heatmap at file level; the port-by-channel
heatmap, rate, charge spectrum and time in frame for a ROC; rate, charge
and time for one channel. A control row of Select widgets picks the
level, ROC, port and channel, and keys do the same without the mouse:
f/r/c switch the level, ]/[ step the ROC, }/{ the port, +/- the channel
(rolling over into the next port like the browser's Prev/Next). Each
plot keeps only the histogram dict of pyevio.web.report and rebuilds its
plotext figure on render, so resizing redraws it; long histograms are
rebinned to the widget width, and the heatmap is drawn as oversampled
colour blocks with the counts printed in the cells and a dash on ports
the ROC's map does not list.
pyevio.gui.export turns one histogram dict into a PNG through matplotlib
(Agg, so it works on a farm node): s saves the focused plot into the
working directory, S every plot of the level, with names like
synthetic_charge_roc7_port1_ch3.png. The module also holds the pure
helpers (series, rebin, occupancy grid, ticks) the pane and the tests
share.
Two Textual details: the pane takes the focus when its tab is shown so
its bindings work at once, and focused plots or Selects release the
focus when hidden; Textual's blur would hand it to a sibling in the
hidden tab and TabbedContent would switch the tab back. Reading the
occupancy depth from the selection rather than the data shape fixes the
export and the heatmap of a file without stream frames. I41.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XDaEfG6VfRLmUbbHW3Uit5
A Rust library that reads what pyevio reads, over any std::io::Read without seeking and with one record in memory: v6 files, bare v6 record streams and the SRO hardware stream of 8-word block headers, in both byte orders, with LZ4, LZ4_BEST and gzip records decompressed. Words stay in file byte order and are decoded on read; a record owns one buffer and events and banks are slices into it. Every record, event, bank, segment and tagsegment length is checked against its container, so corrupt data ends a walk with an error value instead of a panic. A cut final record keeps its complete events and reports the truncated_tail check; garbage after the last record reports bad_record_header; a trailer's index is never read as events; evio-library v4 and v1-3 files are refused with the version error pyevio raises. The fadc250 module walks streaming frames as pyevio.decoders.fadc250_stream does: module per port from the 0x41 map, hits only for fADC250 ports, other modules counted, empty banks counted, the AIS counted and never trusted. The format comes from docs/evio6_file_structure.md and docs/stream_decoding.md. Dependencies are lz4_flex (safe decoder) and flate2 only. Q40, D27. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XDaEfG6VfRLmUbbHW3Uit5
evio-count prints records, events, trailer, layout, the file checks and the throughput of a file or stdin; with --fadc250 it walks every stream frame and decodes every hit word, printing hits for the complete records and the salvaged tail, other-module words, empty banks, AIS counts and a per-ROC table. evio-dump prints record headers and one event's bank tree with the first words in hex. tests/golden.rs walks all 46 rows of the evio-optim golden manifest (checked in as a TSV so cargo test needs no Python): the 39 readable rows give the manifest event count and layout with every bank tree walked, the 7 v4 and v1-3 rows end with the version error. tests/corpus.rs, behind EVIO_CORPUS=1, reproduces the 8 converted corpus files and the raw run-791 files: 86,794 events with truncated_tail (1967400, 531760, 2) and 764,009,774 fADC250 hits (763,992,122 in the complete blocks) for file 00000, 86,805 events with (1960136, 1939708, 10) and 764,140,261 hits (764,052,526 complete) for file 00001, matching pyevio and the independent decoders. One thread decodes every hit of the 15.5 GB raw file in 3.1 s (4.9 GB/s) and scans its records in 1.6 s (9.6 GB/s). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XDaEfG6VfRLmUbbHW3Uit5
The Records pane's status line and the Structure pane's node-path line kept the ids records-placeholder and structure-placeholder from the app skeleton only because tests/gui/test_app.py read them. They are now records-status and structure-path, in the CSS and the queries of both panes and in the three tests that read them; the comment in records.py that explained the old name is gone. No behavior changes. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XDaEfG6VfRLmUbbHW3Uit5
docs/console_gui.md describes pyevio gui the way docs/web_viewer.md describes pyevio web: install, start, the shared analysis cache, the keys of every scope, one section per tab, the PNG export and the limitations of character plots; a Known problems section notes that the analysis of an uncached file fails under Textual with more than one worker and gives --workers 1 as the way around it (F63, fixed in the next commit). The screenshots are SVG exports of the app on sro_000791.evio.00000.v6, four at 140x45 and the two Plots shots at 260x50, the width at which the occupancy heatmap prints its counts; tests/gui/make_screenshots.py regenerates them (it analyzes the file outside the app, then drives the pilot). The README lists both viewers in its overview and the CLI block; the changelog gains the pyevio gui entry, the removal of pyevio ui and the move of textual out of the core dependencies. I42. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XDaEfG6VfRLmUbbHW3Uit5
In the real `pyevio gui`, the analysis of a file that was not in the cache failed with "ValueError: bad value(s) in fds_to_keep" whenever it ran with more than one worker, which is the default. Textual replaces sys.stderr with an object whose fileno() is -1 while the app runs; the multiprocessing resource tracker passes sys.stderr.fileno() to the process it spawns on first use, and fork_exec rejects the -1. The web server never captures stderr, so neither it nor the pilot tests (one worker) showed it. stream_stats.start_worker_services() starts the resource tracker and the forkserver once; `pyevio gui` calls it before the app starts, and compute_stream_stats calls it before creating its pool, pointing sys.stderr at the process's original stderr for the start when the current one has no usable descriptor. A regression test runs a fresh interpreter with a captured stderr and two workers; the "Known problems" section the documentation had to carry is gone. F63. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XDaEfG6VfRLmUbbHW3Uit5
… record count The open screen's hint promised Backspace and Ctrl+A unconditionally, but the path input has the focus at start and binds both keys itself; they reach the screen only with the tree focused. The hint says so. While the analysis still scans the record headers, the status strip said "analyzing 0 of 0 records"; it says "scanning the records" until the total is known. A test predicate whose first clause was always true now waits on the session alone. F66, F67. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XDaEfG6VfRLmUbbHW3Uit5
rust/ becomes a workspace of evio-stream and evio-tui; the lock file moves to the root and the release profile (debug = 1) moves with it, the only place cargo reads profiles from. The library's code and dependencies are unchanged. evio-tui FILE scans the file on a background thread with RecordReader and lists the records as they arrive, with the full header of the selected one and the checks in the status line; Enter opens a record by seeking a fresh reader to the offset noted in the scan (a record header starts every record in both the v6 and the SRO layouts, so no layout hint is needed) and lists its events; Enter again shows the bank tree and the words of an event, and f the fADC250 frame summary. State, formatting and the tree are separate from the drawing and are what the tests drive: unit tests, TestBackend renders of the golden files, and a harness on the corpus files (record 7000 opens in under a millisecond on the converted and the raw file). The rich viewers stay in Python (D27). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XDaEfG6VfRLmUbbHW3Uit5
Describe the workspace, the viewer's screens and keys, how random access works without a library change, the measured timings from the logged run, and the workspace-wide test commands. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XDaEfG6VfRLmUbbHW3Uit5
One paragraph in the README overview and one changelog line name rust/evio-stream (the streaming reader, its tools and the rulers it passes) and rust/evio-tui, and say they are independent of the Python wheel; rust/README.md holds the details. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XDaEfG6VfRLmUbbHW3Uit5
The occupancy heatmap of the console viewer is now laid out by the pane as a Rich text instead of a plotext matrix_plot. F64: at the screenshot size plotext's canvas had 15 lines for 16 ports, merged two ports onto one line and overstruck their centred text() labels (93.6k under 55k read 955kk); every count is now written into its own cell. F65: cells are a whole number of lines and columns with the leftovers as margins, so all rows and columns are the same size; a plot lower than its row count pairs rows as half blocks, lower than half says what it needs. F68: the charge spectrum draws log10(1 + count) with ticks at the real counts (1, 10, 100, 1k, ...) and the label "hits (log)", since plotext's own log scale raises on empty bins and reads ylim in log units; the PNG export is unchanged. Tests for the three cases, docs updated, the two plots screenshots regenerated. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XDaEfG6VfRLmUbbHW3Uit5
The header still said F1—F62 while the ledger runs to F68, and the new public function behind the F63 fix had no entry. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XDaEfG6VfRLmUbbHW3Uit5
…elog F69: comments, docstrings, two test names, pyproject.toml and CHANGELOG.md cited finding, question and decision numbers (F15, Q2, D4, I17), phase names and paths under the untracked space/ workspace. A reader of the repository cannot resolve them. Every site now states the fact itself; the changelog entries stand without numbers. Gate: uv run pytest -q, 237 passed, 5 skipped. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XDaEfG6VfRLmUbbHW3Uit5
The test prints "37 rows read, 7 rows rejected"; the README claimed 46 and 39. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XDaEfG6VfRLmUbbHW3Uit5
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XDaEfG6VfRLmUbbHW3Uit5
`pyevio web` takes a file as well as a folder: the file's folder is served and the printed URL carries ?open=<name>, which the page opens on load. The empty page says so. A favicon.svg is linked from the page and served at /favicon.ico too, so a browser's first request no longer logs a 404. Tests: the path split (folder vs file), /favicon.ico and /favicon.svg answer 200 with SVG, and a Playwright page with ?open= shows the file without a click. Gate: 240 passed, 5 skipped. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XDaEfG6VfRLmUbbHW3Uit5
Closed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add textual and web guis