Add pyevio web viewer

README.md

@@ -128,6 +128,20 @@ pyevio hex sample.evio 0x100 --bytes --endian='>'
 pyevio ui sample.evio
 ```
 
+### GUI Viewer
+
+```bash
+# Serve the viewer at http://localhost:8765, file browser starts in /path/to/runs
+pyevio gui /path/to/runs
+
+# File-based EVIO is experimental: skip the streaming decode, show structure tree only
+pyevio gui /path/to/runs --mode=file
+```
+
+Browse a file in the browser: file checks, channel occupancy, rate, charge and timing plots for CODA streaming fADC250 data, and a structure tree down to single hit words. Decoding runs in the browser. Only CODA streaming fADC250 EVIO is tested; file-based EVIO may not decode well yet. See [docs/web_viewer.md](docs/web_viewer.md).
+
+![GUI viewer overview](docs/images/web_viewer_overview.png)
+
 ## Understanding EVIO File Structure
 
 EVIO files are organized hierarchically:

docs/web_viewer.md

@@ -0,0 +1,73 @@
+# GUI viewer
+
+`pyevio gui` opens a page for browsing an EVIO v6 file: file checks, channel occupancy, rate, charge and timing plots, and a clickable structure tree from the file header down to single fADC hit words. Decoding runs in the browser, and the file is never uploaded.
+
+Plots are written for CODA streaming fADC250 files (0xFF60 built time frames); this is the only EVIO flavor tested so far. File-based (triggered) EVIO uses a different decoding method that hasn't been validated here and may hit hiccups on large or unexpected layouts — pass `--mode=file` to skip the streaming-specific decode and show the structure tree only.
+
+## Start it
+
+```bash
+pyevio gui                       # file browser starts in the current folder
+pyevio gui /path/to/runs         # or in another folder
+pyevio gui --port 9000 --no-browser
+pyevio gui /path/to/runs --mode=file   # file-based EVIO: structure tree only (experimental)
+```
+
+Pick a file with **Browse folders**, **Open .evio file**, or by dragging it onto the page. A 93 MB file loads in under a second. The whole file is held in browser memory, so multi-GB files may not fit.
+
+The page also works without the server: open `pyevio/gui/index.html` directly. Only the folder browser needs `pyevio gui`.
+
+**Files on a remote machine** (e.g. a farm node): run `pyevio gui /path/to/runs --no-browser` there, connect with `ssh -L 8765:localhost:8765 user@host`, and open http://localhost:8765 locally. The server listens on localhost only.
+
+## What it shows
+
+### Overview
+
+Totals, file checks (last-record bits, missing event index, compressed records, frame gaps, out-of-order frames) and notable channels.
+
+![Overview tab](images/web_viewer_overview.png)
+
+### Plots
+
+Three levels, from the whole file down to one channel.
+
+**File:** hits per frame, and total hit rate over time. Gaps show missing frames.
+
+![File-level plots](images/web_viewer_plots_file.png)
+
+**Per ROC:** one block per crate with its board × channel occupancy map, hit rate over time, charge spectrum and hit time within the frame. All maps share one color scale, and charts of the same kind share one y-axis, so crates can be compared directly.
+
+![Per-ROC plots](images/web_viewer_plots_roc.png)
+
+**Per channel:** pick ROC → port → channel, step with Prev / Next, or click a cell in a ROC map.
+
+![Per-channel plots](images/web_viewer_plots_channel.png)
+
+### Structure
+
+A collapsible tree, 100 records per page, with jump-to-record. Each node shows its decoded fields and raw words.
+
+![Structure tab](images/web_viewer_structure.png)
+
+## Data format assumed
+
+```
+Record (header 14 w) → Event 0xFF60 (built time frame, 65,536 ns)
+  ├─ 0xFF31 stream info: segment 0x32 = frame, timestamp; segment 0x42 = ROC list
+  └─ ROC bank (tag = ROC id)
+       ├─ 0xFF30 ROC stream info: segment 0x31 = frame, timestamp; segment 0x41 = payload ports
+       └─ payload bank (tag = port): length 1 → empty; length 1+N → N hit words
+hit word: t = bits[30:17] × 4 ns, ch = bits[16:13], q = bits[12:0]
+```
+
+The hit-word layout was inferred from the data and checked for consistency, not taken from a spec. The Overview tab warns if a file breaks those checks. Compressed (LZ4/gzip) records are skipped in the browser and reported.
+
+## Convert to Parquet
+
+[examples/fadc250_stream_to_parquet.py](../examples/fadc250_stream_to_parquet.py) decodes the same hits with pyevio and numpy into two Parquet tables, for analysis outside the browser (requires `pip install pyarrow`):
+
+```bash
+python examples/fadc250_stream_to_parquet.py run.evio   # → out/run.hits.parquet, out/run.frames.parquet
+```
+
+Hits (`frame, roc, port, ch, t_ns, q`) are sorted by `(frame, t_ns)`. Rows `hit_offset … hit_offset + n_hits` of the hits table belong to that frame.

examples/fadc250_stream_to_parquet.py

@@ -0,0 +1,147 @@
+"""Decode CODA streaming fADC250 hits from an EVIO v6 file into Parquet.
+
+Container access goes through pyevio. The hit words are decoded with numpy.
+Requires pyarrow (pip install pyarrow). Browse the same files with `pyevio gui`.
+
+Hit word layout (big-endian u32):
+  bits 30-17 time in 4 ns ticks from frame start
+  bits 16-13 channel
+  bits 12-0  charge q
+
+Usage:
+  python fadc250_stream_to_parquet.py run.evio            # writes out/run.hits.parquet + out/run.frames.parquet
+  python fadc250_stream_to_parquet.py runs/*.evio --out out --force
+
+Read the output:
+  hits   = pq.read_table("out/run.hits.parquet")    # frame, roc, port, ch, t_ns, q (sorted by frame, t_ns)
+  frames = pq.read_table("out/run.frames.parquet")  # frame, timestamp_ns, n_hits, hit_offset
+"""
+import argparse
+import sys
+import time
+from pathlib import Path
+
+import numpy as np
+import pyarrow as pa
+import pyarrow.parquet as pq
+from pyevio import EvioFile
+
+DECODER_VERSION = "1"
+HIT_FORMAT = "fadc250-stream-v1: t=bits[30:17]*4ns ch=bits[16:13] q=bits[12:0]"
+
+
+# ---------------------------------------------------------------- decoding
+def decode(path):
+    """Return (hits dict of numpy arrays, frames dict, stats dict)."""
+    f = EvioFile(str(path))
+    mm = f.mm
+    dt = np.dtype(f.header.endian + "u4")
+    words, b_frame, b_roc, b_port, b_n = [], [], [], [], []     # raw hit words + one entry per non-empty payload bank
+    frames, ts_list, nhits, payload_words = [], [], [], 0
+    skipped = 0
+    for ri in range(f.record_count):
+        for e in f.get_record(ri).get_events():
+            w = np.frombuffer(mm, dtype=dt, count=e.length // 4, offset=e.offset)
+            if (w[1] >> 16) != 0xFF60 or (w[3] >> 16) != 0xFF31:
+                skipped += 1
+                continue
+            frame = int(w[5])
+            ts = int(w[6]) | (int(w[7]) << 32)
+            i = 2 + int(w[2]) + 1                      # skip stream-info bank
+            nh = 0
+            while i < len(w):
+                end = i + int(w[i]) + 1
+                roc = int(w[i + 1] >> 16)
+                j = i + 2
+                if (w[j + 1] >> 16) == 0xFF30:         # skip ROC stream-info bank
+                    j += int(w[j]) + 1
+                while j < end:
+                    plen = int(w[j])
+                    payload_words += plen - 1
+                    if plen > 1:
+                        words.append(w[j + 2:j + plen + 1])
+                        b_frame.append(frame); b_roc.append(roc); b_port.append(int(w[j + 1] >> 16)); b_n.append(plen - 1)
+                        nh += plen - 1
+                    j += plen + 1
+                i = end
+            frames.append(frame)
+            ts_list.append(ts)
+            nhits.append(nh)
+    h = np.concatenate(words).astype(np.uint32) if words else np.empty(0, np.uint32)
+    del words
+    b_n = np.asarray(b_n, dtype=np.int64)
+    hits = {
+        "frame": np.repeat(np.asarray(b_frame, np.uint32), b_n),
+        "roc": np.repeat(np.asarray(b_roc, np.uint16), b_n),
+        "port": np.repeat(np.asarray(b_port, np.uint8), b_n),
+        "ch": ((h >> 13) & 0xF).astype(np.uint8),
+        "t_ns": (((h >> 17) & 0x3FFF) * 4).astype(np.uint16),
+        "q": (h & 0x1FFF).astype(np.uint16),
+    }
+    # sort by frame, then time in frame (the event builder delivers some frames out of order)
+    order = np.lexsort((hits["t_ns"], hits["frame"]))
+    hits = {k: v[order] for k, v in hits.items()}
+    frames, ts_list, nhits = map(np.asarray, (frames, ts_list, nhits))
+    out_of_order = int((np.diff(frames) < 0).sum())
+    order = np.argsort(frames, kind="stable")
+    frames, ts_list, nhits = frames[order], ts_list[order], nhits[order]
+    offsets = np.concatenate([[0], np.cumsum(nhits)[:-1]]) if len(nhits) else np.empty(0)
+    frame_tab = {"frame": frames.astype(np.uint32), "timestamp_ns": ts_list.astype(np.uint64),
+                 "n_hits": nhits.astype(np.uint32), "hit_offset": offsets.astype(np.uint64)}
+    stats = {"records": f.record_count, "frames": len(frames), "hits": len(h), "skipped_events": skipped,
+             "out_of_order": out_of_order, "payload_words": payload_words}
+    return hits, frame_tab, stats
+
+
+def validate(hits, frames, stats):
+    """Cheap self-consistency checks; raises AssertionError on failure."""
+    assert stats["hits"] == stats["payload_words"], "hit count != sum of (payload bank length - 1)"
+    assert int(frames["n_hits"].sum()) == stats["hits"], "per-frame hit counts do not add up"
+    if stats["hits"]:
+        assert hits["t_ns"].max() < 65536, "hit time outside the 65.5 us frame"
+    f = frames["frame"]
+    if len(f):
+        return {"missing_frames": int(f[-1] - f[0] + 1 - len(np.unique(f))), "duplicate_frames": int(len(f) - len(np.unique(f)))}
+    return {}
+
+
+def convert(path, out_dir, force=False):
+    path, out_dir = Path(path), Path(out_dir)
+    hits_p = out_dir / f"{path.stem}.hits.parquet"
+    frames_p = out_dir / f"{path.stem}.frames.parquet"
+    if not force and hits_p.exists() and hits_p.stat().st_mtime > path.stat().st_mtime:
+        print(f"skip  {path.name} (already converted)")
+        return
+    t0 = time.time()
+    hits, frames, stats = decode(path)
+    checks = validate(hits, frames, stats)
+    meta = {"source_file": path.name, "source_bytes": str(path.stat().st_size), "decoder_version": DECODER_VERSION,
+            "hit_format": HIT_FORMAT, **{k: str(v) for k, v in {**stats, **checks}.items()}}
+    out_dir.mkdir(parents=True, exist_ok=True)
+    for table, p in ((hits, hits_p), (frames, frames_p)):
+        t = pa.table(table)
+        t = t.replace_schema_metadata({k.encode(): v.encode() for k, v in meta.items()})
+        pq.write_table(t, p, compression="zstd")
+    print(f"done  {path.name}: {stats['frames']:,} frames, {stats['hits']:,} hits, "
+          f"{stats['out_of_order']:,} out-of-order, {checks} -> {hits_p.name} "
+          f"({hits_p.stat().st_size / 1e6:.1f} MB) in {time.time() - t0:.1f} s")
+
+
+def main(argv=None):
+    ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
+    ap.add_argument("files", nargs="+", help=".evio files to convert")
+    ap.add_argument("--out", default="out", help="output folder (default: ./out)")
+    ap.add_argument("--force", action="store_true", help="re-convert even if the Parquet is up to date")
+    args = ap.parse_args(argv)
+    failed = 0
+    for p in args.files:
+        try:
+            convert(p, args.out, args.force)
+        except Exception as e:  # keep going with the other files
+            failed += 1
+            print(f"FAIL  {p}: {e}", file=sys.stderr)
+    return 1 if failed else 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

pyevio/cli/__init__.py

@@ -12,6 +12,7 @@
 from pyevio.cli.hex import hex_command
 from pyevio.cli.ui import ui_command
 from pyevio.cli.ana import ana_command
+from pyevio.cli.gui import gui_command
 
 
 @click.group(invoke_without_command=True)
@@ -41,6 +42,7 @@ def cli(ctx, verbose):
 cli.add_command(hex_command)
 cli.add_command(ui_command)
 cli.add_command(ana_command)
+cli.add_command(gui_command)
 
 
 # Entry point for the CLI

pyevio/cli/gui.py

@@ -0,0 +1,21 @@
+import click
+
+
+@click.command(name="gui")
+@click.argument("start_dir", type=click.Path(exists=True, file_okay=False), default=".")
+@click.option('--port', '-p', default=8765, show_default=True, help="Port to serve on (localhost only)")
+@click.option('--no-browser', is_flag=True, help="Do not open a browser tab")
+@click.option('--mode', type=click.Choice(["stream", "file"]), default="stream", show_default=True,
+              help="EVIO flavor to assume: 'stream' for CODA streaming fADC250 data (tested), "
+                   "'file' for triggered/file-based EVIO (experimental, structure tree only)")
+def gui_command(start_dir, port, no_browser, mode):
+    """Browse EVIO files in a browser page: checks, plots and structure tree.
+
+    START_DIR is the folder the file browser opens in (default: current directory).
+
+    Only CODA streaming fADC250 EVIO is fully tested. File-based EVIO uses a
+    different decoding path and may not work well; pass --mode=file to skip
+    the streaming-specific decoding and show the structure tree only.
+    """
+    from pyevio.gui import serve
+    serve(start_dir, port, open_browser=not no_browser, mode=mode)

pyevio/evio_file.py

@@ -6,6 +6,7 @@
 
 from pyevio.file_header import FileHeader
 from pyevio.record import Record
+from pyevio.record_header import RecordHeader
 from pyevio.event import Event
 
 
@@ -63,7 +64,7 @@ def __del__(self):
                 self.mm.close()
             if hasattr(self, 'file') and self.file and not self.file.closed:
                 self.file.close()
-        except (ValueError, AttributeError):
+        except (ValueError, AttributeError, BufferError):
             # Ignore errors during cleanup
             pass
 
@@ -96,8 +97,10 @@ def _scan_record_offsets(self) -> List[int]:
                 # Move to next record
                 offset += record.size
 
-                # Stop if this was the last record
-                if record.header.is_last_record:
+                # Stop if this was the last record. Some CODA writers set the
+                # last-record bit on every record, so keep going if another
+                # record header follows.
+                if record.header.is_last_record and not self._is_record_at(offset):
                     break
 
             except Exception as e:
@@ -107,6 +110,13 @@ def _scan_record_offsets(self) -> List[int]:
 
         return record_offsets
 
+    def _is_record_at(self, offset: int) -> bool:
+        """Check whether a record header (magic word in word 7) starts at offset."""
+        if offset + RecordHeader.HEADER_SIZE > self.file_size:
+            return False
+        magic = struct.unpack(self.header.endian + 'I', self.mm[offset + 28:offset + 32])[0]
+        return magic == RecordHeader.MAGIC_NUMBER
+
     @property
     def record_count(self) -> int:
         """Get the number of records in this file."""