Support bounded curation and full-source replay for long recordings

design/README.md

@@ -57,3 +57,19 @@ and section 11 for integration and manual acceptance.
 
 The implementation and user documentation are complete on the unreleased phy
 2.2 branch. Final large-dataset smoke testing and save/reopen validation remain.
+
+## Long-recording curation (proposed)
+
+The [long-recording curation Phy-side design contract](large-recording-curation.md)
+describes bounded local datasets for ~1000-hour sorting datasets, the three spike-index
+spaces, optional upstream waveform evidence, full-duration metrics, replay safety,
+and the implementation stages. The accompanying synthetic 1000-hour
+fixture and reference tests are in `phy/apps/tests/large_recording_fixtures.py`
+and `phy/apps/tests/test_large_recording_contract.py`.
+
+This is a proposal and a reference test contract, **not yet** a `prepare-large`
+implementation or an upstream export-format agreement.
+
+The [integration audit and plan](large-recording-integration-plan.md) refines
+the contract with correctness findings, module boundaries, ordered commits in one PR and
+acceptance gates. Follow its gates before enabling editable bounded sessions.

design/large-recording-validation.md

@@ -0,0 +1,77 @@
+# Long-recording implementation validation
+
+This record summarizes the current implementation checks and their limits. The
+results validate the measured code paths and fixtures described here. They do
+not establish capacity for every source layout or scientific suitability for
+chronic curation.
+
+## Automated checks
+
+On 2026-10-13, `make test-full` completed with **756 passed and 1
+skipped** (496 non-application tests and 260 application tests). Final channel
+geometry and nonidentity waveform-mapping additions passed a further focused
+source/preparation run of **33 tests**. Lint, format checking, package build,
+and the strict documentation build also passed. The dedicated bounded GUI lifecycle file passed all three
+regressions: reentrant observer rollback with the GUI attached, merge-proposal
+review through undo/redo/save/reopen, and pending full metrics for threshold
+children in the default bounded views.
+
+The reference long-recording fixture is deliberately sparse: 32 spikes spread
+over a 1000-hour sample clock. It checks long `int64` coordinates, tied times,
+source-row mapping, waveform remapping, and loader compatibility. It is not a
+billion-spike capacity test, a realistic chronic recording, or evidence of
+biological identity across a recording.
+
+## Synthetic preparation and replay runs
+
+The reproducible benchmark is `uv run python
+tools/benchmark_long_recording.py`. It writes synthetic source arrays in
+batches, with no raw voltage data, and runs preparation and replay in separate
+processes. Both runs below used Python 3.13.3, NumPy 2.4.3, Linux
+`x86_64`/glibc 2.39, 30 kHz sampling, 1000 units, a 1000-hour sample axis,
+100,000-row batches, 30-second windows, 3600-second strata, a 100,000-spike
+budget, a 512 MiB output budget, seed 0, and 3600-second summary bins. The
+oracle independently streamed every source and output assignment row and
+matched the expected merge in both cases.
+
+| Synthetic source spikes | Source files | Bounded package | Bounded spikes | Curated output | Prepare time / peak RSS | Replay time / peak RSS | Oracle rows |
+| ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
+| 250,000 | 4,032,892 B | 8,625,460 B | 2,389 | 5,038,086 B | 1.902 s / 215,773,184 B | 0.114 s / 214,847,488 B | 250,000 |
+| 2,000,000 | 32,032,892 B | 9,157,120 B | 16,676 | 40,038,086 B | 1.227 s / 214,433,792 B | 0.292 s / 215,285,760 B | 2,000,000 |
+
+Each measurement is a single run, not a throughput curve. Preparation and
+replay durations exclude child-process startup; peak RSS is the process high
+water mark. The source is a regular synthetic distribution with tiny templates
+and no stored waveforms, raw data, irregular acquisition gaps, or real feature
+loads. These figures therefore do not predict every preparation cost or GUI
+interaction on a chronic dataset. The full JSON reports are retained at
+`/tmp/phy-long-recording-benchmark-250k.json` and
+`/tmp/phy-long-recording-benchmark-2m.json` on the validation workstation.
+
+## Ordinary-dataset smoke comparison
+
+A separate three-run comparison used fresh subprocesses, configs, and caches in
+the same Python environment. It compared baseline `a30d5c65` with current
+`f212f069e5b5b19469cfa73781dac08de093d9a1` on a 12,000-spike, two-cluster
+dataset with 1,024 stored waveform rows of shape `[40, 4]` and dense template
+features. Counts and output dimensions matched exactly.
+
+| Measure (median of 3) | Baseline | Current |
+| --- | ---: | ---: |
+| Controller startup | 0.03879 s | 0.02822 s |
+| Waveform provider | 0.00353 s | 0.00293 s |
+| Feature provider | 0.00372 s | 0.00302 s |
+
+This small case showed no regression; the apparent gains are within the range
+where run-to-run noise matters. The comparison did not measure Qt rendering,
+raw-voltage I/O, or large-dataset selection churn. Its report is retained at
+`/tmp/phy-standard-recording-comparison.json` on the validation workstation.
+
+## Remaining acceptance work
+
+Real chronic-recording review by domain curators remains outstanding. It must
+cover drift, rare and bursting units, artifacts, gaps, waveform coverage,
+intentionally incorrect merges, withheld contiguous windows, and full-data
+diagnostics where available. Software integrity checks and synthetic throughput
+do not establish that the evidence is scientifically sufficient or that a
+cluster represents one biological unit throughout 1000 hours.

docs/api.md

@@ -10117,6 +10117,13 @@ Component that brings manual clustering facilities to a GUI:
     Whether automatic navigation and similar-cluster selection skip noise and MUA
     clusters. The default is True.
 
+* `clustering_factory : callable, optional`
+    Construct the clustering model. Defaults to Clustering; bounded curation
+    supplies a guarded implementation with the same constructor interface.
+
+* `cluster_meta_factory : callable, optional`
+    Construct metadata from groups and labels. Defaults to create_cluster_meta.
+
 **Events**
 
 When this component is attached to a GUI, the following events are emitted:
@@ -13197,6 +13204,15 @@ Create a trace view.
 
 ---
 
+#### TemplateController.create_unit_stability_view
+
+
+**`TemplateController.create_unit_stability_view(self)`**
+
+Optionally inspect selected-unit activity over the loaded recording.
+
+---
+
 #### TemplateController.create_waveform_view
 
 

docs/changelog.md

@@ -11,10 +11,27 @@ been included in a stable release. The current entries cover all user-visible
 changes committed since 23 July 2026; test-only commits are represented by the
 behavior they verify rather than listed separately.
 
+### Long recordings
+
+- Added `prepare-large` and `apply-curation` for bounded shared-window evidence,
+  full-duration exposure-aware summaries, semantic merges/labels/source-amplitude
+  splits, and replay into a new complete sorting.
+- Added Unit Stability, precise window-relative amplitude/feature inspection,
+  sampled interval/correlogram views, and explicit full-versus-evidence counts.
+  Standard datasets retain their existing curation workflow and default layout.
+- Bounded saves use validated atomic generations and reject conflicting saves
+  from stale sessions. Unsupported selected-row splits are blocked.
+- Sparse cluster IDs no longer require dense allocations during clustering.
+- Bounded amplitude plots label seconds from window start and display the
+  window's recording-time range above the plot.
+
 ### Documentation
 
 - Documented uv-first installation of exact previous phy releases in separate
   environments, including how to intentionally replace a tool installation.
+- Added a long-recording workflow guide covering bounded evidence preparation,
+  supported curation, full-sorting replay, and limits on interpreting sampled
+  measurements.
 
 ### Added
 

docs/cli.md

@@ -25,6 +25,35 @@ GUI commands accept:
 --clear-cache     remove and rebuild the dataset .phy cache
 ```
 
+## Long recordings
+
+Prepare bounded evidence, curate it with `template-gui`, then replay supported
+actions into a new full sorting. See the [long-recording guide](long-recordings.md)
+for input requirements, interpretation limits, and the complete workflow.
+
+```bash
+phy prepare-large /data/recording/params.py /data/recording-bounded \
+  --duration 3600000 \
+  --valid-intervals /data/recording/valid-intervals.npy \
+  --max-spikes 2000000 \
+  --max-bytes 536870912 \
+  --window-seconds 30 \
+  --stratum-seconds 3600 \
+  --seed 0 \
+  --bin-seconds 3600 \
+  --batch-size 100000
+
+phy template-gui /data/recording-bounded/params.py
+
+phy apply-curation /data/recording/params.py /data/recording-bounded \
+  /data/recording-curated --batch-size 100000
+```
+
+`--duration` is acquisition duration in seconds. `--valid-intervals` reads an
+integer `.npy` array of `[start, end)` sample pairs. `--max-bytes` caps the
+prepared package size, not process memory. The output directories must be new;
+the original sorting remains the replay parent.
+
 ## Raw trace viewer
 
 ```bash

docs/long-recordings.md

@@ -0,0 +1,183 @@
+# Curating long recordings
+
+Phy's long-recording workflow prepares a bounded evidence dataset for interactive
+review, then replays supported curation actions against the complete local
+sorting. Preparation reads source arrays in batches; it does not load the full
+sorting into the GUI. The source dataset is retained unchanged, and replay
+writes a separate output dataset.
+
+This workflow is intended for a Phy sorting with one fixed-rate, monotonic sample
+clock. Multi-segment recordings are not supported. Supply acquisition duration
+and valid acquisition intervals when they are not already recorded in the
+dataset metadata. Duration describes exposure; Phy cannot infer acquisition
+gaps from periods with no spikes. Intervals use half-open sample coordinates
+`[start, end)`.
+
+The screenshots below show the actual bounded GUI views using a deterministic
+synthetic eight-hour sorting. The overview plots full-duration rates and
+available source means; gray marks the synthetic acquisition gap and cyan marks
+the retained evidence windows. The evidence view places windowed amplitudes at
+top left, sampled ISI at top right, sampled correlograms at middle right, and
+features below. They illustrate the interface and its sampling context; they
+are not biological data or scientific examples.
+
+![Full-duration Unit Stability with a synthetic acquisition gap](images/long-recording-overview.png)
+
+![Windowed source amplitudes, features, sampled ISI and correlogram](images/long-recording-evidence.png)
+
+Recreate these images from source with Qt's X11 platform and software OpenGL
+(the script writes a temporary synthetic source and bounded package):
+
+```bash
+QT_QPA_PLATFORM=xcb LIBGL_ALWAYS_SOFTWARE=1 \
+  xvfb-run -a -s '-screen 0 2400x1600x24 +extension GLX' \
+  uv run python tools/capture_long_recording_views.py
+```
+
+## Prepare bounded evidence
+
+Run `prepare-large` with the source `params.py` and a new output directory:
+
+```bash
+phy prepare-large /data/recording/params.py /data/recording-bounded \
+  --duration 3600000 \
+  --valid-intervals /data/recording/valid-intervals.npy \
+  --max-spikes 2000000 \
+  --max-bytes 536870912 \
+  --window-seconds 30 \
+  --stratum-seconds 3600 \
+  --seed 0 \
+  --bin-seconds 3600 \
+  --batch-size 100000
+```
+
+`--duration` is in seconds and must agree with declared acquisition metadata.
+The `.npy` valid-interval file contains an integer array of shape `[W, 2]` with
+ordered, nonoverlapping sample pairs.
+Without it, the full declared duration is treated as valid exposure. Do not use
+that default when the recording contains acquisition gaps.
+
+`--max-bytes` limits the bounded output package size, including its summaries and
+window plan. It is not a process-memory or peak-RSS limit. The source unit IDs
+must fit nonnegative `int32`; sample coordinates and full-source spike row IDs
+use `int64`.
+
+Preparation chooses shared time windows under the spike and byte budgets. The
+window length, strata, and seed make the plan reproducible; inspect the saved
+window plan in the output package when documenting or repeating an analysis.
+Windows that exceed the budget are omitted, so a high-density interval can
+reduce retained evidence. Preparation does not shorten an over-budget window or
+silently thin spikes inside it. A survey of shared windows is supplemented by
+deterministic rescue evidence for units missed by the survey. This improves the
+chance of seeing rare units, but makes the evidence selection unequal: rescue
+windows must not be treated as a probability sample.
+
+Preparation records exact full-recording spike counts and rates using valid
+acquisition exposure. If source scalar amplitudes or depths are available, it
+also records per-bin finite-value means and availability counts. It does not
+invent missing measurements or convert source values to physical units. Stored
+waveforms are included only where available in the source; raw traces are not
+required when stored waveform data are sufficient for the views being used.
+Producer-supplied per-unit metrics and annotations remain inherited metadata;
+Phy does not recalculate those values for merged or split units. Use the
+full-count, rate, amplitude-mean, and depth-mean summaries produced by this
+workflow for the measurements they explicitly cover.
+
+## Review and curate
+
+Open the bounded output with the normal GUI command:
+
+```bash
+phy template-gui /data/recording-bounded/params.py
+```
+
+The bounded manifest explicitly selects this mode. Bounded mode has its own
+configuration namespace and default view layout; it does not replace the normal
+Phy layout or restore ordinary saved dock geometry. The GUI keeps full-recording
+counts and rates distinct from counts in selected evidence rows. Unit Stability
+shows time-binned counts/rates, available source scalar-amplitude means,
+evidence windows, and acquisition gaps. Missing measurements and units without
+evidence remain visible as unavailable.
+
+Evidence Amplitude shows raw source scalar amplitudes from the currently
+selected evidence window. The Feature view also uses window-relative seconds.
+The amplitude plot labels this axis as **Seconds from window start** and shows
+the window's recording-time range above the plot.
+Both views label the current window and reset their time origin when it changes.
+To inspect another retained window, choose **View > Inspect evidence window**
+and enter its zero-based index. Unit Stability keeps the full-recording time
+context visible while you inspect individual windows.
+
+The Sampled ISI view excludes intervals crossing retained-window boundaries.
+The Sampled Correlogram view shows raw pair counts from retained windows; it
+does not apply a full-recording rate baseline. Both views are explicitly labeled
+as sampled evidence. Its central bin counts unordered within-unit pairs and
+combines both temporal orientations for cross-unit pairs whose absolute delay
+is less than one bin, so tied sample times do not depend on source row order.
+For exact pair counts, each unit may have at most 46,340 spikes in an evidence
+window; the view reports an error for a larger per-unit window. Prepare shorter
+windows if this limit is reached.
+
+Bounded curation supports whole-unit merges, cluster labels, and a source
+amplitude threshold action. Use the GUI action **Split by source amplitude**
+with one unit selected and a threshold in the source amplitude units. The action
+uses the source `amplitudes.npy` scalar for every spike during replay; nonfinite
+values stay with the remainder. Child units show full-data metrics as pending
+until the replayed sorting is re-prepared, because their complete counts and
+trends have not yet been calculated. Merge and label actions are journaled with
+the bounded assignments and metadata in an atomic checkpoint generation.
+Reopening loads the last complete generation. A reopened session starts a new
+undo baseline. Each save checks that the checkpoint has not changed since this
+session opened. If another session has saved first, the stale save fails; reopen
+the bounded dataset before continuing to avoid overwriting that generation.
+
+Lasso, feature, and waveform splits remain unavailable in bounded mode because
+the selected rows do not define how unseen source spikes should be assigned.
+Standard, unbounded datasets retain their existing curation behavior.
+
+## Unit Stability on ordinary recordings
+
+Unit Stability can also be opened as an optional view for an ordinary dataset.
+It evaluates only the selected unit and bins its loaded spikes over the recording
+duration. Since ordinary datasets do not supply valid acquisition intervals to
+this view, its rate assumes continuous acquisition; the plot labels that
+assumption. The ordinary default view layout remains unchanged. Use the bounded
+manifest workflow above when known acquisition gaps must be represented
+explicitly.
+
+## Apply to the complete sorting
+
+After saving the bounded session, replay its journal against the exact source
+assignment revision used during preparation:
+
+```bash
+phy apply-curation /data/recording/params.py /data/recording-bounded \
+  /data/recording-curated --batch-size 100000
+```
+
+Replay checks the source identity and parent assignment revision, streams the
+complete assignment array in batches, and writes a new full sorting. It rejects
+a journal prepared from a different source revision. Keep the original source
+and bounded package; they establish the lineage of the curated output. To review
+the curated full sorting with bounded evidence again, run `prepare-large` on
+the curated output to produce a fresh package and revision-specific summaries.
+This replay and re-preparation step also resolves full-data counts and trends
+for threshold-split children.
+
+## Interpreting evidence
+
+Full counts and exposure-based rates describe the complete sorting. Window plots,
+waveforms, feature scatterplots, and other bounded measurements describe only
+the retained evidence. Survey and rescue selection can overrepresent some
+periods or units, and waveform availability can vary across units and time.
+These views do not establish full-recording quality, contamination, refractory
+violations, or continuous biological identity. Use them to guide review, not as
+scientific claims about unseen spikes. Long-duration coordinates preserve the
+source sample clock; silence alone is never evidence that the acquisition was
+inactive.
+
+For reproducible synthetic capacity measurements, run
+`uv run python tools/benchmark_long_recording.py`. The benchmark reports
+preparation and replay time and peak memory in separate processes and checks
+replay against a streamed oracle. It is capped at two million synthetic spikes;
+its results are not a substitute for measurements on real chronic datasets.

mkdocs.yml

@@ -17,6 +17,7 @@ nav:
   - How-to and customization:
       - Configuration and state: configuration.md
       - Performance and spike sampling: performance.md
+      - Curating long recordings: long-recordings.md
       - Change keyboard shortcuts: keyboard_customization.md
       - Plugin introduction: customization.md
       - Plugin examples: plugins.md