feat: a record of another spec stays readable as stored data

docs/developer/README.md

@@ -1,6 +1,6 @@
 # ESS data-reduction framework
 
-**Status: `ess.apps.Workspace`, which `local()` and `open_folder()` make in the caller's process, implements this design, including freeze, `select=`, and persist to a fake store or to scipp HDF5 files in a folder (`ess.apps.folder`). Not implemented: a hosted workspace reached with `connect()`, `client.publish`, software versions in provenance, spec digests ([ADR 0009](adr/0009-a-record-of-another-spec-stays-readable.md)), and catalogue identifiers as file identities ([ADR 0010](adr/0010-a-file-is-identified-by-its-catalogue-identifier.md)). The first release combines no runs and tunes no parameters interactively, so it needs neither stages nor accumulators.**
+**Status: `ess.apps.Workspace`, which `local()` and `open_folder()` make in the caller's process, implements this design, including freeze, `select=`, and persist to a fake store or to scipp HDF5 files in a folder (`ess.apps.folder`). A record of another spec than the one bound reads as stored scipp data ([ADR 0009](adr/0009-a-record-of-another-spec-stays-readable.md)). Not implemented: a hosted workspace reached with `connect()`, `client.publish`, software versions in provenance, and catalogue identifiers as file identities ([ADR 0010](adr/0010-a-file-is-identified-by-its-catalogue-identifier.md)). The first release combines no runs and tunes no parameters interactively, so it needs neither stages nor accumulators.**
 
 This page gives an overview of the framework: its components, what each promises the others, and the ideas that client code is built from.
 The details live one level down:
@@ -121,8 +121,10 @@ for run in client.files.watch(Selector(scan='17')): ...   # existing ones first,
 ```
 
 **Store.** The *store* keeps persisted outputs outside the workspace's memory, also after the workspace has closed ([`ess.apps.protocol`](../../packages/essapps/src/ess/apps/protocol.py)).
-It reads a value back by the type that the spec declares for the output, so a workspace reads the persisted outputs of the specs it runs.
-An output of a record of another spec, such as an older version, is read as plain scipp data ([ADR 0009](adr/0009-a-record-of-another-spec-stays-readable.md)).
+It reads a value back by the type that the spec declares for the output, or as stored, in scipp's types, without one.
+A record names its spec by name, version, and digest, the digest of the spec's models, which the history logs.
+An output of a record of the bound spec, one the workspace runs with the same three, is read by its declared type.
+An output of a record of another spec, such as an older version or the same version with changed models, is read as stored, an array as it is and anything else as a `DataGroup`, and a new request may not reference it ([ADR 0009](adr/0009-a-record-of-another-spec-stays-readable.md)).
 
 **Controller.** The workspace does not decide what to run.
 A *controller* is client code that decides over time what to submit, push, and release, such as the cells of a notebook, the callbacks of an application's screen, or the trigger loop of automatic reduction.
@@ -244,6 +246,7 @@ A workspace that starts from its history log gives the same changes at the same
 Accumulators and pushes are not in the feed.
 Neither is `client.release`, nor any other way for a value to go away: the history does not hold lifetimes, and another client may release a value that this client shows.
 So a client that shows outputs asks `client.readable` for the records it shows, and a read can still raise `NotReadable`.
+It asks `client.bound(records)` which records are of the bound spec, since only their outputs have the classes of `client.spec(record.spec)` ([ADR 0009](adr/0009-a-record-of-another-spec-stays-readable.md)).
 
 ## Stages and accumulators
 

docs/developer/adr/0009-a-record-of-another-spec-stays-readable.md

@@ -1,6 +1,6 @@
 # ADR 0009: A record names its spec by name, version, and digest, and a record of another spec stays readable
 
-- Status: accepted, not implemented. On `main`, an output of a record whose spec has no binding raises `NotReadable`, and a stored group is read into the bound outputs class if no required field is missing.
+- Status: accepted, implemented (`ess.apps.workspace`, log format 3), but for what Open names.
 - Deciders: Simon
 - Date: 2026-11-13
 
@@ -26,15 +26,14 @@ What must reach another workspace, or outlive the history, is published.
 ## Decision
 
 Callers write `SpecId(name, version)` or pass the spec.
-A request made with a spec whose digest differs from that of the spec bound under the same name and version is refused at submission.
 
 The *digest* of a spec covers the name, type, and requiredness of every field of its params and outputs models, including the fields of groups inside them.
 Titles, descriptions, and defaults are not part of it.
 A record stores its params with defaults filled in, so a changed default does not change what an old record means.
 The digest cannot see a change of what a workflow computes.
 The version remains the author's statement of any change a consumer can observe: the params, the outputs, or what the workflow computes (`WorkflowSpec`).
 
-Every event that names a spec, a submission or the opening of an accumulator, carries the digest of each spec it names that the log does not hold yet under that name and version, or holds with another digest.
+Every submission, a freeze's included, carries the digest of each spec its records name that the log does not hold yet under that name and version, or holds with another digest.
 The digest of a record's spec is the last one logged for its name and version before the record.
 A submission, abridged:
 
@@ -44,15 +43,16 @@ A submission, abridged:
 
 A record is *of the bound spec* if the workspace has a binding with the same name, version, and digest.
 Otherwise it is *of another spec*: the version was bumped, the models changed, or no binding is given.
+`client.bound(records)` tells an application which records are of the bound spec.
 A record of another spec stays in the history, and is readable without the classes that made it:
 
 | Of a record of another spec | What a client gets |
 |---|---|
 | its request | name and version, and the params as the stored dict, with files and references |
-| an output | the stored scipp data: a `DataArray`, or a `DataGroup` by field name |
+| an output | the stored scipp data (`Store.read`): a `DataArray` as it is, a group as a `DataGroup` by field name, a number as `DataGroup({'value': 0.5})`, a list as a `DataGroup` with keys `'0'`, `'1'`, ... |
 | `describe` and `select=` | work on that data |
 | a new request that references its output | refused at submission with `SubmitError`, since the output's type cannot be checked |
-| a restart while it is pending | it fails with a reason that says its spec is not bound, and so does a pending record that reads an output of it |
+| a restart while it is pending | the restart rules of [ADR 0002](0002-a-value-lives-while-something-keeps-it.md) come first: it is cancelled if it persists nothing and no pending record that persists reads it, and it fails if it had started. If it would run, it fails with a reason that says its spec is not bound, and so does a pending record that would run and reads an output of it |
 
 A technique package's tests compare each spec's digest with one kept in the package, so a change of a model without a new version fails in the author's CI (`ess.spec.testing`).
 

packages/essapps/docs/batch.md

@@ -263,7 +263,7 @@ $B reduce iofq.yaml runs.csv --label water-series
 
 `{'@latest': label, output: name}` is read when the command starts, as the output of the newest record under the label, of `member` if given, whoever submitted it. It is for a template, not for a cell of a table. The record of each row holds the ID that it was read as. A reference by ID is `{'@ref': output, record: ..., output: ...}`.
 
-One command holds the folder at a time. If a command was killed, the next one first finishes the rows that had not started. A row that was running fails, since its workflow may have been what ended the process. Reduce again with `--remaining` to submit every row that has not completed, the failed ones included. A workspace reads a persisted output by the type that its spec declares, so only one that runs the spec reads it. Nothing removes the files of the store.
+One command holds the folder at a time. If a command was killed, the next one first finishes the rows that had not started, but fails those of a spec it does not bind with the same version and models. A row that was running fails, since its workflow may have been what ended the process. Reduce again with `--remaining` to submit every row that has not completed, the failed ones included. A command that binds another version of a spec, or the same version with changed models, reads the old results as plain scipp data, and shows the rows of another version as not reduced. Nothing removes the files of the store.
 
 ## The UI
 

packages/essapps/src/ess/apps/client.py

@@ -451,8 +451,9 @@ def submit(
         SubmitError
             If a request cannot run: an invalid value, an unknown spec or file, a
             reference at a field that the spec does not mark ``Referenced[...]``, a
-            reference that may not be read, or one whose data do not fit its
-            field. No request of the call is submitted. Each of its ``errors`` names
+            reference that may not be read, one to an output of a record of another
+            spec than the one bound, or one whose data do not fit its field. No
+            request of the call is submitted. Each of its ``errors`` names
             the field at fault by its position and the request by its key or index
             (see :class:`~ess.apps.errors.FieldError`).
         """
@@ -646,8 +647,8 @@ def describe(self, record: Record, name: str) -> dict[str, Any]:
     @_shaped
     def readable(self, records: list[Record]) -> list[tuple[str, ...]]:
         """
-        The outputs of each record, by name, that a plain submission may reference
-        now. A picker of records calls it at each refresh::
+        The outputs of each record, by name, that may be read now. A picker of
+        records calls it at each refresh::
 
             for record, names in zip(records, client.readable(records)):
                 ...  # offer record.ref(name) for each name
@@ -656,6 +657,23 @@ def readable(self, records: list[Record]) -> list[tuple[str, ...]]:
         """
         return self._workspace.readable([r.id for r in records])
 
+    @_shaped
+    def bound(self, records: list[Record]) -> list[bool]:
+        """
+        Whether each record is of the bound spec: the workspace has a binding
+        with the same name, version, and digest as the record's spec. The
+        outputs of any other record are read as stored data and cannot be
+        referenced
+        (``docs/developer/adr/0009-a-record-of-another-spec-stays-readable.md``).
+        A view asks for the spec of a bound record only::
+
+            for record, bound in zip(records, client.bound(records)):
+                spec = client.spec(record.spec) if bound else None
+
+        See :meth:`~ess.apps.protocol.WorkspaceProtocol.bound`.
+        """
+        return self._workspace.bound([r.id for r in records])
+
     def records(
         self, *, label: str | None = None, ids: Sequence[str] | None = None
     ) -> list[Record]:

packages/essapps/src/ess/apps/folder.py

@@ -59,13 +59,13 @@ def write(self, record: str, output: str, value: Any) -> None:
         path.parent.mkdir(parents=True, exist_ok=True)
         hdf5.write(value, path)
 
-    def read(self, record: str, output: str, tp: Any) -> Any:
+    def read(self, record: str, output: str, tp: Any | None) -> Any:
         path = self.path(record, output)
         if not path.is_file():
             raise LookupError(f'the store holds no {output} of {record}')
         try:
             return hdf5.read(tp, path)
-        except ValueError as error:  # written under other outputs of the spec
+        except ValueError as error:  # does not fit tp
             raise LookupError(str(error)) from None
 
 

packages/essapps/src/ess/apps/hdf5.py

@@ -4,27 +4,20 @@
 An output as a scipp HDF5 file, which ``scipp.io.load_hdf5`` reads. It needs
 scipp and h5py.
 
-scipp writes arrays, ``DataGroup``, numbers, and text. So a value is written as
-these, and read back by the type that the spec declares for the output::
-
-    as the workflow returned it          in the file
-    ---------------------------          -----------
-    a scipp array                        the array
-    a group (a pydantic model)           a DataGroup of its fields
-    dict[str, X]                         a DataGroup of its items
-    list[X]                              a DataGroup by index, '0', '1', ...
-    None, as a field or an item          left out
-    a member of an enum                  its value
-    a number, text, or a bool alone      a DataGroup with the key 'value'
+scipp writes arrays, ``DataGroup``, numbers, and text. So a file holds a value
+in its form as stored, which :meth:`ess.apps.protocol.Store.read` tables, and a
+value is read back by the type that the spec declares for the output, or
+without a type as the file holds it.
 
 A value that has no such form is refused with ``TypeError`` when it is written:
 a date, a tuple, ``None`` in a list, an object of any other class, and a
 ``DataGroup`` that holds one of these. A group is read back with the fields
 that were written, not validated again, as an instance of the declared type:
 a field the file lacks is ``None`` if ``None`` is a value of its type (an
 optional type, ``Any``, or ``object``), else its default.
-A file that lacks a required field, or holds a value an enum lacks, was written
-under other outputs of the spec, and reading it raises ``ValueError``.
+A file that lacks a required field, or holds a value an enum lacks, holds no
+value of the declared type, as a file changed or damaged outside the workspace
+would, and reading it raises ``ValueError``.
 """
 
 from __future__ import annotations
@@ -107,26 +100,35 @@ def _typed(tp: Any, data: Any) -> Any:
     return data
 
 
-def write(value: Any, path: Path) -> None:
-    """Write ``value`` to ``path``, as the module says."""
+def stored(value: Any) -> Any:
+    """``value`` as :func:`write` writes it, and :func:`read` without a type reads."""
     plain = _plain(value)
     if not isinstance(plain, (*_ARRAYS, sc.DataGroup)):
         plain = sc.DataGroup({_ALONE: plain})
-    sc.io.save_hdf5(plain, path)
+    return plain
+
+
+def write(value: Any, path: Path) -> None:
+    """Write ``value`` to ``path``, as the module says."""
+    sc.io.save_hdf5(stored(value), path)
 
 
-def read(tp: Any, path: Path) -> Any:
+def read(tp: Any | None, path: Path) -> Any:
     """
-    The value of type ``tp`` that :func:`write` wrote to ``path``. Raises
-    ``ValueError`` if the file holds no value of ``tp``, as the module says.
+    The value of type ``tp`` that :func:`write` wrote to ``path``, or, if
+    ``tp`` is ``None``, the file's data as scipp reads it, as the module's
+    table says. Raises ``ValueError`` if the file holds no value of ``tp``, as
+    the module says.
     """
     data = sc.io.load_hdf5(path)
+    if tp is None:
+        return data
     plain_type, _ = field_types.optional(tp)
     alone = plain_type in _ALONE_TYPES or _is(plain_type, Enum)
     try:
         return _typed(tp, data[_ALONE] if alone else data)
     except ValueError as error:
         raise ValueError(
             f'{path} holds no valid {getattr(tp, "__name__", tp)}: {error}. '
-            "The spec's outputs changed since the value was written."
+            'It may have been changed or damaged outside the workspace.'
         ) from None

packages/essapps/src/ess/apps/listing.py

@@ -15,7 +15,7 @@
 
 from .client import Client
 from .files import Selector
-from .records import Record, Status
+from .records import Record, SpecId, Status
 
 
 class Listing:
@@ -30,8 +30,12 @@ class Listing:
     rest from ``previous``, which stays as it is. The records are oldest
     first. Labels and the outputs offered are worked out from all the records
     in memory, so a label that depends on the other records needs no read.
-    ``specs`` holds the spec of each record by its id, ``status`` its status,
-    and ``readable`` the names of its outputs that a request may reference.
+    ``specs`` holds the spec of each record of the bound spec
+    (:meth:`~ess.apps.Client.bound`) by the record's id, ``status`` the status
+    of each record, and ``readable`` the names of its outputs that may be read
+    (:meth:`~ess.apps.Client.readable`). A record of another spec has no
+    entry in ``specs``: it is labelled by its spec's name and version, its
+    outputs by their names, and none of them is offered to reference.
 
     It stands in for two queries the client lacks (scipp/essapps#95). The
     client lists no records by the type of their outputs (gap 1), so
@@ -56,18 +60,20 @@ def __init__(self, client: Client, previous: Listing | None = None) -> None:
         old = [] if previous is None else previous.records
         known = {r.id for r in old}
         self.records = old + [r for r in changed if r.id not in known]
-        specs = {} if previous is None else {r.spec: previous.specs[r.id] for r in old}
-        for spec in {r.spec for r in changed} - specs.keys():
-            specs[spec] = client.spec(spec)
-        self.specs = {r.id: specs[r.spec] for r in self.records}
+        specs = {SpecId.of(spec): spec for spec in client.specs()}
+        self.specs = {} if previous is None else dict(previous.specs)
         self.status = {} if previous is None else dict(previous.status)
         self.readable = {} if previous is None else dict(previous.readable)
         statuses = client.status(changed)
         readable = client.readable(changed)
+        bound = client.bound(changed)
+        self.specs |= {
+            r.id: specs[r.spec] for r, b in zip(changed, bound, strict=True) if b
+        }
         self.status |= {r.id: s for r, s in zip(changed, statuses, strict=True)}
         self.readable |= {r.id: n for r, n in zip(changed, readable, strict=True)}
         self.labels = _labels(
-            {r.id: (self.specs[r.id].title, r, (r.id[:6],)) for r in self.records}
+            {r.id: (self.title(r), r, (r.id[:6],)) for r in self.records}
         )
         self._output_labels = _labels(
             {
@@ -83,6 +89,7 @@ def __init__(self, client: Client, previous: Listing | None = None) -> None:
                 self.specs[record.id].outputs.model_fields[name].annotation,
             )
             for record in reversed(self.records)
+            if record.id in self.specs
             for name in self.readable[record.id]
         ]
 
@@ -91,13 +98,30 @@ def pending(self) -> list[Record]:
         """The records listed as pending."""
         return [r for r in self.records if self.status[r.id] is Status.PENDING]
 
+    def title(self, record: Record) -> str:
+        """
+        The title of a record's spec; its name and version, ``centre/v1``, if
+        the record is of another spec.
+        """
+        spec = self.specs.get(record.id)
+        return str(record.spec) if spec is None else spec.title
+
     def type_name(self, record: Record, name: str) -> str:
-        """The name of the type of an output, without ``| None``."""
-        info = self.specs[record.id].outputs.model_fields[name]
+        """
+        The name of the type of an output, without ``| None``; the output's
+        name if its record is of another spec.
+        """
+        if (spec := self.specs.get(record.id)) is None:
+            return name
+        info = spec.outputs.model_fields[name]
         return field_types.type_name(field_types.value_type(info.annotation))
 
     def is_group(self, record: Record, name: str) -> bool:
-        """Whether an output is a group, which ``describe`` gives by field."""
+        """
+        Whether an output of a record of the bound spec is a group, which
+        ``describe`` gives by field. Raises ``KeyError`` for a record of another
+        spec: only its stored value says whether it is a ``DataGroup``.
+        """
         info = self.specs[record.id].outputs.model_fields[name]
         return field_types.is_group(field_types.value_type(info.annotation))