feat: a file is identified by its catalogue identifier, or else by its path relative to the folder

docs/developer/README.md

@@ -1,6 +1,6 @@
 # ESS data-reduction framework
 
-**Status: `ess.apps.Workspace`, which `local()` and `open_folder()` make in the caller's process, implements this design, including freeze, `select=`, and persist to a fake store or to scipp HDF5 files in a folder (`ess.apps.folder`). A record of another spec than the one bound reads as stored scipp data ([ADR 0009](adr/0009-a-record-of-another-spec-stays-readable.md)). Not implemented: a hosted workspace reached with `connect()`, `client.publish`, software versions in provenance, and catalogue identifiers as file identities ([ADR 0010](adr/0010-a-file-is-identified-by-its-catalogue-identifier.md)). The first release combines no runs and tunes no parameters interactively, so it needs neither stages nor accumulators.**
+**Status: `ess.apps.Workspace`, which `local()` and `open_folder()` make in the caller's process, implements this design, including freeze, `select=`, and persist to a fake store or to scipp HDF5 files in a folder (`ess.apps.folder`). A record of another spec than the one bound reads as stored scipp data ([ADR 0009](adr/0009-a-record-of-another-spec-stays-readable.md)). A file is identified by its catalogue identifier, or else by its path ([ADR 0010](adr/0010-a-file-is-identified-by-its-catalogue-identifier.md)). Not implemented: a hosted workspace reached with `connect()`, `client.publish`, and software versions in provenance. The first release combines no runs and tunes no parameters interactively, so it needs neither stages nor accumulators.**
 
 This page gives an overview of the framework: its components, what each promises the others, and the ideas that client code is built from.
 The details live one level down:
@@ -113,6 +113,7 @@ It keeps nothing: every client of a workspace reads, references, and releases th
 Every call that starts work returns once the workspace has *logged* it in its history; the work happens later.
 
 **File source.** The workspace finds files, and their metadata, through its *file source*, which is given when the workspace is made.
+At submission it resolves a name, such as `file(run=60339)`, to the file's *identity*, which the record holds: `pid:20.500.12269/<uuid>` for a file the catalogue lists, and else its path, `path:masks/main.xml` relative to the workspace's folder if the file is in it ([ADR 0010](adr/0010-a-file-is-identified-by-its-catalogue-identifier.md)).
 A client sees only the files the workspace's proposal may read:
 
 ```python

docs/developer/adr/0009-a-record-of-another-spec-stays-readable.md

@@ -1,6 +1,6 @@
 # ADR 0009: A record names its spec by name, version, and digest, and a record of another spec stays readable
 
-- Status: accepted, implemented (`ess.apps.workspace`, log format 3), but for what Open names.
+- Status: accepted, implemented (`ess.apps.workspace`, with a new format of the log), but for what Open names.
 - Deciders: Simon
 - Date: 2026-11-13
 

docs/developer/adr/0010-a-file-is-identified-by-its-catalogue-identifier.md

@@ -1,6 +1,6 @@
 # ADR 0010: A file is identified by its catalogue identifier, or else by its path
 
-- Status: accepted, not implemented. On `main`, `FolderFiles` gives every file the identity `path:<absolute path>`.
+- Status: accepted, implemented (`ess.apps.FolderFiles`, and LoKI's `read_metadata`, which forms the identifier from a raw file), but for what Open names. No file source asks the catalogue: `FolderFiles` takes a file's `pid` from its table.
 - Deciders: Simon
 - Date: 2026-11-13
 
@@ -40,8 +40,10 @@ A person still writes `file(run=60339)`.
 - A moved folder, and a second file source, give a raw file the same identity.
 - Provenance lists raw files as the catalogue names them.
 - A file source reads the identifier from a raw file, or gets it from the catalogue.
+- A file first listed without its catalogue identifier and later with it changes identity: a trigger loop reduces it again, and the records made before name an identity the file source no longer knows.
 - Identities are in the log, so changing them changes the log's format ([ADR 0004](0004-history-is-append-only-lists.md#decision)).
 
 ## Open
 
 - Whether every raw file in user operation carries the identifier, and whether a file the catalogue lists can still be unreadable. To be asked of the teams that run the file writer and the catalogue.
+- A catalogue dataset that holds several files cannot be represented, since two rows with one `pid` are refused. Raw files have one file per dataset.

packages/essapps/docs/batch.md

@@ -202,7 +202,7 @@ A cell that is not text is its value, and `None` or `NaN` is an empty cell, so o
 
 **The table.** The first column holds the key of each row; it may also be a blank, as `sample` above. A column named as a blank holds its cells. A column of any other name is info, which comes back as it is; a column named `key`, as `frames.table_of` writes the key, is not info. Two mistakes are refused: a blank that is no column, such as a misspelt header, and a column named as a parameter that is not a blank, whose cells no row would use. An empty cell takes the parameter's default.
 
-**A cell** is read as JSON if it is JSON, so `1.5`, `null`, and `{"x": 0, "y": 1}` are values, and anything else is text. Text that is JSON is written in quotes, such as `"12"` for a sample name at a string parameter; a `.csv` file doubles them, `"""12"""`. At a parameter that is a file field, or a list of them, a whole number names a run, `pid:...` a file by its kind, and other text a path, relative to the working directory. A number with a leading zero, such as `0342`, is not JSON, so it is read as a path: write `342`. The same holds for the values of the template's file. A file field inside a group takes the JSON form, `{"@ref": "file", "file": "run:60339"}`. Every file is one that the workspace's file source knows.
+**A cell** is read as JSON if it is JSON, so `1.5`, `null`, and `{"x": 0, "y": 1}` are values, and anything else is text. Text that is JSON is written in quotes, such as `"12"` for a sample name at a string parameter; a `.csv` file doubles them, `"""12"""`. At a parameter that is a file field, or a list of them, a whole number names a run, text with a kind, such as `pid:...` or `path:...`, a file as the record holds it, and other text a path, relative to the working directory. A table that the batch writes holds each file as the record does, by its identity: `pid:20.500.12269/<uuid>` for a raw file the catalogue lists, and `path:masks/mask.xml`, relative to the workspace's folder, for any other file in it. A number with a leading zero, such as `0342`, is not JSON, so it is read as a path: write `342`. The same holds for the values of the template's file. A file field inside a group takes the JSON form, `{"@ref": "file", "file": "run:60339"}`. Every file is one that the workspace's file source knows.
 
 **`reduce`** returns the table with three columns, which it replaces if the table has them: `status`, the ID of the `record`, and `failure`, the reason. So its result can be its next table. With `--remaining` it leaves out the rows whose newest record under the label completed. It compares neither cells nor settings: after changing either, reduce without `--remaining`, or under another label. The `status` of a row is `completed`, `failed`, or `cancelled`.
 
@@ -227,7 +227,7 @@ log.jsonl      history
 store/         every output of every row, <record>/<output>.h5
 ```
 
-`files.csv` is the table of `FolderFiles`, a stand-in for a catalogue: a row per file with its path, relative to the folder, and its metadata as text. A run number in a template or a table is looked up here.
+`files.csv` is the table of `FolderFiles`, a stand-in for a catalogue: a row per file with its path, relative to the folder, and its metadata as text. A run number in a template or a table is looked up here. A file with a `pid`, which `read_metadata` gives a raw file that carries the catalogue's identifier, is identified by it, and any other file in the folder by its path relative to the folder. So a folder that is moved, with all it holds, keeps its records, its tables, and its results.
 
 ```
 file,run,kind,role
@@ -249,7 +249,7 @@ run,role,sample
 ,direct_beam,          <- with a column file: a file that is no run
 ```
 
-`read_metadata` is a function of the instrument package, from the path of a file to its metadata, or `None` for a file that is not the instrument's (`ess.apps.folder.scan_folder`). The runs come first, by their number, since the order of the rows is the order of measurement. A file that cannot be read is left out, with a warning. Two files of one run are refused.
+`read_metadata` is a function of the instrument package, from the path of a file to its metadata, or `None` for a file that is not the instrument's (`ess.apps.folder.scan_folder`). The runs come first, by their number, since the order of the rows is the order of measurement. A file that cannot be read is left out, with a warning. Two files of one run are refused. Running `files` again after files gained a `pid` changes their identity: a trigger loop reduces them again, and earlier records name files the folder no longer knows.
 
 A file says its kind, its run, and perhaps its sample, but not what a run is for. A row source that pairs runs reads the `role`, and a field such as the direct beam requires it. The run table gives it (`frames.described`): a row describes the file of its `run`, or the file at its `file` if it has no run. A run of the table that has no file in the folder is ignored, so the table may cover more than the folder holds. A file that no row describes keeps what it says of itself, and where both have a column, a cell of the run table that is not empty wins. Two rows for one run are refused. Without `--with`, the roles are typed into `files.csv`, so `files` then refuses to write over a table that exists.
 

packages/essapps/src/ess/apps/batch/export.py

@@ -32,7 +32,8 @@ def write_outputs(client: Client, record: Record, directory: Path) -> dict[str,
     """
     Write each output that a completed ``record`` returned to ``directory``, and
     the record itself, which names the files and the values they were computed
-    from::
+    from. It names a file by its identity: its catalogue identifier, or its
+    path, relative to the workspace's folder if the file is in it::
 
         iofq.h5            what scipp.io.load_hdf5 reads (see ess.apps.hdf5)
         _record.json       the record

packages/essapps/src/ess/apps/batch/text.py

@@ -16,9 +16,10 @@
 A file has no JSON form that a person would write, so at a parameter that is a
 file field, or a list of them, a file is named by text (see :func:`named_file`)::
 
-    60339               the run 60339
-    /data/mask.xml      a path
-    pid:20.500/abc      anything else by its kind, as ``FileRef.file`` holds it
+    60339                   the run 60339
+    /data/mask.xml          a path
+    pid:20.500.12269/abc    anything else by its kind, as ``FileRef.file`` holds it
+    path:masks/mask.xml     a file of the workspace's folder, by its identity
 
 Only a parameter is read this way, not a file field inside a group, which takes
 the JSON form of a file, as the record holds it.
@@ -87,7 +88,7 @@ def named_file(name: Any) -> Any:
         return name
     if _KIND.match(name):
         return FileRef(file=name)
-    return file(path=Path(name).absolute())
+    return file(path=name)
 
 
 def _takes_files(annotation: Any, metadata: tuple[Any, ...]) -> bool:

packages/essapps/src/ess/apps/files.py

@@ -6,10 +6,13 @@
 A person names a file by what they know: a run number, a path, or a PID.
 The workspace asks its file source to resolve such a name to the file's
 identity when a request is submitted, so a record names the file itself.
+A file the catalogue lists is identified by its catalogue identifier, any
+other file by its path (see :class:`FolderFiles`).
 """
 
 from __future__ import annotations
 
+import os
 from collections.abc import Iterable, Mapping
 from pathlib import Path
 from typing import Any
@@ -24,12 +27,18 @@ def file(
     A file by its run number, its path, or the PID of the SciCat dataset
     that holds it, as a request writes it. The workspace resolves it to the
     file's identity when the request is submitted, and the record holds that
-    identity in place of the name. Exactly one of the three is given::
+    identity in place of the name: ``pid:...`` for a file the catalogue lists,
+    and ``path:...`` for any other (see :class:`FolderFiles`). Exactly one of
+    the three is given::
 
         file(run=60339)
         file(path='/home/user/data/run1.h5')
         file(pid='20.500.12269/vanadium')
 
+    A relative path is relative to the working directory, and is made absolute
+    here. A path relative to the workspace's folder is written only as the
+    identity a record holds, ``FileRef(file='path:masks/main.xml')``.
+
     A string where a file field is expected is refused: write ``file(path=...)``.
     The record holds no metadata; read it with :meth:`ess.apps.Files.metadata`.
     """
@@ -38,7 +47,9 @@ def file(
     if len(named) != 1:
         raise ValueError(f'name a file by exactly one of {list(given)}')
     ((kind, value),) = named.items()
-    return FileRef.at(value) if kind == 'path' else FileRef(file=f'{kind}:{value}')
+    if kind == 'path':
+        return FileRef.at(Path(value).absolute())
+    return FileRef(file=f'{kind}:{value}')
 
 
 class Selector:
@@ -86,6 +97,14 @@ def __repr__(self) -> str:
         return f'Selector({self.fields})'
 
 
+def catalogue_identifier(uuid: str) -> str:
+    """
+    The PID under which the catalogue lists the raw file that carries ``uuid``
+    in ``entry/entry_identifier_uuid``: ``20.500.12269/<uuid>``.
+    """
+    return f'20.500.12269/{uuid}'
+
+
 def readable(metadata: dict[str, Any], proposal: str) -> bool:
     """Whether requests of ``proposal`` may read a file with this metadata."""
     return metadata.get('proposal', proposal) == proposal
@@ -96,57 +115,105 @@ class FolderFiles:
     A file source over the files of a folder, with a table a person fills.
 
     The table has a row per file: ``file``, its name in the folder, and its
-    metadata, such as ``run``, ``kind``, and ``role``. A row without ``kind`` is
-    a raw file (see :class:`Selector`), and without ``proposal`` is open to all.
-    The order of the rows is the order of measurement::
+    metadata, such as ``run``, ``kind``, ``role``, and ``pid``. A row without
+    ``kind`` is a raw file (see :class:`Selector`), and without ``proposal`` is
+    open to all. The order of the rows is the order of measurement::
 
         FolderFiles('/data/test', [
             {'file': 'run-1.nxs', 'run': 1, 'role': 'sample'},
-            {'file': 'mask.xml', 'kind': 'mask'},
-        ])
+            {'file': 'masks/main.xml', 'kind': 'mask'},
+        ], root='/data/test')
+
+    A file with a ``pid``, one the catalogue lists, is identified by it, written
+    in full, ``<prefix>/<suffix>``, so every source that lists the file gives it
+    one identity. Any other file is identified by its real path, with symbolic
+    links resolved: relative to the real path of ``root`` if it is inside it,
+    and absolute if not or without ``root``::
 
-    A file is identified by its path, and named by its path, its ``run``, or its
-    ``pid``.
+        pid:20.500.12269/<uuid>     a row with pid 20.500.12269/<uuid>
+        path:masks/main.xml         a file inside root
+        path:/data/other/run-1.nxs  a file outside root
+
+    ``root`` is the folder of a workspace (see :func:`ess.apps.folder.folder_files`),
+    so that moving the folder keeps the identity of each file in it. A file is
+    named by its identity, its ``run``, its ``pid``, or its absolute path, also
+    one through a symbolic link.
 
     Raises
     ------
     FileNotFoundError
         If the folder lacks a file of the table.
     ValueError
-        If two rows name one file, or give one run or one PID.
+        If two rows name one file, give one run or one PID, or a PID is not
+        written in full.
     """
 
     def __init__(
-        self, directory: Path | str, table: Iterable[Mapping[str, Any]]
+        self,
+        directory: Path | str,
+        table: Iterable[Mapping[str, Any]],
+        *,
+        root: Path | str | None = None,
     ) -> None:
+        root = None if root is None else Path(os.path.realpath(root))
+        self._paths: dict[FileRef, Path] = {}
         self._metadata: dict[FileRef, dict[str, Any]] = {}
         self._names: dict[FileRef, FileRef] = {}
         for row in table:
             metadata = dict(row)
-            path = Path(directory).absolute() / metadata.pop('file')
+            path = Path(os.path.realpath(Path(directory, metadata.pop('file'))))
             if not path.is_file():
                 raise FileNotFoundError(path)
-            ref = FileRef.at(path)
-            if ref in self._metadata:
+            if FileRef.at(path) in self._names:
                 raise ValueError(f'two rows name the file {path}')
+            if root is not None and path.is_relative_to(root):
+                local = FileRef.at(path.relative_to(root))
+            else:
+                local = FileRef.at(path)
+            pid = metadata.get('pid')
+            if pid is not None and not _written_in_full(str(pid)):
+                raise ValueError(
+                    f'the pid {pid!r} of {path} is not written in full: a catalogue '
+                    'identifier is written with its prefix, such as '
+                    '20.500.12269/<uuid>'
+                )
+            ref = local if pid is None else FileRef(file=f'pid:{pid}')
+            names = {ref, local, FileRef.at(path)}
+            if 'run' in metadata:
+                names.add(FileRef(file=f'run:{metadata["run"]}'))
+            for name in names:
+                if name in self._names:
+                    raise ValueError(f'two rows give {name}')
+                self._names[name] = ref
+            self._paths[ref] = path
             self._metadata[ref] = metadata
-            for kind in ('run', 'pid'):
-                if kind in metadata:
-                    name = FileRef(file=f'{kind}:{metadata[kind]}')
-                    if name in self._names:
-                        raise ValueError(f'two rows give {name}')
-                    self._names[name] = ref
 
     def resolve(self, name: FileRef) -> FileRef:
-        if name in self._metadata:
-            return name
+        kind, _, value = name.file.partition(':')
+        if kind == 'path' and os.path.isabs(value):
+            name = FileRef.at(os.path.realpath(value))
         return self._names[name]
 
     def path(self, ref: FileRef) -> Path:
-        return Path(ref.file.removeprefix('path:'))
+        return self._paths[self._known(ref)]
 
     def metadata(self, ref: FileRef) -> dict[str, Any]:
-        return dict(self._metadata[ref])
+        return dict(self._metadata[self._known(ref)])
+
+    def _known(self, ref: FileRef) -> FileRef:
+        """
+        ``ref``, if it is the identity of a file of the table. A record may hold
+        one that is not, of a file moved or given a ``pid`` since it was made.
+        """
+        if ref not in self._paths:
+            raise KeyError(f'the file source does not know the file {ref.file}')
+        return ref
 
     def list(self, selector: Selector) -> list[FileRef]:
         return [r for r, m in self._metadata.items() if selector.matches(m)]
+
+
+def _written_in_full(pid: str) -> bool:
+    """Whether ``pid`` has the form ``<prefix>/<suffix>`` of a catalogue identifier."""
+    prefix, slash, suffix = pid.partition('/')
+    return bool(prefix and slash and suffix)