diff --git a/docs/developer/README.md b/docs/developer/README.md index dd8bb54..b4fa521 100644 --- a/docs/developer/README.md +++ b/docs/developer/README.md @@ -1,6 +1,6 @@ # ESS data-reduction framework -**Status: `ess.apps.Workspace`, which `local()` and `open_folder()` make in the caller's process, implements this design, including freeze, `select=`, and persist to a fake store or to scipp HDF5 files in a folder (`ess.apps.folder`). Not implemented: a hosted workspace reached with `connect()`, `client.publish`, and software versions in provenance. The first release combines no runs and tunes no parameters interactively, so it needs neither stages nor accumulators.** +**Status: `ess.apps.Workspace`, which `local()` and `open_folder()` make in the caller's process, implements this design, including freeze, `select=`, and persist to a fake store or to scipp HDF5 files in a folder (`ess.apps.folder`). Not implemented: a hosted workspace reached with `connect()`, `client.publish`, software versions in provenance, spec digests ([ADR 0009](adr/0009-a-record-of-another-spec-stays-readable.md)), and catalogue identifiers as file identities ([ADR 0010](adr/0010-a-file-is-identified-by-its-catalogue-identifier.md)). The first release combines no runs and tunes no parameters interactively, so it needs neither stages nor accumulators.** This page gives an overview of the framework: its components, what each promises the others, and the ideas that client code is built from. The details live one level down: @@ -122,6 +122,7 @@ for run in client.files.watch(Selector(scan='17')): ... # existing ones first, **Store.** The *store* keeps persisted outputs outside the workspace's memory, also after the workspace has closed ([`ess.apps.protocol`](../../packages/essapps/src/ess/apps/protocol.py)). It reads a value back by the type that the spec declares for the output, so a workspace reads the persisted outputs of the specs it runs. +An output of a record of another spec, such as an older version, is read as plain scipp data ([ADR 0009](adr/0009-a-record-of-another-spec-stays-readable.md)). **Controller.** The workspace does not decide what to run. A *controller* is client code that decides over time what to submit, push, and release, such as the cells of a notebook, the callbacks of an application's screen, or the trigger loop of automatic reduction. @@ -293,7 +294,7 @@ It is one step of the path a value takes from one history to another ([deploymen Not visible in the code of notebooks, applications, or workflow packages: - how history and outputs are stored, moved, and dropped -- how a run number or path becomes a file identity +- how a file source finds the file that a run number or path names, though provenance shows the identity it gives the file ([ADR 0010](adr/0010-a-file-is-identified-by-its-catalogue-identifier.md)) - when and where a request runs, and where a stage or a driver lives - the format of persisted outputs and the store's layout - how a process that runs unattended bounds the memory of a workspace, and when it closes an idle one (open questions 8 and 12) diff --git a/docs/developer/adr/0002-a-value-lives-while-something-keeps-it.md b/docs/developer/adr/0002-a-value-lives-while-something-keeps-it.md index 042c4d9..7602841 100644 --- a/docs/developer/adr/0002-a-value-lives-while-something-keeps-it.md +++ b/docs/developer/adr/0002-a-value-lives-while-something-keeps-it.md @@ -46,7 +46,14 @@ A client chooses per output, with `persist=` at submission or at `client.freeze` The store is then the only keeper, as batch reduction and the trigger loop need. A failed write fails the record, so a controller sees it like any other failed record. The store keys a value by record ID and output name, so a record holds no file name and a write changes no record. -A persisted output stays readable after an upgrade removes its record's spec version, since the record stays in history. +A persisted output stays readable after an upgrade removes its record's spec version, since the record stays in history, and is read as plain scipp data ([ADR 0009](0009-a-record-of-another-spec-stays-readable.md)). + +**A workspace reads and writes persisted outputs through `Store`**, by record, output name, and type. +A file format belongs to each stored value, so a store may hold several. +A value's file format must let one array be read without the others, and the sizes, unit, and coordinates without the data, since `client.describe` reads no array and `select=` reads a part. +scipp's HDF5 layout does. +`Store` reads a value whole. +Reading a part is added with the first output that needs it. **The same rules hold in the user's process and on the service.** What differs: @@ -75,6 +82,7 @@ Each pending record is - **One workspace for many proposals.** The core would separate the history, values, and files of each proposal, and check every call against them. A service gets the separation between proposals by running one workspace per proposal instead. - **Record objects keep values, as dask futures do.** `client.records()`, `client.latest()`, and provenance each return a new object for the same record, so no single object could own the value. A dict of records (D2, kept for the morning check) would keep every value, and so would Jupyter's `Out[n]`. - **Persisting a kept record later (`client.persist`).** A person could keep a result after seeing it without computing it again. No story or application needs it. A write that ends after its record has finished has no record to fail, so its failure would need a channel of its own, such as an error raised when the workspace closes. It needs two more kinds of event in history. It can be added when a story needs it. +- **Running a started record again once after a restart.** A reduction that exhausts memory would end the process a second time, and history would have to count the attempts. - **Keep only on request (`keep=True`).** The flag must be given at submission, since a value dropped when its request finishes cannot be kept later without a race. Forgetting it shows only when a later `client.output` or reference is refused. - **Views first, and a record made only at persist.** A read returns a view, a handle without a record. It needs a second handle type, records of upstream views that have no values, which a later `persist` cannot complete, and the states of accumulators kept inside views. A record is one entry in history, so making it lazily saves nothing worth that. - **Sessions or nested scopes.** A session reopened by name is lost when the workspace restarts, and needs names and an expiry. A scope is a `with` block, which a notebook cannot hold across cells. diff --git a/docs/developer/adr/0004-history-is-append-only-lists.md b/docs/developer/adr/0004-history-is-append-only-lists.md index a210f96..db64097 100644 --- a/docs/developer/adr/0004-history-is-append-only-lists.md +++ b/docs/developer/adr/0004-history-is-append-only-lists.md @@ -1,4 +1,4 @@ -# ADR 0004: History is append-only lists, dropped per proposal once it is idle +# ADR 0004: History is append-only lists, and a history no person holds is dropped once its proposal is idle - Status: accepted - Deciders: Simon @@ -28,27 +28,30 @@ A read of a state makes no record. ## Decision -History is four lists, each only appended to: +History is five lists, each only appended to: | List | One item per | Holds | |---|---|---| | records | record | ID, time, proposal, submitter, request, output names, label, member | | accumulators | opened accumulator | ID, proposal, template, and the template's values typed with defaults filled in | +| starts | record whose work began | record ID | | finishes | finished record | record ID, status, failure message, the names of optional outputs that the workflow did not return | | pushes | push into an accumulator | accumulator ID, one row per table | -- A record never changes. Its status is its finish, and a record without one is pending. +- A record never changes. Its status is its finish, and a record without one is pending. A record with a start and no finish is running, and at a restart it fails ([ADR 0002](0002-a-value-lives-while-something-keeps-it.md#decision)). - The record of a freeze ([ADR 0006](0006-the-unit-is-an-accumulating-workflow.md)) is stored as the accumulator and its number of pushes. Its request is listed from the pushes list when it is read: its rows are those of the first n pushes. So the record of a freeze takes a constant amount of history, and reads of a state add nothing. - A persist request is part of the record. The outcome of the write is the record's finish ([ADR 0002](0002-a-value-lives-while-something-keeps-it.md)). -- Output values are not history ([ADR 0002](0002-a-value-lives-while-something-keeps-it.md)). History names no file. -- Once a proposal has been idle ([`ess.apps.protocol`](../../../packages/essapps/src/ess/apps/protocol.py)) for the retention period, days to weeks as the deployment sets it, its history is dropped as a whole. An output needed for longer is published (README, [Provenance and publication](../README.md#provenance-and-publication)). +- Output values are not history ([ADR 0002](0002-a-value-lives-while-something-keeps-it.md)). History names no file of the store. +- A history that a process holds for a proposal without a person is dropped as a whole once the proposal has been idle ([`ess.apps.protocol`](../../../packages/essapps/src/ess/apps/protocol.py)) for the retention period, days to weeks as the deployment sets it. Today that process is the automatic reduction, and it would be a hosted workspace if one is built. An output needed for longer is published (README, [Provenance and publication](../README.md#provenance-and-publication)). +- The framework deletes nothing in a folder that a person opened. An application may offer to clear a folder when it closes. - How a deployment stores the lists is its choice. `ess.apps.Workspace` appends them to one event log, in memory or in a file of JSON lines. +- A log file starts with `{"format": N}`. A log reads only files of its own format, and leaves a file it refuses as it was. The number changes with any change to the events or to the files of the store. Older formats are not read. - The changes to a proposal's records have one order: the *change feed*, the ID of a record at each of its items in the records and finishes. A client reads it from a position (`client.changes`), so that it reads only the records that changed. It holds no events, and neither accumulators nor pushes, so a workspace keeps this order however it stores the lists. ## Alternatives considered - **History defined as one event log, with records and labels as views**, indexes rebuilt from the log. The lists hold the same history. The log adds one order across all lists, and outside storage only the change feed needs an order, of the records alone. It would tie a hosted workspace to a log. -- **History kept as long as the proposal.** That may be forever, a second record of what ran next to SciCat. +- **A history held without a person kept as long as the proposal.** That may be forever, a second record of what ran next to SciCat. - **Each record dropped by age, keeping the older records that a kept record reads.** The trigger loop would reduce again every file whose records were dropped. It would need a memory of its own, such as a set of handled files per rule. - **A proposal's history dropped a fixed time after its last record.** A trigger loop that sees no new file for that long loses its rule's records and then reduces every file again. A client still in use would lose the records it reads. - **The record of a freeze stored with every row, in storage that shares repeated rows.** Sharing saves bytes, but every layer above the store still handles each row twice, once in its push and once in the record. @@ -63,10 +66,11 @@ History is four lists, each only appended to: - A workspace that restarts does not know when its proposal was last called, so it counts idle time from its start. - `ess.apps.Workspace` never drops history. - Its log appends one line per change, so a submission is stored whole or not at all. A line cut short by a crash is dropped when the file is read. -- Old event formats must stay readable as long as the log is kept. +- A folder written in an older format is refused. This is acceptable before the first release, and is decided again after it. - A workspace that rebuilds its indexes of records from the log at start holds every record of its proposal in memory. - A restart keeps history, not values, so pending work goes on only if what it reads was persisted ([ADR 0002](0002-a-value-lives-while-something-keeps-it.md)). ## Open +- **A file that arrives after its proposal's history was dropped.** How the automatic reduction then knows which files it has handled. - **The form of the record of a freeze.** Only a freeze makes a record of a state, and an accumulator is frozen again only after a freeze has failed or been cancelled. A record of a freeze stored with its rows would hold each row about twice, once in its push and once in the record. Whether storing it as the accumulator and its number of pushes still earns its place is open. diff --git a/docs/developer/adr/0009-a-record-of-another-spec-stays-readable.md b/docs/developer/adr/0009-a-record-of-another-spec-stays-readable.md new file mode 100644 index 0000000..e229424 --- /dev/null +++ b/docs/developer/adr/0009-a-record-of-another-spec-stays-readable.md @@ -0,0 +1,74 @@ +# ADR 0009: A record names its spec by name, version, and digest, and a record of another spec stays readable + +- Status: accepted, not implemented. On `main`, an output of a record whose spec has no binding raises `NotReadable`, and a stored group is read into the bound outputs class if no required field is missing. +- Deciders: Simon +- Date: 2026-11-13 + +## Context + +A workspace made with `open_folder` keeps its history and its persisted outputs in a folder, and a later process reads them, possibly with another version of the technique package installed. +An outputs class can change under the same spec version: + +```python +# iofq/v1 wrote Centre(x=0.5, y=0.25); the package now has, still under v1: +class Centre(BaseModel): + x_mm: float = 0.0 # renamed from x + y: float + +client.output(old_record, 'centre') # Centre(x_mm=0.0, y=0.25), no error +``` + +A version bump instead makes the old outputs unreadable, and a batch shows their rows as not reduced. + +Each deployment has its own workspaces, so a proposal has several histories ([deployments.md](../deployments.md#one-proposal-several-histories)). +What must reach another workspace, or outlive the history, is published. + +## Decision + +Callers write `SpecId(name, version)` or pass the spec. +A request made with a spec whose digest differs from that of the spec bound under the same name and version is refused at submission. + +The *digest* of a spec covers the name, type, and requiredness of every field of its params and outputs models, including the fields of groups inside them. +Titles, descriptions, and defaults are not part of it. +A record stores its params with defaults filled in, so a changed default does not change what an old record means. +The digest cannot see a change of what a workflow computes. +The version remains the author's statement of any change a consumer can observe: the params, the outputs, or what the workflow computes (`WorkflowSpec`). + +Every event that names a spec, a submission or the opening of an accumulator, carries the digest of each spec it names that the log does not hold yet under that name and version, or holds with another digest. +The digest of a record's spec is the last one logged for its name and version before the record. +A submission, abridged: + +```json +{"kind": "submitted", "specs": {"iofq/v1": "9f2c..."}, "records": [{"id": "r2", "request": {"spec": {"name": "iofq", "version": 1}, "params": {"bins": 4}}}]} +``` + +A record is *of the bound spec* if the workspace has a binding with the same name, version, and digest. +Otherwise it is *of another spec*: the version was bumped, the models changed, or no binding is given. +A record of another spec stays in the history, and is readable without the classes that made it: + +| Of a record of another spec | What a client gets | +|---|---| +| its request | name and version, and the params as the stored dict, with files and references | +| an output | the stored scipp data: a `DataArray`, or a `DataGroup` by field name | +| `describe` and `select=` | work on that data | +| a new request that references its output | refused at submission with `SubmitError`, since the output's type cannot be checked | +| a restart while it is pending | it fails with a reason that says its spec is not bound, and so does a pending record that reads an output of it | + +A technique package's tests compare each spec's digest with one kept in the package, so a change of a model without a new version fails in the author's CI (`ess.spec.testing`). + +## Alternatives considered + +- **The digest as the spec's identity**, `SpecId(name, digest)`. Callers, templates, and saved settings would name a digest. A person writes and reads the version, and the digest only has to detect a change. +- **Refusing a folder whose spec changed.** A person could open the folder only with the old package installed, and could not compare results across the change. +- **Keeping each spec's JSON schema in the log**, so that old params show with their titles and descriptions. Not needed while the stored names and values are readable. +- **A spec bound only for reading old outputs** into their classes. Not needed while plain scipp data is enough to show and compare an old result. + +## Consequences + +- A result of another spec cannot feed a new request. A person who needs it as an input reduces again, or keeps the old spec installed. +- An edit to a title or a description changes no digest, so no record becomes one of another spec. +- What the digest covers is part of the log's format ([ADR 0004](0004-history-is-append-only-lists.md#decision)). + +## Open + +- A batch should show a row whose record is of another spec as no longer current, while its result stays readable, so that results of two versions of a spec can be compared. How, and how that relates to "stale", is scipp/essapps#191. diff --git a/docs/developer/adr/0010-a-file-is-identified-by-its-catalogue-identifier.md b/docs/developer/adr/0010-a-file-is-identified-by-its-catalogue-identifier.md new file mode 100644 index 0000000..6c8531e --- /dev/null +++ b/docs/developer/adr/0010-a-file-is-identified-by-its-catalogue-identifier.md @@ -0,0 +1,47 @@ +# ADR 0010: A file is identified by its catalogue identifier, or else by its path + +- Status: accepted, not implemented. On `main`, `FolderFiles` gives every file the identity `path:`. +- Deciders: Simon +- Date: 2026-11-13 + +## Context + +A person names a file as `file(run=60339)`, and the file source resolves the name to the file's *identity*, which the record keeps and provenance lists. +With paths as identities: + +- A moved folder changes every identity, so a trigger loop reduces its runs again. +- The same run seen through two file sources has two identities. + +The catalogue is SciCat. +A derived dataset in the catalogue lists its inputs by *catalogue identifier*. +A raw file carries `entry/entry_identifier_uuid`, and the catalogue lists it as `20.500.12269/` ([data.md](../../requirements/data.md)). + +## Decision + +| File | Identity | +|---|---| +| a file the catalogue lists | `pid:20.500.12269/` | +| any other file, inside the workspace's folder | `path:masks/main.xml`, relative to the folder | +| any other file, outside it | `path:/data/other/run-1.nxs` | + +The catalogue identifier is written in full. +For a raw file, a file source without access to the catalogue forms it from the identifier the file carries. +An *other file* is one the catalogue does not list, such as a mask, a test file, or a file of another facility. +A workspace without a folder, made with `local`, gives absolute paths. +A person still writes `file(run=60339)`. + +## Alternatives considered + +- **Paths throughout.** A moved folder and a second file source change the identity of a raw file, and provenance would not name inputs as the catalogue does. +- **A hash of the content.** A file would be read whole to be identified, and an hour of DREAM at its highest rate is about 2 TB ([data.md](../../requirements/data.md)). + +## Consequences + +- A moved folder, and a second file source, give a raw file the same identity. +- Provenance lists raw files as the catalogue names them. +- A file source reads the identifier from a raw file, or gets it from the catalogue. +- Identities are in the log, so changing them changes the log's format ([ADR 0004](0004-history-is-append-only-lists.md#decision)). + +## Open + +- Whether every raw file in user operation carries the identifier, and whether a file the catalogue lists can still be unreadable. To be asked of the teams that run the file writer and the catalogue. diff --git a/docs/developer/adr/index.md b/docs/developer/adr/index.md index 879f48e..1a5dfe3 100644 --- a/docs/developer/adr/index.md +++ b/docs/developer/adr/index.md @@ -9,8 +9,10 @@ The format follows [scipp's ADR convention](https://github.com/scipp/scipp/tree/ In reading order: -- [ADR 0004: History is append-only lists, dropped per proposal once it is idle](0004-history-is-append-only-lists.md). Records never change, and output values are not history. +- [ADR 0004: History is append-only lists, and a history no person holds is dropped once its proposal is idle](0004-history-is-append-only-lists.md). Records never change, and output values are not history. - [ADR 0002: A value lives while something keeps it, and nothing is written unless persisted](0002-a-value-lives-while-something-keeps-it.md). The workspace of a proposal keeps what its clients ask for, work that nothing keeps is cancelled, and only persisted records continue after a restart. +- [ADR 0009: A record names its spec by name, version, and digest, and a record of another spec stays readable](0009-a-record-of-another-spec-stays-readable.md). An output of a record whose spec is not bound is read as plain scipp data. +- [ADR 0010: A file is identified by its catalogue identifier, or else by its path](0010-a-file-is-identified-by-its-catalogue-identifier.md). A moved folder or a second file source leaves the identity of a raw file unchanged. - [ADR 0006: Runs are combined by an accumulator, not by combining the outputs of requests](0006-the-unit-is-an-accumulating-workflow.md). State n gives the outputs of the plain request over the first n pushes, and a freeze makes a record of the plain request of the logged state, state n for the n pushes logged so far. - [ADR 0003: An accumulator's driver adds in place, and push n+1 waits for the readers of state n](0003-accumulators-add-in-place.md). `client.output` copies a state and makes no record, a request reads an accumulator only through the record of its freeze, and freeze needs no copy. - [ADR 0007: Distributions are split by who depends on them and when they are released](0007-packages-split-by-dependencies.md). essspec, essapps, and essinstruments; code that needs dependencies only some users install is a subpackage behind an extra. diff --git a/docs/developer/prior-art/snakemake.md b/docs/developer/prior-art/snakemake.md index a12fe26..e10f793 100644 --- a/docs/developer/prior-art/snakemake.md +++ b/docs/developer/prior-art/snakemake.md @@ -36,7 +36,7 @@ The code is the text of the shell command or of the rule's Python block, so a co For us: provenance lists software versions ([README, Provenance and publication](../README.md#provenance-and-publication)), not yet implemented, and what a finish records is open ([ADR 0002, Open](../adr/0002-a-value-lives-while-something-keeps-it.md#open)). Package names and versions, as Snakemake's list holds, is a grain that cosmetic edits do not change. The metadata format is at version 7, and each rerun check skips records written before the version that first stored its field reliably ([persistence](https://github.com/snakemake/snakemake/blob/4523356973a83aa237a121717ee81dd2e09175da/src/snakemake/persistence/__init__.py#L588-L622)). -That is one way to keep old event formats readable as long as the log is kept ([ADR 0004](../adr/0004-history-is-append-only-lists.md#consequences)). +That is one way to read old formats. A log here reads only files of its own format ([ADR 0004](../adr/0004-history-is-append-only-lists.md#decision)). **Outputs are deleted on failure; logs survive, but only what the author redirects into them.** Snakemake deletes a failed job's outputs, so a half-written file is never taken for a result.