Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
86 changes: 57 additions & 29 deletions docs/developer/README.md

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion docs/developer/accumulators.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ Both make records that are identical to the records of plain requests with the s
The workspace keeps a stage or an accumulator until a client releases it or the workspace closes.
Every client that holds a stage or an accumulator may call through it, push into it, and read it.
No client call lists them or reopens one by its ID, and they are not in the changes.
So a second connection, such as a reloaded web page, reaches the stages and accumulators of the first only if it is handed them ([README, open question 9](README.md#open-questions)).
So a second client of a workspace reaches the stages and accumulators of the first only if it is handed them ([README, open question 9](README.md#open-questions)).
Making one over a binding that does not serve it is refused.
[ADR 0006](adr/0006-the-unit-is-an-accumulating-workflow.md) and [ADR 0003](adr/0003-accumulators-add-in-place.md) give the reasons; [bindings.md](../../packages/essspec/docs/bindings.md) gives the binding's side of the contract.

Expand Down
2 changes: 1 addition & 1 deletion docs/developer/automatic-reduction.md
Original file line number Diff line number Diff line change
Expand Up @@ -71,7 +71,7 @@ def stitch(client: Client, run: FileRef) -> Record:
```

The label belongs to the rule: any record under it counts, whoever submitted it and whether or not it failed.
A record that a notebook submits under a rule's label stops the rule from reducing the files it names, so manual work uses labels of its own.
A record that another client of the workspace submits under a rule's label stops the rule from reducing the files it names, so manual work uses labels of its own.
A failed record counts so that the loop does not retry a failed file, which could fail again at every step; running it again is the user's decision (D2).

A file whose function raises `LookupError` or `ValueError`, or whose request is refused with `SubmitError`, is skipped: the rule goes on with its other new files, and its status names each skipped file and why.
Expand Down
110 changes: 110 additions & 0 deletions docs/developer/deployments.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,110 @@
# Deployments: which process holds what

The terms *workspace*, *client*, *controller*, *binding*, and *history* are defined in [README.md](README.md#components).
An *application* is a program that holds a client: a controller, and a *screen* if a person uses it.
In each deployment of the first release, one Python process holds the application, its client, and the workspace:

```
screen ──(a)──▶ application ──(b)──▶ workspace ──(c)──▶ binding
```

| Boundary | Interface | Between two processes? |
|---|---|---|
| (a) screen to application | the application's own: HTTP for a web page, the channel of the widget library for widgets | yes, when a browser draws the screen |
| (b) application to workspace | the client | no |
| (c) workspace to binding | `Binding`, `Driver` | no: 4 threads of the same process |

A screen never calls a workspace.
What a screen shows is computed in Python next to the data, and only the result is sent to the screen: a plot function returns a figure, and a form is built from the spec's params model.
So the client is an interface for Python code, and the HTTP interface of a page belongs to that page.

## Notebook

```
VISA machine or laptop, one process
┌───────────────────────────────────────────────┐
│ notebook cells │
│ Client │
│ Workspace local(...): history in memory │
│ open_folder(directory): folder │
│ 4 threads run the bindings │
└───────────────────────────────────────────────┘
```

One process holds the cells, the client, and the workspace.
An application of widgets in a notebook is the same case: its controller runs in the notebook's process, and the browser only draws the widgets.
This deployment is implemented.

## Technique application with a batch page

```
browser VISA machine, one process per folder
┌─────────────┐ HTTP ┌───────────────────────────────────────────────┐
│ page: │ ──────────▶ │ web server rows, edits, settings (memory) │
│ selection, │ the page's │ batch logic, Client │
│ filter, tab │ own API │ Workspace open_folder(directory) │
└─────────────┘ │ 4 threads run the bindings │
└───────────────────────────────────────────────┘
folder: files.csv log.jsonl store/<record>/<output>.h5
```

The page holds only what it shows, and every rule is in the Python process.
The command `essapps-batch` opens the same folder when the application is not running.
Two users of one proposal have two folders.
The batch logic (`ess.apps.batch`), the command, a view in Panel, and the batch of LoKI (`ess.instruments.loki.batch`) are implemented.
The web page (`ess.apps.batch.web`) is experimental and runs as a demo.

## Automatic reduction

```
DMSC server, one process per instrument
┌──────────────────────────────────────────────────────────────────┐
│ loop: for each new file, submit what the file needs │
│ page to configure the reduction │
│ configuration: per instrument, adjusted per proposal │
│ a Client and a Workspace per proposal: history and store on disk │
│ threads run the bindings │
└──────────────────────────────────────────────────────────────────┘
│ reads new raw files │ writes result files, creates
▼ through a file source ▼ derived datasets in SciCat
```

An instrument has one stream of new files, so it has one automatic reduction.
The process holds a workspace for each proposal that has new files.
It runs with no page open, and users find its results through SciCat, not by connecting to it.
For the first release, the same process may run on a VISA machine ([requirements](../requirements/users.md)).
A scientist who needs a variant of the automatic reduction runs a batch.
Of this deployment, only the loop exists (`TriggerLoop`, [automatic-reduction.md](automatic-reduction.md)).

## One proposal, several histories

A workspace holds one proposal.
A proposal has several workspaces, each with its own history: one in the automatic reduction, and one for each folder or notebook of a user.

```
automatic reduction user A's batch folder user B's notebook
history 1 history 2 history 3
└───────────── a value passes only as a file ───────────┘
```

A reference names a record of the same history.
So a batch of user A cannot reference the beam centre that the automatic reduction computed, and two users of a proposal do not see each other's records.
A value reaches another history as a file:

```
record's output ──▶ written ──▶ named ──▶ published
│
record in another history ◀── loaded ◀── listed
```

| Step | What happens |
|---|---|
| written | a writer turns an output of a type into a file of a format, such as NXcanSAS for a SANS curve |
| named | the file gets a folder and a name |
| published | a derived dataset is created in SciCat, with its input datasets and software |
| listed | a file source lists the file with a kind, such as `mask` |
| loaded | a spec reads the file and gives the value as an output |

The inputs that instrument scientists prepare for later proposals, such as masks and calibrations, take the same path.
Parts of the path exist: a writer for HDF5 (`ess.apps.batch.export`), a file source that lists such files (`ess.instruments.loki.files`), and a spec that loads mask files (`MASKS`).
No value takes the whole path yet, and the path has no design ([README, open question 11](README.md#open-questions)).
1 change: 1 addition & 0 deletions docs/developer/system-stories.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ These stories check what the system must provide beyond the API of [README.md](R
Each has an actor, a goal, and the property to check.
Where a story shares a goal with an API story in [user-stories.md](user-stories.md), it has the same identifier.
How the workspace keeps history and values is described in its docstrings (`ess.apps.workspace`).
Stories that name *the service* describe a hosted workspace, which the users of a proposal share. No deployment of the first release has one ([deployments.md](deployments.md)).

## S. Small stories

Expand Down
7 changes: 6 additions & 1 deletion docs/developer/user-stories.md
Original file line number Diff line number Diff line change
Expand Up @@ -82,6 +82,11 @@ A story may add a toy spec to the list of toy specs; it must take runs directly,
Fixtures: `client` is a client for proposal `p1`.
In the notebook stories it is `local(proposal='p1', files=files, bind=...)`, a client of a *local workspace*, the workspace in the user's process that `local()` makes ([ADR 0002](adr/0002-a-value-lives-while-something-keeps-it.md)).
In the batch and automatic stories, sections D and E, it is `connect(url, proposal='p1')`, a client of the service at `url` ([ADR 0002](adr/0002-a-value-lives-while-something-keeps-it.md)), which is designed and not implemented.
The service is a hosted workspace, which the users of a proposal share.
No deployment of the first release has one ([deployments.md](deployments.md)): a batch application and the automatic reduction each hold their own workspace.
There, a second client such as `user = connect()` in E1 and E4 is a client in the same process, and a user in another process reads the results as files.
A client made the next day, as in D2, is `open_folder` on the same folder.
Logging in as another user, `user=` in G5, has no counterpart.
In the story tests, every client of `p1`, in the notebook stories and in sections D and E alike, is a `Client` of one `ess.apps.Workspace`, with a fake store in memory.
In the stories, `connect(proposal=..., user=...)` is `connect(url, ...)` to the same service, by default for `client`'s proposal and user; `user=` stands for logging in as another user.
`other` is a client of the workspace of proposal `p2`.
Expand Down Expand Up @@ -911,7 +916,7 @@ The provenance of the sum reaches that record.
What the design leaves open or defers, with the stories each item affects.

- **Removing a file** (system story A4): deferred, together with whether the outputs derived from it go too.
- **The service** (sections D and E): designed and not implemented; the format of each persisted output type, the layout of the store, and how the service bounds and ends its workspaces are open ([ADR 0002](adr/0002-a-value-lives-while-something-keeps-it.md); README.md open question 8).
- **The service** (sections D and E): designed and not implemented, and not part of a deployment of the first release ([deployments.md](deployments.md)); the format of each persisted output type, the layout of the store, and how the service bounds and ends its workspaces are open ([ADR 0002](adr/0002-a-value-lives-while-something-keeps-it.md); README.md open question 8).
- **Looks that compute** (B4, D7): a selection only indexes; a projection or a thick slice summed waits for the plotting work; README.md open question 4.
- **Persist at submission** (D1, D2, D3, D6, E1 to E4): whether `persist=` should also keep the values in the workspace; README.md open question 6.
- **Labels and members** (D1, D2): labels and members on records are tentative; README.md open question 3.
Expand Down
3 changes: 2 additions & 1 deletion docs/requirements/glossary.md
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,8 @@ Physics terms such as Q, I(Q), or S(Q, E) are not explained.
- **ess packages**: the reduction workflows in the scipp/ess repository (esssans, essreflectometry, ...), built with sciline (a workflow library) on scipp (labelled arrays); **plopp** is the team's plotting library.
- **VISA**: ESS remote desktops (virtual machines) with JupyterLab, created by a user for a proposal.
- **DMSC**: the ESS Data Management and Scientific Computing division; it runs the cluster and the data services, and the team that builds this framework belongs to it.
- **Batch-reduction service** and **automatic-reduction service**: long-running programs on DMSC servers that reduce data for many users; DMSC keeps them running, and users and instrument scientists configure them ([users](users.md)).
- **Automatic-reduction service**: a long-running program on a DMSC server that reduces the new runs of an instrument; DMSC keeps it running, and instrument scientists and users configure it ([users](users.md)).
- **Batch-reduction service**: the program that reduces a batch of runs for a user. An application that the user starts on a VISA machine serves as well as a long-running program on DMSC servers for many users. *Simon, 2026-11-13*
- **ISIS**, **SNS**, **PSI**: neutron sources in the UK, the USA, and Switzerland; Amor (PSI) and Larmor, Sans2d, Zoom, Offspec (ISIS) were used to prototype ess code.
- **Mantid**: the reduction software used at ISIS and SNS.
- **Slurm**: the job scheduler of the DMSC cluster.
Expand Down
3 changes: 2 additions & 1 deletion docs/requirements/users.md
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,8 @@ Results go to SciCat and must say what produced them, but the framework must not

- A configured reduction workflow is applied to every new run. *Simon, 2026-09-04*
- Reducing a growing series again at every new run, such as all angles of one sample so far, must be possible, but not as the only behaviour. *Simon, 2026-09-28*
- Batch and automatic reduction, and large work spread over the cluster, run as services, so leaks and peak memory must be bounded by how the services work, not by users freeing memory. *Simon, 2026-10-02*
- Automatic reduction, and large work spread over the cluster, run as services. Batch reduction runs as a service or as an application that a user starts on a VISA machine. In each, leaks and peak memory must be bounded by how the program works, not by users freeing memory. *Simon, 2026-10-02 and 2026-11-13*
- For a first release, automatic reduction may also run as an application on a VISA machine. *Simon, 2026-11-13*

### Provenance and the catalogue

Expand Down
Loading