Skip to content

feat: a file is identified by its catalogue identifier, or else by its path relative to the folder - #195

Merged
SimonHeybrock merged 2 commits into
mainfrom
184-file-identity
Nov 13, 2026
Merged

SimonHeybrock merged 2 commits into
mainfrom
184-file-identity

Conversation

@SimonHeybrock

Copy link
Copy Markdown
Member

Stacked on #194 (base branch 184-record-of-another-spec), which is stacked on #193 and #192. Merge those first, in order, or this PR lands in its base branch and not in main.

Implements ADR 0010: a file is identified by its catalogue identifier, or else by its path.

The problem. FolderFiles gave every file the identity path:<absolute path>. Moving a workspace's folder changed every identity, so a trigger loop reduced its runs again and a batch's rows no longer matched their records. The same raw file seen through two file sources had two identities.

The rule. FolderFiles takes root=, the workspace's folder, which folder_files passes:

pid:20.500.12269/<uuid>     a table row with a pid: a file the catalogue lists
path:masks/main.xml         any other file inside root
path:/data/other/run-1.nxs  any other file outside root, or without root (local())

A person names a file as before, by run, by pid, or by path. A relative path a person writes, in file(path=...), in a Path param, or in a batch's text form, is relative to the working directory. The folder-relative form exists only as the identity a record holds. Only the file source's path() turns an identity into a path.

LoKI's read_metadata gives the pid of a raw file that carries entry/entry_identifier_uuid. The prefix 20.500.12269 is written in one place, ess.apps.files.catalogue_identifier. The files of the detector test at Larmor carry no identifier and are identified by path.

The log format is 4, since records hold identities.

What a move does now. A test renames a folder and opens it again: the records' files match the file source, the outputs read, the trigger loop finds nothing to do, and the batch's table is unchanged.

Costs to be aware of.

  • A file first listed without a pid and later with one changes identity. Running essapps-batch files again after files gained identifiers has this effect, and a trigger loop then reduces those files again. ADR 0010 and batch.md say so.
  • File cells of a table the batch writes show pid:20.500.12269/<uuid> where they showed a path. The row key still shows the run, and the web and Panel views show run N. Writing the run number in those cells needs the file's metadata where the table is written, so it is left for a follow-up.
  • An exported _record.json with a folder-relative identity does not name the folder.
  • Two rows with one pid are refused, so a catalogue dataset that holds several files cannot be listed. Raw files have one file per dataset. Noted under Open in the ADR.

Review. A fresh reviewer probed moved folders, symlinks, and every reader of an identity. Fixed from it: a relative file(path=...) was taken against the folder while the batch's text form took it against the working directory; a folder reached through a symlink with a ../ row named a file that does not exist (real paths are used now); a pid not written in full was accepted (now refused, the prefix is not guessed); a record resumed after its file left the file source failed with KeyError: FileRef(...) as its reason (now a sentence that names the file).

🤖 Generated with Claude Code

SimonHeybrock and others added 2 commits November 13, 2026 08:20
…s path relative to the folder

FolderFiles gave every file the identity path:<absolute path>, so moving a
workspace's folder changed every identity: a trigger loop reduced its runs
again, and a table that a batch wrote named files that no longer resolved.
The same raw file seen through two file sources had two identities, and
provenance named raw files by path, not as the catalogue does.

A row with a pid is identified by it, pid:20.500.12269/<uuid>. Any other
file is identified by its path, relative to the given root if it is inside
it, absolute otherwise. folder_files passes the workspace's folder as the
root; a FolderFiles without one, as with local(), gives absolute paths.
LoKI's read_metadata gives the pid of a raw file that carries
entry/entry_identifier_uuid, through catalogue_identifier in ess.apps.files.

Records hold file identities, so the log's format is 4.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…al paths; a pid is written in full

file(path=...) and a Path in params are made absolute against the working
directory, as the batch's text form reads them; the folder-relative form is
only the identity a record holds. FolderFiles identifies a file by its real
path, so a folder reached through a symbolic link resolves '../' rows, and
looks up an absolute name by its real path. A pid not of the form
<prefix>/<suffix> is refused. path() and metadata() raise KeyError naming an
identity the source does not know, and a record that reads such a file fails
with that sentence.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
@SimonHeybrock SimonHeybrock self-assigned this Nov 13, 2026
Base automatically changed from 184-record-of-another-spec to main November 13, 2026 10:06
@SimonHeybrock
SimonHeybrock merged commit 5c32677 into main Nov 13, 2026
2 checks passed
@SimonHeybrock
SimonHeybrock deleted the 184-file-identity branch November 13, 2026 10:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant