Repository navigation
feat: a file is identified by its catalogue identifier, or else by its path relative to the folder - #195
Merged
Conversation
…s path relative to the folder FolderFiles gave every file the identity path:<absolute path>, so moving a workspace's folder changed every identity: a trigger loop reduced its runs again, and a table that a batch wrote named files that no longer resolved. The same raw file seen through two file sources had two identities, and provenance named raw files by path, not as the catalogue does. A row with a pid is identified by it, pid:20.500.12269/<uuid>. Any other file is identified by its path, relative to the given root if it is inside it, absolute otherwise. folder_files passes the workspace's folder as the root; a FolderFiles without one, as with local(), gives absolute paths. LoKI's read_metadata gives the pid of a raw file that carries entry/entry_identifier_uuid, through catalogue_identifier in ess.apps.files. Records hold file identities, so the log's format is 4. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…al paths; a pid is written in full file(path=...) and a Path in params are made absolute against the working directory, as the batch's text form reads them; the folder-relative form is only the identity a record holds. FolderFiles identifies a file by its real path, so a folder reached through a symbolic link resolves '../' rows, and looks up an absolute name by its real path. A pid not of the form <prefix>/<suffix> is refused. path() and metadata() raise KeyError naming an identity the source does not know, and a record that reads such a file fails with that sentence. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
5 of 7 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #194 (base branch
184-record-of-another-spec), which is stacked on #193 and #192. Merge those first, in order, or this PR lands in its base branch and not inmain.Implements ADR 0010: a file is identified by its catalogue identifier, or else by its path.
The problem.
FolderFilesgave every file the identitypath:<absolute path>. Moving a workspace's folder changed every identity, so a trigger loop reduced its runs again and a batch's rows no longer matched their records. The same raw file seen through two file sources had two identities.The rule.
FolderFilestakesroot=, the workspace's folder, whichfolder_filespasses:A person names a file as before, by run, by
pid, or by path. A relative path a person writes, infile(path=...), in aPathparam, or in a batch's text form, is relative to the working directory. The folder-relative form exists only as the identity a record holds. Only the file source'spath()turns an identity into a path.LoKI's
read_metadatagives thepidof a raw file that carriesentry/entry_identifier_uuid. The prefix20.500.12269is written in one place,ess.apps.files.catalogue_identifier. The files of the detector test at Larmor carry no identifier and are identified by path.The log format is 4, since records hold identities.
What a move does now. A test renames a folder and opens it again: the records' files match the file source, the outputs read, the trigger loop finds nothing to do, and the batch's table is unchanged.
Costs to be aware of.
pidand later with one changes identity. Runningessapps-batch filesagain after files gained identifiers has this effect, and a trigger loop then reduces those files again. ADR 0010 andbatch.mdsay so.pid:20.500.12269/<uuid>where they showed a path. The row key still shows the run, and the web and Panel views showrun N. Writing the run number in those cells needs the file's metadata where the table is written, so it is left for a follow-up._record.jsonwith a folder-relative identity does not name the folder.pidare refused, so a catalogue dataset that holds several files cannot be listed. Raw files have one file per dataset. Noted under Open in the ADR.Review. A fresh reviewer probed moved folders, symlinks, and every reader of an identity. Fixed from it: a relative
file(path=...)was taken against the folder while the batch's text form took it against the working directory; a folder reached through a symlink with a../row named a file that does not exist (real paths are used now); apidnot written in full was accepted (now refused, the prefix is not guessed); a record resumed after its file left the file source failed withKeyError: FileRef(...)as its reason (now a sentence that names the file).🤖 Generated with Claude Code