Repository navigation
Log format number, stored groups checked on read, and a linear Client.wait - #185
Merged
Merged
Conversation
A log file starts with a header such as {"format": 1}, written when the
file is created. A file whose first line is not the header of the format
this version reads is refused with an error naming the file, the format
found (or that it has none), and the format read. The number is bumped
on any change to the events or to the files of the store, since stored
values are found through the log. Files without a header are refused:
no migration.
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
A group read back from the store was built without checks, so a file written before the spec's outputs gained a required field read back with that field as None. Reading such a file raises ValueError naming the file, the type, and the missing field, and saying that the spec's outputs changed since the value was written. client.output raises it, and a request that references the output fails with it. The group is still not validated again: a validator need not give the same value when it runs on its own result. So only absent required fields (and enum values) are caught, not a field whose type changed. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
wait asked the status of every record after every change in the feed, so waiting for n records took time quadratic in n. It keeps the records still pending and, after each change, asks only about those that are pending and changed. 4000 trivial records: 14.6 s to 1.9 s until all finished (1.7 s without wait). Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
5 of 7 tasks
Log(path) cut a last line short and wrote a header into a file with no complete line before checking the header, so a refused file lost its last line and a file without a newline was emptied and accepted. The header is now checked first, and written only into an empty file or one holding the beginning of the header. A header naming the format as true or 1.0 is refused: only the integer is accepted. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…e no longer holds FolderStore.read let hdf5.read's ValueError through, which callers take for a misuse of select or a refused call. It now raises LookupError with that message, the store protocol's case for a value it no longer holds: a client's read raises NotReadable and a request that references the output fails. The workspace keeps the store's message in NotReadable, so both name the missing field. A required field typed Any or object that held None is left out when written, and is read back as None instead of being refused as missing. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Step 0 of #184: three small changes that do not depend on any open design decision.
The log names its format. A log file starts with a header line,
{"format": 1}, and a log refuses a file whose header is another format or missing. The number covers the events and the files of the store, since stored values are found through the log. Motivation:open_folderkeeps a log and a store in a user's folder, while spec identity, file identity, labels, and the stored form of a request are still open (#80, #175, #177, #179). With the number, a change to any of them is a bump that refuses old folders with a message, where today it would misread them.A file the log refuses is left as it was. This PR adds the mechanism only. Whether a later format reads older folders or drops them is not decided here, and ADR 0004 is unchanged. One consequence to be aware of: a
log.jsonlwritten before this change has no header and is refused.A stored group that lacks a required field is refused. A group was read back with
model_construct, so a file written when the outputs class had fewer fields gave the added field asNone, without an error, and a binding received it:A request that references such an output fails its record with the same reason. The store reports such a value as one it no longer holds (
LookupError), so callers that handleNotReadablehandle this too. Limits: a group is still not validated again when read (a validator need not give the same value when run on its own result, which an existing test states), so a field whose type changed is not caught, and an added field that has a default reads as that default. The complete answer is spec identity (#80).Client.waitasks only about records that changed. It asked the status of every record after every change. Measured with trivial requests: 1000 records 1.34 s → 0.46 s, 4000 records 14.6 s → 1.9 s (0.40 s and 1.7 s is the time the work takes without waiting).Not included: refusing a pipeline parameter that no output depends on.
PipelineBindingaccepts such a blank on purpose (a test and the module docstring say so), so this needs a decision first.🤖 Generated with Claude Code