Repository navigation
feat: fail a record that was running when its workspace ended - #189
Merged
Merged
Conversation
… run again A workspace started from an existing log ran a pending record again if it persisted outputs. A binding that ends the process while it runs, as an out-of-memory kill does, left its record pending, so the process ended again at every start of the workspace, and no record said why. The log now has a `started` event. A worker appends it when it begins a record's work, once what the record reads is there, before anything is read: for plain requests, calls through a stage, and the record of a freeze alike. A record waiting for another has not started. At the next start, a pending record that started is failed with "the process ended while it ran", and the records that read it fail as readers of a failed record do. Pending records that had not started follow the rules as before: they run if they persist outputs, or persisted work reads them, and are cancelled otherwise. A person submits a failed record's request again to run it. Every record logs `started`, also one that persists nothing: such a record runs again at a restart when persisted work reads it, so it needs the event as well, and one rule for all records is simpler to state. Closing a workspace waits for its pending work, so an orderly stop leaves no record started without finishing. The log format goes from 1 to 2. A folder or log of format 1 is refused with the existing message. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
At a restart, a started record that no persisted work reads was cancelled with "the workspace restarted", so nothing recorded why the process ended, for example after an out-of-memory kill of an interactive compute. The restart now checks `started` first: every record that started and did not finish fails, whatever reads it. The reason is "the workspace ended while it ran", since a workspace also stops without its process ending, when a log append fails. The tests copy the log only once the record is inside its workflow, so they pin down that `started` is written before the binding runs. The docs on restarts, upgrades, and `reduce --remaining` state the rule. `Store.write` no longer promises to replace a value, since no record writes an output twice. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
It wrote a log by hand in which a record finished without having started, and checked that the record reads as finished. That case occurs in the current format for a record cancelled before it ran, and other tests cover it through a real workspace. Written by hand it looked like reading an older log, which is not supported: a file of another format is refused. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
# Conflicts: # packages/essapps/tests/persist_test.py
5 of 7 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A record that was running when its workspace ended is failed at the next start, not run again. Part of step 3 of #184 (what is durable); decided ahead of the ADR because it stands on its own.
The problem. A workspace made with
open_folderkeeps its history inlog.jsonl. A reduction that ends the process, as an out-of-memory kill does, left its record pending. The next open ran it again if it persisted outputs, so the process died at every start, and no record said why.The rule. The log gets a
startedevent, written when the workspace begins a record's work, before any input is read. At the next start, a record that started and did not finish fails:{"kind": "started", "record": "r2"} {"kind": "finished", "record": "r2", "status": "failed", "failure": "the workspace ended while it ran"}The second line is written at the next start. A record that reads it fails as any record that reads a failed record. A pending record that had not started follows the rules as before: it runs if it persists outputs and what it reads is there. A person or the application submits a failed request again;
essapps-batch reduce --remainingdoes so for a batch. The reason stays text, and no kinds of failure are introduced.Log format 2. The format number changes, since the log has a new event. Nothing reads format 1: the log reads only files of its own format and refuses any other, as before this change.
An orderly stop is unaffected. Closing a workspace waits until every record has finished, so nothing is left started. A SIGTERM is not an orderly stop today: nothing in the repository handles the signal, so a process stopped by
docker stopor systemd fails its running records at the next start, where it used to run them again. A service that must upgrade without failing running reductions has to close its workspaces on SIGTERM. System story H2 says so now.Review. A fresh reviewer checked the change with probes and mutations. Fixed from it: a started record that no persisted work reads was cancelled, not failed, which hid the cause in exactly the case of an interactive reduction that ran out of memory; no test noticed when
startedwas written after the workflow returned; the reason said "the process ended", which is wrong when the workspace stops after a failed log write. It found no problem in the moment of the event, restarts of chains, locking, the change feed, or the format refusal.Docs. ADR 0002,
batch.md, system story H2, and one prior-art page stated that running records run again; they state the rule above.Store.writeno longer promises that a write replaces an earlier one, since no record writes twice.This touches
workspace.pyandpersist_test.py, as #188 does; whichever merges second needs a merge ofmain.🤖 Generated with Claude Code