diff --git a/design/workflow-step.md b/design/workflow-step.md index e9ff094298..cef0c1b903 100644 --- a/design/workflow-step.md +++ b/design/workflow-step.md @@ -41,9 +41,30 @@ unless `continue_on_error: true` is set (an explicit abort always stops). The engine calls `validate()` during workflow validation but does not automatically validate a definition passed to `execute()`. Guard invalid configurations in `execute()` too, returning a failed result rather than a -successful default or an unhandled exception. Resume restarts the current -top-level step; a pause inside nested steps re-runs their parent and nested -body. Design side effects accordingly. +successful default or an unhandled exception. + +New runs persist an execution tree. Resume replays completed occurrences into +their expression contexts without calling their implementations, then continues +at the unfinished occurrence. Selected branches, custom expansions, loop +iterations, fan-out items, and bound workflow definitions therefore remain +frozen across resume. This is intentionally forward-only: new resume inputs do +not reinterpret or repeat completed work. Legacy states enter the tree once +through their saved top-level index. + +The common occurrence runner owns start, checkpoint, replay, and finalization. +Step implementations return results or expansions; they do not manage persisted +lifecycle state. Internal workflow-call and registered-step paths share the +same checked transitions and post-checkpoint notifications. Per-occurrence +activity belongs to the execution tree, never to the shared step instance. + +Fan-out items have independent expression contexts. Their internal results do +not enter the shared parent context; the parent receives the qualified item +result and the fan-out step's ordered `output.results`. Fan-out item aliases are +reporting-only; a `fan-in` `wait_for` targets declared step IDs (the fan-out +step's own `id`) and reads its ordered `output.results`, never a generated item +alias. A side effect performed +before its completion checkpoint can still repeat after an interruption, so the +guarantee is at-least-once rather than exactly-once. The registry holds one shared instance per type. Concurrent `fan-out` can invoke that instance from multiple threads: keep execution stateless and diff --git a/docs/reference/workflows.md b/docs/reference/workflows.md index 2ebe4ff5e6..a68e0345c3 100644 --- a/docs/reference/workflows.md +++ b/docs/reference/workflows.md @@ -67,9 +67,17 @@ specify workflow resume | `-i` / `--input` | Updated input values as `key=value` (repeatable) | | `--json` | Emit the resume outcome as a single JSON object | -Resumes a paused or failed workflow run from the exact step where it stopped. Useful after responding to a gate step or fixing an issue that caused a failure. - -Supplied `--input` values are merged over the run's stored inputs and re-validated against the workflow's input types, then the blocked step is re-run with the updated values. This lets a run continue with information that only became available after it paused, or with a corrected value after a failure: +Resumes a paused or failed workflow run from its persisted execution state. A +`running` run is not resumable. Resume replays completed work into its +expression contexts without running step implementations again, then continues +at the unfinished occurrence. This lets a run continue with information that +only became available after it paused, or with a corrected value after a +failure. + +Supplied `--input` values are merged over the run's stored root inputs and +re-validated against the workflow's input types. Unknown root input names are +ignored. Updated values affect only unfinished work; completed steps and calls +are not reinterpreted or repeated: ```bash specify workflow resume --input cmd="exit 0" @@ -579,6 +587,162 @@ specify workflow run speckit -i spec="Build a kanban board with drag-and-drop ta | `do-while` | Execute at least once, then loop on condition | | `fan-out` | Dispatch a step for each item in a list | | `fan-in` | Aggregate results from a fan-out step | +| `workflow` | Call an installed workflow with private inputs and declared outputs | + +### Workflow composition + +A `workflow` step executes an installed, enabled workflow in the current project +as a private scope of the same run. Targets can be literal IDs or expressions; +the resolved string must match the ID exactly, including case and whitespace. + +```yaml +inputs: + target: {type: string, required: true} + report: {type: string, required: true} +steps: + - id: investigate + type: workflow + workflow: "{{ inputs.target }}" + input: + report: "{{ inputs.report }}" +``` + +The included workflow sees only declared inputs passed through `input` and its +own step results. It does not inherit the caller's `inputs`, step results, +fan-out `item`/`fan_in` values, or workflow defaults. Unknown input names, +missing required values, and invalid types/enums fail the call. Existing child +defaults and `integration: auto` resolution apply. Values cross back only +through explicit declarations: + +```yaml +outputs: + report: + value: "{{ steps.analyze.output.stdout }}" +``` + +The caller reads `{{ steps.investigate.output.report }}`. Output includes +`workflow` and `status`; failures include `error` when the failed operation +reported one, and an abort includes `aborted: true`. These names and +`integration`, `model`, `options`, and `input` are reserved. Returned values +must be JSON-safe. Private inputs, step records, and logs are not part of the +return value. No separate child run is created. + +`continue_on_error: true` on a workflow call handles a returned child failure +and call-boundary contract failures, such as an unavailable target, invalid +mapped input, cycle, depth limit, or invalid declared output. It does not catch +step exceptions, expression errors (including `from_json` errors in `input:` or +`outputs:`), interruptions, or checkpoint failures; those propagate exactly as +they do at the root. An unavailable step implementation is terminal where the +step occurs; at a workflow call it is a reported child failure, whether it is +detected while binding the target or later. Pauses and explicit aborts always +stop execution. + +Nested composition is allowed, but repeated workflow IDs on the active call +path are cycles. Diamonds are allowed. The maximum included depth is 16, with +the root at depth zero. + +### Calling a Child Gate + +Only root-declared inputs can be supplied to `workflow resume --input`. Map a +root input through each workflow boundary when a child gate uses it as a +`verdict_input`: + +```yaml +# Parent workflow +inputs: + approval: + type: string + default: "" +steps: + - id: review-release + type: workflow + workflow: release-notes + input: + approval: "{{ inputs.approval }}" +``` + +```yaml +# Installed release-notes workflow +inputs: + approval: + type: string + default: "" +steps: + - id: review + type: gate + message: "Approve the release notes?" + options: [approve, reject] + verdict_input: approval +``` + +The initial run pauses if `approval` is empty. Its structured status identifies +the nested gate and its enclosing call with `gate.scope_path`. Resume the root +run, not the child, to continue it: + +```bash +specify workflow status --json +specify workflow resume --input approval=approve +``` + +A gate with `verdict_input` is not supported inside a fan-out item, including +through one or more workflow calls. The `inside_fan_out` runtime condition is +preserved across workflow boundaries. + +### Defaults and Errors at the Call Boundary + +Parent `integration`, `model`, and `options` defaults apply only to steps in +the parent workflow. A child uses its own defaults or automatic resolution. To +make a value common to both, declare it as a child input and map it explicitly. +The call result intentionally has an empty `input` field and does not expose +the child's private inputs, step records, or logs. + +An initial binding or output-finalization contract failure can be handled with +`continue_on_error`. A failure while rebinding an incomplete call during +`workflow resume --input` is different: it propagates, leaves the call and its +child subtree unchanged, and can be retried with corrected root inputs. + +### Execution identity and resume + +Each step occurrence owns a record in a persisted execution tree. An authored +step ID is a local expression alias, not a global execution ID. Fan-out items +have independent alias contexts and ordered item results. Public reporting uses +qualified occurrence IDs where needed, for example `fan:template:0` for a +fan-out item and `loop:step:1` for a later loop iteration. Qualified IDs are +not expression names. Fan-out item aliases are reporting-only: a `fan-in` +`wait_for` references declared step IDs, in particular the fan-out step's own +`id`, whose ordered item results are available as +`steps..output.results`. + +New runs persist selected branches, dynamic custom-step expansions, loop +iterations, and fan-out items. Resume retains completed work without +reevaluating already selected branches. Workflow targets and overlay-resolved +definitions remain bound even if installations change. An unbound call still +checks that its target is installed and enabled when execution reaches it. + +Ordinary resume retains bound inputs. Explicit `--input` rebinds reached, +incomplete calls through their original mappings: newly mapped values override +the prior binding, while values not mapped again retain their bound values. +Completed calls retain their results. Failed output evaluation retries only +finalization, without repeating completed child commands. + +Snapshots are stored as YAML strings inside the private JSON execution tree, +preserving YAML scalar types. Inputs and results remain JSON values. The +snapshot freezes workflow definitions and expansions, not files under +`context.workflow_dir` or step implementations; missing resources can still +cause ordinary step failures. Legacy runs without a tree enter through their +saved root index, then use tree-backed resume. Resume is available only for +runs in the `paused` or `failed` state. + +`current_step_index` is the root-sequence index; `current_step_id` is the +occurrence ID of the active leaf, matching the event `step_id` (for example +`fan:item:0` for a fan-out item or `loop:body:1` for a later loop iteration). +Inside a workflow call, IDs are relative to the called workflow. Structured +run/status output includes `workflow_scopes` +summaries when calls exist and reports an active nested gate with its +`scope_path`. Private log events add `workflow_id` and `execution_path` to the +qualified `step_id`. A step emits completion after its checkpoint; containers +emit before their children, while replayed completed occurrences emit no step +events. > **Security note:** a `shell` step runs a local command with **your** privileges. There is no capability sandbox — `requires` is an advisory pre-condition block (spec-kit version, integrations), not a runtime gate, so it does **not** restrict what a step can do. In particular there is no `requires.permissions` capability gate: it is rejected by validation precisely because it would imply a sandbox that does not exist. Review any catalog or downloaded workflow before running it, and use a `gate` step to require explicit approval before sensitive or destructive shell commands. @@ -692,7 +856,9 @@ Each workflow run persists its state at `.specify/workflows/runs//`: - `inputs.json` — resolved input values - `log.jsonl` — step-by-step execution log -This enables `specify workflow resume` to continue from the exact step where a run was paused (e.g., at a gate) or failed. +This enables `specify workflow resume` to replay completed occurrences and +continue at the unfinished occurrence where a run paused (for example, at a +gate) or failed. ### Gate Verdict Inputs diff --git a/src/specify_cli/workflows/__init__.py b/src/specify_cli/workflows/__init__.py index 2bb3de56a5..dca51280e5 100644 --- a/src/specify_cli/workflows/__init__.py +++ b/src/specify_cli/workflows/__init__.py @@ -49,6 +49,7 @@ def _register_builtin_steps() -> None: from .step.fan_in import FanInStep from .step.fan_out import FanOutStep from .step.gate import GateStep + from .step.workflow import WorkflowStep from .step.if_then import IfThenStep from .step.init import InitStep from .step.prompt import PromptStep @@ -62,6 +63,7 @@ def _register_builtin_steps() -> None: _register_step(FanInStep()) _register_step(FanOutStep()) _register_step(GateStep()) + _register_step(WorkflowStep()) _register_step(IfThenStep()) _register_step(InitStep()) _register_step(PromptStep()) diff --git a/src/specify_cli/workflows/_commands.py b/src/specify_cli/workflows/_commands.py index 241bddb775..925456f021 100644 --- a/src/specify_cli/workflows/_commands.py +++ b/src/specify_cli/workflows/_commands.py @@ -947,6 +947,12 @@ def _workflow_run_payload(state: Any) -> dict[str, Any]: error = _failed_step_error(state) if error is not None: payload["error"] = error + if getattr(state, "execution", None): + from ._execution import scope_summaries + + scopes = scope_summaries(state.execution, state.status.value) + if scopes: + payload["workflow_scopes"] = scopes return payload @@ -985,21 +991,35 @@ def _gate_outcome(state: Any) -> dict[str, Any] | None: if getattr(state.status, "value", state.status) not in ("paused", "aborted"): return None step = (getattr(state, "step_results", None) or {}).get(state.current_step_id) + step_id = state.current_step_id + scope_path = None + if getattr(state, "execution", None): + from ._execution import active_step + + active = active_step(state.execution) + if active is None: + return None + path, node, step_id = active + scope_path = path[:-1] + step = node.get("result") if not isinstance(step, dict) or not _is_gate_step(step): return None output = step.get("output") or {} - # `message`, `options`, and `choice` may be non-string YAML literals in an - # unvalidated workflow (GateStep coerces none of them for the payload), so + # `message`, `options`, and `choice` may be non-string YAML literals in + # legacy or synthetic records, so # normalise all three for a stable JSON schema: message → str, options → # list[str] | None, choice → str | None (None means no decision yet). message = output.get("message") choice = output.get("choice") - return { - "step_id": state.current_step_id, + detail = { + "step_id": step_id, "message": None if message is None else str(message), "options": _normalize_gate_options(output.get("options")), "choice": None if choice is None else str(choice), } + if scope_path: + detail["scope_path"] = scope_path + return detail def _normalize_gate_options(options: Any) -> list[str] | None: diff --git a/src/specify_cli/workflows/_execution.py b/src/specify_cli/workflows/_execution.py new file mode 100644 index 0000000000..f4a100142d --- /dev/null +++ b/src/specify_cli/workflows/_execution.py @@ -0,0 +1,948 @@ +"""A persisted execution tree, traversed identically on first execution and resume. + +An occurrence owns its result and descendants. Authored names are only aliases +in an expression context; tree positions distinguish repeated executions. +""" + +from __future__ import annotations + +from concurrent.futures import ThreadPoolExecutor +from copy import deepcopy +from dataclasses import dataclass, replace +import threading +from typing import Any + +import yaml + +from .base import StepContext, StepResult, StepStatus +from .composition import ( + CallError, + bind_inputs, + evaluate_outputs, + resolve_target, + validate_runtime_call, +) +from .expressions import evaluate_condition, evaluate_expression + +HALTING = {"paused", "failed", "aborted"} +EXECUTION_VERSION = 2 + +def unknown_step_error(kind): + return f"Unknown step type: {kind!r}" + + +class CheckpointError(RuntimeError): + """Persistence failed; reload the authoritative disk checkpoint before retrying.""" + + +def sequence(steps: list[dict[str, Any]]) -> dict[str, Any]: + return { + "source": yaml.safe_dump(steps, sort_keys=False), + "nodes": [{"phase": "ready"} for _ in steps], + } + + +def occurrences(steps: list[dict[str, Any]]) -> dict[str, Any]: + """Create occurrence storage that shares its parent's immutable source.""" + return {"nodes": [{"phase": "ready"} for _ in steps]} + + +def new_execution( + steps: list[dict[str, Any]], offset: int, initial: dict[str, Any] +) -> dict[str, Any]: + """Create the persisted root execution tree.""" + return { + "version": EXECUTION_VERSION, + "sequence": sequence(steps), + "offset": max(0, offset), + "initial": deepcopy(initial), + } + + +def steps_of(seq: dict[str, Any]) -> Any: + """Read the persisted YAML source for one execution sequence.""" + return yaml.safe_load(seq["source"]) + + +LOOP_TYPES = frozenset({"while", "do-while"}) +TRANSITIONS = { + "begin": (frozenset({"ready", "blocked", "children", "outputs"}), frozenset()), + "expand": (frozenset({"ready", "blocked"}), frozenset({"result", "children", "template"})), + "bind": (frozenset({"ready", "blocked"}), frozenset({"binding", "children"})), + "rebind": (frozenset({"children", "blocked", "outputs"}), frozenset({"binding"})), + "iterate": (frozenset({"children"}), frozenset({"children"})), + "outputs": (frozenset({"children", "blocked", "outputs"}), frozenset()), + "finish": (frozenset({"ready", "blocked", "children", "outputs"}), frozenset({"result", "outcome", "error"})), + "settle": (frozenset({"children"}), frozenset({"outcome", "error", "fan_results"})), + "leave": (frozenset({"ready", "blocked", "children", "outputs", "done"}), frozenset()), +} + + +def shares_source(kind, index): + """Whether child ``index`` reads its steps from the parent occurrence.""" + return kind in {"workflow", "fan-out"} or (kind in LOOP_TYPES and index > 0) + + +def child_steps(config, node, index): + """Read a child sequence from its position in the parent occurrence.""" + kind = config.get("type") + if not shares_source(kind, index): + return steps_of(node["children"][index]) + if kind == "workflow": + return yaml.safe_load(node["binding"]["definition"])["steps"] + if kind == "fan-out": + template = yaml.safe_load(node["template"]) + return [{"id": "item", **template}] + return steps_of(node["children"][0]) + + +def child_path(config, path, item): + """Return a child's reporting path from its parent occurrence.""" + return ( + [*path, str(item)] + if config.get("type") in {"fan-out"} | LOOP_TYPES + else path + ) + + +def walk_execution( + seq, + path=(), + *, + children_first=False, + skip_done=False, + steps=None, + loop_alias=None, +): + """Yield execution occurrences with one shared path and ID rule. + + Each item is ``(config, node, path, step_id)``; ``step_id`` is the + occurrence ID used by events, relative to the enclosing workflow scope. + """ + steps = steps_of(seq) if steps is None else steps + for index, (config, node) in enumerate(zip(steps, seq["nodes"])): + if skip_done and node["phase"] == "done": + continue + name = config.get("id", f"step-{index}") + here = [*path, name] + qualified = qualified_id(name, loop_alias) + kind = config.get("type", "command") + if not children_first: + yield config, node, here, qualified + for item, child in enumerate(node.get("children", [])): + yield from walk_execution( + child, + child_path(config, here, item), + children_first=children_first, + skip_done=skip_done, + steps=child_steps(config, node, item), + loop_alias=( + (qualified, item) + if kind == "fan-out" + else loop_alias_for(kind, qualified, item) + ), + ) + if children_first: + yield config, node, here, qualified + + +def check_sequence(seq, steps=None, *, shared=False): + """Walk a stored sequence, checking every occurrence and its descendants.""" + if not isinstance(seq, dict) or (shared and "source" in seq): + raise ValueError("Invalid execution sequence") + if shared: + if not isinstance(steps, list): + raise ValueError("Invalid shared execution source") + else: + if not isinstance(seq.get("source"), str): + raise ValueError("Invalid execution sequence") + steps = steps_of(seq) + nodes = seq.get("nodes") + if ( + not isinstance(steps, list) + or not all(isinstance(s, dict) for s in steps) + or not isinstance(nodes, list) + or len(nodes) != len(steps) + ): + raise ValueError("Invalid execution sequence length or steps") + for step, node in zip(steps, nodes): + check_node(step, node) + for index, child in enumerate(node.get("children", [])): + if shares_source(step.get("type"), index): + check_sequence(child, child_steps(step, node, index), shared=True) + else: + check_sequence(child) + + +def check_node(step, node, *, changed=None, new_children=()): + """Validate one occurrence and its direct children without walking descendants. + + On a transition, unchanged YAML sources and existing child shapes were + checked when they were created or loaded. Load checks everything. + """ + if not isinstance(node, dict) or node.get("phase") not in { + "ready", "children", "outputs", "blocked", "done", + }: + raise ValueError("Invalid execution phase") + result = node.get("result") + if result is not None and ( + not isinstance(result, dict) + or result.get("status") not in {s.value for s in StepStatus} + or not isinstance(result.get("output"), dict) + ): + raise ValueError("Invalid execution result") + if node["phase"] == "done" and result is None: + raise ValueError("Completed execution lacks a result") + if "active" in node and type(node["active"]) is not bool: + raise ValueError("Invalid execution activity") + if node["phase"] == "done" and node.get("active"): + raise ValueError("Completed execution is still active") + if node.get("outcome", "completed") not in {"completed", *HALTING}: + raise ValueError("Invalid execution outcome") + if node["phase"] == "done" and node.get("outcome", "completed") != "completed": + raise ValueError("Completed execution has a blocking outcome") + if node["phase"] == "blocked" and ( + result is None or result["status"] not in {"failed", "paused"} + or node.get("outcome") not in HALTING + ): + raise ValueError("Blocked execution lacks a blocking result") + children = node.get("children", []) + if not isinstance(children, list): + raise ValueError("Invalid execution children") + kind = step.get("type") + if "fan_results" in node and ( + kind != "fan-out" or not isinstance(node["fan_results"], list) + ): + raise ValueError("Invalid fan-out aggregate results") + if "template" in node and (kind != "fan-out" or not isinstance(node["template"], str)): + raise ValueError("Invalid fan-out template") + binding = node.get("binding") + if binding is not None: + if kind != "workflow" or not isinstance(binding, dict): + raise ValueError("Invalid workflow binding") + if changed is None or "binding" in changed: + source = binding.get("definition") + if not isinstance(source, str): + raise ValueError("Invalid bound workflow definition or inputs") + bound_definition = yaml.safe_load(source) + if ( + not isinstance(bound_definition, dict) + or not isinstance(bound_definition.get("workflow"), dict) + or bound_definition["workflow"].get("id") != binding.get("workflow") + or not isinstance(binding.get("inputs"), dict) + or "workflow_dir" not in binding + or not isinstance(binding["workflow_dir"], (str, type(None))) + or len(children) != 1 + or not isinstance(bound_definition.get("steps"), list) + ): + raise ValueError("Invalid bound workflow definition or inputs") + elif kind == "workflow" and children: + raise ValueError("Workflow children require a binding") + if kind == "fan-out" and children and (changed is None or "template" in changed): + source = node.get("template") + template = yaml.safe_load(source) if isinstance(source, str) else None + if not isinstance(template, dict): + raise ValueError("Invalid fan-out template") + if changed is None or new_children: + shared_steps = None + for index, child in enumerate(children): + if changed is not None and not any(child is fresh for fresh in new_children): + continue + if shares_source(kind, index): + if shared_steps is None: + shared_steps = child_steps(step, node, index) + check_sequence_shape(child, shared_steps, shared=True) + else: + check_sequence_shape(child) + handled_child_failure = ( + kind == "workflow" and step.get("continue_on_error") is True + and node.get("outcome") == "completed" and result is not None + and result["status"] == "failed" + ) + if node["phase"] == "done" and not handled_child_failure: + for child in children: + if any(nested["phase"] != "done" for nested in child["nodes"]): + raise ValueError("Completed execution has unfinished children") + if node["phase"] == "outputs" and binding is None: + raise ValueError("Output finalization requires a workflow binding") + if node["phase"] == "children" and ( + not children or (binding is None and result is None) + ): + raise ValueError("Expanded execution lacks its children or result") + if node["phase"] == "ready" and (result is not None or children or binding): + raise ValueError("Unstarted execution already has progress") + if kind == "fan-out" and children: + items = (result or {}).get("output", {}).get("items") + if not isinstance(items, list) or len(items) != len(children): + raise ValueError("Fan-out items do not match execution children") + + +def check_sequence_shape(seq, steps=None, *, shared=False): + """Check a newly attached child sequence without visiting its nodes' descendants.""" + if not isinstance(seq, dict) or (shared and "source" in seq): + raise ValueError("Invalid execution sequence") + if shared: + if not isinstance(steps, list): + raise ValueError("Invalid shared execution source") + else: + if not isinstance(seq.get("source"), str): + raise ValueError("Invalid execution sequence") + steps = steps_of(seq) + nodes = seq.get("nodes") + if ( + not isinstance(steps, list) or not all(isinstance(s, dict) for s in steps) + or not isinstance(nodes, list) or len(nodes) != len(steps) + ): + raise ValueError("Invalid execution sequence length or steps") + + +def validate_execution(tree: Any, *, workflow_steps=None, current_step_index=None) -> None: + """Validate stored structure without importing a project's custom steps.""" + + try: + if not isinstance(tree, dict) or tree.get("version") != EXECUTION_VERSION: + raise ValueError("Unsupported execution version") + offset = tree.get("offset", 0) + if ( + type(offset) is not int + or offset < 0 + or not isinstance(tree.get("initial", {}), dict) + ): + raise ValueError("Invalid execution offset or initial aliases") + check_sequence(tree["sequence"]) + if workflow_steps is not None: + if offset >= len(workflow_steps) or offset > current_step_index: + raise ValueError("Invalid execution offset for workflow position") + if steps_of(tree["sequence"]) != workflow_steps[offset:]: + raise ValueError("Invalid execution state: root sequence differs from workflow snapshot") + except (KeyError, TypeError, yaml.YAMLError, RecursionError) as exc: + raise ValueError(f"Invalid execution state: {exc}") from exc + + +def active_step(tree): + """First unfinished leaf in execution order, including nested workflow scopes. + + Returns ``(path, node, step_id)``. + """ + for _, node, path, step_id in walk_execution( + tree["sequence"], children_first=True, skip_done=True + ): + return path, node, step_id + return None + + +def scope_summaries(tree, run_status="running"): + """Report workflow boundaries without exposing private inputs or results. + + Only an unfinished scope containing the active occurrence inherits a + terminal run status; completed scopes retain their recorded result. + """ + summaries = [] + active = active_step(tree) + active_path = active[0] if active is not None else () + for _, node, path, _ in walk_execution(tree["sequence"]): + binding = node.get("binding") + if binding: + output = node.get("result", {}).get("output", {}) + status = output.get("status") + if status is None: + status = ( + run_status + if run_status in HALTING and active_path[:len(path)] == path + else "running" + ) + summaries.append( + { + "scope_path": path, + "workflow_id": binding["workflow"], + "status": status, + } + ) + return summaries + + +def child_context_for_call(parent, binding, definition): + """Create the private context for one bound workflow call.""" + return StepContext( + inputs=binding["inputs"], + project_root=parent.project_root, + run_id=parent.run_id, + is_resume=parent.is_resume, + inside_fan_out=parent.inside_fan_out, + workflow_dir=binding["workflow_dir"], + default_integration=definition.default_integration, + default_model=definition.default_model, + default_options=definition.default_options, + ) + + +def qualified_id(name, loop_alias=None): + """Return the public occurrence ID of a loop-iteration or fan-out-item step.""" + if loop_alias is None: + return name + return f"{loop_alias[0]}:{name}:{loop_alias[1]}" + + +def loop_alias_for(kind, qualified, iteration): + """Qualify direct body steps of later loop iterations (4.6).""" + return (qualified, iteration) if kind in LOOP_TYPES and iteration else None + + +def result_view(node): + """Public result, distinct from the frozen result used to execute children.""" + result = node["result"] + if "fan_results" in node: + return {**result, "output": {**result["output"], "results": node["fan_results"]}} + return result + + +@dataclass(frozen=True) +class Occurrence: + """One invocation, never stored on a shared step implementation.""" + + config: dict + node: dict + context: StepContext + ancestry: tuple + path: tuple + public: bool + qualified: str + + +@dataclass(frozen=True) +class SubtreeResult: + outcome: str + error: str | None + outputs: list | None = None + + +@dataclass(frozen=True) +class MissingImplementation: + error: str + + +class Execution: + def __init__(self, engine, state, registry, *, rebind=False): + self.engine, self.state, self.registry = engine, state, registry + self.rebind = rebind + + def project(self, context, name, result, *, public, qualified): + """Apply an occurrence result to its local and public views.""" + context.steps[name] = result + if public: + self.state.step_results[name] = result + if qualified != name: + context.steps[qualified] = result + self.state.step_results[qualified] = result + + def project_alias(self, context, name, result): + """Project a fan-out item result into its enclosing workflow scope.""" + context.steps[name] = result + + def transition( + self, + operation, + occurrence, + changes=None, + *, + publish=False, + announce=True, + ): + """The only writer of occurrence progress and its checkpointed views. + + Phase is the continuation point; active marks entry into that continuation. + Own-step results and subtree outcomes deliberately remain distinct. + Validate the candidate before changing memory or writing anything. + """ + node = occurrence.node + with self.state._lock: + if self.state._checkpoint_failed: + raise CheckpointError("A previous checkpoint failed") + phases, fields = TRANSITIONS[operation] + if node["phase"] not in phases: + raise ValueError(f"Invalid {operation} transition from {node['phase']}") + if set(changes or {}) - fields: + raise ValueError(f"Invalid fields for {operation} transition") + candidate = {**node, **(changes or {})} + if operation in {"expand", "bind"}: + candidate["phase"] = "children" + elif operation == "outputs": + candidate["phase"] = "outputs" + elif operation in {"finish", "settle"}: + candidate["phase"] = ( + "done" if candidate["outcome"] == "completed" + else "blocked" if operation == "finish" else "children" + ) + if operation == "begin": + candidate["active"] = True + elif operation in {"finish", "settle", "leave"}: + candidate["active"] = False + previous = node.get("children", []) + new_children = () + if "children" in (changes or {}): + existing = {id(child) for child in previous} + new_children = tuple( + child for child in candidate["children"] if id(child) not in existing + ) + # The same node rules are checked when a checkpoint is loaded. + check_node(occurrence.config, candidate, changed=set(changes or ()), + new_children=new_children) + node.update(candidate) + if operation == "begin": + self.state.current_step_id = occurrence.qualified + if publish and "result" in node: + self.project( + occurrence.context, + occurrence.config["id"], + result_view(node), + public=occurrence.public, + qualified=occurrence.qualified, + ) + # Leave is memory-only and best-effort; the run-level handler + # checkpoints the unwound tree on an exception. + # Do not insert a second checkpoint between completion and its event. + if operation != "leave": + self.state.save() + if announce: + self.notify(operation, occurrence, publish=publish) + + def notify(self, operation, occurrence, *, publish): + """Post-checkpoint notifications; never responsible for persistence.""" + config, node = occurrence.config, occurrence.node + kind = config.get("type", "command") + args = occurrence.qualified, occurrence.path, occurrence.ancestry + if operation == "begin": + label = kind if kind == "workflow" else config.get("command", "") or kind + self.emit("step_started", *args, type=kind, callback_label=label) + elif operation in {"expand", "finish"}: + result = node["result"] + if publish: + self.emit("step_completed", *args, status=result["status"]) + if result["status"] == "failed": + event = { + "aborted": "workflow_aborted", + "completed": "step_continue_on_error", + }.get(node.get("outcome"), "step_failed") + self.emit(event, *args, error=result.get("error")) + + def emit( + self, + event, + qualified, + path, + ancestry, + *, + callback_label=None, + **fields, + ): + """Emit one step event and its optional start callback.""" + entry = {"event": event, "step_id": qualified, **fields} + if len(ancestry) > 1: + entry.update(execution_path=list(path), workflow_id=ancestry[-1]) + self.state.append_log(entry) + if callback_label is not None and self.engine.on_step_start is not None: + with self.engine._callback_lock: + self.engine.on_step_start(qualified, callback_label) + + def run( + self, + seq, + context, + ancestry, + *, + path=(), + public=True, + root=False, + loop_alias=None, + steps=None, + ): + steps = steps_of(seq) if steps is None else steps + for index, (config, node) in enumerate(zip(steps, seq["nodes"])): + config = {"id": f"step-{index}", **config} + name = config["id"] + occurrence = (*path, index) + qualified = qualified_id(name, loop_alias) + if root and node["phase"] != "done": + with self.state._lock: + self.state.current_step_index = index + self.state.execution.get( + "offset", 0 + ) + outcome = self.step( + config, + node, + context, + ancestry, + occurrence, + public, + qualified, + ) + if outcome in HALTING: + return outcome, node.get("error") + return "completed", None + + def run_children(self, config, node, context, ancestry, path, public, qualified): + """Run or replay persisted child sequences in order.""" + kind = config.get("type", "command") + for iteration, child in enumerate(node.get("children", [])): + outcome, error = self.run( + child, + context, + ancestry, + path=(*path, iteration), + public=public, + loop_alias=loop_alias_for(kind, qualified, iteration), + steps=child_steps(config, node, iteration), + ) + if outcome in HALTING: + return outcome, error + return "completed", None + + def step(self, config, node, context, ancestry, path, public, qualified): + name = config.get("id", "step-0") + kind = config.get("type", "command") + if node["phase"] == "done": + self.project( + context, name, result_view(node), public=public, qualified=qualified + ) + if kind == "fan-out" and node.get("children"): + self.fan_out( + config, + node, + context, + ancestry, + path, + public, + qualified, + sequential=True, + ) + elif kind not in {"workflow", "fan-out"}: + self.run_children( + config, node, context, ancestry, path, public, qualified + ) + return "completed" + if node.get("outcome") == "aborted": + # Replay the result the aborted occurrence projected when it ran, so + # its fan-out item keeps its published output (see run_item). + if node.get("result") is not None: + self.project( + context, name, result_view(node), public=public, qualified=qualified + ) + return "aborted" + + occurrence = Occurrence(config, node, context, ancestry, path, public, qualified) + calls = kind == "workflow" + try: + # Workflow calls are announced on every entry, including resumed bound calls. + self.transition( + "begin", occurrence, + announce=node["phase"] in {"ready", "blocked"} or calls, + ) + result = self.workflow(occurrence) if calls else self.execute_step(occurrence) + return self.finish(occurrence, result) + except BaseException: + if node.get("active"): + try: + self.transition("leave", occurrence) + except Exception: + pass # A crash-shaped active node is valid on resume. + raise + + def execute_step(self, occurrence): + config, node, context = occurrence.config, occurrence.node, occurrence.context + ancestry, path = occurrence.ancestry, occurrence.path + public, qualified = occurrence.public, occurrence.qualified + name, kind = config["id"], config.get("type", "command") + + if node["phase"] in {"ready", "blocked"}: + impl = self.registry.get(kind) + if impl is None: + # As on main: terminal, only step_failed, no projected result. + # The node keeps its result so resume can retry after reinstalling. + error = unknown_step_error(kind) + return MissingImplementation(error) + result = impl.execute(config, context) + expansion = {} + if kind == "fan-out": + # Definitions belong to the tree, never to JSON result records. + output = dict(result.output) + template = output.pop("step_template", {}) + expansion["template"] = yaml.safe_dump(template, sort_keys=False) + result = replace(result, output=output) + if result.status in {StepStatus.FAILED, StepStatus.PAUSED}: + return result + children = [sequence(result.next_steps)] if result.next_steps else [] + if kind == "fan-out": + children = ( + [occurrences([template]) for _ in result.output.get("items", [])] + if template + else [] + ) + data = self.record(config, result, context) + if not children: + if kind == "fan-out": + result.output = {**result.output, "results": []} + return result + self.transition( + "expand", occurrence, + {"result": data, "children": children, **expansion}, + publish=True, + ) + else: + self.project( + context, name, result_view(node), public=public, qualified=qualified + ) + + if kind == "fan-out": + outcome, error, outputs = self.fan_out( + config, node, context, ancestry, path, public, qualified + ) + else: + outcome, error = self.run_children( + config, node, context, ancestry, path, public, qualified + ) + if outcome == "completed" and kind in LOOP_TYPES: + limit = config.get("max_iterations", 10) + if isinstance(limit, bool) or not isinstance(limit, int) or limit < 1: + limit = 10 + while len(node["children"]) < limit and evaluate_condition( + config.get("condition", False), context + ): + child = occurrences(steps_of(node["children"][0])) + self.transition("iterate", occurrence, {"children": [*node["children"], child]}) + outcome, error = self.run( + child, + context, + ancestry, + path=(*path, len(node["children"]) - 1), + public=public, + loop_alias=loop_alias_for( + kind, qualified, len(node["children"]) - 1 + ), + steps=child_steps(config, node, len(node["children"]) - 1), + ) + if outcome in HALTING: + break + return SubtreeResult(outcome, error, outputs if kind == "fan-out" else None) + + @staticmethod + def record(config, result, context): + output = result.output + data = { + "type": config.get("type", "command"), + "integration": output.get("integration") + or config.get("integration") + or context.default_integration, + "model": output.get("model") + or config.get("model") + or context.default_model, + "options": output.get("options") or config.get("options", {}), + "input": output.get("input") or config.get("input", {}), + "output": output, + "status": result.status.value, + "error": result.error, + } + if config.get("type") == "workflow": + # A call result exposes neither caller defaults nor private inputs (4.3). + data.update(integration=None, model=None, options={}, input={}) + if data["type"] == "command" and "integration_args" in output: + data.update( + integration_args=output["integration_args"], + integration_options=output["integration_options"], + ) + return data + + def finish(self, occurrence, result): + config, context = occurrence.config, occurrence.context + if isinstance(result, SubtreeResult): + self.transition( + "settle", occurrence, + {"outcome": result.outcome, "error": result.error, + **({"fan_results": result.outputs} if result.outputs is not None else {})}, + publish=result.outputs is not None, + ) + return result.outcome + missing = isinstance(result, MissingImplementation) + if missing: + result = StepResult(StepStatus.FAILED, error=result.error) + outcome = "completed" + if result.status == StepStatus.PAUSED: + outcome = "paused" + elif result.status == StepStatus.FAILED: + outcome = "aborted" if result.output.get("aborted") else "failed" + if not missing and outcome == "failed" and config.get("continue_on_error") is True: + outcome = "completed" + self.transition( + "finish", occurrence, + { + "result": self.record(config, result, context), + "outcome": outcome, + "error": result.error, + }, + publish=not missing, + ) + return outcome + + def workflow(self, occurrence): + from .engine import WorkflowDefinition, workflow_dir_for + + config, node, context = occurrence.config, occurrence.node, occurrence.context + ancestry, path = occurrence.ancestry, occurrence.path + binding = node.get("binding") + target = binding["workflow"] if binding else config.get("workflow") + if binding is None: + try: + validate_runtime_call(config) + target = evaluate_expression(target, context) + definition = resolve_target(self.state.project_root, target, ancestry) + binding = { + "workflow": target, + "definition": yaml.safe_dump(definition.data, sort_keys=False), + "inputs": bind_inputs(self.engine, definition, config, context), + "workflow_dir": workflow_dir_for(definition), + } + self.transition( + "bind", occurrence, + { + "binding": binding, + "children": [occurrences(definition.steps)], + }, + ) + except CallError as exc: + target = target if isinstance(target, str) else repr(target) + return StepResult( + StepStatus.FAILED, + output={"workflow": target, "status": "failed", "error": str(exc)}, + error=str(exc), + ) + else: + definition = WorkflowDefinition(yaml.safe_load(binding["definition"])) + if self.rebind: + inputs = bind_inputs( + self.engine, definition, config, context, binding["inputs"] + ) + binding = {**binding, "inputs": inputs} + self.transition("rebind", occurrence, {"binding": binding}) + child_context = child_context_for_call(context, binding, definition) + outcome, error = self.run( + node["children"][0], + child_context, + (*ancestry, target), + path=(*path, "workflow"), + public=False, + steps=child_steps(config, node, 0), + ) + output = {"workflow": target, "status": outcome} + if outcome == "completed": + self.transition("outputs", occurrence) + try: + output.update(evaluate_outputs(definition, child_context)) + except CallError as exc: + output.update(status="failed", error=str(exc)) + return StepResult(StepStatus.FAILED, output=output, error=str(exc)) + elif outcome == "aborted": + output["aborted"] = True + if error is not None: + output["error"] = error + status = ( + StepStatus.COMPLETED + if outcome == "completed" + else StepStatus.PAUSED + if outcome == "paused" + else StepStatus.FAILED + ) + return StepResult(status, output=output, error=error) + + def fan_out( + self, config, node, context, ancestry, path, public, qualified, *, sequential=False + ): + output = node["result"]["output"] + items = output.get("items", []) + try: + workers = max(1, int(output.get("max_concurrency", 1))) + except (TypeError, ValueError, OverflowError): + workers = 1 + if sequential: + workers = 1 + workers = min(workers, len(items)) + initial = deepcopy(context.steps) + # Children always consume the frozen expansion result. Aggregate outputs + # belong to the reporting view, never to the item execution context. + for key in {config.get("id", "step-0"), qualified}: + if key in initial: + initial[key] = deepcopy(node["result"]) + halted = threading.Event() + template_name = child_steps(config, node, 0)[0]["id"] + + def run_item(index): + local = replace( + context, steps=deepcopy(initial), item=items[index], inside_fan_out=True + ) + local_name = template_name + inherited = local.steps.get(local_name) + child = node["children"][index] + outcome, error = self.run( + child, + local, + ancestry, + path=(*path, "item", index), + public=False, + loop_alias=(qualified, index), + steps=child_steps(config, node, index), + ) + if outcome in HALTING: + halted.set() + record = child["nodes"][0].get("result") + if record is not None: + record = result_view(child["nodes"][0]) + # Expose only results projected by the item traversal. Missing step + # implementations keep an internal retry record but publish nothing. + if local.steps.get(local_name) is inherited: + record = None + if record is not None: + # Projection only (E2): persisted by the next commit, rebuilt on replay. + with self.state._lock: + self.project_alias( + context, + f"{qualified}:{template_name}:{index}", + record, + ) + if public: + self.state.step_results[ + f"{qualified}:{template_name}:{index}" + ] = record + return outcome, error, (record or {}).get("output", {}) + + results = [] + if workers <= 1: + for index in range(len(items)): + outcome, error, value = run_item(index) + results.append(value) + if outcome in HALTING: + return outcome, error, results + return "completed", None, results + + def run_item_guarded(index): + try: + return run_item(index) + except BaseException: + halted.set() + raise + + with ThreadPoolExecutor(max_workers=workers) as pool: + futures = {i: pool.submit(run_item_guarded, i) for i in range(workers)} + for index in range(len(items)): + try: + outcome, error, value = futures.pop(index).result() + except BaseException: + for future in futures.values(): + future.cancel() + raise + results.append(value) + if outcome in HALTING: + for future in futures.values(): + future.cancel() + return outcome, error, results + following = index + workers + if following < len(items) and not halted.is_set(): + futures[following] = pool.submit(run_item_guarded, following) + return "completed", None, results diff --git a/src/specify_cli/workflows/command_resume.py b/src/specify_cli/workflows/command_resume.py index 3be24f9582..585d531aec 100644 --- a/src/specify_cli/workflows/command_resume.py +++ b/src/specify_cli/workflows/command_resume.py @@ -20,6 +20,7 @@ def workflow_resume( ): """Resume a paused or failed workflow run.""" from . import load_custom_steps + from ._execution import CheckpointError from .engine import RunState, WorkflowEngine project_root = cli._require_specify_project() @@ -76,7 +77,7 @@ def workflow_resume( pre_state.installed_origin_tracked = True try: pre_state.save() - except OSError as exc: + except (OSError, CheckpointError) as exc: err.print(f"[red]Resume failed:[/red] {cli._escape_markup(str(exc))}") raise cli.typer.Exit(1) diff --git a/src/specify_cli/workflows/command_status.py b/src/specify_cli/workflows/command_status.py index d9af4b48d6..97471eb10c 100644 --- a/src/specify_cli/workflows/command_status.py +++ b/src/specify_cli/workflows/command_status.py @@ -90,6 +90,15 @@ def workflow_status( s, "white" ) cli.console.print(f" [{sc}]●[/{sc}] {step_id}: {s}") + if state.execution: + from ._execution import scope_summaries + + for scope in scope_summaries(state.execution, state.status.value): + path = " / ".join(scope["scope_path"]) + cli.console.print( + f" {cli._escape_markup(path)} → " + f"{cli._escape_markup(scope['workflow_id'])}: {scope['status']}" + ) else: runs = engine.list_runs() diff --git a/src/specify_cli/workflows/composition.py b/src/specify_cli/workflows/composition.py new file mode 100644 index 0000000000..c7eff18028 --- /dev/null +++ b/src/specify_cli/workflows/composition.py @@ -0,0 +1,176 @@ +"""Typed workflow-call boundary; execution and persistence belong to the engine.""" + +from __future__ import annotations + +import json +from pathlib import Path +from typing import Any + +from .base import StepContext +from .expressions import evaluate_expression + +RESERVED_OUTPUT_NAMES = frozenset( + { + "workflow", + "status", + "error", + "aborted", + "integration", + "model", + "options", + "input", + } +) +MAX_COMPOSITION_DEPTH = 16 + + +class CallError(ValueError): + """A treatable violation of the workflow-call boundary.""" + + +def require_json(value: Any) -> None: + """Reject lossy/non-JSON values, including cycles, before checkpointing.""" + try: + encoded = json.dumps(value, allow_nan=False) + if json.loads(encoded) != value: + raise ValueError("JSON encoding changes the value") + except (TypeError, ValueError, RecursionError) as exc: + raise ValueError(f"Value is not JSON-safe: {exc}") from exc + + +def validate_call(config: dict[str, Any]) -> list[str]: + from .engine import _ID_PATTERN + from .overlay.schema import _RESERVED_WORKFLOW_IDS + + errors = [] + target = config.get("workflow") + if not isinstance(target, str): + errors.append("'workflow' must be a string") + elif "{{" not in target and ( + not _ID_PATTERN.fullmatch(target) or target in _RESERVED_WORKFLOW_IDS + ): + errors.append("'workflow' must be an exact, safe, non-reserved workflow ID") + mapping = config.get("input", {}) + if not isinstance(mapping, dict) or any(not isinstance(k, str) for k in mapping): + errors.append("'input' must be a mapping with string keys") + else: + try: + require_json(mapping) + except ValueError as exc: + errors.append(str(exc)) + return errors + + +def validate_runtime_call(config: dict[str, Any]) -> None: + """Raise a call-boundary error for an invalid authored call shape.""" + errors = validate_call(config) + if errors: + raise CallError("; ".join(errors)) + + +def validate_outputs(outputs: Any) -> list[str]: + from .engine import _ID_PATTERN + + if not isinstance(outputs, dict): + return ["'outputs' must be a mapping"] + errors = [] + for name, entry in outputs.items(): + if not isinstance(name, str) or not _ID_PATTERN.fullmatch(name): + errors.append(f"Output {name!r} must be a safe identifier") + elif name in RESERVED_OUTPUT_NAMES: + errors.append(f"Output {name!r} is reserved") + if not isinstance(entry, dict) or set(entry) != {"value"}: + errors.append(f"Output {name!r} must contain exactly 'value'") + else: + try: + require_json(entry["value"]) + except ValueError as exc: + errors.append(f"Output {name!r}: {exc}") + return errors + + +def resolve_target(project_root: Path, target: Any, ancestry: tuple[str, ...]): + from .catalog import WorkflowRegistry + from .engine import _ID_PATTERN, validate_workflow + from .overlay import WorkflowResolver + from .overlay.schema import _RESERVED_WORKFLOW_IDS + + try: + if not isinstance(target, str) or not _ID_PATTERN.fullmatch(target): + raise CallError("Workflow target must be an exact safe workflow ID") + if target in _RESERVED_WORKFLOW_IDS: + raise CallError(f"Workflow {target!r} is reserved") + if target in ancestry: + raise CallError( + f"Workflow composition cycle: {' -> '.join((*ancestry, target))}" + ) + if len(ancestry) > MAX_COMPOSITION_DEPTH: + raise CallError( + f"Workflow composition exceeds maximum depth {MAX_COMPOSITION_DEPTH}" + ) + metadata = WorkflowRegistry(project_root).get(target) + if not isinstance(metadata, dict): + raise CallError(f"Workflow {target!r} is not installed") + if not metadata.get("enabled", True): + raise CallError(f"Workflow {target!r} is disabled") + definition = WorkflowResolver(project_root).resolve(target) + if definition.id != target: + raise CallError( + f"Workflow {target!r} resolves to mismatched ID {definition.id!r}" + ) + errors = validate_workflow(definition) + if errors: + raise CallError(f"Invalid workflow {target!r}: {'; '.join(errors)}") + return definition + except CallError: + raise + except (OSError, ValueError) as exc: + raise CallError(str(exc)) from exc + + +def bind_inputs( + engine, + definition, + config: dict[str, Any], + context: StepContext, + previous: dict[str, Any] | None = None, +): + mapping = config.get("input", {}) + if not isinstance(mapping, dict) or any(not isinstance(k, str) for k in mapping): + raise CallError("'input' must be a mapping with string keys") + provided = { + key: evaluate_expression(value, context) for key, value in mapping.items() + } + try: + require_json(provided) + unknown = provided.keys() - definition.inputs.keys() + if unknown: + raise CallError( + f"Undeclared inputs for workflow {definition.id!r}: {sorted(unknown)}" + ) + resolved = ( + engine._resolve_inputs(definition, provided) + if previous is None + else engine._merge_inputs(definition, previous, provided) + ) + require_json(resolved) + return resolved + except CallError: + raise + except (TypeError, ValueError, RecursionError) as exc: + raise CallError(str(exc)) from exc + + +def evaluate_outputs(definition, context: StepContext) -> dict[str, Any]: + errors = validate_outputs(definition.outputs) + if errors: + raise CallError("; ".join(errors)) + output = { + name: evaluate_expression(entry["value"], context) + for name, entry in definition.outputs.items() + } + try: + require_json(output) + return output + except (TypeError, ValueError, RecursionError) as exc: + raise CallError(str(exc)) from exc diff --git a/src/specify_cli/workflows/engine.py b/src/specify_cli/workflows/engine.py index d7ad0fb857..50be13033e 100644 --- a/src/specify_cli/workflows/engine.py +++ b/src/specify_cli/workflows/engine.py @@ -10,14 +10,12 @@ from __future__ import annotations -import dataclasses import json import os import re import tempfile import threading import uuid -from concurrent.futures import Future, ThreadPoolExecutor from datetime import datetime, timezone from pathlib import Path from typing import Any @@ -28,7 +26,7 @@ default_integration_key, try_read_integration_json, ) -from .base import RunStatus, StepContext, StepResult, StepStatus +from .base import RunStatus, StepContext # -- Workflow Definition -------------------------------------------------- @@ -86,6 +84,7 @@ def __init__(self, data: dict[str, Any], source_path: Path | None = None) -> Non # Inputs self.inputs: dict[str, Any] = data.get("inputs", {}) + self.outputs: Any = data.get("outputs", {}) # Steps self.steps: list[dict[str, Any]] = data.get("steps", []) @@ -118,6 +117,13 @@ def from_string(cls, content: str) -> WorkflowDefinition: return cls(data) +def workflow_dir_for(definition: WorkflowDefinition) -> str | None: + """Return the resolved parent directory of a workflow definition's source.""" + if definition.source_path is None: + return None + return str(definition.source_path.resolve().parent) + + # -- Workflow Validation -------------------------------------------------- # ID format: lowercase alphanumeric with hyphens @@ -148,7 +154,7 @@ def _get_valid_step_types() -> set[str]: return set(STEP_REGISTRY.keys()) return { "command", "shell", "prompt", "gate", "if", "init", "slot", - "switch", "while", "do-while", "fan-out", "fan-in", + "switch", "while", "do-while", "fan-out", "fan-in", "workflow", } @@ -191,7 +197,9 @@ def validate_workflow(definition: WorkflowDefinition) -> list[str]: An empty list means the workflow is valid. """ - errors: list[str] = [] + from .composition import validate_outputs + + errors: list[str] = validate_outputs(definition.outputs) # -- Schema version --------------------------------------------------- # str() so an unquoted ``schema_version: 1.0`` (YAML float) is accepted — @@ -703,14 +711,18 @@ def __init__( self.current_step_index = 0 self.current_step_id: str | None = None self.step_results: dict[str, dict[str, Any]] = {} + self.execution: dict[str, Any] | None = None + self._checkpoint_failed = False # Guards step_results mutation and save() so a concurrent fan-out cannot # mutate the dict while save() is serializing it (which would raise # "dictionary changed size during iteration"). - self._lock = threading.Lock() + self._lock = threading.RLock() + # Reentrant so an execution transition can update views and call save() + # under the same lock. Step implementations run outside that lock. # Serializes append_log's list append + log.jsonl write so concurrent - # fan-out workers cannot interleave or corrupt log lines. Kept separate - # from _lock so frequent logging never contends with state saves; since - # append_log is never called while _lock is held, the two never nest. + # fan-out workers cannot interleave or corrupt log lines. append_log + # takes _lock first to observe a checkpoint failure atomically, so the + # order is always _lock -> _log_lock and the two cannot deadlock. self._log_lock = threading.Lock() self.inputs: dict[str, Any] = {} self.workflow_dir: str | None = None @@ -723,28 +735,6 @@ def __init__( def runs_dir(self) -> Path: return self.project_root / ".specify" / "workflows" / "runs" / self.run_id - def record_step_result(self, step_id: str, data: dict[str, Any]) -> None: - """Record one step's result under the run lock. - - Routing the mutation through the lock keeps it from racing a concurrent - ``save()`` that is iterating ``step_results`` (e.g. during a concurrent - fan-out). For a sequential run this is an uncontended lock. - """ - with self._lock: - self.step_results[step_id] = data - - def set_step_output(self, step_id: str, output: Any) -> None: - """Replace an already-recorded step's ``output`` under the run lock. - - Fan-out updates its parent step's output after the items have run; - routing that nested mutation through the lock keeps it from racing a - ``save()`` serializing ``step_results`` — the same invariant - ``record_step_result`` provides for the top-level assignment. - """ - with self._lock: - if step_id in self.step_results: - self.step_results[step_id]["output"] = output - def save(self) -> None: """Persist current state to disk. @@ -757,6 +747,10 @@ def save(self) -> None: runs_dir.mkdir(parents=True, exist_ok=True) with self._lock: + if self._checkpoint_failed: + from ._execution import CheckpointError + + raise CheckpointError("A previous checkpoint failed; reload the run") # Stamp updated_at inside the lock so the timestamp matches the # snapshot this thread serializes (concurrent savers don't race it). self.updated_at = datetime.now(timezone.utc).isoformat() @@ -773,9 +767,21 @@ def save(self) -> None: "created_at": self.created_at, "updated_at": self.updated_at, "error": self.error, + "inputs": self.inputs, } - self._atomic_write_json(runs_dir / "state.json", state_data) - self._atomic_write_json(runs_dir / "inputs.json", {"inputs": self.inputs}) + if self.execution is not None: + state_data["execution"] = self.execution + try: + self._atomic_write_json(runs_dir / "state.json", state_data) + self._atomic_write_json(runs_dir / "inputs.json", {"inputs": self.inputs}) + except Exception as exc: + # Only write failures poison the instance. A graceful interrupt + # (KeyboardInterrupt) propagates unchanged so execute()/resume() + # pause the run, as on main; the pause re-saves from memory. + from ._execution import CheckpointError + + self._checkpoint_failed = True + raise CheckpointError(str(exc)) from exc @staticmethod def _atomic_write_json(path: Path, data: dict[str, Any]) -> None: @@ -900,9 +906,14 @@ def load(cls, run_id: str, project_root: Path) -> RunState: state.created_at = state_data.get("created_at", "") state.updated_at = state_data.get("updated_at", "") state.error = state_data.get("error") + state.execution = state_data.get("execution") + if "execution" in state_data: + from ._execution import validate_execution + + validate_execution(state.execution) inputs_path = runs_dir / "inputs.json" - if inputs_path.exists(): + if "inputs" not in state_data and inputs_path.exists(): with open(inputs_path, encoding="utf-8") as f: inputs_data = json.load(f) if not isinstance(inputs_data, dict): @@ -916,6 +927,11 @@ def load(cls, run_id: str, project_root: Path) -> RunState: ) state.inputs = inputs + if "inputs" in state_data: + if not isinstance(state_data["inputs"], dict): + raise ValueError("Invalid run inputs: 'inputs' must be a JSON object") + state.inputs = state_data["inputs"] + return state def append_log(self, entry: dict[str, Any]) -> None: @@ -923,14 +939,21 @@ def append_log(self, entry: dict[str, Any]) -> None: Held under ``_log_lock`` so concurrent fan-out workers serialize their list append and ``log.jsonl`` write rather than interleaving lines. + The checkpoint-failure check shares ``_lock`` with ``save()``, so no + worker can log once any checkpoint of this run has failed. """ entry["timestamp"] = datetime.now(timezone.utc).isoformat() runs_dir = self.runs_dir runs_dir.mkdir(parents=True, exist_ok=True) - with self._log_lock: - self.log_entries.append(entry) - with open(runs_dir / "log.jsonl", "a", encoding="utf-8") as f: - f.write(json.dumps(entry) + "\n") + with self._lock: + if self._checkpoint_failed: + from ._execution import CheckpointError + + raise CheckpointError("A previous checkpoint failed; reload the run") + with self._log_lock: + self.log_entries.append(entry) + with open(runs_dir / "log.jsonl", "a", encoding="utf-8") as f: + f.write(json.dumps(entry) + "\n") # -- Workflow Engine ------------------------------------------------------ @@ -1069,11 +1092,7 @@ def execute( # Resolve inputs resolved_inputs = self._resolve_inputs(definition, inputs or {}) state.inputs = resolved_inputs - workflow_dir = ( - str(definition.source_path.resolve().parent) - if definition.source_path is not None - else None - ) + workflow_dir = workflow_dir_for(definition) state.workflow_dir = workflow_dir state.status = RunStatus.RUNNING state.save() @@ -1097,6 +1116,8 @@ def execute( state.save() return state except Exception as exc: + if state._checkpoint_failed: + raise state.status = RunStatus.FAILED state.error = str(exc) state.append_log({"event": "workflow_failed", "error": str(exc)}) @@ -1152,6 +1173,15 @@ def resume( ) raise ValueError(msg) + if state.execution is not None: + from ._execution import validate_execution + + validate_execution( + state.execution, + workflow_steps=definition.steps, + current_step_index=state.current_step_index, + ) + dispatch_default_errors = _dispatch_default_errors(definition) if dispatch_default_errors: raise ValueError(" ".join(dispatch_default_errors)) @@ -1159,8 +1189,7 @@ def resume( # Merge any newly-supplied inputs over the persisted ones and # re-validate through the same typing path as the initial run. if inputs: - merged = {**state.inputs, **inputs} - state.inputs = self._resolve_inputs(definition, merged) + state.inputs = self._merge_inputs(definition, state.inputs, inputs) # Restore context context = StepContext( @@ -1190,6 +1219,7 @@ def resume( self._execute_steps( remaining_steps, context, state, STEP_REGISTRY, step_offset=step_offset, + rebind=bool(inputs), ) except KeyboardInterrupt: state.status = RunStatus.PAUSED @@ -1197,6 +1227,8 @@ def resume( state.save() return state except Exception as exc: + if state._checkpoint_failed: + raise state.status = RunStatus.FAILED state.error = str(exc) state.append_log({"event": "resume_failed", "error": str(exc)}) @@ -1209,22 +1241,6 @@ def resume( state.save() return state - @staticmethod - def _record_result( - context: StepContext, state: RunState, step_id: str, data: dict[str, Any] - ) -> None: - """Record a step result into both the live context and persistent state. - - ``record_step_result`` writes ``state.step_results`` under the run lock. - On a resume run ``context.steps`` *is* that same dict, so that locked - write is the only one needed; mirror into ``context.steps`` separately - only when it is a distinct object (a fresh run), to avoid an unlocked - mutation of the shared dict that could race a concurrent ``save()``. - """ - if context.steps is not state.step_results: - context.steps[step_id] = data - state.record_step_result(step_id, data) - def _execute_steps( self, steps: list[dict[str, Any]], @@ -1233,429 +1249,28 @@ def _execute_steps( registry: dict[str, Any], *, step_offset: int = 0, + rebind: bool = False, ) -> None: - """Execute a list of steps sequentially.""" - for i, step_config in enumerate(steps): - step_id = step_config.get("id", f"step-{i}") - step_type = step_config.get("type", "command") - - state.current_step_id = step_id - if step_offset >= 0: - state.current_step_index = step_offset + i - state.save() - - state.append_log( - {"event": "step_started", "step_id": step_id, "type": step_type} - ) + """Execute or resume the persisted tree (legacy indices adapt once).""" + from ._execution import Execution, active_step, new_execution + from copy import deepcopy - # Log progress — use the engine's on_step_start callback if set, - # otherwise stay silent (library-safe default). - label = step_config.get("command", "") or step_type - if self.on_step_start is not None: - with self._callback_lock: - self.on_step_start(step_id, label) - - step_impl = registry.get(step_type) - if not step_impl: - state.status = RunStatus.FAILED - state.error = f"Unknown step type: {step_type!r}" - state.append_log( - { - "event": "step_failed", - "step_id": step_id, - "error": f"Unknown step type: {step_type!r}", - } - ) - state.save() - return - - result: StepResult = step_impl.execute(step_config, context) - - # Record step results — prefer resolved values from step output - step_data = { - "type": step_type, - "integration": result.output.get("integration") - or step_config.get("integration") - or context.default_integration, - "model": result.output.get("model") - or step_config.get("model") - or context.default_model, - "options": result.output.get("options") - or step_config.get("options", {}), - "input": result.output.get("input") - or step_config.get("input", {}), - "output": result.output, - "status": result.status.value, - "error": result.error, - } - if step_type == "command" and "integration_args" in result.output: - step_data["integration_args"] = result.output["integration_args"] - step_data["integration_options"] = result.output[ - "integration_options" - ] - self._record_result(context, state, step_id, step_data) - - state.append_log( - { - "event": "step_completed", - "step_id": step_id, - "status": result.status.value, - } - ) - - # Handle gate pauses - if result.status == StepStatus.PAUSED: - state.status = RunStatus.PAUSED - state.save() - return - - # Handle failures - if result.status == StepStatus.FAILED: - # Gate abort (output.aborted) maps to ABORTED status. - # Aborts are deliberate operator decisions, so - # `continue_on_error` does NOT override them — that flag - # is for transient/expected step failures only. - if result.output.get("aborted"): - state.status = RunStatus.ABORTED - state.error = result.error - state.append_log( - { - "event": "workflow_aborted", - "step_id": step_id, - } - ) - state.save() - return - - # `continue_on_error: true` lets the pipeline route - # around the failure instead of halting. The step - # result (including exit_code, stderr, status) is - # still recorded so a downstream `if` or `switch` - # can branch on it (or a `gate` can surface it to the - # operator via message interpolation). Log a single, - # unambiguous event per failure resolution — either - # the run continued past it, or it halted. - # - # Use identity comparison (`is True`) rather than - # truthiness so that only a literal boolean enables - # the behaviour, even if validation was skipped. - # Validation rejects non-bool values at parse time, - # but `WorkflowEngine.execute()` does not auto-validate - # (see `WorkflowEngine.load_workflow`, whose docstring - # explicitly notes "not yet validated; call - # `validate_workflow()` or `engine.validate()` - # separately"), so a caller passing an unvalidated - # definition could otherwise see truthy non-bool - # values like the string `"true"` silently change - # run semantics. - if step_config.get("continue_on_error") is True: - state.append_log( - { - "event": "step_continue_on_error", - "step_id": step_id, - "error": result.error, - } - ) - state.save() - continue - - state.status = RunStatus.FAILED - state.error = result.error - state.append_log( - { - "event": "step_failed", - "step_id": step_id, - "error": result.error, - } - ) - state.save() - return - - # Execute nested steps (from control flow) - # NOTE: Nested steps run with step_offset=-1 so they don't - # update current_step_index. If a nested step pauses, - # resume will re-run the parent step and its nested body. - # A step-path stack for exact nested resume is a future - # enhancement. - if result.next_steps: - self._execute_steps( - result.next_steps, context, state, registry, - step_offset=-1, - ) - if state.status in ( - RunStatus.PAUSED, - RunStatus.FAILED, - RunStatus.ABORTED, - ): - return - - # Loop iteration: while/do-while re-evaluate after body - if step_type in ("while", "do-while"): - from .expressions import evaluate_condition - - max_iters = step_config.get("max_iterations") - # A bool is an int in Python (isinstance(True, int) is True - # and True == 1), so a bool max_iterations would slip past - # the int check and cap the loop at range(0)==1 iteration - # instead of the default. Exclude bools, mirroring the - # while/do-while validators and the continue_on_error guard. - if ( - isinstance(max_iters, bool) - or not isinstance(max_iters, int) - or max_iters < 1 - ): - max_iters = 10 - condition = step_config.get("condition", False) - for _loop_iter in range(max_iters - 1): - if not evaluate_condition(condition, context): - break - # Namespace nested step IDs per iteration - # so logs and state keys are unique. - # Execute one step at a time and alias each - # result back to the unprefixed key so that - # later steps in the same body and the loop - # condition see the latest values. - for ns_idx, ns in enumerate(result.next_steps): - ns_copy = dict(ns) - orig = ns_copy.get("id") - base_id = orig or f"step-{ns_idx}" - ns_copy["id"] = f"{step_id}:{base_id}:{_loop_iter + 1}" - self._execute_steps( - [ns_copy], context, state, registry, - step_offset=-1, - ) - if state.status in ( - RunStatus.PAUSED, - RunStatus.FAILED, - RunStatus.ABORTED, - ): - return - if orig and ns_copy["id"] in context.steps: - self._record_result( - context, state, orig, - context.steps[ns_copy["id"]], - ) - - # Fan-out: execute the nested step template once per item. Honors - # max_concurrency — <=1 runs sequentially (default, historical - # behavior); >1 runs up to that many items concurrently. Either way - # results are assembled in item order under the - # parentId:templateId:index id grammar. - if step_type == "fan-out": - items = result.output.get("items", []) - template = result.output.get("step_template", {}) - if template and items: - fan_out_results = self._run_fan_out( - items, template, step_id, context, state, registry, - result.output.get("max_concurrency", 1), - ) - context.item = None - # Preserve original output and add collected results - fan_out_output = dict(result.output) - fan_out_output["results"] = fan_out_results - # set_step_output updates the recorded dict under the run lock; - # context.steps[step_id] is that same object, so it reflects the - # change too — no separate (unlocked) context mutation needed. - state.set_step_output(step_id, fan_out_output) - if state.status in ( - RunStatus.PAUSED, - RunStatus.FAILED, - RunStatus.ABORTED, - ): - return - else: - # Empty items or no template — normalize output - result.output["results"] = [] - state.set_step_output(step_id, result.output) - - def _run_fan_out( - self, - items: list[Any], - template: dict[str, Any], - step_id: str, - context: StepContext, - state: RunState, - registry: dict[str, Any], - max_concurrency: Any, - ) -> list[Any]: - """Run a fan-out template once per item; return per-item outputs in item order. - - ``max_concurrency`` <= 1 (the default) runs items sequentially, identical - to the historical fan-out behavior. ``max_concurrency`` > 1 runs items on a - bounded thread pool using a sliding submission window of that size: at most - that many items are ever in flight, and no new item is launched once the run - has reached a halting status, so a halt cannot keep starting queued work. - - Results are always returned in item order (never completion order). On a - halt (PAUSED/FAILED/ABORTED) the returned prefix is the items up to and - including the first item *in item order* whose own execution halted the run - — identical to the sequential path. Later items that have not yet started - are cancelled; any already running are allowed to finish but their outputs - are ignored. Halt is attributed per item from that item's recorded result - (not the shared run status, which a concurrently-running later item may have - already flipped), so the prefix never drops the actual halting item. - - ``max_concurrency`` is coerced with ``int()``; a value that cannot be - coerced (``None``, a non-numeric string, ``.inf``/``.nan``, …) or that - coerces to <= 1 runs sequentially, while a numeric string like ``"4"`` or - a float like ``4.0`` is honored. - """ - if not items: - return [] - - halting = (RunStatus.PAUSED, RunStatus.FAILED, RunStatus.ABORTED) + if state.execution is None: + state.execution = new_execution(steps, step_offset, context.steps) + state.save() + context.steps = deepcopy(state.execution.get("initial", {})) + tree = state.execution["sequence"] + executor = Execution(self, state, registry, rebind=rebind) try: - workers = max(1, int(max_concurrency)) - except (TypeError, ValueError, OverflowError): - # OverflowError: int(float("inf")) — a YAML ``max_concurrency: .inf`` - # would otherwise crash the whole run instead of falling back. - workers = 1 - # Never spin up more workers than there is work — bounds a user-controlled - # max_concurrency from over-allocating threads. - workers = min(workers, len(items)) - - base_id = template.get("id", "item") - - def item_id(idx: int) -> str: - # Per-item ID grammar: parentId:templateId:index. - return f"{step_id}:{base_id}:{idx}" - - def run_item(idx: int, item_ctx: StepContext) -> Any: - item_step = dict(template) - item_step["id"] = item_id(idx) - self._execute_steps( - [item_step], item_ctx, state, registry, step_offset=-1, - ) - # Read back through the context that was actually executed against, - # not the outer closure — clearer and robust if StepContext copying - # ever stops sharing the steps dict by reference. - return item_ctx.steps.get(item_step["id"], {}).get("output", {}) - - # Sequential path — identical to the historical behavior. - if workers <= 1: - results: list[Any] = [] - previous_item = context.item - previous_inside_fan_out = context.inside_fan_out - context.inside_fan_out = True - try: - for item_idx, item_val in enumerate(items): - context.item = item_val - results.append(run_item(item_idx, context)) - if state.status in halting: - break - finally: - context.item = previous_item - context.inside_fan_out = previous_inside_fan_out - return results - - # Concurrent path — bounded sliding window; results assembled in item order. - n = len(items) - slots: list[Any] = [None] * n - - def run_isolated(idx: int) -> Any: - # Each item runs against its own context copy so context.item is not - # clobbered across threads; the shared steps dict is written only on the - # disjoint parentId:templateId:index key (GIL-safe on distinct keys). - return run_item( - idx, - dataclasses.replace( - context, - item=items[idx], - inside_fan_out=True, - ), + outcome, error = executor.run( + tree, context, (state.workflow_id,), root=True ) - - def item_halt_status(idx: int) -> RunStatus | None: - # If THIS item's own execution halted the run, return the resulting run - # status; else None. Decided from the item's own recorded result, not - # the shared run status, so a later item's concurrent halt is never - # misattributed here. Mirrors the sequential mapping: PAUSED -> PAUSED; - # FAILED -> ABORTED when aborted, else FAILED, unless continue_on_error - # routes around it. - rec = context.steps.get(item_id(idx)) - if rec is None: - # Ran but recorded nothing — only when the item failed before - # record_step_result (e.g. an unknown step type returns early). - # Every item runs the same template, so the shared run status is - # this item's own outcome; attribute the halt to it. - return state.status if state.status in halting else None - status = rec.get("status") - if status == StepStatus.PAUSED.value: - return RunStatus.PAUSED - if status == StepStatus.FAILED.value: - out = rec.get("output") or {} - if out.get("aborted"): - return RunStatus.ABORTED - if template.get("continue_on_error") is not True: - return RunStatus.FAILED - return None - - # (halting item index, its run status) once a halt is attributed. - halt: tuple[int, RunStatus] | None = None - collected = 0 - with ThreadPoolExecutor(max_workers=workers) as pool: - futures: dict[int, Future] = {} - next_submit = 0 - for idx in range(n): - # Refill the window: keep <= workers in flight, and stop launching - # new items once the run is halting so a halt cannot keep starting - # queued work. Already-submitted futures are still collected in - # item order below. - while ( - next_submit < n - and len(futures) < workers - and state.status not in halting - ): - futures[next_submit] = pool.submit(run_isolated, next_submit) - next_submit += 1 - - fut = futures.pop(idx, None) - if fut is None: - # Safety net: the window submits indices in order and the loop - # breaks at the first halting item, so every collected index has - # an in-flight future. Stop cleanly rather than raise if a future - # change ever breaks that invariant. - break - try: - slots[idx] = fut.result() - except Exception: - # A genuine exception escaping a step (not a normal step - # FAILED, which sets state.status) must not be masked: cancel - # outstanding work and re-raise — with a bare ``raise`` so the - # original traceback is preserved — so the engine marks the run - # failed instead of reporting a vacuous completion. The pool's - # __exit__ still joins any already-running workers. - for other in futures.values(): - other.cancel() - raise - collected = idx + 1 - halt_status = item_halt_status(idx) - if halt_status is not None: - # First halting item in item order: include it (slots[idx] is - # already set), record its status, and cancel everything pending. - halt = (idx, halt_status) - for other in futures.values(): - other.cancel() - break - - if halt is not None: - halted_at, halted_status = halt - # A later in-flight item may have overwritten state.status before the - # pool joined; restore the halting item's own outcome so the final run - # status matches the sequential semantics. - state.status = halted_status - # Restore the halting item's error so it matches the terminal - # status — a concurrent item may have overwritten state.error - # before the pool joined. Assign unconditionally when a record - # exists (even when the halting item's own error is falsy) so a - # third-party step returning FAILED with no message never inherits - # an unrelated concurrent item's error; this mirrors the sequential - # path, which sets state.error = result.error verbatim. - halt_rec = context.steps.get(item_id(halted_at)) - if isinstance(halt_rec, dict): - state.error = halt_rec.get("error") - return slots[: halted_at + 1] - return slots[:collected] + finally: + active = active_step(state.execution) + if active is not None: + state.current_step_id = active[2] + state.status = RunStatus.RUNNING if outcome == "completed" else RunStatus(outcome) + state.error = error def _resolve_inputs( self, @@ -1722,6 +1337,15 @@ def _resolve_inputs( resolved[name] = self._coerce_input(name, value, coerce_input_def) return resolved + def _merge_inputs( + self, + definition: WorkflowDefinition, + previous: dict[str, Any], + updates: dict[str, Any], + ) -> dict[str, Any]: + """Resolve updates over an existing input snapshot.""" + return self._resolve_inputs(definition, {**previous, **updates}) + def _resolve_default(self, name: str, default: Any) -> Any: """Resolve special default sentinels against project state. diff --git a/src/specify_cli/workflows/step/fan_in/__init__.py b/src/specify_cli/workflows/step/fan_in/__init__.py index ddcc2afcbb..ca5967ada4 100644 --- a/src/specify_cli/workflows/step/fan_in/__init__.py +++ b/src/specify_cli/workflows/step/fan_in/__init__.py @@ -86,6 +86,25 @@ def execute(self, config: dict[str, Any], context: StepContext) -> StepResult: output={"results": []}, ) + # Generated fan-out item aliases (``fan:template:0``) are reporting-only: + # declared step IDs cannot contain ':' (engine validation rejects them), + # so a ':' entry can never match a real step. The engine does not + # auto-validate before ``execute``, so reject it loudly here too rather + # than silently joining a stale or empty item result. + aliased_entries = [w for w in wait_for if ":" in w] + if aliased_entries: + first = aliased_entries[0] + return StepResult( + status=StepStatus.FAILED, + error=( + f"Fan-in step {config.get('id', '?')!r}: 'wait_for' entry " + f"{first!r} looks like a fan-out item alias; wait on the " + f"fan-out step id instead — its 'output.results' holds the " + f"ordered item results." + ), + output={"results": []}, + ) + # Collect results from referenced steps results = [] for step_id in wait_for: diff --git a/src/specify_cli/workflows/step/fan_out/__init__.py b/src/specify_cli/workflows/step/fan_out/__init__.py index d961170c8c..5161377c24 100644 --- a/src/specify_cli/workflows/step/fan_out/__init__.py +++ b/src/specify_cli/workflows/step/fan_out/__init__.py @@ -15,7 +15,7 @@ class FanOutStep(StepBase): setting ``context.item`` for each iteration. ``max_concurrency`` controls parallelism: ``<= 1`` (the default) runs items sequentially, while ``> 1`` runs up to that many items concurrently - on a bounded thread pool (see ``WorkflowEngine._run_fan_out``). + on a bounded thread pool. """ type_key = "fan-out" @@ -28,12 +28,10 @@ def execute(self, config: dict[str, Any], context: StepContext) -> StepResult: # The engine does not auto-validate step config (see # ``WorkflowEngine.load_workflow``). On a COMPLETED fan-out it reads the - # ``step_template`` back out and, when it is truthy, calls - # ``template.get("id", ...)`` in ``_run_fan_out``. A truthy non-mapping - # ``step`` (a scalar or list authoring mistake) would crash the whole - # run with AttributeError there — the engine invokes ``execute`` and - # ``_run_fan_out`` with no surrounding try/except. ``validate`` already - # rejects a non-mapping ``step``; fail this step loudly on an + # ``step_template`` back out and calls ``template.get("id", ...)`` while + # expanding items. A truthy non-mapping ``step`` (a scalar or list + # authoring mistake) would crash the whole run with AttributeError there. + # ``validate`` already rejects a non-mapping ``step``; fail this step loudly on # unvalidated run instead, mirroring the ``items`` guard below. An empty # or absent ``step`` defaults to ``{}`` (falsy) and the engine's # ``if template and items`` skips fan-out, so it stays valid here. diff --git a/src/specify_cli/workflows/step/gate/__init__.py b/src/specify_cli/workflows/step/gate/__init__.py index 5aac060c0f..86dec595b2 100644 --- a/src/specify_cli/workflows/step/gate/__init__.py +++ b/src/specify_cli/workflows/step/gate/__init__.py @@ -2,6 +2,7 @@ from __future__ import annotations +import json import re import sys from pathlib import Path @@ -40,6 +41,12 @@ def execute(self, config: dict[str, Any], context: StepContext) -> StepResult: message = config.get("message", "Review required.") if isinstance(message, str) and "{{" in message: message = evaluate_expression(message, context) + # Keep JSON values typed; only values the JSON checkpoint cannot + # persist (e.g. native YAML dates) are stored as text. + try: + json.dumps(message) + except (TypeError, ValueError): + message = str(message) options = config.get("options", ["approve", "reject"]) on_reject = config.get("on_reject", "abort") diff --git a/src/specify_cli/workflows/step/workflow/__init__.py b/src/specify_cli/workflows/step/workflow/__init__.py new file mode 100644 index 0000000000..0c8f1df328 --- /dev/null +++ b/src/specify_cli/workflows/step/workflow/__init__.py @@ -0,0 +1,17 @@ +"""Installed workflow calls are scoped execution handled by the engine.""" + +from ...base import StepBase, StepResult, StepStatus +from ...composition import validate_call + + +class WorkflowStep(StepBase): + type_key = "workflow" + + def validate(self, config): + return [*super().validate(config), *validate_call(config)] + + def execute(self, config, context): + return StepResult( + status=StepStatus.FAILED, + error="Workflow calls require the workflow engine", + ) diff --git a/tests/specify_cli/workflows/test_command_resume.py b/tests/specify_cli/workflows/test_command_resume.py index c4f0c050c0..21c12716d8 100644 --- a/tests/specify_cli/workflows/test_command_resume.py +++ b/tests/specify_cli/workflows/test_command_resume.py @@ -296,6 +296,31 @@ def test_resume_preload_io_error_is_reported_cleanly( assert "Resume failed" in result.output assert "permission [denied]" in result.output + def test_resume_legacy_migration_checkpoint_error_is_reported_cleanly( + self, project_dir, monkeypatch + ): + from typer.testing import CliRunner + from specify_cli import app + from specify_cli.workflows._execution import CheckpointError + from specify_cli.workflows.engine import RunState + + monkeypatch.chdir(project_dir) + runner = CliRunner() + run_id = self._install_and_run_gated(runner, app, project_dir) + state_path = project_dir / ".specify" / "workflows" / "runs" / run_id / "state.json" + data = json.loads(state_path.read_text(encoding="utf-8")) + data["installed_workflow_id"] = None + data.pop("installed_origin_tracked", None) + state_path.write_text(json.dumps(data), encoding="utf-8") + monkeypatch.setattr(RunState, "save", lambda _: (_ for _ in ()).throw(CheckpointError("disk full"))) + + result = runner.invoke(app, ["workflow", "resume", run_id]) + + assert result.exit_code != 0 + assert result.exception is None or isinstance(result.exception, SystemExit) + assert "Resume failed" in result.output + assert "disk full" in result.output + @pytest.mark.parametrize("malformation", ["non-object", "missing-run-id"]) def test_resume_preload_rejects_malformed_state_cleanly( self, project_dir, monkeypatch, malformation diff --git a/tests/specify_cli/workflows/test_command_status.py b/tests/specify_cli/workflows/test_command_status.py index 0a7b50ac13..73994d5b22 100644 --- a/tests/specify_cli/workflows/test_command_status.py +++ b/tests/specify_cli/workflows/test_command_status.py @@ -95,6 +95,87 @@ def test_status_json_single_and_list(self, project_dir): ) assert any(r["run_id"] == rid for r in listing["runs"]) + def test_composed_gate_status_and_resume(self, project_dir): + from specify_cli.workflows.catalog import WorkflowRegistry + + child_dir = project_dir / ".specify" / "workflows" / "child" + child_dir.mkdir(parents=True) + (child_dir / "workflow.yml").write_text( + "workflow: {id: child, name: Child}\n" + "inputs:\n verdict: {type: string, default: ''}\n" + "steps:\n - {id: review, type: gate, message: Review, verdict_input: verdict}\n", + encoding="utf-8", + ) + WorkflowRegistry(project_dir).add("child", {"enabled": True}) + root = self._write_wf( + project_dir, + "workflow: {id: parent, name: Parent}\n" + "inputs:\n verdict: {type: string, default: ''}\n" + "steps:\n - id: call\n type: workflow\n workflow: child\n" + " input: {verdict: '{{ inputs.verdict }}'}\n", + "parent", + ) + run = json.loads(self._invoke(project_dir, ["workflow", "run", str(root), "--json"]).stdout) + assert run["status"] == "paused", run + status = json.loads(self._invoke(project_dir, ["workflow", "status", run["run_id"], "--json"]).stdout) + assert status["gate"] == run["gate"] + assert status["gate"]["scope_path"] == ["call"] + assert status["workflow_scopes"] == [{"scope_path": ["call"], "workflow_id": "child", "status": "paused"}] + human = self._invoke(project_dir, ["workflow", "status", run["run_id"]]) + assert "child: paused" in human.stdout + resumed = self._invoke(project_dir, ["workflow", "resume", run["run_id"], "--input", "verdict=approve", "--json"]) + assert resumed.exit_code == 0 + assert json.loads(resumed.stdout)["status"] == "completed" + + @pytest.mark.parametrize("failure, expected", [ + (KeyboardInterrupt, "paused"), + (RuntimeError, "failed"), + ]) + def test_status_reports_interrupted_call_in_json_and_text( + self, project_dir, monkeypatch, failure, expected + ): + from specify_cli.workflows import STEP_REGISTRY + from specify_cli.workflows.base import StepBase + from specify_cli.workflows.catalog import WorkflowRegistry + from specify_cli.workflows.engine import WorkflowDefinition, WorkflowEngine + + class Explode(StepBase): + type_key = "explode" + + def execute(self, config, context): + raise failure("boom") + + monkeypatch.setitem(STEP_REGISTRY, "explode", Explode()) + child_dir = project_dir / ".specify" / "workflows" / "child" + child_dir.mkdir(parents=True) + (child_dir / "workflow.yml").write_text( + "workflow: {id: child, name: Child}\n" + "steps:\n - {id: work, type: explode}\n", + encoding="utf-8", + ) + WorkflowRegistry(project_dir).add("child", {"enabled": True}) + engine = WorkflowEngine(project_dir) + root = WorkflowDefinition({ + "workflow": {"id": "parent", "name": "Parent"}, + "steps": [{"id": "call", "type": "workflow", "workflow": "child"}], + }) + if failure is RuntimeError: + with pytest.raises(RuntimeError, match="boom"): + engine.execute(root, run_id="interrupted-call") + else: + engine.execute(root, run_id="interrupted-call") + + status = self._invoke(project_dir, [ + "workflow", "status", "interrupted-call", "--json", + ]) + assert status.exit_code == 0, status.output + assert json.loads(status.stdout)["workflow_scopes"] == [ + {"scope_path": ["call"], "workflow_id": "child", "status": expected} + ] + human = self._invoke(project_dir, ["workflow", "status", "interrupted-call"]) + assert human.exit_code == 0, human.output + assert f"child: {expected}" in human.stdout + class TestWorkflowCliAlignment: diff --git a/tests/test_workflows.py b/tests/test_workflows.py index acfdaa0b3b..3d77b82f46 100644 --- a/tests/test_workflows.py +++ b/tests/test_workflows.py @@ -4191,7 +4191,7 @@ def test_execute_non_dict_step_fails_loudly(self): ``validate`` rejects a non-dict ``step``, but the engine's ``execute()`` does not auto-validate (see ``WorkflowEngine.load_workflow``). On a COMPLETED fan-out the engine reads ``step_template`` back out and, when - it is truthy, calls ``template.get("id", ...)`` in ``_run_fan_out``. A + it is truthy, calls ``template.get("id", ...)`` while expanding items. A truthy non-mapping ``step`` (a scalar or list authoring mistake) raised AttributeError there and took down the whole run. Mirrors the fan-out non-list ``items`` guard and the switch non-dict ``cases`` guard. @@ -4422,24 +4422,16 @@ def test_validate_accepts_mapping_or_absent_output(self): class TestFanOutConcurrency: - """Fan-out honors max_concurrency (WorkflowEngine._run_fan_out).""" + """Fan-out honors max_concurrency through WorkflowEngine.execute().""" @staticmethod - def _build(tmp_path, on_item=None): - """Wire an engine + run state to a probe step that echoes context.item. - - Per-item output is ``{"seen": }`` so order and per-thread item - isolation are checkable. ``on_item(item)`` may run a side effect and - optionally return a StepStatus to override COMPLETED (or raise). - """ - from specify_cli.workflows.base import ( - RunStatus, - StepBase, - StepContext, - StepResult, - StepStatus, - ) - from specify_cli.workflows.engine import RunState, WorkflowEngine + def _run( + tmp_path, monkeypatch, items, max_concurrency, on_item=None, template=None + ): + """Run a public fan-out workflow with a probe that echoes ``context.item``.""" + from specify_cli.workflows import STEP_REGISTRY + from specify_cli.workflows.base import StepBase, StepResult, StepStatus + from specify_cli.workflows.engine import WorkflowDefinition, WorkflowEngine class _ProbeStep(StepBase): type_key = "probe" @@ -4452,35 +4444,41 @@ def execute(self, config, context): status = override return StepResult(status=status, output={"seen": context.item}) - engine = WorkflowEngine(project_root=tmp_path) - context = StepContext() - state = RunState(run_id="r", workflow_id="w", project_root=tmp_path) - state.status = RunStatus.RUNNING - template = {"id": "impl", "type": "probe"} - return engine, context, state, {"probe": _ProbeStep()}, template - - def _run(self, tmp_path, items, max_concurrency, on_item=None): - engine, context, state, registry, template = self._build(tmp_path, on_item) - results = engine._run_fan_out( - items, template, "fan", context, state, registry, max_concurrency + monkeypatch.setitem(STEP_REGISTRY, "probe", _ProbeStep()) + template = template or {"id": "impl", "type": "probe"} + state = WorkflowEngine(tmp_path).execute( + WorkflowDefinition( + { + "workflow": {"id": "fan-out-test", "name": "Fan-out Test"}, + "steps": [ + { + "id": "fan", + "type": "fan-out", + "items": items, + "max_concurrency": max_concurrency, + "step": template, + } + ], + } + ) ) - return results, state + return state.step_results["fan"]["output"]["results"], state - def test_sequential_default_preserves_order(self, tmp_path): - results, _ = self._run(tmp_path, list(range(5)), 1) + def test_sequential_default_preserves_order(self, tmp_path, monkeypatch): + results, _ = self._run(tmp_path, monkeypatch, list(range(5)), 1) assert results == [{"seen": i} for i in range(5)] - def test_concurrent_runs_all_items_in_item_order(self, tmp_path): - results, _ = self._run(tmp_path, list(range(10)), 4) + def test_concurrent_runs_all_items_in_item_order(self, tmp_path, monkeypatch): + results, _ = self._run(tmp_path, monkeypatch, list(range(10)), 4) assert results == [{"seen": i} for i in range(10)] - def test_sequential_and_concurrent_agree(self, tmp_path): + def test_sequential_and_concurrent_agree(self, tmp_path, monkeypatch): items = [{"n": i} for i in range(8)] - seq, _ = self._run(tmp_path, items, 1) - con, _ = self._run(tmp_path, items, 4) + seq, _ = self._run(tmp_path, monkeypatch, items, 1) + con, _ = self._run(tmp_path, monkeypatch, items, 4) assert seq == con == [{"seen": {"n": i}} for i in range(8)] - def test_shuffled_completion_preserves_item_order(self, tmp_path): + def test_shuffled_completion_preserves_item_order(self, tmp_path, monkeypatch): # Determinism keystone: completion order is forced to the exact REVERSE of # item order by an event chain (no sleeps) — item i blocks until item i+1 # has finished, so item 0 completes LAST — yet results must still be in @@ -4500,11 +4498,11 @@ def on_item(item): done[item].set() return None - results, _ = self._run(tmp_path, list(range(n)), n, on_item) + results, _ = self._run(tmp_path, monkeypatch, list(range(n)), n, on_item) assert results == [{"seen": i} for i in range(n)] assert completion == list(reversed(range(n))) - def test_concurrency_is_real(self, tmp_path): + def test_concurrency_is_real(self, tmp_path, monkeypatch): import threading # Deterministic proof of real parallelism (no wall-clock threshold to @@ -4518,30 +4516,35 @@ def on_item(item): barrier.wait() return None - results, _ = self._run(tmp_path, list(range(n)), n, on_item) + results, _ = self._run(tmp_path, monkeypatch, list(range(n)), n, on_item) assert results == [{"seen": i} for i in range(n)] @pytest.mark.parametrize( "bad", [0, -1, None, "abc", 1.0, float("inf"), float("nan")] ) - def test_invalid_max_concurrency_coerces_to_sequential(self, tmp_path, bad): + def test_invalid_max_concurrency_coerces_to_sequential( + self, tmp_path, monkeypatch, bad + ): # float("inf") -> int() raises OverflowError (not TypeError/ValueError); # it must fall back to sequential like any other uncoercible value, not # crash the run. - results, _ = self._run(tmp_path, list(range(4)), bad) + results, _ = self._run(tmp_path, monkeypatch, list(range(4)), bad) assert results == [{"seen": i} for i in range(4)] - def test_string_max_concurrency_is_honored(self, tmp_path): - results, _ = self._run(tmp_path, list(range(4)), "2") + def test_string_max_concurrency_is_honored(self, tmp_path, monkeypatch): + results, _ = self._run(tmp_path, monkeypatch, list(range(4)), "2") assert results == [{"seen": i} for i in range(4)] - def test_context_item_isolation_across_threads(self, tmp_path): + def test_context_item_isolation_across_threads(self, tmp_path, monkeypatch): items = [{"id": f"x{i}"} for i in range(6)] - results, _ = self._run(tmp_path, items, 6) + results, _ = self._run(tmp_path, monkeypatch, items, 6) assert [r["seen"]["id"] for r in results] == [f"x{i}" for i in range(6)] @pytest.mark.parametrize("max_concurrency", [1, 2]) - def test_marks_item_context_as_inside_fan_out(self, tmp_path, max_concurrency): + def test_marks_item_context_as_inside_fan_out( + self, tmp_path, monkeypatch, max_concurrency + ): + from specify_cli.workflows import STEP_REGISTRY from specify_cli.workflows.base import StepBase, StepResult, StepStatus class _ContextProbeStep(StepBase): @@ -4553,28 +4556,27 @@ def execute(self, config, context): output={"inside_fan_out": context.inside_fan_out}, ) - engine, context, state, _registry, _template = self._build(tmp_path) - results = engine._run_fan_out( + monkeypatch.setitem(STEP_REGISTRY, "context-probe", _ContextProbeStep()) + results, _ = self._run( + tmp_path, + monkeypatch, ["a", "b"], - {"id": "probe", "type": "context-probe"}, - "fan", - context, - state, - {"context-probe": _ContextProbeStep()}, max_concurrency, + template={"id": "probe", "type": "context-probe"}, ) assert results == [ {"inside_fan_out": True}, {"inside_fan_out": True}, ] - assert context.inside_fan_out is False - def test_empty_items(self, tmp_path): - results, _ = self._run(tmp_path, [], 4) + def test_empty_items(self, tmp_path, monkeypatch): + results, _ = self._run(tmp_path, monkeypatch, [], 4) assert results == [] - def test_concurrent_halt_status_not_clobbered_by_later_item(self, tmp_path): + def test_concurrent_halt_status_not_clobbered_by_later_item( + self, tmp_path, monkeypatch + ): # Item 1 PAUSES (first halting item in order); item 3 FAILS while in # flight. The final run status must be the halting item's (PAUSED), never # a later item's (FAILED) that raced after it — matching sequential. @@ -4587,22 +4589,24 @@ def on_item(item): return StepStatus.FAILED return None - results, state = self._run(tmp_path, list(range(4)), 4, on_item) + results, state = self._run(tmp_path, monkeypatch, list(range(4)), 4, on_item) assert results == [{"seen": 0}, {"seen": 1}] assert state.status == RunStatus.PAUSED - def test_halt_on_failure_sequential_returns_prefix(self, tmp_path): + def test_halt_on_failure_sequential_returns_prefix(self, tmp_path, monkeypatch): from specify_cli.workflows.base import RunStatus, StepStatus def on_item(item): return StepStatus.FAILED if item == 2 else None - results, state = self._run(tmp_path, list(range(5)), 1, on_item) + results, state = self._run(tmp_path, monkeypatch, list(range(5)), 1, on_item) assert len(results) == 3 # items 0,1,2 ran; 3,4 never dispatched assert results[2] == {"seen": 2} assert state.status == RunStatus.FAILED - def test_halt_on_failure_concurrent_includes_halting_item(self, tmp_path): + def test_halt_on_failure_concurrent_includes_halting_item( + self, tmp_path, monkeypatch + ): # The concurrent prefix must match the sequential one: items up to and # INCLUDING the failing item (2), never a short prefix that drops it just # because a later in-flight item flipped the shared run status first. @@ -4611,11 +4615,11 @@ def test_halt_on_failure_concurrent_includes_halting_item(self, tmp_path): def on_item(item): return StepStatus.FAILED if item == 2 else None - results, state = self._run(tmp_path, list(range(6)), 4, on_item) + results, state = self._run(tmp_path, monkeypatch, list(range(6)), 4, on_item) assert results == [{"seen": 0}, {"seen": 1}, {"seen": 2}] assert state.status == RunStatus.FAILED - def test_concurrent_restores_halting_item_error(self, tmp_path): + def test_concurrent_restores_halting_item_error(self, tmp_path, monkeypatch): # After a concurrent fan-out halts, the run-level error must be the first # halting item's own error (parity with the sequential path), even when a # later concurrent item failed with a different error AND the halting @@ -4648,15 +4652,15 @@ def execute(self, config, context): status=StepStatus.COMPLETED, output={"seen": item} ) - engine, context, state, _registry, _template = self._build(tmp_path) - engine._run_fan_out( + from specify_cli.workflows import STEP_REGISTRY + + monkeypatch.setitem(STEP_REGISTRY, "err-probe", _ErrorProbe()) + _results, state = self._run( + tmp_path, + monkeypatch, ["ok0", "halt", "ok2", "leak"], - {"id": "impl", "type": "err-probe"}, - "fan", - context, - state, - {"err-probe": _ErrorProbe()}, 4, + template={"id": "impl", "type": "err-probe"}, ) assert state.status == RunStatus.FAILED @@ -4665,7 +4669,9 @@ def execute(self, config, context): # error verbatim, even when falsy. assert state.error == "" - def test_continue_on_error_item_does_not_halt_concurrent(self, tmp_path): + def test_continue_on_error_item_does_not_halt_concurrent( + self, tmp_path, monkeypatch + ): # A failing item whose template sets continue_on_error must NOT truncate # the fan-out: every item still runs and is returned in order. from specify_cli.workflows.base import StepStatus @@ -4673,41 +4679,38 @@ def test_continue_on_error_item_does_not_halt_concurrent(self, tmp_path): def on_item(item): return StepStatus.FAILED if item == 2 else None - engine, context, state, registry, template = self._build(tmp_path, on_item) - template["continue_on_error"] = True - results = engine._run_fan_out( - list(range(5)), template, "fan", context, state, registry, 4 + results, _ = self._run( + tmp_path, + monkeypatch, + list(range(5)), + 4, + on_item, + template={"id": "impl", "type": "probe", "continue_on_error": True}, ) assert results == [{"seen": i} for i in range(5)] - def test_unknown_template_type_halts_concurrent_like_sequential(self, tmp_path): + def test_unknown_template_type_halts_concurrent_like_sequential( + self, tmp_path, monkeypatch + ): # A template whose type isn't registered fails fast and records no result; # the concurrent path must still attribute the halt to the first item and # return the same prefix as sequential — never run on as if completed. - from specify_cli.workflows.base import RunStatus, StepContext - from specify_cli.workflows.engine import RunState, WorkflowEngine - - def fresh(): - state = RunState(run_id="r", workflow_id="w", project_root=tmp_path) - state.status = RunStatus.RUNNING - return WorkflowEngine(project_root=tmp_path), StepContext(), state - template = {"id": "impl", "type": "does-not-exist"} - e1, c1, s1 = fresh() - seq = e1._run_fan_out(list(range(5)), template, "fan", c1, s1, {}, 1) - e2, c2, s2 = fresh() - con = e2._run_fan_out(list(range(5)), template, "fan", c2, s2, {}, 4) + seq, s1 = self._run(tmp_path, monkeypatch, list(range(5)), 1, template=template) + con, s2 = self._run(tmp_path, monkeypatch, list(range(5)), 4, template=template) assert seq == con == [{}] # halted at the first item; rest never returned + from specify_cli.workflows.base import RunStatus + assert s1.status == s2.status == RunStatus.FAILED - def test_first_exception_cancels_and_reraises(self, tmp_path): + def test_first_exception_cancels_and_reraises(self, tmp_path, monkeypatch): def on_item(item): if item == 0: raise ValueError("boom") return None with pytest.raises(ValueError, match="boom"): - self._run(tmp_path, list(range(4)), 2, on_item) + self._run(tmp_path, monkeypatch, list(range(4)), 2, on_item) class TestFanInWaitForValidation: @@ -4831,6 +4834,26 @@ def test_non_string_wait_for_entry_is_rejected(self): "must be step-id strings" in e and "int" in e for e in errors ) + def test_non_ascii_fan_out_item_index_is_rejected(self): + errors = self._errors(""" +workflow: + id: wf + name: wf + version: "1.0.0" +steps: + - id: fan + type: fan-out + items: [one] + step: + id: item + type: command + command: speckit.implement + - id: collect + type: fan-in + wait_for: [fan:item:١] +""") + assert any("unknown or not-yet-declared step id 'fan:item:١'" in e for e in errors) + # ===== Workflow Definition Tests ===== @@ -7784,6 +7807,59 @@ def test_unknown_step_type_sets_run_error(self, project_dir): assert state.status == RunStatus.FAILED assert state.error == "Unknown step type: 'definitely-not-a-real-step'" + def test_unknown_step_type_ignores_continue_on_error(self, project_dir): + """A missing implementation is terminal even with recovery enabled.""" + from specify_cli.workflows.engine import WorkflowDefinition, WorkflowEngine + from specify_cli.workflows.base import RunStatus + + definition = WorkflowDefinition.from_string(""" +schema_version: "1.0" +workflow: + id: "unknown-type-recovery" + name: "Unknown Type Recovery" + version: "1.0.0" +steps: + - id: mystery + type: definitely-not-a-real-step + continue_on_error: true + - id: after + type: shell + run: "echo should-not-run" +""") + state = WorkflowEngine(project_dir).execute(definition) + + assert state.status == RunStatus.FAILED + assert state.error == "Unknown step type: 'definitely-not-a-real-step'" + assert "after" not in state.step_results + events = [entry["event"] for entry in state.log_entries] + assert "step_failed" in events + assert "step_continue_on_error" not in events + + def test_unknown_step_type_events_match_main(self, project_dir): + """Only step_started and step_failed, and no projected step result.""" + from specify_cli.workflows.engine import WorkflowDefinition, WorkflowEngine + from specify_cli.workflows.base import RunStatus + + definition = WorkflowDefinition.from_string(""" +schema_version: "1.0" +workflow: + id: "unknown-type-events" + name: "Unknown Type Events" + version: "1.0.0" +steps: + - id: mystery + type: definitely-not-a-real-step +""") + state = WorkflowEngine(project_dir).execute(definition) + + assert state.status == RunStatus.FAILED + assert [ + entry["event"] + for entry in state.log_entries + if entry.get("step_id") == "mystery" + ] == ["step_started", "step_failed"] + assert "mystery" not in state.step_results + # ===== State Persistence Tests ===== diff --git a/tests/workflows/test_composition_execution.py b/tests/workflows/test_composition_execution.py new file mode 100644 index 0000000000..9c6f669212 --- /dev/null +++ b/tests/workflows/test_composition_execution.py @@ -0,0 +1,3412 @@ +"""Behavioral contracts for scoped calls and durable execution occurrences.""" + +from collections import Counter +from datetime import date +import json +import threading + +import pytest +import yaml + +from specify_cli.workflows import STEP_REGISTRY +from specify_cli.workflows.base import RunStatus, StepBase, StepResult, StepStatus +from specify_cli.workflows.engine import ( + RunState, + WorkflowDefinition, + WorkflowEngine, + validate_workflow, +) + + +def definition(name, steps, **fields): + return WorkflowDefinition( + {"workflow": {"id": name, "name": name}, "steps": steps, **fields} + ) + + +def install(root, child, enabled=True): + from specify_cli.workflows.catalog import WorkflowRegistry + + directory = root / ".specify" / "workflows" / child.id + directory.mkdir(parents=True, exist_ok=True) + (directory / "workflow.yml").write_text( + yaml.safe_dump(child.data), encoding="utf-8" + ) + registry = WorkflowRegistry(root) + registry.add(child.id, {"version": "1.0.0", "enabled": enabled}) + return directory + + +def call(target="child", **extra): + return {"id": "call", "type": "workflow", "workflow": target, **extra} + + +@pytest.fixture +def probe(monkeypatch): + counts = Counter() + + class Probe(StepBase): + type_key = "probe" + + def execute(self, config, context): + from specify_cli.workflows.expressions import evaluate_expression + + counts[config["id"]] += 1 + value = evaluate_expression(config.get("value"), context) + status = config.get("status", "completed") + if config.get("await") and not context.inputs.get("approve"): + status = "paused" + return StepResult( + StepStatus(status), output={"value": value, **config.get("output", {})} + ) + + monkeypatch.setitem(STEP_REGISTRY, "probe", Probe()) + return counts + + +def test_declared_output_and_scope_isolation(tmp_path, probe): + child = definition( + "child", + [ + {"id": "inspect", "type": "probe", "value": "{{ inputs.value }}"}, + { + "id": "private", + "type": "probe", + "value": "{{ steps.parent.output.value }}", + }, + ], + inputs={"value": {"type": "string"}}, + outputs={ + "value": {"value": "{{ steps.inspect.output.value }}"}, + "hidden": {"value": "{{ steps.private.output.value }}"}, + }, + ) + install(tmp_path, child) + state = WorkflowEngine(tmp_path).execute( + definition( + "parent", + [ + {"id": "parent", "type": "probe", "value": "secret"}, + call(input={"value": "mapped"}), + { + "id": "consume", + "type": "probe", + "value": "{{ steps.call.output.value }}", + }, + ], + ) + ) + assert state.status == RunStatus.COMPLETED + assert state.step_results["call"]["output"] == { + "workflow": "child", + "status": "completed", + "value": "mapped", + "hidden": None, + } + assert state.step_results["consume"]["output"]["value"] == "mapped" + assert "inspect" not in state.step_results + assert len(list((tmp_path / ".specify/workflows/runs").iterdir())) == 1 + + +def test_called_workflow_dir_resolves_definition_symlink(tmp_path, monkeypatch, probe): + import specify_cli.workflows._execution as execution + + resolved_dir = tmp_path / "resolved-child" + resolved_dir.mkdir() + symlink_dir = tmp_path / "linked-child" + symlink_dir.symlink_to(resolved_dir, target_is_directory=True) + child = definition( + "child", + [{"id": "path", "type": "probe", "value": "{{ context.workflow_dir }}"}], + outputs={"workflow-dir": {"value": "{{ steps.path.output.value }}"}}, + ) + child.source_path = symlink_dir / "workflow.yml" + monkeypatch.setattr(execution, "resolve_target", lambda *_: child) + + state = WorkflowEngine(tmp_path).execute(definition("parent", [call()])) + + assert state.status == RunStatus.COMPLETED, state.error + assert state.step_results["call"]["output"]["workflow-dir"] == str( + resolved_dir + ) + + +def test_concurrent_nested_calls_keep_downstream_aliases_local( + tmp_path, monkeypatch, probe +): + install( + tmp_path, + definition( + "child", + [{"id": "work", "type": "probe"}], + inputs={"value": {"type": "number"}}, + outputs={"value": {"value": "{{ inputs.value }}"}}, + ), + ) + barrier = threading.Barrier(2, timeout=5) + consumed = {} + + class Consume(StepBase): + type_key = "consume" + + def execute(self, config, context): + barrier.wait() + consumed[context.item] = context.steps["call"]["output"]["value"] + return StepResult(StepStatus.COMPLETED) + + monkeypatch.setitem(STEP_REGISTRY, "consume", Consume()) + state = WorkflowEngine(tmp_path).execute( + definition( + "parent", + [ + { + "id": "spread", + "type": "fan-out", + "items": [1, 2], + "max_concurrency": 2, + "step": { + "id": "branch", + "type": "if", + "condition": True, + "then": [ + call(input={"value": "{{ item }}"}), + {"id": "consume", "type": "consume"}, + ], + }, + } + ], + ) + ) + assert state.status == RunStatus.COMPLETED + assert consumed == {1: 1, 2: 2} + items = RunState.load(state.run_id, tmp_path).execution["sequence"]["nodes"][0][ + "children" + ] + results = [ + item["nodes"][0]["children"][0]["nodes"][0]["result"]["output"]["value"] + for item in items + ] + assert results == [1, 2] + assert "call" not in state.step_results + + +def test_nested_resume_freezes_branch_binding_and_completed_prefix(tmp_path, probe): + child = definition( + "child", + [ + {"id": "prepare", "type": "probe"}, + {"id": "wait", "type": "probe", "await": True}, + ], + inputs={"approve": {"type": "boolean", "default": False}}, + ) + directory = install(tmp_path, child) + root = definition( + "parent", + [ + { + "id": "route", + "type": "if", + "condition": "{{ inputs.choose }}", + "then": [call(input={"approve": "{{ inputs.approve }}"})], + "else": [{"id": "wrong", "type": "probe"}], + } + ], + inputs={ + "choose": {"type": "boolean", "default": True}, + "approve": {"type": "boolean", "default": False}, + }, + ) + state = WorkflowEngine(tmp_path).execute(root) + assert state.status == RunStatus.PAUSED + (directory / "workflow.yml").write_text("invalid", encoding="utf-8") + state = WorkflowEngine(tmp_path).resume(state.run_id, {"choose": False}) + assert state.status == RunStatus.PAUSED + state = WorkflowEngine(tmp_path).resume(state.run_id, {"approve": True}) + assert state.status == RunStatus.COMPLETED + assert probe == {"prepare": 1, "wait": 3} + + +@pytest.mark.parametrize( + "status,aborted,handled,expected", + [ + ("failed", False, False, "failed"), + ("failed", False, True, "completed"), + ("failed", True, True, "aborted"), + ("paused", False, True, "paused"), + ], +) +def test_call_outcomes(tmp_path, probe, status, aborted, handled, expected): + install( + tmp_path, + definition( + "child", + [ + { + "id": "work", + "type": "probe", + "status": status, + "output": {"aborted": aborted}, + }, + ], + ), + ) + state = WorkflowEngine(tmp_path).execute( + definition( + "parent", + [ + call(continue_on_error=handled), + {"id": "after", "type": "probe"}, + ], + ) + ) + assert state.status.value == expected + assert probe["after"] == (expected == "completed") + assert state.step_results["call"]["output"]["status"] == ( + "aborted" if aborted else status + ) + + +@pytest.mark.parametrize( + "target", [" child", "child\n", "CHILD", "../child", "runs", 123, date(2026, 1, 1)] +) +def test_invalid_targets_are_handled_failures(tmp_path, probe, target): + state = WorkflowEngine(tmp_path).execute( + definition("parent", [call(target, continue_on_error=True)]) + ) + assert state.status == RunStatus.COMPLETED + assert state.step_results["call"]["status"] == "failed" + RunState.load(state.run_id, tmp_path) + + +@pytest.mark.parametrize( + "mapping", + [{"unknown": "x"}, {"value": []}, {"value": date(2026, 1, 1)}, [1], None], +) +def test_invalid_inputs_do_not_execute_child(tmp_path, probe, mapping): + install( + tmp_path, + definition( + "child", + [{"id": "work", "type": "probe"}], + inputs={"value": {"type": "string", "required": True}}, + ), + ) + state = WorkflowEngine(tmp_path).execute( + definition("parent", [call(input=mapping)]) + ) + assert state.status == RunStatus.FAILED + assert not probe + RunState.load(state.run_id, tmp_path) + + +def test_child_exception_propagates_despite_continue_on_error(tmp_path, monkeypatch): + class Explode(StepBase): + type_key = "explode" + + def execute(self, config, context): + raise RuntimeError("child exploded") + + monkeypatch.setitem(STEP_REGISTRY, "explode", Explode()) + install(tmp_path, definition("child", [{"id": "work", "type": "explode"}])) + + with pytest.raises(RuntimeError, match="child exploded"): + WorkflowEngine(tmp_path).execute( + definition("parent", [call(continue_on_error=True)]) + ) + + +@pytest.mark.parametrize("location", ["input", "output"]) +def test_call_expression_errors_propagate_despite_continue_on_error( + tmp_path, probe, location +): + child = definition( + "child", + [{"id": "work", "type": "probe"}], + inputs={"value": {"type": "string"}}, + outputs=( + {"value": {"value": "{{ inputs.value | from_json }}"}} + if location == "output" + else {} + ), + ) + install(tmp_path, child) + config = call( + input={ + "value": ( + "{{ inputs.value | from_json }}" if location == "input" else "not json" + ) + }, + continue_on_error=True, + ) + + with pytest.raises(ValueError, match="from_json: invalid JSON"): + WorkflowEngine(tmp_path).execute( + definition( + "parent", + [config], + inputs={"value": {"type": "string", "default": "not json"}}, + ) + ) + + +@pytest.mark.parametrize("handled", [False, True]) +def test_output_failure_retries_only_finalization( + tmp_path, monkeypatch, probe, handled +): + import specify_cli.workflows._execution as execution + from specify_cli.workflows.composition import CallError + + install(tmp_path, definition("child", [{"id": "work", "type": "probe"}])) + original = execution.evaluate_outputs + monkeypatch.setattr( + execution, + "evaluate_outputs", + lambda *_: (_ for _ in ()).throw(CallError("bad output")), + ) + state = WorkflowEngine(tmp_path).execute( + definition("parent", [call(continue_on_error=handled)]) + ) + assert state.status == ( + RunStatus.COMPLETED if handled else RunStatus.FAILED + ) + if handled: + assert probe["work"] == 1 + return + state = WorkflowEngine(tmp_path).resume(state.run_id) + assert state.status == RunStatus.FAILED + monkeypatch.setattr(execution, "evaluate_outputs", original) + state = WorkflowEngine(tmp_path).resume(state.run_id) + assert state.status == RunStatus.COMPLETED + assert probe["work"] == 1 + + +def test_output_expression_failure_retries_only_finalization(tmp_path, probe): + install( + tmp_path, + definition( + "child", + [{"id": "work", "type": "probe"}], + inputs={"value": {"type": "string"}}, + outputs={"value": {"value": "{{ inputs.value | from_json }}"}}, + ), + ) + root = definition( + "parent", + [call(input={"value": "{{ inputs.value }}"})], + inputs={"value": {"type": "string", "default": "not json"}}, + ) + + with pytest.raises(ValueError, match="from_json: invalid JSON"): + WorkflowEngine(tmp_path).execute(root, run_id="output-expression") + + state = WorkflowEngine(tmp_path).resume( + "output-expression", + {"value": '{"ok": true}'}, + ) + + assert state.status == RunStatus.COMPLETED + assert probe["work"] == 1 + assert state.step_results["call"]["output"]["value"] == {"ok": True} + + +@pytest.mark.parametrize("status", [RunStatus.PAUSED, RunStatus.FAILED]) +def test_resume_accepts_paused_and_failed_tree_backed_runs(tmp_path, probe, status): + state = WorkflowEngine(tmp_path).execute( + definition( + "parent", + [{"id": "wait", "type": "probe", "status": status.value}], + ) + ) + assert state.status == status + + resumed = WorkflowEngine(tmp_path).resume(state.run_id) + + assert resumed.status == status + + +def test_resume_rejects_running_tree_backed_run_without_writes(tmp_path, probe): + state = WorkflowEngine(tmp_path).execute( + definition("parent", [{"id": "wait", "type": "probe", "await": True}]) + ) + path = state.runs_dir / "state.json" + data = json.loads(path.read_text()) + data["status"] = RunStatus.RUNNING.value + path.write_text(json.dumps(data)) + before = path.read_bytes() + + with pytest.raises(ValueError, match="Cannot resume run .* 'running'"): + WorkflowEngine(tmp_path).resume(state.run_id) + + assert path.read_bytes() == before + + +def test_legacy_resume_adapts_once(tmp_path, probe): + root = definition( + "parent", + [ + {"id": "before", "type": "probe"}, + {"id": "wait", "type": "probe", "await": True}, + ], + inputs={"approve": {"type": "boolean", "default": False}}, + ) + state = WorkflowEngine(tmp_path).execute(root) + path = state.runs_dir / "state.json" + data = json.loads(path.read_text()) + del data["execution"] + path.write_text(json.dumps(data)) + state = WorkflowEngine(tmp_path).resume(state.run_id) + assert state.status == RunStatus.PAUSED + state = WorkflowEngine(tmp_path).resume(state.run_id, {"approve": True}) + assert state.status == RunStatus.COMPLETED + assert probe == {"before": 1, "wait": 3} + + +def test_version_one_tree_rejected_without_writes(tmp_path, probe): + state = WorkflowEngine(tmp_path).execute( + definition("parent", [{"id": "wait", "type": "probe", "await": True}]) + ) + path = state.runs_dir / "state.json" + data = json.loads(path.read_text()) + data["execution"]["version"] = 1 + path.write_text(json.dumps(data)) + before = {p.name: p.read_bytes() for p in state.runs_dir.iterdir()} + + with pytest.raises(ValueError, match="Unsupported execution version"): + WorkflowEngine(tmp_path).resume(state.run_id) + + assert {p.name: p.read_bytes() for p in state.runs_dir.iterdir()} == before + + +@pytest.mark.parametrize( + "mutation", + [ + lambda tree: tree.update(version=99), + lambda tree: tree["sequence"].update(nodes=[]), + lambda tree: tree["sequence"]["nodes"][0].update(phase="nonsense"), + lambda tree: tree["sequence"]["nodes"][0]["binding"].pop("workflow_dir"), + lambda tree: tree["sequence"]["nodes"][0]["binding"].update( + workflow_dir=42 + ), + lambda tree: tree["sequence"]["nodes"][0]["binding"].update( + workflow_dir={} + ), + lambda tree: tree["sequence"]["nodes"][0]["binding"].update( + definition=42 + ), + ], +) +def test_bad_checkpoint_rejected_without_writes(tmp_path, probe, mutation): + install( + tmp_path, + definition( + "child", + [{"id": "wait", "type": "probe", "await": True}], + inputs={"approve": {"type": "boolean", "default": False}}, + ), + ) + state = WorkflowEngine(tmp_path).execute( + definition( + "parent", + [call(input={"approve": "{{ inputs.approve }}"})], + inputs={"approve": {"type": "boolean", "default": False}}, + ) + ) + path = state.runs_dir / "state.json" + data = json.loads(path.read_text()) + mutation(data["execution"]) + path.write_text(json.dumps(data)) + before = path.read_bytes() + with pytest.raises(ValueError): + WorkflowEngine(tmp_path).resume(state.run_id) + assert path.read_bytes() == before + + +@pytest.mark.parametrize("index", [0, 1]) +def test_unbound_workflow_children_rejected_without_writes(tmp_path, probe, index): + install(tmp_path, definition("child", [{"id": "work", "type": "probe"}])) + state = WorkflowEngine(tmp_path).execute( + definition( + "parent", + [ + call(id="first"), + call(id="second"), + {"id": "wait", "type": "probe", "await": True}, + ], + inputs={"approve": {"type": "boolean", "default": False}}, + ) + ) + assert state.status == RunStatus.PAUSED + path = state.runs_dir / "state.json" + data = json.loads(path.read_text()) + data["execution"]["sequence"]["nodes"][index].pop("binding") + path.write_text(json.dumps(data)) + before = path.read_bytes() + + with pytest.raises(ValueError): + WorkflowEngine(tmp_path).resume(state.run_id, {"approve": True}) + + assert path.read_bytes() == before + + +def test_completed_container_with_unfinished_child_rejected(tmp_path, probe): + state = WorkflowEngine(tmp_path).execute( + definition( + "parent", + [ + { + "id": "route", + "type": "if", + "condition": True, + "then": [{"id": "work", "type": "probe"}], + }, + {"id": "wait", "type": "probe", "await": True}, + ], + inputs={"approve": {"type": "boolean", "default": False}}, + ) + ) + path = state.runs_dir / "state.json" + data = json.loads(path.read_text()) + data["execution"]["sequence"]["nodes"][0]["children"][0]["nodes"][0] = { + "phase": "ready" + } + path.write_text(json.dumps(data)) + before = path.read_bytes() + + with pytest.raises(ValueError): + WorkflowEngine(tmp_path).resume(state.run_id, {"approve": True}) + + assert path.read_bytes() == before + assert probe["work"] == 1 + + +@pytest.mark.parametrize("child_phase", ["ready", "blocked"]) +def test_completed_workflow_call_with_unfinished_child_rejected( + tmp_path, probe, child_phase +): + install( + tmp_path, + definition( + "child", + [{"id": "work", "type": "probe"}, {"id": "wait", "type": "probe", "await": True}], + inputs={"approve": {"type": "boolean", "default": False}}, + ), + ) + state = WorkflowEngine(tmp_path).execute( + definition( + "parent", + [call(input={"approve": "{{ inputs.approve }}"}), {"id": "pause", "type": "probe", "await": True}], + inputs={"approve": {"type": "boolean", "default": False}}, + ) + ) + path = state.runs_dir / "state.json" + data = json.loads(path.read_text()) + call_node = data["execution"]["sequence"]["nodes"][0] + call_node.update(phase="done", outcome="completed") + if child_phase == "ready": + call_node["children"][0]["nodes"][1] = {"phase": "ready"} + path.write_text(json.dumps(data)) + before = path.read_bytes() + + with pytest.raises(ValueError, match="Completed execution has unfinished children"): + WorkflowEngine(tmp_path).resume(state.run_id, {"approve": True}) + + assert path.read_bytes() == before + + +def test_handled_failed_workflow_call_can_retain_blocked_child(tmp_path, probe): + install( + tmp_path, + definition( + "child", + [ + {"id": "fail", "type": "probe", "status": "failed"}, + {"id": "unreached", "type": "probe"}, + ], + ), + ) + + state = WorkflowEngine(tmp_path).execute( + definition("parent", [call(continue_on_error=True)]) + ) + + assert state.status == RunStatus.COMPLETED + assert state.step_results["call"]["status"] == "failed" + assert RunState.load(state.run_id, tmp_path).status == RunStatus.COMPLETED + + +def test_bound_call_without_source_path_resumes_with_null_workflow_dir( + tmp_path, monkeypatch, probe +): + import specify_cli.workflows._execution as execution + + child = definition( + "child", + [{"id": "wait", "type": "probe", "await": True}], + inputs={"approve": {"type": "boolean", "default": False}}, + ) + monkeypatch.setattr(execution, "resolve_target", lambda *_: child) + state = WorkflowEngine(tmp_path).execute( + definition( + "parent", + [call(input={"approve": "{{ inputs.approve }}"})], + inputs={"approve": {"type": "boolean", "default": False}}, + ) + ) + assert state.status == RunStatus.PAUSED + assert state.execution["sequence"]["nodes"][0]["binding"]["workflow_dir"] is None + + state = WorkflowEngine(tmp_path).resume(state.run_id, {"approve": True}) + + assert state.status == RunStatus.COMPLETED + + +@pytest.mark.parametrize("failure_after_replace", [False, True]) +def test_checkpoint_failure_leaves_running_run_not_resumable( + tmp_path, monkeypatch, probe, failure_after_replace +): + from specify_cli.workflows._execution import CheckpointError + + original = RunState._atomic_write_json + failed = False + + def write(path, data): + nonlocal failed + record = data.get("step_results", {}).get("work") + if path.name == "state.json" and record and not failed: + failed = True + if failure_after_replace: + original(path, data) + raise OSError("checkpoint failure") + original(path, data) + + monkeypatch.setattr(RunState, "_atomic_write_json", staticmethod(write)) + with pytest.raises(CheckpointError, match="checkpoint failure"): + WorkflowEngine(tmp_path).execute( + definition("parent", [{"id": "work", "type": "probe"}]), run_id="fault" + ) + disk = json.loads( + (tmp_path / ".specify/workflows/runs/fault/state.json").read_text() + ) + node = disk["execution"]["sequence"]["nodes"][0] + assert node["phase"] == ("done" if failure_after_replace else "ready") + assert ("work" in disk["step_results"]) is failure_after_replace + with pytest.raises( + ValueError, match="Cannot resume run 'fault' with status 'running'" + ): + WorkflowEngine(tmp_path).resume("fault") + assert probe["work"] == 1 + + +def _interrupt_state_write(monkeypatch, when): + """Raise ``KeyboardInterrupt`` from the first ``state.json`` write matching *when*.""" + original = RunState._atomic_write_json + fired = [] + + def write(path, data): + if path.name == "state.json" and not fired and when(data): + fired.append(path) + raise KeyboardInterrupt + original(path, data) + + monkeypatch.setattr(RunState, "_atomic_write_json", staticmethod(write)) + return fired + + +def _logged_events(state): + log = (state.runs_dir / "log.jsonl").read_text(encoding="utf-8").splitlines() + return [json.loads(line)["event"] for line in log] + + +@pytest.mark.parametrize("scope", ["root", "workflow-call"]) +def test_interrupt_during_checkpoint_pauses_run(tmp_path, monkeypatch, probe, scope): + steps = [{"id": "first", "type": "probe"}, {"id": "second", "type": "probe"}] + if scope == "workflow-call": + install(tmp_path, definition("child", steps)) + steps = [call()] + # Interrupt the checkpoint that marks ``second`` active, after ``first`` ran. + fired = _interrupt_state_write( + monkeypatch, lambda data: data.get("current_step_id") == "second" + ) + + state = WorkflowEngine(tmp_path).execute(definition("parent", steps)) + + # A graceful interrupt is not a checkpoint failure: it reaches the engine's + # pause path, as on main, even inside a called workflow. + assert fired + assert state.status == RunStatus.PAUSED + assert RunState.load(state.run_id, tmp_path).status == RunStatus.PAUSED + assert _logged_events(state)[-1] == "workflow_interrupted" + assert probe == {"first": 1} + + state = WorkflowEngine(tmp_path).resume(state.run_id) + + assert state.status == RunStatus.COMPLETED + assert probe == {"first": 1, "second": 1} + + +def test_interrupt_during_resume_checkpoint_pauses_run(tmp_path, monkeypatch, probe): + root = definition( + "parent", + [ + {"id": "wait", "type": "probe", "await": True}, + {"id": "next", "type": "probe"}, + ], + inputs={"approve": {"type": "boolean", "default": False}}, + ) + state = WorkflowEngine(tmp_path).execute(root) + assert state.status == RunStatus.PAUSED + # Interrupt the checkpoint that records ``wait`` as completed on resume. + fired = _interrupt_state_write( + monkeypatch, + lambda data: data.get("step_results", {}).get("wait", {}).get("status") + == "completed", + ) + + state = WorkflowEngine(tmp_path).resume(state.run_id, {"approve": True}) + + assert fired + assert state.status == RunStatus.PAUSED + assert RunState.load(state.run_id, tmp_path).status == RunStatus.PAUSED + assert _logged_events(state)[-1] == "workflow_interrupted" + + state = WorkflowEngine(tmp_path).resume(state.run_id) + + # The interrupted checkpoint's transition is saved by the pause, so the + # completed ``wait`` is not run a third time. + assert state.status == RunStatus.COMPLETED + assert probe == {"wait": 2, "next": 1} + + +@pytest.mark.parametrize("failure", [KeyboardInterrupt, RuntimeError]) +def test_leave_failure_preserves_step_exception(tmp_path, monkeypatch, failure): + from specify_cli.workflows._execution import Execution + + class Explode(StepBase): + type_key = "explode" + + def execute(self, config, context): + raise failure("boom") + + original_transition = Execution.transition + + def transition(self, operation, occurrence, *args, **kwargs): + if operation == "leave": + raise ValueError("leave failed") + return original_transition(self, operation, occurrence, *args, **kwargs) + + monkeypatch.setitem(STEP_REGISTRY, "explode", Explode()) + monkeypatch.setattr(Execution, "transition", transition) + engine = WorkflowEngine(tmp_path) + root = definition("parent", [{"id": "work", "type": "explode"}]) + if failure is KeyboardInterrupt: + state = engine.execute(root, run_id="leave-failure") + assert state.status == RunStatus.PAUSED + assert _logged_events(state)[-1] == "workflow_interrupted" + else: + with pytest.raises(RuntimeError, match="boom"): + engine.execute(root, run_id="leave-failure") + state = RunState.load("leave-failure", tmp_path) + assert state.error == "boom" + + +def test_checkpoint_failure_stops_concurrent_fan_out_logging(tmp_path, monkeypatch): + import specify_cli.workflows._execution as execution + from specify_cli.workflows._execution import CheckpointError + + items = {} + first_committed = threading.Event() + second_failed = threading.Event() + + class Sync(StepBase): + type_key = "sync" + + def execute(self, config, context): + items[threading.current_thread()] = context.item + if context.item == 2: + assert first_committed.wait(5) + return StepResult(output={"value": context.item}) + + original_transition = execution.Execution.transition + original_notify = execution.Execution.notify + original_write = RunState._atomic_write_json + + def transition(self, *args, **kwargs): + item = items.get(threading.current_thread()) + try: + original_transition(self, *args, **kwargs) + except CheckpointError: + if item == 2: + second_failed.set() + raise + + def notify(self, operation, occurrence, **kwargs): + if items.get(threading.current_thread()) == 1 and operation == "finish": + # Item 1 committed and released the lock; item 2 now fails its + # checkpoint before item 1 emits its completion event. + first_committed.set() + assert second_failed.wait(5) + return original_notify(self, operation, occurrence, **kwargs) + + def write(path, data): + if items.get(threading.current_thread()) == 2: + raise OSError("disk full") + original_write(path, data) + + monkeypatch.setitem(STEP_REGISTRY, "sync", Sync()) + monkeypatch.setattr(execution.Execution, "transition", transition) + monkeypatch.setattr(execution.Execution, "notify", notify) + monkeypatch.setattr(RunState, "_atomic_write_json", staticmethod(write)) + with pytest.raises(CheckpointError): + WorkflowEngine(tmp_path).execute( + definition( + "parent", + [ + { + "id": "fan", + "type": "fan-out", + "items": [1, 2], + "max_concurrency": 2, + "step": {"type": "sync"}, + } + ], + ), + run_id="concurrent-fault", + ) + + assert second_failed.is_set() + log = tmp_path / ".specify/workflows/runs/concurrent-fault/log.jsonl" + events = [json.loads(line) for line in log.read_text().splitlines()] + item_events = [ + (entry["event"], entry["step_id"]) + for entry in events + if entry.get("step_id", "").startswith("fan:") + ] + assert sorted(item_events) == [ + ("step_started", "fan:item:0"), + ("step_started", "fan:item:1"), + ] + + +def test_completion_log_failure_does_not_replay_committed_step( + tmp_path, monkeypatch, probe +): + original = RunState.append_log + failed = False + + def log(self, entry): + nonlocal failed + if entry["event"] == "step_completed" and not failed: + failed = True + raise OSError("log failed") + original(self, entry) + + monkeypatch.setattr(RunState, "append_log", log) + with pytest.raises(OSError, match="log failed"): + WorkflowEngine(tmp_path).execute( + definition("parent", [{"id": "work", "type": "probe"}]), run_id="log" + ) + state = WorkflowEngine(tmp_path).resume("log") + assert state.status == RunStatus.COMPLETED + assert probe["work"] == 1 + + +def test_container_completion_log_failure_does_not_replay_expansion( + tmp_path, monkeypatch, probe +): + original = RunState.append_log + failed = False + expanded = 0 + + class Expand(StepBase): + type_key = "expand" + + def execute(self, config, context): + nonlocal expanded + expanded += 1 + return StepResult( + StepStatus.COMPLETED, + next_steps=[{"id": "work", "type": "probe"}], + ) + + def log(self, entry): + nonlocal failed + if entry == { + "event": "step_completed", + "step_id": "expand", + "status": "completed", + } and not failed: + failed = True + raise OSError("log failed") + original(self, entry) + + monkeypatch.setitem(STEP_REGISTRY, "expand", Expand()) + monkeypatch.setattr(RunState, "append_log", log) + with pytest.raises(OSError, match="log failed"): + WorkflowEngine(tmp_path).execute( + definition("parent", [{"id": "expand", "type": "expand"}]), + run_id="container-log", + ) + + state = WorkflowEngine(tmp_path).resume("container-log") + + assert state.status == RunStatus.COMPLETED + assert expanded == 1 + assert probe["work"] == 1 + + +@pytest.mark.parametrize("kind", ["if", "while", "do-while", "fan-out"]) +def test_expansion_resume_preserves_completed_work(tmp_path, probe, kind): + body = [ + {"id": "prepare", "type": "probe"}, + {"id": "wait", "type": "probe", "await": True}, + ] + config = {"id": "outer", "type": kind, "condition": True, "max_iterations": 2} + if kind == "if": + config["then"] = body + elif kind == "fan-out": + config.update( + items=[1, 2], + max_concurrency=2, + step={"id": "branch", "type": "if", "condition": True, "then": body}, + ) + else: + config["steps"] = body + state = WorkflowEngine(tmp_path).execute( + definition( + "parent", + [config], + inputs={"approve": {"type": "boolean", "default": False}}, + ) + ) + assert state.status == RunStatus.PAUSED + state = WorkflowEngine(tmp_path).resume(state.run_id, {"approve": True}) + assert state.status == RunStatus.COMPLETED + assert probe["prepare"] == (1 if kind == "if" else 2) + + +def test_replay_restores_fan_out_item_aliases_from_completed_if(tmp_path, probe): + state = WorkflowEngine(tmp_path).execute( + definition( + "parent", + [ + { + "id": "route", + "type": "if", + "condition": True, + "then": [ + { + "id": "fan", + "type": "fan-out", + "items": [1], + "step": { + "id": "template", + "type": "probe", + "value": "{{ item }}", + }, + } + ], + }, + {"id": "wait", "type": "probe", "await": True}, + { + "id": "join", + "type": "fan-in", + "wait_for": ["fan"], + "output": {"merged": "{{ steps.fan.output.results }}"}, + }, + ], + inputs={"approve": {"type": "boolean", "default": False}}, + ) + ) + assert state.status == RunStatus.PAUSED + + state = WorkflowEngine(tmp_path).resume(state.run_id, {"approve": True}) + + assert state.status == RunStatus.COMPLETED + assert state.step_results["fan:template:0"]["output"] == {"value": 1} + assert state.step_results["join"]["output"]["merged"] == [{"value": 1}] + + +def test_resume_restores_completed_fan_out_item_aliases(tmp_path, monkeypatch, probe): + class PauseSecond(StepBase): + type_key = "pause-second" + + def execute(self, config, context): + if context.item == 2 and not context.inputs["approve"]: + return StepResult(StepStatus.PAUSED) + return StepResult(output={"value": context.item}) + + monkeypatch.setitem(STEP_REGISTRY, "pause-second", PauseSecond()) + state = WorkflowEngine(tmp_path).execute( + definition( + "parent", + [ + { + "id": "fan", + "type": "fan-out", + "items": [1, 2], + "step": {"id": "template", "type": "pause-second"}, + }, + { + "id": "join", + "type": "fan-in", + "wait_for": ["fan"], + "output": {"merged": "{{ steps.fan.output.results }}"}, + }, + ], + inputs={"approve": {"type": "boolean", "default": False}}, + ) + ) + assert state.status == RunStatus.PAUSED + + state = WorkflowEngine(tmp_path).resume(state.run_id, {"approve": True}) + + assert state.status == RunStatus.COMPLETED + assert state.step_results["fan:template:0"]["output"] == {"value": 1} + assert state.step_results["fan:template:1"]["output"] == {"value": 2} + assert state.step_results["join"]["output"]["merged"] == [ + {"value": 1}, + {"value": 2}, + ] + + +def test_replay_keeps_private_nested_fan_out_aliases_private(tmp_path, probe): + state = WorkflowEngine(tmp_path).execute( + definition( + "parent", + [ + { + "id": "outer", + "type": "fan-out", + "items": [1], + "step": { + "id": "outer-item", + "type": "fan-out", + "items": [1], + "step": {"id": "inner", "type": "probe"}, + }, + }, + {"id": "wait", "type": "probe", "await": True}, + ], + inputs={"approve": {"type": "boolean", "default": False}}, + ) + ) + assert state.status == RunStatus.PAUSED + + state = WorkflowEngine(tmp_path).resume(state.run_id, {"approve": True}) + + assert state.status == RunStatus.COMPLETED + assert "outer:outer-item:0" in state.step_results + assert "outer:outer-item:0:inner:0" not in state.step_results + + +def test_replay_does_not_execute_completed_fan_out_or_emit_callbacks(tmp_path, probe): + callbacks = [] + engine = WorkflowEngine(tmp_path) + engine.on_step_start = lambda step_id, _label: callbacks.append(step_id) + root = definition( + "parent", + [ + { + "id": "fan", + "type": "fan-out", + "items": [1, 2], + "step": { + "id": "template", + "type": "probe", + "value": "{{ item }}", + }, + }, + {"id": "wait", "type": "probe", "await": True}, + ], + inputs={"approve": {"type": "boolean", "default": False}}, + ) + state = engine.execute(root) + assert state.status == RunStatus.PAUSED + assert probe == {"template": 2, "wait": 1} + callbacks.clear() + + state = engine.resume(state.run_id, {"approve": True}) + + assert state.status == RunStatus.COMPLETED + assert probe == {"template": 2, "wait": 2} + assert callbacks == ["wait"] + assert [ + (entry["event"], entry["step_id"]) + for entry in state.log_entries + if entry["event"] in {"step_started", "step_completed"} + ] == [("step_started", "wait"), ("step_completed", "wait")] + + +def test_fan_out_item_aliases_are_projected_under_run_lock(tmp_path, monkeypatch): + from specify_cli.workflows._execution import Execution + + class PauseSecond(StepBase): + type_key = "pause-second" + + def execute(self, config, context): + if context.item == 2 and not context.inputs["approve"]: + return StepResult(StepStatus.PAUSED) + return StepResult(output={"value": context.item}) + + monkeypatch.setitem(STEP_REGISTRY, "pause-second", PauseSecond()) + state = WorkflowEngine(tmp_path).execute( + definition( + "parent", + [ + { + "id": "fan", + "type": "fan-out", + "items": [1, 2], + "max_concurrency": 2, + "step": {"id": "template", "type": "pause-second"}, + } + ], + inputs={"approve": {"type": "boolean", "default": False}}, + ) + ) + assert state.status == RunStatus.PAUSED + owned = [] + original = Execution.project_alias + + def spy(self, context, name, result): + owned.append(self.state._lock._is_owned()) + return original(self, context, name, result) + + monkeypatch.setattr(Execution, "project_alias", spy) + state = WorkflowEngine(tmp_path).resume(state.run_id, {"approve": True}) + + assert state.status == RunStatus.COMPLETED + assert owned and all(owned) + assert state.step_results["fan:template:0"]["output"] == {"value": 1} + assert state.step_results["fan:template:1"]["output"] == {"value": 2} + + +def test_replay_does_not_start_worker_threads(tmp_path, monkeypatch, probe): + import specify_cli.workflows._execution as execution + + state = WorkflowEngine(tmp_path).execute( + definition( + "parent", + [ + { + "id": "fan", + "type": "fan-out", + "items": [1, 2], + "max_concurrency": 2, + "step": {"id": "template", "type": "probe"}, + }, + {"id": "wait", "type": "probe", "await": True}, + ], + inputs={"approve": {"type": "boolean", "default": False}}, + ) + ) + assert state.status == RunStatus.PAUSED + + class NoThreads: + def __init__(self, *args, **kwargs): + raise AssertionError("replay must not start worker threads") + + monkeypatch.setattr(execution, "ThreadPoolExecutor", NoThreads) + state = WorkflowEngine(tmp_path).resume(state.run_id, {"approve": True}) + + assert state.status == RunStatus.COMPLETED + assert probe == {"template": 2, "wait": 2} + + +def test_fan_out_without_children_survives_replay(tmp_path, probe): + state = WorkflowEngine(tmp_path).execute( + definition( + "parent", + [ + {"id": "fan", "type": "fan-out", "items": [1, 2], "step": {}}, + {"id": "wait", "type": "probe", "await": True}, + ], + inputs={"approve": {"type": "boolean", "default": False}}, + ) + ) + assert state.status == RunStatus.PAUSED + + state = WorkflowEngine(tmp_path).resume(state.run_id, {"approve": True}) + + assert state.status == RunStatus.COMPLETED + assert state.step_results["fan"]["output"]["results"] == [] + + +def test_replay_does_not_evaluate_completed_loop_condition( + tmp_path, monkeypatch, probe +): + import specify_cli.workflows._execution as execution + + state = WorkflowEngine(tmp_path).execute( + definition( + "parent", + [ + { + "id": "loop", + "type": "do-while", + "condition": False, + "steps": [{"id": "work", "type": "probe"}], + }, + {"id": "wait", "type": "probe", "await": True}, + ], + inputs={"approve": {"type": "boolean", "default": False}}, + ) + ) + assert state.status == RunStatus.PAUSED + monkeypatch.setattr( + execution, + "evaluate_condition", + lambda *_: pytest.fail("replay must not evaluate loop conditions"), + ) + + state = WorkflowEngine(tmp_path).resume(state.run_id, {"approve": True}) + + assert state.status == RunStatus.COMPLETED + assert probe == {"work": 1, "wait": 2} + + +def test_replay_does_not_commit_completed_nodes(tmp_path, monkeypatch, probe): + import specify_cli.workflows._execution as execution + + state = WorkflowEngine(tmp_path).execute( + definition( + "parent", + [ + {"id": "done", "type": "probe"}, + {"id": "wait", "type": "probe", "await": True}, + ], + inputs={"approve": {"type": "boolean", "default": False}}, + ) + ) + assert state.status == RunStatus.PAUSED + original = execution.Execution.transition + committed = [] + + def transition(self, operation, occurrence, changes=None, **kwargs): + if occurrence.node["phase"] == "done": + committed.append(occurrence.node) + return original(self, operation, occurrence, changes, **kwargs) + + monkeypatch.setattr(execution.Execution, "transition", transition) + + state = WorkflowEngine(tmp_path).resume(state.run_id, {"approve": True}) + + assert state.status == RunStatus.COMPLETED + assert committed == [] + + +def test_composed_fan_out_joins_container_results(tmp_path, probe): + install( + tmp_path, + definition( + "child", + [ + { + "id": "fan", + "type": "fan-out", + "items": [1, 2], + "step": { + "id": "template", + "type": "probe", + "value": "{{ item }}", + }, + }, + { + "id": "join", + "type": "fan-in", + "wait_for": ["fan"], + "output": {"merged": "{{ steps.fan.output.results }}"}, + }, + ], + outputs={"results": {"value": "{{ steps.join.output.merged }}"}}, + ), + ) + + state = WorkflowEngine(tmp_path).execute(definition("parent", [call()])) + + assert state.status == RunStatus.COMPLETED + assert state.step_results["call"]["output"]["results"] == [ + {"value": 1}, + {"value": 2}, + ] + assert "fan:template:0" not in state.step_results + assert "fan:template:1" not in state.step_results + + +def test_fan_in_rejects_fan_out_item_alias(): + for alias in ("fan:template:0", "fan:template:not-an-item"): + errors = validate_workflow( + definition( + "parent", + [ + { + "id": "fan", + "type": "fan-out", + "items": [1], + "step": {"id": "template", "type": "probe"}, + }, + {"id": "join", "type": "fan-in", "wait_for": [alias]}, + ], + ) + ) + + assert any("unknown or not-yet-declared" in error for error in errors), alias + + +def test_fan_in_rejects_item_alias_at_runtime(): + from specify_cli.workflows.base import StepContext + from specify_cli.workflows.step.fan_in import FanInStep + + result = FanInStep().execute( + {"id": "join", "type": "fan-in", "wait_for": ["fan:template:0"]}, + StepContext(), + ) + + assert result.status == StepStatus.FAILED + assert "fan-out item alias" in (result.error or "") + + +def test_fan_in_container_join_in_loop_sees_current_iteration(tmp_path, monkeypatch, probe): + calls = [] + + class Varying(StepBase): + type_key = "varying" + + def execute(self, config, context): + calls.append(1) + items = [10, 20] if len(calls) == 1 else [30, 40] + return StepResult(output={"items": items}) + + monkeypatch.setitem(STEP_REGISTRY, "varying", Varying()) + state = WorkflowEngine(tmp_path).execute( + definition( + "parent", + [ + { + "id": "loop", + "type": "do-while", + "condition": True, + "max_iterations": 2, + "steps": [ + {"id": "varying", "type": "varying"}, + { + "id": "fan", + "type": "fan-out", + "items": "{{ steps.varying.output.items }}", + "step": { + "id": "template", + "type": "probe", + "value": "{{ item }}", + }, + }, + { + "id": "join", + "type": "fan-in", + "wait_for": ["fan"], + "output": {"merged": "{{ steps.fan.output.results }}"}, + }, + ], + } + ], + ) + ) + + assert state.status == RunStatus.COMPLETED + assert state.step_results["join"]["output"]["merged"] == [ + {"value": 30}, + {"value": 40}, + ] + + +def test_fan_out_in_later_loop_iteration_uses_qualified_aliases(tmp_path, probe): + state = WorkflowEngine(tmp_path).execute( + definition( + "parent", + [ + { + "id": "loop", + "type": "do-while", + "condition": True, + "max_iterations": 2, + "steps": [ + { + "id": "fan", + "type": "fan-out", + "items": [1, 2], + "step": { + "id": "template", + "type": "probe", + "value": "{{ item }}", + }, + } + ], + } + ], + ) + ) + + assert state.status == RunStatus.COMPLETED + assert state.step_results["fan:template:0"]["output"]["value"] == 1 + assert state.step_results["fan:template:1"]["output"]["value"] == 2 + assert state.step_results["loop:fan:1:template:0"]["output"]["value"] == 1 + assert state.step_results["loop:fan:1:template:1"]["output"]["value"] == 2 + + +def test_fan_out_events_and_callbacks_use_qualified_item_ids(tmp_path, probe): + callbacks = [] + engine = WorkflowEngine(tmp_path) + engine.on_step_start = lambda step_id, _label: callbacks.append(step_id) + + state = engine.execute( + definition( + "parent", + [ + { + "id": "fan", + "type": "fan-out", + "items": [1, 2], + "step": {"id": "template", "type": "probe"}, + } + ], + ) + ) + + events = [ + (entry["event"], entry["step_id"]) + for entry in state.log_entries + if entry["event"] in {"step_started", "step_completed"} + ] + assert callbacks == ["fan", "fan:template:0", "fan:template:1"] + assert events == [ + ("step_started", "fan"), + ("step_completed", "fan"), + ("step_started", "fan:template:0"), + ("step_completed", "fan:template:0"), + ("step_started", "fan:template:1"), + ("step_completed", "fan:template:1"), + ] + + +def test_unnamed_fan_out_templates_use_item_id_for_results_and_events(tmp_path, probe): + callbacks = [] + engine = WorkflowEngine(tmp_path) + engine.on_step_start = lambda step_id, _label: callbacks.append(step_id) + + state = engine.execute( + definition( + "parent", + [ + { + "id": "fan", + "type": "fan-out", + "items": [1], + "step": {"type": "probe", "value": "{{ item }}"}, + } + ], + ) + ) + + item_events = [ + entry["step_id"] + for entry in state.log_entries + if entry["event"] in {"step_started", "step_completed"} + and entry["step_id"].startswith("fan:") + ] + assert callbacks == ["fan", "fan:item:0"] + assert item_events == ["fan:item:0", "fan:item:0"] + assert state.step_results["fan:item:0"]["output"] == {"value": 1} + + +def test_loop_events_and_callbacks_use_qualified_iteration_ids(tmp_path, probe): + callbacks = [] + engine = WorkflowEngine(tmp_path) + engine.on_step_start = lambda step_id, _label: callbacks.append(step_id) + + state = engine.execute( + definition( + "parent", + [ + { + "id": "loop", + "type": "do-while", + "condition": True, + "max_iterations": 2, + "steps": [{"id": "body", "type": "probe"}], + } + ], + ) + ) + + events = [ + (entry["event"], entry["step_id"]) + for entry in state.log_entries + if entry["event"] in {"step_started", "step_completed"} + ] + assert callbacks == ["loop", "body", "loop:body:1"] + assert events == [ + ("step_started", "loop"), + ("step_completed", "loop"), + ("step_started", "body"), + ("step_completed", "body"), + ("step_started", "loop:body:1"), + ("step_completed", "loop:body:1"), + ] + + +def test_private_scope_events_include_execution_path_and_workflow_id(tmp_path, probe): + callbacks = [] + engine = WorkflowEngine(tmp_path) + engine.on_step_start = lambda step_id, _label: callbacks.append(step_id) + install(tmp_path, definition("child", [{"id": "work", "type": "probe"}])) + state = engine.execute(definition("parent", [call()])) + + call_entries = [ + entry + for entry in state.log_entries + if entry.get("step_id") == "call" and entry["event"].startswith("step_") + ] + child_entries = [ + entry + for entry in state.log_entries + if entry.get("step_id") == "work" and entry["event"].startswith("step_") + ] + assert all("execution_path" not in entry for entry in call_entries) + assert all("workflow_id" not in entry for entry in call_entries) + assert all(entry["workflow_id"] == "child" for entry in child_entries) + assert all(entry["execution_path"] == [0, "workflow", 0] for entry in child_entries) + assert callbacks == ["call", "work"] + + +def test_completed_workflow_call_replay_emits_no_events(tmp_path, probe): + callbacks = [] + engine = WorkflowEngine(tmp_path) + engine.on_step_start = lambda step_id, _label: callbacks.append(step_id) + install(tmp_path, definition("child", [{"id": "work", "type": "probe"}])) + root = definition( + "parent", + [ + call(), + {"id": "wait", "type": "probe", "await": True}, + ], + inputs={"approve": {"type": "boolean", "default": False}}, + ) + + state = engine.execute(root) + assert state.status == RunStatus.PAUSED + callbacks.clear() + + state = engine.resume(state.run_id, {"approve": True}) + + events = [ + (entry["event"], entry["step_id"]) + for entry in state.log_entries + if entry["event"] in {"step_started", "step_completed"} + ] + assert state.status == RunStatus.COMPLETED + assert callbacks == ["wait"] + assert events == [("step_started", "wait"), ("step_completed", "wait")] + + +def test_unfinished_workflow_call_emits_events_on_resume(tmp_path, probe): + callbacks = [] + engine = WorkflowEngine(tmp_path) + engine.on_step_start = lambda step_id, _label: callbacks.append(step_id) + install( + tmp_path, + definition( + "child", + [{"id": "wait", "type": "probe", "await": True}], + inputs={"approve": {"type": "boolean", "default": False}}, + ), + ) + root = definition( + "parent", + [call(input={"approve": "{{ inputs.approve }}"})], + inputs={"approve": {"type": "boolean", "default": False}}, + ) + + state = engine.execute(root) + assert state.status == RunStatus.PAUSED + callbacks.clear() + + state = engine.resume(state.run_id, {"approve": True}) + + events = [ + (entry["event"], entry["step_id"]) + for entry in state.log_entries + if entry["event"] in {"step_started", "step_completed"} + ] + assert state.status == RunStatus.COMPLETED + assert callbacks == ["call", "wait"] + assert events == [ + ("step_started", "call"), + ("step_started", "wait"), + ("step_completed", "wait"), + ("step_completed", "call"), + ] + + +@pytest.mark.parametrize( + "template, expected_event, status", + [ + ({"status": "failed"}, "step_failed", RunStatus.FAILED), + ( + {"status": "failed", "continue_on_error": True}, + "step_continue_on_error", + RunStatus.COMPLETED, + ), + ( + {"status": "failed", "output": {"aborted": True}}, + "workflow_aborted", + RunStatus.ABORTED, + ), + ], +) +def test_fan_out_failure_events_use_qualified_item_ids( + tmp_path, probe, template, expected_event, status +): + state = WorkflowEngine(tmp_path).execute( + definition( + "parent", + [ + { + "id": "fan", + "type": "fan-out", + "items": [1], + "step": {"id": "template", "type": "probe", **template}, + } + ], + ) + ) + + failures = [ + entry + for entry in state.log_entries + if entry["event"] in { + "step_failed", + "step_continue_on_error", + "workflow_aborted", + } + ] + assert state.status == status + assert len(failures) == 1 + assert failures[0]["step_id"] == "fan:template:0" + assert failures[0]["event"] == expected_event + + +@pytest.mark.parametrize("mode", ["unknown", "disabled", "mismatch", "cycle"]) +def test_resolution_failures_are_call_failures(tmp_path, probe, mode): + if mode != "unknown": + child = definition( + "child", + [call("parent") if mode == "cycle" else {"id": "work", "type": "probe"}], + ) + directory = install(tmp_path, child, enabled=mode != "disabled") + if mode == "mismatch": + child.data["workflow"]["id"] = "other" + (directory / "workflow.yml").write_text(yaml.safe_dump(child.data)) + state = WorkflowEngine(tmp_path).execute( + definition("parent", [call(continue_on_error=True)]) + ) + assert state.status == RunStatus.COMPLETED + assert state.step_results["call"]["status"] == "failed" + assert not probe + + +@pytest.mark.parametrize( + "error, handled", + [ + (ValueError("bad overlay"), True), + (FileNotFoundError("gone"), True), + (RuntimeError("resolver bug"), False), + ], +) +def test_only_resolver_contract_errors_are_call_failures( + tmp_path, monkeypatch, probe, error, handled +): + from specify_cli.workflows.overlay.resolver import WorkflowResolver + + install(tmp_path, definition("child", [{"id": "work", "type": "probe"}])) + + def resolve(self, workflow_id): + raise error + + monkeypatch.setattr(WorkflowResolver, "resolve", resolve) + parent = definition("parent", [call(continue_on_error=True)]) + if handled: + state = WorkflowEngine(tmp_path).execute(parent) + assert state.status == RunStatus.COMPLETED + assert state.step_results["call"]["status"] == "failed" + else: + with pytest.raises(RuntimeError, match="resolver bug"): + WorkflowEngine(tmp_path).execute(parent) + assert not probe + + +@pytest.mark.parametrize( + "child_continue, call_continue", [(False, False), (True, False), (False, True)] +) +def test_unknown_child_step_is_a_reported_call_failure( + tmp_path, monkeypatch, probe, child_continue, call_continue +): + class TemporarilyInstalled(StepBase): + type_key = "temporarily-installed" + + def execute(self, config, context): + return StepResult(StepStatus.COMPLETED) + + monkeypatch.setitem(STEP_REGISTRY, "temporarily-installed", TemporarilyInstalled()) + install( + tmp_path, + definition( + "child", + [ + {"id": "wait", "type": "probe", "await": True}, + { + "id": "missing", + "type": "temporarily-installed", + "continue_on_error": child_continue, + } + ], + inputs={"approve": {"type": "boolean", "default": False}}, + ), + ) + state = WorkflowEngine(tmp_path).execute( + definition( + "parent", + [ + call( + input={"approve": "{{ inputs.approve }}"}, + continue_on_error=call_continue, + ), + {"id": "after", "type": "probe"}, + ], + inputs={"approve": {"type": "boolean", "default": False}}, + ) + ) + assert state.status == RunStatus.PAUSED + monkeypatch.delitem(STEP_REGISTRY, "temporarily-installed") + state = WorkflowEngine(tmp_path).resume(state.run_id, {"approve": True}) + + error = "Unknown step type: 'temporarily-installed'" + events = [entry["event"] for entry in state.log_entries] + # The missing step itself is terminal inside the child, as at the root. + assert [ + entry["event"] + for entry in state.log_entries + if entry.get("step_id") == "missing" and entry.get("workflow_id") == "child" + ] == ["step_started", "step_failed"] + if call_continue: + # At the call boundary it is a reported child failure. + assert state.status == RunStatus.COMPLETED + assert state.step_results["call"]["status"] == "failed" + assert state.step_results["call"]["output"]["error"] == error + assert probe == {"wait": 2, "after": 1} + assert "step_continue_on_error" in events + else: + assert state.status == RunStatus.FAILED + assert state.error == error + assert probe == {"wait": 2} + assert "step_continue_on_error" not in events + + +@pytest.mark.parametrize("call_continue", [False, True]) +def test_unknown_child_step_at_bind_is_a_reported_call_failure( + tmp_path, monkeypatch, probe, call_continue +): + install( + tmp_path, + definition("child", [{"id": "missing", "type": "later-installed"}]), + ) + parent = definition( + "parent", + [call(continue_on_error=call_continue), {"id": "after", "type": "probe"}], + ) + state = WorkflowEngine(tmp_path).execute(parent, run_id="bind") + + assert "invalid type 'later-installed'" in state.step_results["call"]["error"] + assert "binding" not in state.execution["sequence"]["nodes"][0] + if call_continue: + # Same treatment as an implementation that disappears after binding. + assert state.status == RunStatus.COMPLETED + assert state.step_results["call"]["status"] == "failed" + assert probe == {"after": 1} + return + + assert state.status == RunStatus.FAILED + assert not probe + + class LaterInstalled(StepBase): + type_key = "later-installed" + + def execute(self, config, context): + return StepResult(StepStatus.COMPLETED) + + monkeypatch.setitem(STEP_REGISTRY, "later-installed", LaterInstalled()) + state = WorkflowEngine(tmp_path).resume("bind") + + assert state.status == RunStatus.COMPLETED + assert state.step_results["call"]["status"] == "completed" + assert probe == {"after": 1} + + +def test_step_reported_unknown_type_error_is_handled_like_any_failure( + tmp_path, monkeypatch, probe +): + class Mimic(StepBase): + type_key = "mimic" + + def execute(self, config, context): + return StepResult(StepStatus.FAILED, error="Unknown step type: 'mimic'") + + monkeypatch.setitem(STEP_REGISTRY, "mimic", Mimic()) + install(tmp_path, definition("child", [{"id": "work", "type": "mimic"}])) + state = WorkflowEngine(tmp_path).execute( + definition( + "parent", + [ + {"id": "direct", "type": "mimic", "continue_on_error": True}, + call(continue_on_error=True), + {"id": "after", "type": "probe"}, + ], + ) + ) + + assert state.status == RunStatus.COMPLETED + assert state.step_results["direct"]["status"] == "failed" + assert state.step_results["call"]["status"] == "failed" + assert probe == {"after": 1} + + +def test_unknown_step_type_resumes_after_reinstall(tmp_path, monkeypatch, probe): + class Reinstalled(StepBase): + type_key = "temporarily-installed" + + def execute(self, config, context): + return StepResult(StepStatus.COMPLETED) + + state = WorkflowEngine(tmp_path).execute( + definition( + "parent", + [{"id": "missing", "type": "temporarily-installed"}], + ), + run_id="reinstall", + ) + assert state.status == RunStatus.FAILED + + monkeypatch.setitem(STEP_REGISTRY, "temporarily-installed", Reinstalled()) + state = WorkflowEngine(tmp_path).resume("reinstall") + + assert state.status == RunStatus.COMPLETED + assert state.step_results["missing"]["status"] == "completed" + + +@pytest.mark.parametrize("workers", [1, 2]) +@pytest.mark.parametrize("named", [False, True]) +def test_unknown_fan_out_template_step_always_fails_despite_continue_on_error( + tmp_path, monkeypatch, probe, workers, named +): + local_name = "missing" if named else "step-0" + alias_name = "missing" if named else "item" + state = WorkflowEngine(tmp_path).execute( + definition( + "parent", + [ + # An inherited same-name result must not become the item result. + {"id": local_name, "type": "probe", "value": "parent"}, + { + "id": "fan", + "type": "fan-out", + "items": [1, 2], + "max_concurrency": workers, + "step": { + **({"id": "missing"} if named else {}), + "type": "not-installed", + "continue_on_error": True, + }, + }, + {"id": "after", "type": "probe"}, + ], + ) + ) + + assert state.status == RunStatus.FAILED + assert state.error == "Unknown step type: 'not-installed'" + assert probe == {local_name: 1} + assert set(state.step_results) == {local_name, "fan"} + saved = RunState.load(state.run_id, tmp_path) + assert saved.step_results == state.step_results + assert saved.step_results["fan"]["output"]["results"] == [{}] + events = [entry["event"] for entry in state.log_entries] + assert "step_failed" in events + assert "step_continue_on_error" not in events + item_events = {} + for entry in state.log_entries: + if entry.get("step_id", "").startswith("fan:"): + item_events.setdefault(entry["step_id"], []).append(entry["event"]) + assert item_events + assert all( + events == ["step_started", "step_failed"] for events in item_events.values() + ) + + class Reinstalled(StepBase): + type_key = "not-installed" + + def execute(self, config, context): + return StepResult(output={"value": context.item}) + + monkeypatch.setitem(STEP_REGISTRY, "not-installed", Reinstalled()) + state = WorkflowEngine(tmp_path).resume(state.run_id) + + assert state.status == RunStatus.COMPLETED + assert probe == {local_name: 1, "after": 1} + assert state.step_results["fan"]["output"]["results"] == [ + {"value": 1}, {"value": 2} + ] + for index, value in enumerate([1, 2]): + assert state.step_results[f"fan:{alias_name}:{index}"]["output"] == { + "value": value + } + assert RunState.load(state.run_id, tmp_path).step_results == state.step_results + + +def test_diamond_and_depth_limit(tmp_path, probe): + install(tmp_path, definition("leaf", [{"id": "work", "type": "probe"}])) + for name in ("left", "right"): + install(tmp_path, definition(name, [call("leaf")])) + state = WorkflowEngine(tmp_path).execute( + definition( + "parent", + [ + {**call("left"), "id": "left"}, + {**call("right"), "id": "right"}, + ], + ) + ) + assert state.status == RunStatus.COMPLETED + assert probe["work"] == 2 + for index in range(1, 18): + install( + tmp_path, + definition( + f"level-{index}", + [call(f"level-{index + 1}")] + if index < 17 + else [{"id": "too-deep", "type": "probe"}], + ), + ) + state = WorkflowEngine(tmp_path).execute(definition("parent", [call("level-1")])) + assert state.status == RunStatus.FAILED + assert "depth 16" in state.error + assert probe["too-deep"] == 0 + + +def test_output_validation_rejects_reserved_and_yaml_native_values(tmp_path, probe): + from specify_cli.workflows.composition import require_json, validate_outputs + + circular = [] + circular.append(circular) + for value in (circular, date(2026, 1, 1), {1: "key"}, float("nan")): + with pytest.raises(ValueError, match="JSON-safe"): + require_json(value) + for outputs in ( + {"status": {"value": "oops"}}, + {"value": {"wrong": 1}}, + {"value": {"value": date(2026, 1, 1)}}, + ): + assert validate_outputs(outputs) + + +def test_nested_gate_reporting_uses_active_occurrence(tmp_path, probe): + from specify_cli.workflows._commands import _workflow_run_payload + + install( + tmp_path, + definition( + "child", + [{"id": "review", "type": "gate", "message": "Approve", "mode": "manual"}], + ), + ) + state = WorkflowEngine(tmp_path).execute( + definition( + "parent", + [ + { + "id": "route", + "type": "if", + "condition": True, + "then": [call()], + } + ], + ) + ) + assert state.status == RunStatus.PAUSED + payload = _workflow_run_payload(RunState.load(state.run_id, tmp_path)) + # Private scopes report the child-relative occurrence ID (own namespace). + assert payload["current_step_id"] == "review" + assert payload["gate"]["step_id"] == "review" + assert payload["gate"]["scope_path"] == ["route", "call"] + assert payload["workflow_scopes"][0]["status"] == "paused" + + +@pytest.mark.parametrize("failure, expected", [ + (KeyboardInterrupt, "paused"), + (RuntimeError, "failed"), +]) +def test_interrupted_bound_call_reports_run_outcome(tmp_path, monkeypatch, probe, failure, expected): + from specify_cli.workflows._commands import _workflow_run_payload + + class Explode(StepBase): + type_key = "explode" + + def execute(self, config, context): + raise failure("boom") + + monkeypatch.setitem(STEP_REGISTRY, "explode", Explode()) + install(tmp_path, definition("first", [{"id": "done", "type": "probe"}])) + install(tmp_path, definition("child", [{"id": "work", "type": "explode"}])) + engine = WorkflowEngine(tmp_path) + root = definition("parent", [call("first", id="prior"), call()]) + if failure is RuntimeError: + with pytest.raises(RuntimeError, match="boom"): + engine.execute(root, run_id="interrupted-call") + else: + assert engine.execute(root, run_id="interrupted-call").status == RunStatus.PAUSED + + saved = RunState.load("interrupted-call", tmp_path) + payload = _workflow_run_payload(saved) + assert payload["status"] == expected + assert payload["workflow_scopes"] == [ + {"scope_path": ["prior"], "workflow_id": "first", "status": "completed"}, + {"scope_path": ["call"], "workflow_id": "child", "status": expected}, + ] + + +def test_rebind_failure_has_one_failed_caller_outcome(tmp_path, probe): + install( + tmp_path, + definition( + "child", + [{"id": "wait", "type": "probe", "await": True}], + inputs={"approve": {"type": "boolean"}}, + ), + ) + root = definition( + "parent", + [call(input={"approve": "{{ inputs.approve }}"}, continue_on_error=True)], + inputs={"approve": {"type": "string", "default": "false"}}, + ) + state = WorkflowEngine(tmp_path).execute(root) + assert state.status == RunStatus.PAUSED + before = RunState.load(state.run_id, tmp_path).execution["sequence"]["nodes"][0] + + with pytest.raises(ValueError, match="expected a boolean"): + WorkflowEngine(tmp_path).resume(state.run_id, {"approve": "invalid"}) + + failed = RunState.load(state.run_id, tmp_path) + node = failed.execution["sequence"]["nodes"][0] + assert failed.status == RunStatus.FAILED + assert node == before + assert WorkflowEngine(tmp_path).resume( + state.run_id, {"approve": "true"} + ).status == RunStatus.COMPLETED + + +@pytest.mark.parametrize("depth", [1, 2]) +def test_verdict_input_remains_forbidden_through_calls_in_fan_out( + tmp_path, probe, depth +): + child = definition( + "child", + [ + { + "id": "review", + "type": "gate", + "message": "Review", + "verdict_input": "approve", + } + ], + inputs={"approve": {"type": "string", "default": ""}}, + ) + install(tmp_path, child) + if depth == 2: + install( + tmp_path, + definition( + "middle", + [call("child", input={"approve": "{{ inputs.approve }}"})], + inputs={"approve": {"type": "string", "default": ""}}, + ), + ) + target = "middle" if depth == 2 else "child" + state = WorkflowEngine(tmp_path).execute( + definition( + "parent", + [ + { + "id": "fan", + "type": "fan-out", + "items": ["item"], + "step": { + "id": "call", + "type": "workflow", + "workflow": target, + "input": {"approve": ""}, + }, + } + ], + ) + ) + + assert state.status == RunStatus.FAILED + assert "not supported inside fan-out templates" in state.error + + +def test_workflow_result_does_not_inherit_parent_defaults(tmp_path, probe): + install( + tmp_path, + definition( + "child", + [{"id": "work", "type": "probe"}], + workflow={ + "id": "child", + "name": "child", + "integration": "child", + "model": "child-model", + "options": {"x": 2}, + }, + ), + ) + state = WorkflowEngine(tmp_path).execute( + definition( + "parent", + [call()], + workflow={ + "id": "parent", + "name": "parent", + "integration": "parent", + "model": "parent-model", + "options": {"x": 1}, + }, + ) + ) + + call_result = state.step_results["call"] + assert call_result["integration"] is None + assert call_result["model"] is None + assert call_result["options"] == {} + assert call_result["input"] == {} + + +def test_child_context_does_not_inherit_item_or_fan_in(tmp_path, probe): + install( + tmp_path, + definition( + "child", + [ + {"id": "item", "type": "probe", "value": "{{ item }}"}, + {"id": "fan-in", "type": "probe", "value": "{{ fan_in }}"}, + ], + outputs={ + "item": {"value": "{{ steps.item.output.value }}"}, + "fan-in": {"value": "{{ steps.fan-in.output.value }}"}, + }, + ), + ) + + state = WorkflowEngine(tmp_path).execute(definition("parent", [call()])) + + assert state.status == RunStatus.COMPLETED + assert state.step_results["call"]["output"] == { + "workflow": "child", + "status": "completed", + "item": None, + "fan-in": {}, + } + + +def test_rebind_keeps_unmapped_auto_input(tmp_path, probe): + marker = tmp_path / ".specify" / "integration.json" + marker.parent.mkdir() + marker.write_text('{"version": 1, "default_integration": "first"}') + install( + tmp_path, + definition( + "child", + [{"id": "wait", "type": "probe", "await": True}], + inputs={ + "approve": {"type": "boolean", "default": False}, + "integration": {"type": "string", "default": "auto"}, + }, + ), + ) + root = definition( + "parent", + [call(input={"approve": "{{ inputs.approve }}"})], + inputs={"approve": {"type": "boolean", "default": False}}, + ) + state = WorkflowEngine(tmp_path).execute(root) + assert state.status == RunStatus.PAUSED + marker.write_text('{"version": 1, "default_integration": "second"}') + + state = WorkflowEngine(tmp_path).resume(state.run_id, {"approve": True}) + + assert state.status == RunStatus.COMPLETED + binding = state.execution["sequence"]["nodes"][0]["binding"] + assert binding["inputs"] == {"approve": True, "integration": "first"} + + +@pytest.mark.parametrize("change", ["disable", "uninstall"]) +def test_bound_call_uses_snapshot_after_target_changes(tmp_path, probe, change): + child = definition( + "child", + [{"id": "wait", "type": "probe", "await": True}], + inputs={"approve": {"type": "boolean", "default": False}}, + ) + directory = install(tmp_path, child) + root = definition( + "parent", + [call(input={"approve": "{{ inputs.approve }}"})], + inputs={"approve": {"type": "boolean", "default": False}}, + ) + state = WorkflowEngine(tmp_path).execute(root) + assert state.status == RunStatus.PAUSED + if change == "disable": + from specify_cli.workflows.catalog import WorkflowRegistry + + WorkflowRegistry(tmp_path).add("child", {"version": "1.0.0", "enabled": False}) + else: + directory.rename(tmp_path / "removed-child") + + resumed = WorkflowEngine(tmp_path).resume(state.run_id, {"approve": True}) + + assert resumed.status == RunStatus.COMPLETED + + +@pytest.mark.parametrize("change", ["disable", "uninstall"]) +def test_unbound_call_checks_target_when_resume_reaches_it(tmp_path, probe, change): + child = definition("child", [{"id": "work", "type": "probe"}]) + directory = install(tmp_path, child) + root = definition( + "parent", + [ + { + "id": "wait", + "type": "gate", + "message": "Wait", + "verdict_input": "approve", + }, + call(continue_on_error=True), + ], + inputs={"approve": {"type": "string", "default": ""}}, + ) + state = WorkflowEngine(tmp_path).execute(root) + assert state.status == RunStatus.PAUSED + if change == "disable": + from specify_cli.workflows.catalog import WorkflowRegistry + + WorkflowRegistry(tmp_path).add("child", {"version": "1.0.0", "enabled": False}) + else: + directory.rename(tmp_path / "removed-child") + + resumed = WorkflowEngine(tmp_path).resume(state.run_id, {"approve": "approve"}) + + assert resumed.status == RunStatus.COMPLETED + assert resumed.step_results["call"]["status"] == "failed" + assert not probe + + +@pytest.mark.parametrize( + "status,handled", [("failed", False), ("paused", False), ("failed", True)] +) +def test_unsuccessful_expansion_never_executes_children( + tmp_path, monkeypatch, probe, status, handled +): + class Expand(StepBase): + type_key = "expand" + + def execute(self, config, context): + return StepResult( + StepStatus(status), next_steps=[{"id": "wrong", "type": "probe"}] + ) + + monkeypatch.setitem(STEP_REGISTRY, "expand", Expand()) + state = WorkflowEngine(tmp_path).execute( + definition( + "parent", + [ + { + "id": "expand", + "type": "expand", + "continue_on_error": handled, + } + ], + ) + ) + assert state.status.value == ("completed" if handled else status) + assert not probe + RunState.load(state.run_id, tmp_path) + + +def test_custom_expansion_is_frozen_across_resume(tmp_path, monkeypatch, probe): + class Expand(StepBase): + type_key = "expand" + + def execute(self, config, context): + probe["expand"] += 1 + return StepResult( + StepStatus.COMPLETED, + next_steps=[ + {"id": f"prefix-{probe['expand']}", "type": "probe"}, + {"id": "wait", "type": "probe", "await": True}, + ], + ) + + monkeypatch.setitem(STEP_REGISTRY, "expand", Expand()) + root = definition( + "parent", + [{"id": "expand", "type": "expand"}], + inputs={"approve": {"type": "boolean", "default": False}}, + ) + state = WorkflowEngine(tmp_path).execute(root) + state = WorkflowEngine(tmp_path).resume(state.run_id, {"approve": True}) + assert state.status == RunStatus.COMPLETED + assert probe == {"expand": 1, "prefix-1": 1, "wait": 2} + + +def test_native_yaml_definition_and_long_id_roundtrip(tmp_path, probe): + target = "a" * 240 + child = definition( + target, [{"id": "review", "type": "gate", "message": date(2026, 1, 1)}] + ) + install(tmp_path, child) + state = WorkflowEngine(tmp_path).execute(definition("parent", [call(target)])) + assert state.status == RunStatus.PAUSED + saved = RunState.load(state.run_id, tmp_path) + binding = saved.execution["sequence"]["nodes"][0]["binding"] + assert yaml.safe_load(binding["definition"])["steps"][0]["message"] == date( + 2026, 1, 1 + ) + assert WorkflowEngine(tmp_path).resume(state.run_id).status == RunStatus.PAUSED + + +@pytest.mark.parametrize("scope", ["root", "if", "workflow", "fan-out"]) +def test_yaml_native_gate_template_survives_checkpoint_and_resume(tmp_path, monkeypatch, scope): + monkeypatch.setattr("sys.stdin.isatty", lambda: False) + gate = {"id": "wait", "type": "gate", "message": date(2026, 1, 1)} + steps = [gate] + if scope == "if": + steps = [{"id": "branch", "type": "if", "condition": True, "then": steps}] + elif scope == "workflow": + install(tmp_path, definition("child", steps)) + steps = [call()] + elif scope == "fan-out": + steps = [{"id": "fan", "type": "fan-out", "items": [0, 1], "step": gate}] + engine = WorkflowEngine(tmp_path) + state = engine.execute(definition("parent", steps)) + assert state.status == RunStatus.PAUSED + state = engine.resume(state.run_id) + assert state.status == RunStatus.PAUSED + from specify_cli.workflows._execution import walk_execution + + gates = [config for config, _, _, _ in walk_execution(state.execution["sequence"]) + if config.get("type") == "gate"] + assert gates and all(config["message"] == date(2026, 1, 1) for config in gates) + + +@pytest.mark.parametrize("offset", [1, 2, 3]) +def test_resume_rejects_offset_skipping_blocked_step_without_writes(tmp_path, probe, offset): + steps = [{"id": "wait", "type": "probe", "await": True}, + {"id": "later", "type": "probe"}] + engine = WorkflowEngine(tmp_path) + state = engine.execute(definition("parent", steps)) + path = state.runs_dir / "state.json" + data = json.loads(path.read_text()) + data["execution"].update(offset=offset, sequence={ + "source": yaml.safe_dump(steps[offset:]), + "nodes": [{"phase": "ready"} for _ in steps[offset:]], + }) + path.write_text(json.dumps(data)) + before = {p.name: p.read_bytes() for p in state.runs_dir.iterdir()} + with pytest.raises(ValueError, match="offset"): + engine.resume(state.run_id) + assert {p.name: p.read_bytes() for p in state.runs_dir.iterdir()} == before + assert probe == {"wait": 1} + + +@pytest.mark.parametrize("scope", ["step", "if", "workflow", "fan-out"]) +def test_lifecycle_start_is_tree_backed_and_replay_does_not_restart(tmp_path, probe, scope): + from specify_cli.workflows._execution import walk_execution + + work = {"id": "work", "type": "probe"} + steps = [work] + if scope == "if": + steps = [{"id": "branch", "type": "if", "condition": True, "then": steps}] + elif scope == "workflow": + install(tmp_path, definition("child", steps)) + steps = [call()] + elif scope == "fan-out": + steps = [{"id": "fan", "type": "fan-out", "items": [0, 1], + "max_concurrency": 2, "step": work}] + steps.append({"id": "wait", "type": "probe", "await": True}) + engine = WorkflowEngine(tmp_path) + announced = [] + + def started(step_id, label): + saved = RunState.load("lifecycle", tmp_path) + matches = [node for _, node, _, name in walk_execution(saved.execution["sequence"]) + if name == step_id] + assert any(node.get("active") for node in matches) + announced.append(step_id) + + engine.on_step_start = started + state = engine.execute(definition("parent", steps), run_id="lifecycle") + assert state.status == RunStatus.PAUSED + assert all(not node.get("active") for _, node, _, _ in walk_execution(state.execution["sequence"])) + before = probe.copy() + announced.clear() + engine.resume(state.run_id) + assert announced == ["wait"] + assert probe["work"] == before["work"] + + +@pytest.mark.parametrize("mutation", [ + lambda node: node.update(active=True), + lambda node: node.update(active="yes"), + lambda node: node.update(template="- invalid\n"), + lambda node: node.update(fan_results={}), + lambda node: node.update(fan_results={"a": 1}), +]) +def test_lifecycle_rejects_invalid_completed_fan_out_without_writes(tmp_path, probe, mutation): + engine = WorkflowEngine(tmp_path) + state = engine.execute(definition("parent", [ + {"id": "fan", "type": "fan-out", "items": [0], + "step": {"id": "work", "type": "probe"}}, + {"id": "wait", "type": "probe", "await": True}, + ])) + path = state.runs_dir / "state.json" + data = json.loads(path.read_text()) + mutation(data["execution"]["sequence"]["nodes"][0]) + path.write_text(json.dumps(data)) + before = {p.name: p.read_bytes() for p in state.runs_dir.iterdir()} + with pytest.raises(ValueError): + engine.resume(state.run_id) + assert {p.name: p.read_bytes() for p in state.runs_dir.iterdir()} == before + + +@pytest.mark.parametrize("field,value", [("fan_results", []), ("template", "{}\n")]) +def test_non_fan_out_rejects_fan_out_fields(tmp_path, probe, field, value): + state = WorkflowEngine(tmp_path).execute(definition("parent", [ + {"id": "work", "type": "probe"}, + {"id": "wait", "type": "probe", "await": True}, + ])) + path = state.runs_dir / "state.json" + data = json.loads(path.read_text()) + data["execution"]["sequence"]["nodes"][0][field] = value + path.write_text(json.dumps(data)) + before = {p.name: p.read_bytes() for p in state.runs_dir.iterdir()} + + with pytest.raises(ValueError, match="Invalid fan-out"): + WorkflowEngine(tmp_path).resume(state.run_id) + + assert {p.name: p.read_bytes() for p in state.runs_dir.iterdir()} == before + + +@pytest.mark.parametrize("invalid", ["restart-completed", "override-phase", "finish-unfinished", "foreign-field"]) +def test_lifecycle_rejects_invalid_transitions_before_mutation(tmp_path, probe, invalid): + from copy import deepcopy + from specify_cli.workflows._execution import Execution, Occurrence + from specify_cli.workflows.base import StepContext + + config = {"id": "branch", "type": "if", "condition": True, + "then": [{"id": "wait", "type": "probe", "await": True}]} + engine = WorkflowEngine(tmp_path) + state = engine.execute(definition("parent", [config])) + node = state.execution["sequence"]["nodes"][0] + occurrence = Occurrence(config, node, StepContext(), ("parent",), (0,), True, "branch") + executor = Execution(engine, state, STEP_REGISTRY) + if invalid == "restart-completed": + occurrence = Occurrence( + {"id": "done", "type": "probe"}, + {"phase": "done", "result": {"status": "completed", "output": {}}}, + StepContext(), ("parent",), (0,), True, "done", + ) + operation, changes = "begin", {} + elif invalid == "override-phase": + operation, changes = "begin", {"phase": "done"} + elif invalid == "foreign-field": + operation, changes = "iterate", {"result": {"status": "completed", "output": {}}} + else: + operation, changes = "settle", {"outcome": "completed", "error": None} + before_node = deepcopy(occurrence.node) + before_files = {p.name: p.read_bytes() for p in state.runs_dir.iterdir()} + with pytest.raises(ValueError): + executor.transition(operation, occurrence, changes) + assert occurrence.node == before_node + assert {p.name: p.read_bytes() for p in state.runs_dir.iterdir()} == before_files + + +def test_execution_shares_workflow_fan_out_and_loop_sources(tmp_path, probe): + child = definition( + "child", + [{"id": "work", "type": "probe"}], + ) + install(tmp_path, child) + state = WorkflowEngine(tmp_path).execute( + definition( + "parent", + [ + call(), + { + "id": "fan", + "type": "fan-out", + "items": [1, 2], + "step": {"id": "item", "type": "probe"}, + }, + { + "id": "loop", + "type": "do-while", + "condition": True, + "max_iterations": 2, + "steps": [{"id": "body", "type": "probe"}], + }, + {"id": "wait", "type": "probe", "await": True}, + ], + ) + ) + + tree = json.loads((state.runs_dir / "state.json").read_text())["execution"] + call_node, fan_node, loop_node, _ = tree["sequence"]["nodes"] + assert "source" not in call_node["children"][0] + assert "source" not in fan_node["children"][0] + assert "source" not in fan_node["children"][1] + assert "source" in loop_node["children"][0] + assert "source" not in loop_node["children"][1] + assert yaml.safe_load(call_node["binding"]["definition"])["steps"] == [ + {"id": "work", "type": "probe"} + ] + RunState.load(state.run_id, tmp_path) + + +@pytest.mark.parametrize( + "mutation", + [ + lambda tree: tree["sequence"]["nodes"][0]["children"][0].update( + source="[]\n" + ), + lambda tree: tree["sequence"]["nodes"][1]["children"][0].update( + source="[]\n" + ), + lambda tree: tree["sequence"]["nodes"][2]["children"][1].update( + source="[]\n" + ), + lambda tree: tree["sequence"]["nodes"][0]["children"][0]["nodes"].append( + {"phase": "ready"} + ), + ], +) +def test_shared_execution_sources_reject_local_copies(tmp_path, probe, mutation): + child = definition("child", [{"id": "work", "type": "probe"}]) + install(tmp_path, child) + state = WorkflowEngine(tmp_path).execute( + definition( + "parent", + [ + call(), + { + "id": "fan", + "type": "fan-out", + "items": [1], + "step": {"id": "item", "type": "probe"}, + }, + { + "id": "loop", + "type": "do-while", + "condition": True, + "max_iterations": 2, + "steps": [{"id": "body", "type": "probe"}], + }, + {"id": "wait", "type": "probe", "await": True}, + ], + ) + ) + path = state.runs_dir / "state.json" + data = json.loads(path.read_text()) + mutation(data["execution"]) + path.write_text(json.dumps(data)) + before = path.read_bytes() + + with pytest.raises(ValueError, match="Invalid execution sequence"): + WorkflowEngine(tmp_path).resume(state.run_id) + + assert path.read_bytes() == before + + +@pytest.mark.parametrize("items", [1, 4]) +def test_fan_out_saves_once_per_item_transition(tmp_path, monkeypatch, probe, items): + original = RunState.save + saves = 0 + + def save(self): + nonlocal saves + saves += 1 + original(self) + + monkeypatch.setattr(RunState, "save", save) + state = WorkflowEngine(tmp_path).execute( + definition( + "parent", + [ + { + "id": "fan", + "type": "fan-out", + "items": list(range(items)), + "max_concurrency": 1, + "step": {"id": "item", "type": "probe"}, + } + ], + ) + ) + + assert state.status == RunStatus.COMPLETED + # Each started occurrence (fan and items) is checkpointed before it runs, + # and each item result once more before it is done. + assert saves == 2 * items + 6 + + +def test_fan_out_snapshot_size_does_not_scale_with_template_length(tmp_path, probe): + def size(items, length): + root = tmp_path / f"fan-{items}-{length}" + state = WorkflowEngine(root).execute( + definition( + "parent", + [ + { + "id": "fan", + "type": "fan-out", + "items": list(range(items)), + "max_concurrency": 1, + "step": { + "id": "item", + "type": "probe", + "payload": "x" * length, + }, + } + ], + ) + ) + return (state.runs_dir / "state.json").stat().st_size + + small_one, large_one = size(1, 128), size(1, 4096) + small_many, large_many = size(4, 128), size(4, 4096) + + assert large_many - small_many <= large_one - small_one + 256 + + +def test_loop_snapshot_size_does_not_scale_with_body_length(tmp_path, probe): + def size(iterations, length): + root = tmp_path / f"loop-{iterations}-{length}" + state = WorkflowEngine(root).execute( + definition( + "parent", + [ + { + "id": "loop", + "type": "do-while", + "condition": True, + "max_iterations": iterations, + "steps": [ + { + "id": "body", + "type": "probe", + "payload": "x" * length, + } + ], + } + ], + ) + ) + return (state.runs_dir / "state.json").stat().st_size + + small_one, large_one = size(1, 128), size(1, 4096) + small_many, large_many = size(4, 128), size(4, 4096) + + assert large_many - small_many <= large_one - small_one + 256 + + +def test_loop_transition_validation_does_not_rewalk_prior_iterations( + tmp_path, monkeypatch, probe +): + import specify_cli.workflows._execution as execution + + original = execution.yaml.safe_load + counts = [] + + def parse(source): + counts[-1] += 1 + return original(source) + + monkeypatch.setattr(execution.yaml, "safe_load", parse) + for iterations in (20, 40): + counts.append(0) + state = WorkflowEngine(tmp_path / str(iterations)).execute( + definition("parent", [{ + "id": "loop", "type": "do-while", "condition": True, + "max_iterations": iterations, + "steps": [{"id": "body", "type": "probe"}], + }]) + ) + assert state.status == RunStatus.COMPLETED + assert counts[1] < counts[0] * 2.5 + + +def test_tree_backed_resume_has_no_setup_checkpoint(tmp_path, monkeypatch, probe): + state = WorkflowEngine(tmp_path).execute( + definition( + "parent", + [{"id": "wait", "type": "probe", "await": True}], + inputs={"approve": {"type": "boolean", "default": False}}, + ) + ) + original = RunState.save + saves = 0 + + def save(self): + nonlocal saves + saves += 1 + original(self) + + monkeypatch.setattr(RunState, "save", save) + state = WorkflowEngine(tmp_path).resume(state.run_id, {"approve": True}) + + assert state.status == RunStatus.COMPLETED + # Restarting ``wait`` persists it as active before it runs, then its result. + assert saves == 4 + + +def test_exact_depth_limit_is_allowed(tmp_path, probe): + for index in range(1, 17): + install( + tmp_path, + definition( + f"level-{index}", + [call(f"level-{index + 1}")] + if index < 16 + else [{"id": "work", "type": "probe"}], + ), + ) + state = WorkflowEngine(tmp_path).execute(definition("parent", [call("level-1")])) + assert state.status == RunStatus.COMPLETED + assert probe["work"] == 1 + + +@pytest.mark.parametrize( + "template", + [ + {"id": "mixed", "type": "mixed"}, + { + "id": "mixed", + "type": "if", + "condition": True, + "then": [{"id": "inner", "type": "mixed"}], + }, + call("child", id="mixed", input={"n": "{{ item }}"}), + ], + ids=["step", "if-container", "workflow-call"], +) +def test_aborted_fanout_sibling_is_never_restarted(tmp_path, monkeypatch, template): + barrier = threading.Barrier(2, timeout=5) + counts = Counter() + + class Mixed(StepBase): + type_key = "mixed" + + def execute(self, config, context): + item = context.item if context.item is not None else context.inputs["n"] + counts[item] += 1 + if not context.is_resume: + barrier.wait() + if item == 0: + return StepResult( + StepStatus.COMPLETED if context.is_resume else StepStatus.PAUSED + ) + return StepResult(StepStatus.FAILED, output={"aborted": True}) + + monkeypatch.setitem(STEP_REGISTRY, "mixed", Mixed()) + install( + tmp_path, + definition( + "child", + [{"id": "inner", "type": "mixed"}], + inputs={"n": {"type": "number"}}, + ), + ) + root = definition( + "parent", + [ + { + "id": "spread", + "type": "fan-out", + "items": [0, 1], + "max_concurrency": 2, + "step": template, + } + ], + ) + state = WorkflowEngine(tmp_path).execute(root) + assert state.status == RunStatus.PAUSED + state = WorkflowEngine(tmp_path).resume(state.run_id) + assert state.status == RunStatus.ABORTED + assert counts == {0: 2, 1: 1} + # Replaying the aborted item keeps the result it published when it ran. + published = state.step_results["spread:mixed:1"]["output"] + assert published + for results in ( + state.step_results["spread"]["output"]["results"], + RunState.load(state.run_id, tmp_path).step_results["spread"]["output"][ + "results" + ], + ): + assert results[1] == published + + +FAN_OUT_ITEM_TEMPLATES = [ + {"id": "tmpl", "type": "snap"}, + { + "id": "tmpl", + "type": "if", + "condition": True, + "then": [{"id": "inner", "type": "snap"}], + }, + call("child", id="tmpl", input={"n": "{{ item }}"}), +] +FAN_OUT_ITEM_TEMPLATE_IDS = ["step", "if-container", "workflow-call"] + + +@pytest.mark.parametrize("max_concurrency", [1, 3], ids=["sequential", "parallel"]) +@pytest.mark.parametrize( + "template", FAN_OUT_ITEM_TEMPLATES, ids=FAN_OUT_ITEM_TEMPLATE_IDS +) +def test_resumed_fan_out_item_sees_its_uninterrupted_context( + tmp_path, monkeypatch, probe, template, max_concurrency +): + """Exact resume must re-run a fan-out item in the context it has in a run + that never paused. Replay reconstructs item contexts from the checkpoint, + so any difference (for example, a result the live run adds only after its + items) is a live/replay gap.""" + from dataclasses import asdict + + pause = {"enabled": False} + seen = {} + + class Snap(StepBase): + type_key = "snap" + + def execute(self, config, context): + item = context.item if context.item is not None else context.inputs["n"] + view = asdict(context) + del view["run_id"], view["is_resume"] + phase = "resume" if context.is_resume else pause["enabled"] + seen.setdefault(phase, {})[item] = view + if item == 1 and pause["enabled"] and not context.is_resume: + return StepResult(StepStatus.PAUSED, output={"item": item}) + return StepResult(output={"item": item}) + + monkeypatch.setitem(STEP_REGISTRY, "snap", Snap()) + install( + tmp_path, + definition( + "child", + [{"id": "inner", "type": "snap"}], + inputs={"n": {"type": "number"}}, + ), + ) + root = definition( + "parent", + [ + {"id": "setup", "type": "probe", "value": "ready"}, + { + "id": "fan", + "type": "fan-out", + "items": [0, 1, 2], + "max_concurrency": max_concurrency, + "step": template, + }, + ], + ) + state = WorkflowEngine(tmp_path).execute(root) + assert state.status == RunStatus.COMPLETED + pause["enabled"] = True + state = WorkflowEngine(tmp_path).execute(root) + assert state.status == RunStatus.PAUSED + + state = WorkflowEngine(tmp_path).resume(state.run_id) + + assert state.status == RunStatus.COMPLETED + resumed = seen["resume"] + assert 1 in resumed + for item, view in resumed.items(): + assert view == seen[False][item], f"item {item} context differs on resume" + + +def test_resumed_fan_out_items_do_not_see_partial_results(tmp_path, monkeypatch): + seen = [] + + class Look(StepBase): + type_key = "look" + + def execute(self, config, context): + seen.append("results" in context.steps["fan"]["output"]) + if context.item == 1 and not context.is_resume: + return StepResult(StepStatus.PAUSED, output={"item": 1}) + return StepResult(output={"item": context.item}) + + monkeypatch.setitem(STEP_REGISTRY, "look", Look()) + root = definition( + "parent", + [ + { + "id": "fan", + "type": "fan-out", + "items": [0, 1, 2], + "step": {"id": "look", "type": "look"}, + } + ], + ) + state = WorkflowEngine(tmp_path).execute(root) + assert state.status == RunStatus.PAUSED + # The paused fan-out still reports its partial results. + assert state.step_results["fan"]["output"]["results"] == [{"item": 0}, {"item": 1}] + + state = WorkflowEngine(tmp_path).resume(state.run_id) + + assert state.status == RunStatus.COMPLETED + # Items never see the engine-added results: not live, not after resume. + assert seen == [False, False, False, False] + assert state.step_results["fan"]["output"]["results"] == [ + {"item": 0}, + {"item": 1}, + {"item": 2}, + ] + + +NAN_SNAPSHOT = """ +schema_version: "1.0" +workflow: + id: nan-snapshot + name: NaN Snapshot + version: "1.0.0" +inputs: + approve: + type: boolean + default: false +steps: + - id: fan + type: fan-out + items: [1, 2] + max_concurrency: .nan + step: + type: probe + value: "{{ item }}" + - id: wait + type: probe + await: true +""" + + +def test_resume_accepts_native_yaml_nan_in_snapshot(tmp_path, probe): + state = WorkflowEngine(tmp_path).execute( + WorkflowDefinition.from_string(NAN_SNAPSHOT), run_id="nan" + ) + assert state.status == RunStatus.PAUSED + copy = tmp_path / ".specify/workflows/runs/nan/workflow.yml" + data = yaml.safe_load(copy.read_text(encoding="utf-8")) + # Key order is not part of the snapshot identity. + data["steps"][0] = dict(reversed(list(data["steps"][0].items()))) + copy.write_text(yaml.safe_dump(data, sort_keys=False), encoding="utf-8") + + state = WorkflowEngine(tmp_path).resume("nan", {"approve": True}) + + assert state.status == RunStatus.COMPLETED, state.error + assert state.step_results["fan"]["output"]["results"] == [ + {"value": 1}, {"value": 2} + ] + assert probe == {"item": 2, "wait": 2} + + +def test_resume_rejects_changed_workflow_snapshot(tmp_path, probe): + state = WorkflowEngine(tmp_path).execute( + WorkflowDefinition.from_string(NAN_SNAPSHOT), run_id="changed" + ) + assert state.status == RunStatus.PAUSED + copy = tmp_path / ".specify/workflows/runs/changed/workflow.yml" + data = yaml.safe_load(copy.read_text(encoding="utf-8")) + data["steps"][0]["items"] = [1, 2, 3] + copy.write_text(yaml.safe_dump(data, sort_keys=False), encoding="utf-8") + + with pytest.raises(ValueError, match="root sequence differs"): + WorkflowEngine(tmp_path).resume("changed", {"approve": True}) + +def test_paused_fan_out_item_reports_qualified_current_step_id(tmp_path, probe): + callbacks = [] + engine = WorkflowEngine(tmp_path) + engine.on_step_start = lambda step_id, _label: callbacks.append(step_id) + + state = engine.execute( + definition( + "parent", + [ + { + "id": "fan", + "type": "fan-out", + "items": [1], + "step": {"id": "item", "type": "probe", "status": "paused"}, + } + ], + ) + ) + + assert state.status == RunStatus.PAUSED + assert state.current_step_id == "fan:item:0" + assert "fan:item:0" in state.step_results + assert RunState.load(state.run_id, tmp_path).current_step_id == "fan:item:0" + assert callbacks[-1] == "fan:item:0" + + +def test_paused_later_loop_iteration_reports_qualified_current_step_id( + tmp_path, monkeypatch +): + calls = Counter() + + class PauseSecond(StepBase): + type_key = "pause-second" + + def execute(self, config, context): + calls[config["id"]] += 1 + if calls[config["id"]] == 2: + return StepResult(StepStatus.PAUSED) + return StepResult(output={}) + + monkeypatch.setitem(STEP_REGISTRY, "pause-second", PauseSecond()) + state = WorkflowEngine(tmp_path).execute( + definition( + "parent", + [ + { + "id": "loop", + "type": "do-while", + "condition": True, + "max_iterations": 3, + "steps": [{"id": "body", "type": "pause-second"}], + } + ], + ) + ) + + assert state.status == RunStatus.PAUSED + assert state.current_step_id == "loop:body:1" + assert "loop:body:1" in state.step_results + assert RunState.load(state.run_id, tmp_path).current_step_id == "loop:body:1" + + +@pytest.mark.parametrize( + ("steps", "expected"), + [ + ( + [{"id": "first", "type": "probe"}, {"id": "observe", "type": "observe"}], + [("observe", 1)], + ), + ( + [ + {"id": "first", "type": "probe"}, + { + "id": "loop", + "type": "do-while", + "condition": True, + "max_iterations": 2, + "steps": [{"id": "observe", "type": "observe"}], + }, + ], + [("observe", 1), ("loop:observe:1", 1)], + ), + ( + [ + {"id": "first", "type": "probe"}, + { + "id": "fan", + "type": "fan-out", + "items": [1, 2], + "step": {"id": "observe", "type": "observe"}, + }, + ], + [("fan:observe:0", 1), ("fan:observe:1", 1)], + ), + ], + ids=["root", "later-loop-iteration", "fan-out-item"], +) +def test_running_step_is_persisted_before_it_executes( + tmp_path, monkeypatch, probe, steps, expected +): + seen = [] + + class Observe(StepBase): + type_key = "observe" + + def execute(self, config, context): + persisted = RunState.load(context.run_id, tmp_path) + seen.append((persisted.current_step_id, persisted.current_step_index)) + return StepResult(output={}) + + monkeypatch.setitem(STEP_REGISTRY, "observe", Observe()) + + state = WorkflowEngine(tmp_path).execute(definition("parent", steps)) + + assert state.status == RunStatus.COMPLETED + assert seen == expected + + +def _record_persisted_step_on_start(engine, tmp_path, seen): + """Record the persisted ``current_step_id`` whenever a step start is announced.""" + + def on_step_start(step_id, label): + (path,) = (tmp_path / ".specify/workflows/runs").glob("*/state.json") + persisted = json.loads(path.read_text(encoding="utf-8"))["current_step_id"] + seen.append((step_id, label, persisted)) + + engine.on_step_start = on_step_start + return engine + + +def test_workflow_call_is_persisted_before_it_starts(tmp_path, probe): + install(tmp_path, definition("grandchild", [{"id": "leaf", "type": "probe"}])) + install( + tmp_path, + definition( + "child", + [ + {"id": "prepare", "type": "probe"}, + {"id": "nested", "type": "workflow", "workflow": "grandchild"}, + ], + ), + ) + seen = [] + engine = _record_persisted_step_on_start(WorkflowEngine(tmp_path), tmp_path, seen) + + state = engine.execute(definition("parent", [{"id": "first", "type": "probe"}, call()])) + + assert state.status == RunStatus.COMPLETED + # Like every other step, a call (top-level and nested inside a called + # workflow, by its workflow-relative ID) is checkpointed as the active step + # before its start is logged and announced. + assert seen == [ + ("first", "probe", "first"), + ("call", "workflow", "call"), + ("prepare", "probe", "prepare"), + ("nested", "workflow", "nested"), + ("leaf", "probe", "leaf"), + ] + + +def test_resumed_bound_workflow_call_is_persisted_before_it_starts(tmp_path, probe): + install( + tmp_path, + definition( + "child", + [{"id": "wait", "type": "probe", "await": True}], + inputs={"approve": {"type": "boolean", "default": False}}, + ), + ) + root = definition( + "parent", + [call(input={"approve": "{{ inputs.approve }}"})], + inputs={"approve": {"type": "boolean", "default": False}}, + ) + state = WorkflowEngine(tmp_path).execute(root) + assert state.status == RunStatus.PAUSED + assert RunState.load(state.run_id, tmp_path).current_step_id == "wait" + seen = [] + engine = _record_persisted_step_on_start(WorkflowEngine(tmp_path), tmp_path, seen) + + state = engine.resume(state.run_id, {"approve": True}) + + assert state.status == RunStatus.COMPLETED + # The already-bound call skips binding, which previously held the only + # pre-child checkpoint; it must still persist itself before re-announcing. + assert seen == [("call", "workflow", "call"), ("wait", "probe", "wait")] + + +def test_workflow_call_start_checkpoint_failure_logs_no_start(tmp_path, monkeypatch, probe): + from specify_cli.workflows._execution import CheckpointError + + install(tmp_path, definition("child", [{"id": "inner", "type": "probe"}])) + original = RunState._atomic_write_json + + def write(path, data): + if path.name == "state.json" and data.get("current_step_id") == "call": + raise OSError("checkpoint failure") + original(path, data) + + monkeypatch.setattr(RunState, "_atomic_write_json", staticmethod(write)) + callbacks = [] + engine = WorkflowEngine(tmp_path) + engine.on_step_start = lambda step_id, label: callbacks.append(step_id) + + with pytest.raises(CheckpointError, match="checkpoint failure"): + engine.execute( + definition("parent", [{"id": "first", "type": "probe"}, call()]), + run_id="fault", + ) + + runs = tmp_path / ".specify/workflows/runs/fault" + started = [ + entry["step_id"] + for entry in map(json.loads, (runs / "log.jsonl").read_text().splitlines()) + if entry["event"] == "step_started" + ] + disk = json.loads((runs / "state.json").read_text()) + node = disk["execution"]["sequence"]["nodes"][1] + # The failed start checkpoint is not followed by a start event, a callback, + # or target binding. + assert started == ["first"] + assert callbacks == ["first"] + assert disk["current_step_id"] == "first" + assert node["phase"] == "ready" + assert "binding" not in node + assert probe["inner"] == 0 + + +def test_failed_fan_out_item_exception_reports_qualified_current_step_id( + tmp_path, monkeypatch +): + class Boom(StepBase): + type_key = "boom" + + def execute(self, config, context): + raise RuntimeError("boom") + + monkeypatch.setitem(STEP_REGISTRY, "boom", Boom()) + engine = WorkflowEngine(tmp_path) + with pytest.raises(RuntimeError, match="boom"): + engine.execute( + definition( + "parent", + [ + { + "id": "fan", + "type": "fan-out", + "items": [1], + "step": {"id": "blast", "type": "boom"}, + } + ], + ), + run_id="boom", + ) + + loaded = RunState.load("boom", tmp_path) + assert loaded.status == RunStatus.FAILED + assert loaded.current_step_id == "fan:blast:0" + + +def test_paused_fan_out_gate_payload_reports_qualified_step_id(tmp_path): + from specify_cli.workflows._commands import _workflow_run_payload + + state = WorkflowEngine(tmp_path).execute( + definition( + "parent", + [ + { + "id": "fan", + "type": "fan-out", + "items": [1], + "step": {"id": "review", "type": "gate", "message": "Approve"}, + } + ], + ) + ) + + assert state.status == RunStatus.PAUSED + payload = _workflow_run_payload(RunState.load(state.run_id, tmp_path)) + assert payload["current_step_id"] == "fan:review:0" + assert payload["gate"]["step_id"] == "fan:review:0" + assert payload["gate"]["message"] == "Approve" + + +def test_gate_message_keeps_typed_template_result(tmp_path): + from specify_cli.workflows._commands import _workflow_run_payload + + workflow = definition( + "parent", + [{"id": "gate", "type": "gate", "message": "{{ inputs.notice }}"}], + inputs={"notice": {"type": "number"}}, + ) + assert validate_workflow(workflow) == [] + + state = WorkflowEngine(tmp_path).execute(workflow, {"notice": 42}) + + assert state.status == RunStatus.PAUSED + loaded = RunState.load(state.run_id, tmp_path) + message = loaded.step_results["gate"]["output"]["message"] + assert message == 42 + assert isinstance(message, int) + assert _workflow_run_payload(loaded)["gate"]["message"] == "42" + + +def test_gate_message_stores_non_json_literal_as_text(tmp_path): + state = WorkflowEngine(tmp_path).execute( + definition("parent", [{"id": "gate", "type": "gate", "message": date(2026, 1, 1)}]) + ) + + assert state.status == RunStatus.PAUSED + loaded = RunState.load(state.run_id, tmp_path) + assert loaded.step_results["gate"]["output"]["message"] == "2026-01-01" + + +def test_fan_out_worker_exception_stops_dispatch_before_earlier_items_finish( + tmp_path, monkeypatch +): + from types import SimpleNamespace + + from specify_cli.workflows import _execution + + halted_seen = threading.Event() + + class ObservedEvent(threading.Event): + def set(self): + super().set() + halted_seen.set() + + monkeypatch.setattr(_execution, "threading", SimpleNamespace(Event=ObservedEvent)) + started = set() + lock = threading.Lock() + + class Blow(StepBase): + type_key = "blow" + + def execute(self, config, context): + with lock: + started.add(context.item) + if context.item == 2: + raise RuntimeError("boom") + if context.item in {0, 1}: + # Earlier items finish only after the failure halts dispatch. + halted_seen.wait(2) + return StepResult(output={"value": context.item}) + + monkeypatch.setitem(STEP_REGISTRY, "blow", Blow()) + with pytest.raises(RuntimeError, match="boom"): + WorkflowEngine(tmp_path).execute( + definition( + "parent", + [ + { + "id": "fan", + "type": "fan-out", + "items": [0, 1, 2, 3, 4, 5], + "max_concurrency": 3, + "step": {"id": "item", "type": "blow"}, + } + ], + ) + ) + + assert started == {0, 1, 2} diff --git a/workflows/ARCHITECTURE.md b/workflows/ARCHITECTURE.md index 1b06dc2e66..291c924467 100644 --- a/workflows/ARCHITECTURE.md +++ b/workflows/ARCHITECTURE.md @@ -70,16 +70,81 @@ flowchart LR E -- "resume()" --> B ``` -When a `gate` step pauses execution, the engine persists `current_step_index` and all accumulated `step_results`. On `specify workflow resume `, the engine restores the context and continues from the paused step. - -> **Note:** Resume tracking is at the top-level step index only. If a -> nested step (inside `if`/`switch`/`while`) pauses, resume re-runs -> the parent control-flow step and its nested body. A nested step-path -> stack for exact resume is a planned enhancement. +When a `gate` step pauses execution, the engine persists `current_step_index` +and all accumulated `step_results`. On `specify workflow resume `, the +same executor replays completed occurrences into their contexts without +executing them, then continues at the unfinished occurrence. + +New runs use a versioned execution tree. Each occurrence owns its result, +selected child sequences, and optional workflow binding. A binding stores the +target, frozen definition, private inputs, and `workflow_dir`; the called +workflow remains a private scope in the same run. Fan-out items have separate +contexts. Each occurrence, including a workflow call, is checkpointed as the +active step before its start is logged and before it executes, so status +reports it while it runs. Binding and selected expansions are +checkpointed before child side effects, results before an occurrence is done, +and logs after the checkpoint. +Legacy runs enter through their top-level index once. Inputs and tree +transitions share one atomic state checkpoint; the inputs file is a +compatibility mirror. A checkpoint failure prevents further writes by that +executor instance. + +### Occurrence lifecycle + +`Execution.step()` is the common runner for registered steps and workflow calls. +`execute_step()` and `workflow()` return a `StepResult`, a subtree outcome, or an +unknown-implementation failure. The runner alone performs `begin`, `finish`, +`settle`, and best-effort `leave` on exception unwinding; the phase and field +allow-lists in `transition()` reject invalid operations. The public, stateless +`StepBase.execute()` extension contract is unchanged. + +All occurrence mutations pass through `Execution.transition()`. It checks the +allowed source phase and fields, derives the destination phase, validates the +candidate with the same node rules used on load, projects results, and saves +under the run lock. Callers cannot supply a destination phase. + +| Operation | Meaning | +|-----------|---------| +| `begin` | Mark this occurrence active and checkpoint before `step_started` or its callback; a failed checkpoint leaves no start event | +| `expand` / `bind` | Freeze children and their source before child execution | +| `rebind` / `iterate` | Persist updated binding or the next loop occurrence | +| `outputs` | Children finished; declared workflow outputs remain to finalize | +| `finish` / `settle` | Record an own-step result or subtree outcome; clear activity | +| `leave` | Clear activity on exception unwinding; the run handler saves the failure/pause | + +`phase` identifies the continuation point, `active` identifies entered occurrences +(several may be active in a fan-out), and `outcome` identifies a subtree halt or +completion. A container's own result may be completed while its children are +paused. `current_step_id` is a compatibility status view, updated on entry and +reconciled against the tree on exit, rather than a resume cursor. +Status reporting uses a bound call's recorded result when available. If an +interruption or exception leaves the active call unfinished without a result, +its scope inherits the run's paused or failed status; completed calls keep +their own recorded status. + +`notify()` emits events and callbacks only after the corresponding checkpoint. +Persistence is mandatory lifecycle behavior, not a listener. Existing container +events describe completion of their own expansion; calls finish after their +children and declared outputs. Completed replay emits neither events nor saves. +Entering an unfinished container on resume checkpoints activity without repeating +its expansion event. + +Execution schema version 2 stores fan-out templates as shared YAML sources on +their parent occurrence. Raw `step_template` configuration is not published in +persisted step outputs. Frozen expansion results and aggregated `fan_results` +are separate: `result_view()` adds the aggregate for reporting and downstream +steps, while items always receive the frozen expansion view. This also preserves +YAML-native template scalars without putting them in JSON result records. + +Resume validates tree structure, root snapshot, and the legacy offset together, +before any writes. The offset must be within the workflow and no later than the +saved root index. Main-format checkpoints without a tree still adapt once; +private, unreleased version-1 trees are rejected rather than silently interpreted +as the new format. ## Step Types -The engine ships with 12 built-in step types, each in its own subpackage under `src/specify_cli/workflows/step/`: +The engine ships with 13 built-in step types, each in its own subpackage under `src/specify_cli/workflows/step/`: | Type Key | Class | Purpose | Returns `next_steps`? | |----------|-------|---------|-----------------------| @@ -95,6 +160,7 @@ The engine ships with 12 built-in step types, each in its own subpackage under ` | `do-while` | `DoWhileStep` | Loop, always runs body at least once | Yes (always) | | `fan-out` | `FanOutStep` | Dispatch per item over a collection | No (engine expands) | | `fan-in` | `FanInStep` | Aggregate results from fan-out | No | +| `workflow` | `WorkflowStep` | Execute an installed workflow in a private scope | No (engine enters scope) | ## Step Registry diff --git a/workflows/PUBLISHING.md b/workflows/PUBLISHING.md index 2163a136dc..9a62773716 100644 --- a/workflows/PUBLISHING.md +++ b/workflows/PUBLISHING.md @@ -90,7 +90,7 @@ steps: - ✅ `version` follows semantic versioning (X.Y.Z) - ✅ `description` is concise - ✅ All step IDs are unique -- ✅ Step types are valid: `command`, `prompt`, `shell`, `init`, `slot`, `gate`, `if`, `switch`, `while`, `do-while`, `fan-out`, `fan-in` +- ✅ Step types are valid: `command`, `prompt`, `shell`, `init`, `slot`, `gate`, `if`, `switch`, `while`, `do-while`, `fan-out`, `fan-in`, `workflow` - ✅ Required fields present per step type (e.g., `condition` for `if`, `expression` for `switch`) - ✅ Input types are valid: `string`, `number`, `boolean` - ✅ Step IDs do not contain `:` (reserved for engine-generated nested IDs like `parentId:childId`) diff --git a/workflows/README.md b/workflows/README.md index da045bfdf9..4ece1d330c 100644 --- a/workflows/README.md +++ b/workflows/README.md @@ -4,7 +4,11 @@ Workflows are multi-step, resumable automation pipelines defined in YAML. They o ## How It Works -A workflow definition declares a sequence of steps. The engine executes them in order, dispatching commands to AI integrations, running shell commands, evaluating conditions for branching, and pausing at gates for human review. State is persisted after each step, so workflows can be resumed after interruption. +A workflow definition declares a sequence of steps. The engine executes them in +order, dispatching commands to AI integrations, running shell commands, +evaluating conditions for branching, and pausing at gates for human review. +Persisted execution transitions let workflows resume after interruption without +re-running completed work. ```yaml steps: @@ -85,7 +89,10 @@ The bundled `speckit` workflow only declares `spec` (and optional ## Step Types -Workflows support 12 built-in step types: +Workflows support 13 built-in step types, including `workflow` for calling an +installed workflow with private inputs and declared outputs. See +[workflow composition and resume](../docs/reference/workflows.md#workflow-composition) +for the scope and execution identity contracts. ### Command Steps (default) @@ -313,6 +320,11 @@ Aggregate results from fan-out steps: output: {} ``` +`wait_for` references declared step IDs. For a fan-out, use the fan-out step's +own `id` (`parallel-impl` above); its ordered item results are available as +`steps..output.results`. Generated item aliases such as +`fan:template:0` are reporting-only and are not valid `wait_for` targets. + ## Error Handling By default, any step that returns `StepResult(status=StepStatus.FAILED, ...)`