treaty rc12: why, observe, secrets, import, deploy, rollback, restart, data migrate on treaty

CHANGELOG.md

@@ -80,6 +80,30 @@ Notable changes to Cloudfall. The format follows
   A missing playbook, inventory, or role directory exits 2 before Ansible
   starts. `playbook run --check` is now `--dry-run`; Ansible's play log
   streams to stderr. `cloudfall-engine manifest` describes every command
+- **Breaking:** the last eight commands that change state move to treaty:
+  `observe`, `secrets render`, `import render`, `import render-api`,
+  `deploy`, `rollback`, `restart` and `data migrate` (treaty #183 lets
+  their camelCase keys through). Each keeps its `data` keys, `status`
+  included, and adds `effect`. `deploy`, `rollback`, `restart` and
+  `data migrate` still change nothing without `--yes`; their plan answers
+  `effect: noop` and writes the result keys as `null`, and the result
+  writes the plan keys (`wouldRun`, `instruction`) as `null`, since treaty
+  answers one object per command. `observe` exits 91 `INCOMPLETE` (was 1)
+  with the run in `data` when a server gave no snapshot. Engine steps,
+  sops and the Render API go through treaty: `doctor` checks sops, and
+  `import render-api` honours `--proxy` and marks its answer as content
+  from outside. Failures that were exit 2 are 4 `PRECONDITION` with the
+  old code in `error.context.code`. A key a command writes only sometimes
+  is `null` when absent, so `inventory show` writes `data.ansible` as
+  `null` for a project. Still on argparse: `changelog`, `operator run`
+  (treaty #175) and `dashboard serve`
+- **Breaking:** `why` runs on treaty, `--format html` included (treaty
+  #179): the page is drawn from the answer's JSON document, as before. Its
+  verdict moves to `data.status`; a bad `--since` or `--until` is 2
+  `ARG_ERROR` with `why_instant_invalid` in `error.context.code`, and a
+  record that cannot be read 86 `RECORD_INVALID`. treaty offers a custom
+  format on every command (#209), so `--format html` on any other treaty
+  command renders its `data` as a page of JSON
 - **Breaking:** seven commands that run Ansible move to treaty: `health`,
   `backup run`, `backup verify`, `operations propose`, `operations approve`,
   `operator approve` and `migrate`. Ansible runs through treaty in the
@@ -116,24 +140,22 @@ Notable changes to Cloudfall. The format follows
   root `--help` and `--version` stay on argparse for now. Their answer is a
   treaty envelope (`ok`, `data`, `error`, `meta`, `warnings`): the verdict
   moves from the top-level `status` to `data.status`, and the other `data`
-  keys are unchanged. `inventory show` writes `data.ansible` as `{}` for a
+  keys are unchanged. `inventory show` writes `data.ansible` as `null` for a
   project. Failures are an envelope on stdout with their own exit codes:
   79 `PROJECT_INVALID`, 80 `CONFIG_INVALID` (resources or evidence files),
   85 `INVENTORY_UNREADABLE`, 86 `RECORD_INVALID`, 5 `NOT_FOUND` for an
   undeclared operation or a missing proposal, 2 `ARG_ERROR` for bad input;
   the old snake_case code is in `error.context.code`. `audit` exits 83
-  `DRIFT` (was 1) or 84 `UNKNOWN` (was 3) with the report in `data`; until
-  treaty #181 is fixed, that report's lists come back sorted rather than in
-  the order the servers and checks were declared. Pick
+  `DRIFT` (was 1) or 84 `UNKNOWN` (was 3) with the report in `data`. Pick
   the format with `--format json`: `--output` after one of these commands
   is refused, since treaty's `--output` names a file. Relative paths still
   resolve against the project, without changing the working directory
 - `cloudfall-engine` runs Ansible and git through treaty's `ctx.run`
-  (treaty 1.0.0rc10), so `--timeout` and Ctrl-C stop the whole process
+  (treaty 1.0.0rc12), so `--timeout` and Ctrl-C stop the whole process
   group, and secrets are redacted from the play log. The play log streams
-  to stderr at a terminal; elsewhere, such as under an agent or in CI, it
-  shows only with `-v`, one JSON line per log line. A failed playbook
-  carries the last 4096 characters of the log in `error.context.output`.
+  to stderr as plain text, as before, terminal or not; `--quiet` silences
+  it. A failed playbook carries the last 4096 characters of the log in
+  `error.context.output`, marked as untrusted content from the hosts.
   `playbook run` reports `meta.dry_run: true` under `--dry-run`. A git step
   that runs past its 600-second limit now exits 10 `TIMEOUT`, not 81 with
   `artifact_git_timeout`. `cloudfall-engine doctor` checks for

engine/README.md

@@ -56,10 +56,11 @@ uv run cloudfall-engine playbook run engine/ansible/playbooks/proxy.yml \
 ```
 
 `--dry-run` (Ansible's check mode), `--diff`, `--syntax-check`, `--limit`,
-`--tags`, and `--extra-vars` pass through to `ansible-playbook`. At a terminal,
-Ansible's play log streams to stderr as it runs; elsewhere, such as under an
-agent or in CI, pass `-v` to see it. stdout carries one JSON document when the
-run ends, and a failed run keeps the end of the log in `error.context.output`.
+`--tags`, and `--extra-vars` pass through to `ansible-playbook`. Ansible's play
+log streams to stderr as it runs, as plain text whether or not a terminal is
+attached (`--quiet` silences it). stdout carries one JSON document when the run
+ends, and a failed run keeps the end of the log in `error.context.output`,
+marked as untrusted content from the hosts.
 `cloudfall-engine doctor` checks that ansible-playbook and git are installed.
 A project uses this instead of
 setting `ANSIBLE_CONFIG` against a checkout; the `task` targets in this

engine/src/cloudfall_engine/cli.py

@@ -3,7 +3,7 @@
 Built on treaty: every run answers one JSON envelope on stdout, a failure
 carries a declared exit code, and ``cloudfall-engine manifest`` describes
 each command. Ansible and git run through ``ctx.run``; Ansible's play log
-streams to stderr on a terminal or under ``-v``.
+streams to stderr as plain text.
 """
 
 import json
@@ -23,7 +23,18 @@
 )
 from cloudfall.resources import default_engine_directory, default_schema_directory
 from cloudfall.validation import ConfigValidationError, validate_config
-from treaty import App, Arg, Ctx, Exit, Flag, Out, ParseError, Subprocess, Timeout
+from treaty import (
+    App,
+    Arg,
+    Ctx,
+    Exit,
+    External,
+    Flag,
+    Out,
+    ParseError,
+    Subprocess,
+    Timeout,
+)
 
 from cloudfall_engine.ansible_inventory import render_ansible_inventory
 from cloudfall_engine.artifact import (
@@ -88,8 +99,8 @@
     retryable=False,
     side_effects="partial",
     suggestion=(
-        "read the end of the Ansible log in error.context.output, or run "
-        "again with -v to stream all of it; fix the failing task, and run again"
+        "read the Ansible log on stderr, or its end in error.context.output; "
+        "fix the failing task, and run again"
     ),
 )
 
@@ -392,13 +403,14 @@ class Played:
     "run",
     description=(
         "Run one bundled or project playbook against the inventory; "
-        "the Ansible log streams to stderr at a terminal or under -v"
+        "the Ansible log streams to stderr"
     ),
     danger_level="mutating",
     exit_codes=["PRECONDITION", "PLAYBOOK_FAILED"],
     timeout=None,
     subprocess=Subprocess("ansible-playbook"),
     required_tools={"ansible-playbook": ANSIBLE_MINIMUM_VERSION},
+    child_log=True,
     external=False,
     supports_raw_payload=True,
     examples=[
@@ -423,9 +435,10 @@ def run_playbook(args: RunArgs, ctx: Ctx) -> Played:
             error.detail, context={"code": error.code}
         ) from error
     # Stdout belongs to the envelope, so treaty streams Ansible's play log to
-    # stderr line by line, redacted, and keeps its tail for a failure.
+    # stderr as plain text, redacted, terminal or not (a timer's journal or a
+    # CI log keeps it), and keeps its tail for a failure.
     completed = ctx.run(
-        command.argv, env=command.environment, check=False, stream=True
+        command.argv, env=command.environment, check=False, stream="always"
     )
     if completed.returncode != 0:
         returncode = completed.returncode
@@ -435,7 +448,8 @@ def run_playbook(args: RunArgs, ctx: Ctx) -> Played:
             context={
                 "returncode": returncode,
                 "playbook": str(run.playbook),
-                "output": completed.stdout,
+                # The hosts wrote this text: the agent gets it marked untrusted.
+                "output": External(completed.stdout),
             },
         )
     return Played(_effect(args), run.playbook, run.inventory_file)

pyproject.toml

@@ -20,7 +20,7 @@ dependencies = [
     "ansible-core>=2.21,<2.22",
     "jsonschema[format]>=4.23,<5",
     "pyyaml>=6,<7",
-    "treaty==1.0.0rc10",
+    "treaty==1.0.0rc12",
 ]
 
 [project.urls]