Repository navigation
Decision records take a host's changed flag from its last task only, so verify can pass a run that changed the host #76
Description
What happens
_preview in sdk/src/cloudfall/decision.py reads the per-host files the Ansible tree callback leaves in ANSIBLE_CALLBACK_TREE_DIR. That callback rewrites a host's file after every task, so the file holds the last task's result only. A host whose earlier tasks changed it but whose last task reported ok is recorded as unchanged.
Evidence
examples/disk-full-incident/recordings/disk-full-approved/shop-logrotate-20261008204351, a real run on 2026-10-08:
- Execution log
PLAY RECAP:hz1 : ok=5 changed=2(deleted three 8G rotated logs, truncated the live log) - Decision record:
execution.changed: [],execution.unchanged: ["hz1"] - The check run has the same shape: the diff shows the deletions, yet
check.changedis[]
The last task, "Install a logrotate rule for the worker", reported ok, and that is what was kept.
Why it matters
The verify step decides VERIFIED from the same fields: the second run must change nothing. A playbook whose verify run changes something in an early task and ends with an unchanged task is recorded VERIFIED. The gate also shows "no host would change" in a proposal that would change the host, which is what a person approves against.
In this recording verify was right by luck (changed=0 in its recap).
Fix direction
Take per-host totals from the play stats, not the tree files, for example a small callback in the engine that writes each host's changed, failed and unreachable counts from v2_playbook_on_stats. Re-check --check runs, executions and verify runs. recording.py replays the same fields and needs to follow.
🤖 Generated with Claude Code
Activity
romamo commentedon Oct 11, 2026
It already produced a wrong verdict on a real host. The
agent investigateproving run on 2026-10-08 (Nemotron 3 Super, cx23), decisionshop-logrotate-20261008162054:PLAY RECAP:hz1 : ok=3 changed=1 skipped=1. "Truncate the live worker log" reportedchangedagain: the playbook was not idempotent yetverify.changed: [], statusverified, verdict "the run succeeded and the verify step changed nothing"The last task, "Install a logrotate rule for the worker", reported
ok, so the host counted as unchanged. The playbook was made idempotent later the same day, and the run recorded afterwards (recordings/disk-full-approved) haschanged=0in its verify recap. Cloudfall's verdict was wrong, though, and nothing in the record shows it.shipmill-romamo commentedon Oct 11, 2026
Triage: implement, for the next release candidate.
Confirmed on
main(c29874a):_previewinsdk/src/cloudfall/decision.pysorts each host intochangedorunchangedfrom the file thetreecallback leaves for it, and that callback rewrites the file after every task. Only the last task's result survives. The--checkpreview, the execution and the verify run all go through_preview, so all three are affected, and the verify verdict (the verify step changed nothing) is read from the same field.The plan:
v2_playbook_on_stats, one JSON file per host with itschanged,failedandunreachablecounts (oktoo).run_commandenables it next to (or in place of)treeand points it at the same per-run directory_previewreads those files: a host ischangedwhen itschangedcount is above 0. The output schema stays as it is (changedandunchangedare host lists); they now mean what they always claimed torecording.pywrites what the new callback would have written, so replay keeps working. Old recordings keep their stored fields: they are records of what Cloudfall said at the time and are not rewrittenokis recordedchangedin the check, the execution and the verify step, and such a verify run is notVERIFIED. A test drives the real callback throughansible-playbookagainstlocalhostif the suite already runs Ansible; otherwise the callback is unit-tested on a stats objectOut of scope: correcting the two decision records that already carry the wrong value (
shop-logrotate-20261008162054and thedisk-full-approvedrecording). #77 reports the first one in the proving-run write-up. I'll link the PR here.shipmill-romamo commentedon Oct 11, 2026
The fix is in #80. A callback that ships with the SDK (
cloudfall_host_stats) writes each host's recap counts when the play ends, and the check preview, the execution and the verify run all readchangedfrom those counts. Ansible'streecallback is no longer used, and nothing else read it. The record schema is unchanged. A verify run that changed a host in an early task is now recordedfailed, notverified. The new end-to-end test checks this with a realansible-playbookonlocalhost.Trade-off:
ANSIBLE_CALLBACK_PLUGINSreplaces the configured path, so the run reads the team's callback path (the env, then theiransible.cfg, then Ansible's default) through ansible-core'sConfigManagerand puts the SDK's directory in front of it. That is an internal ansible-core API, kept inansible_api.pybeside the inventory reader, which already depends on ansible-core internals. The stored recordings keep their old values. This issue closes when #80 merges.shipmill-romamo commentedon Oct 11, 2026
Released in v0.7.0rc1.
shipmill-romamo commentedon Oct 20, 2026
Released in v0.7.0.
uv tool install cloudfall==0.7.0