[romamo]

What

• agent investigate --replay DIR: DIR is a decisions directory from a run on real hosts. Each check-mode call answers with the output and per-host results recorded for the same playbook, --limit and --extra-vars, through the existing CheckRunner seam, so validation, decision records and the gate are exactly the live ones. A call the recording does not hold fails with "holds no run of ..."; nothing is invented. The record names the recording under spec.replay (schema updated)
• A mutating operation whose check failed now gives the model the end of its output (live too: Lightning never saw No space left on device)
• examples/disk-full-incident/recordings/disk-full: 10 recorded call variants from the evaluation on the broken cx23 (84K, no addresses or secrets)

Why

The hosted hackathon demo runs a live Nemotron against this recording instead of a real host, and the same mechanism lets anyone try an agent on a real incident repeatedly, as a regression test.

Proven

Against the recording with live Nemotron on Token Factory, no host: Super found /var/log/shop and proposed shop-logrotate then postgresql-restart in 5 turns; Lightning proposed the wipe and ran out of turns, as on the real host.

Checks

• uv run pytest: 393 passed (new CLI test: a replay answers from the recording while the playbook would fail if run, and an unrecorded call fails)
• uv run ruff check ., uv run mypy: clean

🤖 Generated with Claude Code

https://claude.ai/code/session_01APTdZZw8Jw9PQF4TD55wMv

[shipmill-romamo]

Review: MERGE AFTER SMALL FIX (fix pushed as ac10916)

Checked

• Recording keys runs by playbook, --limit pattern and --extra-vars read back from the same run_command argv the live path builds, so a replay goes through the same validation, decision store and gate
• Unrecorded calls fail with exit 3 and say so; nothing is invented
• _tool_result now hands the model the check output of a failed mutating proposal (through host_text, so long output keeps only its end)
• The 10 recorded runs hold no addresses, home paths or secrets
• Schema: spec.replay is optional, so older investigation records stay valid

Fixed (ac10916)

• A --replay path that was missing, or a directory holding no decision record, loaded as an empty recording, and the model spent all its turns on calls that each failed "holds no run". It now exits PRECONDITION (recording_empty) before the model is asked, with a regression test for both cases and a line in the CHANGELOG entry

Follow-up, not blocking

• Decisions proposed during a replay land in --decisions like live ones and carry no marker that their check output was played back; only the investigation record names the replay. An operator could approve one against a check that never ran on the host. Worth marking the decision record (or writing replayed decisions to a separate default directory) in a later change

Checks (locally, on the PR head and on the head merged with main): uv run pytest 395 passed, ruff check, mypy, cloudfall config validate --project config/examples, uv lock --check, all clean