Skip to content

Report the agent proving run on the disk-full incident - #77

Merged
shipmill-romamo[bot] merged 1 commit into
mainfrom
docs/proving-run-agent
Oct 11, 2026
Merged

shipmill-romamo[bot] merged 1 commit into
mainfrom
docs/proving-run-agent

Conversation

@romamo

@romamo romamo commented Oct 11, 2026

Copy link
Copy Markdown
Collaborator

docs/proving-runs/2026-10-08-disk-full-agent.md: Nemotron 3 Super on Token Factory works the real disk-full incident through cloudfall agent investigate. It took 5 turns and 21 s, with 14.9k tokens. It found /var/log/shop and proposed shop-logrotate and then postgresql-restart. A person approved, and PostgreSQL came back.

The report is honest about two things:

  • Cloudfall got a verdict wrong in that run. The rotation's verify run truncated the log again (changed=1 in its recap), yet the decision was recorded verified, because only the last task's result is read (Decision records take a host's changed flag from its last task only, so verify can pass a run that changed the host #76)
  • The 100% → 5% figure comes from the rerun on a fresh host with the idempotent playbook (recordings/disk-full-approved), not from the agent's own run. The README, the evaluation README and the demo page claimed otherwise, or said "the same host", and now say so. The demo page change ships with the next demo redeploy

🤖 Generated with Claude Code

https://claude.ai/code/session_01APTdZZw8Jw9PQF4TD55wMv

Nemotron 3 Super worked the real incident in five turns and proposed the
right fix; a person approved it and PostgreSQL came back. The report also
records the verdict Cloudfall got wrong there: a verify run that changed
the host was recorded verified (#76).

The 100% to 5% disk figure comes from the rerun on a fresh host with the
idempotent playbook, not from the agent's own run, so the README, the
evaluation README and the demo page now say so.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01APTdZZw8Jw9PQF4TD55wMv
@shipmill-romamo
shipmill-romamo Bot merged commit 57a5d90 into main Oct 11, 2026
1 check passed
@shipmill-romamo
shipmill-romamo Bot deleted the docs/proving-run-agent branch October 11, 2026 18:37
@shipmill-romamo shipmill-romamo Bot added this to the 0.7.0 milestone Oct 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant