Repository navigation
Disk-full incident example and a Nemotron evaluation on it - #33
Merged
Merged
Conversation
A one-host Ansible repository with an operations catalog over its playbooks and a scenario that fills the disk with unrotated worker logs, so PostgreSQL goes down for a reason that is not the symptom. The catalog holds a destructive trap (dropping the cluster). This is the incident agent investigate was proven on. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01APTdZZw8Jw9PQF4TD55wMv
Ten runs per model through agent investigate on a real broken host. Super finds the cause and the fix 10 of 10; Ultra tries to approve itself 2 of 10; Nano names the symptom but picks the fix 9 of 10; Lightning proposes dropping the database 10 of 10 and never answers. The gate kept all of it a proposal. The example's logrotate rule is written through shell, so its check mode works on a disk with no free byte, and a decision whose check failed is no longer reported as unrecorded. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01APTdZZw8Jw9PQF4TD55wMv
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
examples/disk-full-incident: a one-host Ansible repository, an operations catalog (2 read, 4 mutating, 1 destructive trap) andscenario/break.yml, which fills the disk with unrotated worker logs so PostgreSQL goes down. The README walks through it end to endevaluations/nemotron-incident:run.pydrivescloudfall agent investigateagainst the broken host and scores the investigation records; README with results; raw rows per roundagent investigate: a decision whose check failed is no longer listed underunrecorded(regression test added)Results, 10 runs per model on a real cx23
With the gate, none of it reached the host.
Found along the way
Once the worker took the last free byte, check mode of a
copytask failed (Ansible stages files in a remote temp directory). The example now writes its logrotate rule throughshell; round 1 is kept and its caveat explained in the README.Checks
uv run pytest: 392 passeduv run ruff check .(evaluation scripts allowed outside a package),uv run mypy: clean🤖 Generated with Claude Code
https://claude.ai/code/session_01APTdZZw8Jw9PQF4TD55wMv