Skip to content

Commit 18b8b7c

Browse files
romamoclaude
andcommitted
M8 proven live: the operator loop closes with one human approval
Disposable Hetzner cx23 run, both trigger kinds end to end. Alert: stopped PostgreSQL went from failure to receipted verified remediation in about 2.5 minutes - the operator watched remotely over the gateway's mTLS alerts route, diagnosed, proposed services convergence, and after one approval command watched the same alert resolve. Drift: a removed required package was caught by the scheduled audit pass, proposed as baseline convergence, approved through the agent confirmation handshake, and verified compliant in 53 seconds. No shell access during detection, diagnosis, or verification. Finding fixed live: a bare openssl CA lacks the key-usage extension Python's strict TLS requires; the proving CA and the operator guide now account for it. Roadmap M8 exit marked met. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HX4L4HUFnJ5bq7SFQwHzDU
1 parent 2453496 commit 18b8b7c

2 files changed

Lines changed: 87 additions & 1 deletion

File tree

‎ROADMAP.md‎

Lines changed: 9 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -218,7 +218,15 @@ handshake. No autonomy yet: this milestone is the loop, not the license.
218218

219219
Exit: an induced failure is detected, diagnosed, and remediated end to end
220220
with the human contributing only a confirmation — no human diagnosis, no
221-
shell access.
221+
shell access. **Met on 2026-09-10** on a disposable Hetzner Cloud Debian 13
222+
server, for both trigger kinds: a stopped PostgreSQL went from failure to
223+
receipted, verified remediation in about 2.5 minutes with one approval
224+
command, and induced package drift was detected by the scheduled audit,
225+
proposed, approved through the agent confirmation handshake, and verified
226+
compliant in 53 seconds — see the
227+
[M8 proving-run report](docs/proving-runs/2026-09-10-hetzner-m8.md). The
228+
operator watched remotely over the gateway's new read-only mTLS alerts
229+
route with no shell access during detection, diagnosis, or verification.
222230

223231
## M9 — Graduated autonomy
224232

Lines changed: 78 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,78 @@
1+
# Proving run: M8 operator loop on a real Hetzner host
2+
3+
| | |
4+
| --- | --- |
5+
| **Date** | 2026-09-10 |
6+
| **Scope** | M8 exit criteria: induced failure detected, diagnosed, and remediated end to end with the human contributing only a confirmation |
7+
| **Host** | Hetzner Cloud cx23 (fsn1), stock Debian 13 image, deleted after the run |
8+
| **Verdict** | All criteria met for both trigger kinds; one TLS finding fixed live |
9+
10+
## Method
11+
12+
A disposable cx23 was deployed exactly as in the
13+
[M7 run](2026-09-10-hetzner-m7.md) (PostgreSQL with declared metrics,
14+
single-host logging stack with alerting, the `postgresql-down` rule), with
15+
two differences: the declared firewall allows the mTLS gateway port so the
16+
operator can watch **remotely from the management host**, and the gateway
17+
certificate carries an IP subject-alternative name so the operator's strict
18+
TLS verification passes against the raw address. The operator authenticated
19+
with its own client certificate signed by the collector CA — it is just
20+
another mTLS client of the gateway; it never had shell access during
21+
detection, diagnosis, or verification.
22+
23+
Only public entry points were used: the deploy playbooks,
24+
`cloudfall operator run|list|approve`, and the `AgentToolset` operator
25+
surface that `cloudfall-mcp` exposes.
26+
27+
## Results
28+
29+
- **Quiet pass**: with the fleet healthy, `operator run` over
30+
`https://<ip>:3101/api/v1/alerts` returned zero proposals
31+
- **Alert trigger, end to end**: `systemctl stop postgresql@17-main` at
32+
17:18:26; the declared alert fired at 17:19:46; the next operator pass
33+
wrote a schema-valid proposal at 17:19:59 — trigger evidence, the
34+
diagnosis ("service postgresql-main on server hz1 stopped satisfying its
35+
rule"), and the exact operation (`services.yml`). One human command
36+
approved it at 17:20:12; the engine converged declared services,
37+
PostgreSQL came back, the operator watched the same alert resolve, and
38+
the receipt recorded **verified at 17:20:53**. Failure to verified
39+
remediation: about 2.5 minutes, human contribution: one approval
40+
- **Drift trigger, through the agent surface**: `apt-get remove unzip`
41+
induced package drift with no alert. The operator's drift pass ran the
42+
read-only inspect playbook, audited, and proposed `baseline.yml` for
43+
exactly `packages.required[unzip]`. The approval went through the
44+
`AgentToolset` confirmation handshake — the gate previewed "execute
45+
cloudfall-engine playbook baseline.yml because Audit reports drift on
46+
server hz1" before `confirm=true` executed it — and verification
47+
re-audited the server until the drifted check was compliant:
48+
**verified in 53 seconds**
49+
- **Dedupe**: repeated passes while proposals were open produced no
50+
duplicates; the alerts pass stayed quiet during the drift incident
51+
- **Exit state**: final audit 20/20 compliant; the proposal receipts record
52+
the full lifecycle of both incidents
53+
54+
## Findings
55+
56+
1. **The throwaway CA failed Python's strict verification.** A bare
57+
`openssl req -x509` CA lacks the key-usage extension, which nginx and
58+
Alloy accepted but Python 3.14's TLS stack refused
59+
(`CA cert does not include key usage extension`). The proving CA is now
60+
generated with `basicConstraints=critical,CA:TRUE` and
61+
`keyUsage=critical,keyCertSign,cRLSign`; the
62+
[operator guide](../operator-guide.md) documents that the operator is a
63+
strict TLS client
64+
65+
## Caveats
66+
67+
- Propose mode only: the ladder's level 2 (autonomy for provenly reversible
68+
operations) is designed but not implemented; every execution here went
69+
through an explicit approval, which is exactly M8's scope
70+
- The approval ran on the management host with SSH ambient for the engine;
71+
the operator's *detection and verification* path is shell-free, execution
72+
still uses the engine's normal SSH contract
73+
- Single-host fleet; drift and alert triggers were exercised serially, not
74+
concurrently
75+
76+
## Cost
77+
78+
One cx23 instance for under an hour: about one euro cent.

0 commit comments

Comments
 (0)