|
| 1 | +# Proving run: M8 operator loop on a real Hetzner host |
| 2 | + |
| 3 | +| | | |
| 4 | +| --- | --- | |
| 5 | +| **Date** | 2026-09-10 | |
| 6 | +| **Scope** | M8 exit criteria: induced failure detected, diagnosed, and remediated end to end with the human contributing only a confirmation | |
| 7 | +| **Host** | Hetzner Cloud cx23 (fsn1), stock Debian 13 image, deleted after the run | |
| 8 | +| **Verdict** | All criteria met for both trigger kinds; one TLS finding fixed live | |
| 9 | + |
| 10 | +## Method |
| 11 | + |
| 12 | +A disposable cx23 was deployed exactly as in the |
| 13 | +[M7 run](2026-09-10-hetzner-m7.md) (PostgreSQL with declared metrics, |
| 14 | +single-host logging stack with alerting, the `postgresql-down` rule), with |
| 15 | +two differences: the declared firewall allows the mTLS gateway port so the |
| 16 | +operator can watch **remotely from the management host**, and the gateway |
| 17 | +certificate carries an IP subject-alternative name so the operator's strict |
| 18 | +TLS verification passes against the raw address. The operator authenticated |
| 19 | +with its own client certificate signed by the collector CA — it is just |
| 20 | +another mTLS client of the gateway; it never had shell access during |
| 21 | +detection, diagnosis, or verification. |
| 22 | + |
| 23 | +Only public entry points were used: the deploy playbooks, |
| 24 | +`cloudfall operator run|list|approve`, and the `AgentToolset` operator |
| 25 | +surface that `cloudfall-mcp` exposes. |
| 26 | + |
| 27 | +## Results |
| 28 | + |
| 29 | +- **Quiet pass**: with the fleet healthy, `operator run` over |
| 30 | + `https://<ip>:3101/api/v1/alerts` returned zero proposals |
| 31 | +- **Alert trigger, end to end**: `systemctl stop postgresql@17-main` at |
| 32 | + 17:18:26; the declared alert fired at 17:19:46; the next operator pass |
| 33 | + wrote a schema-valid proposal at 17:19:59 — trigger evidence, the |
| 34 | + diagnosis ("service postgresql-main on server hz1 stopped satisfying its |
| 35 | + rule"), and the exact operation (`services.yml`). One human command |
| 36 | + approved it at 17:20:12; the engine converged declared services, |
| 37 | + PostgreSQL came back, the operator watched the same alert resolve, and |
| 38 | + the receipt recorded **verified at 17:20:53**. Failure to verified |
| 39 | + remediation: about 2.5 minutes, human contribution: one approval |
| 40 | +- **Drift trigger, through the agent surface**: `apt-get remove unzip` |
| 41 | + induced package drift with no alert. The operator's drift pass ran the |
| 42 | + read-only inspect playbook, audited, and proposed `baseline.yml` for |
| 43 | + exactly `packages.required[unzip]`. The approval went through the |
| 44 | + `AgentToolset` confirmation handshake — the gate previewed "execute |
| 45 | + cloudfall-engine playbook baseline.yml because Audit reports drift on |
| 46 | + server hz1" before `confirm=true` executed it — and verification |
| 47 | + re-audited the server until the drifted check was compliant: |
| 48 | + **verified in 53 seconds** |
| 49 | +- **Dedupe**: repeated passes while proposals were open produced no |
| 50 | + duplicates; the alerts pass stayed quiet during the drift incident |
| 51 | +- **Exit state**: final audit 20/20 compliant; the proposal receipts record |
| 52 | + the full lifecycle of both incidents |
| 53 | + |
| 54 | +## Findings |
| 55 | + |
| 56 | +1. **The throwaway CA failed Python's strict verification.** A bare |
| 57 | + `openssl req -x509` CA lacks the key-usage extension, which nginx and |
| 58 | + Alloy accepted but Python 3.14's TLS stack refused |
| 59 | + (`CA cert does not include key usage extension`). The proving CA is now |
| 60 | + generated with `basicConstraints=critical,CA:TRUE` and |
| 61 | + `keyUsage=critical,keyCertSign,cRLSign`; the |
| 62 | + [operator guide](../operator-guide.md) documents that the operator is a |
| 63 | + strict TLS client |
| 64 | + |
| 65 | +## Caveats |
| 66 | + |
| 67 | +- Propose mode only: the ladder's level 2 (autonomy for provenly reversible |
| 68 | + operations) is designed but not implemented; every execution here went |
| 69 | + through an explicit approval, which is exactly M8's scope |
| 70 | +- The approval ran on the management host with SSH ambient for the engine; |
| 71 | + the operator's *detection and verification* path is shell-free, execution |
| 72 | + still uses the engine's normal SSH contract |
| 73 | +- Single-host fleet; drift and alert triggers were exercised serially, not |
| 74 | + concurrently |
| 75 | + |
| 76 | +## Cost |
| 77 | + |
| 78 | +One cx23 instance for under an hour: about one euro cent. |
0 commit comments