Run site.yml from CI, one box at a time, with lockout checks

[TakGN]

site.yml now runs from CI the same way platform.yml does. It needs no new secrets. routing.yml, bootstrap.yml, content.yml and k3s-upgrade.yml deliberately stay manual for now.

What happens on merge

Once the checks pass, a dry run (--check --diff) of the merged commit writes a per-box, per-task table to the run summary. The real run then waits for a required reviewer to approve it in production. If a merge touches both playbooks, site.yml goes first and platform.yml's dry run waits until it has been applied. A failed or rejected site.yml stops platform.yml. If a newer merge supersedes the pending site.yml apply, platform.yml still goes ahead, so its change is not lost. For manual runs, workflow_dispatch now takes playbook (site/platform) and run (none/check/apply), plus limit.

Safeguards against lockouts

• One box at a time. production_rollout now governs site.yml as well: one box per batch, and a box that fails ends the run with the remaining boxes untouched. I verified this in check mode by forcing a failure on sing-1; lith-1 and chic-1 never entered the play.
• After each box, whatever --tags ran, and in dry runs too:
  • a fresh SSH login (no multiplexed socket) as ipni with sudo -n
  • every key in admin_pubkey_file still in authorized_keys (a negative test with a missing key fails as expected)
  • sshd -t on the config sshd would load at its next start
• CI never builds a box. CI sets site_require_hardened, so a box that refuses the admin key fails the run instead of falling back to root. I tested this with a bogus admin user.
• CI does not fetch cluster-admin kubeconfigs (k3s_write_local_kubeconfig: false). The apt cache refresh no longer reports changed. Together these keep the dry-run table limited to real changes.

Current dry run (all three boxes, CI's vars, from this branch)

Box	Task	Changes
sing-1	common : Apply sysctl tuning	1
sing-1	hardening : Install the admin SSH public keys	1
lith-1	common : Apply sysctl tuning	1
lith-1	hardening : Install the admin SSH public keys	1
chic-1	common : Apply sysctl tuning	1
chic-1	hardening : Install the admin SSH public keys	1

• sysctl: net.ipv4.ip_local_reserved_ports gains 8000, 8443 (from Install Argo CD on lith-1 and open the ports monitoring needs #2) and 9800-9801 (from Reserve ports 9800-9801 for the monitoring Traefik #5), which never reached the boxes. Reserving a port only affects how the kernel picks outbound source ports, so existing listeners and connections are untouched.
• authorized_keys: the same 5 keys on every box. nymd's duplicate line is removed and comments are normalised. I checked by key material that all 5 committed keys are present on all three boxes before and after.

Nothing changes in sshd, ufw or k3s, and there are no restarts. This merge will offer exactly this apply.

Other changes

• scripts/ci-platform.sh becomes scripts/ci-playbook.sh <site|platform> <check|apply> [limit].
• Fixes site.yml from a macOS controller: the local probe play cached the controller's Homebrew Python as each box's interpreter.
• README ("Running playbooks from CI"), AGENTS.md and the production_rollout comment are updated.

Known trade-off

site-main and platform-main are separate concurrency groups, so a site.yml apply and a platform.yml apply from different merges can overlap. A single shared group would let one playbook's newer merge cancel the other playbook's pending apply. This is documented in the README.

Lint: yamllint, ansible-lint (production profile, 0 failures), syntax-check of all six playbooks, shellcheck and actionlint all pass locally.

🤖 Generated with Claude Code