From c36dd1bbd9e63ae199b7fc943e78e0135131752b Mon Sep 17 00:00:00 2001 From: Takwa Date: Tue, 13 Oct 2026 19:04:34 +0200 Subject: [PATCH 1/3] Run site.yml from CI, one box at a time, with lockout checks site.yml now follows the same flow as platform.yml: the checks, a dry run of the merged commit written to the run summary, then the real run once a required reviewer approves it in `production`. When a merge touches both, site.yml goes first and platform.yml's dry run waits for it. Safeguards, since site.yml touches sshd, ufw, the admin keys and k3s: - production_rollout now governs site.yml too: one box per batch, and a failed box ends the run with the rest untouched. - After each box, whatever --tags ran: a fresh SSH login as the admin user with sudo, every key in admin_pubkey_file still in authorized_keys, and `sshd -t`. - CI sets site_require_hardened: a box that refuses the admin key fails the run rather than falling back to root, so CI never builds a box. - CI does not fetch kubeconfigs (k3s_write_local_kubeconfig), and the apt cache refresh no longer reports a change, so dry runs list real changes. scripts/ci-platform.sh becomes scripts/ci-playbook.sh , and workflow_dispatch takes `playbook` and `run` instead of `platform`. Also fixes site.yml from a macOS controller: the local probe play cached the controller's Python as each box's interpreter. Co-Authored-By: Claude Opus 5.5 --- .github/workflows/ci.yml | 193 ++++++++++++++++----- AGENTS.md | 9 + README.md | 96 ++++++---- group_vars/ipfs_nodes/main.yml | 10 +- roles/common/tasks/main.yml | 3 + roles/k3s/tasks/main.yml | 6 + scripts/{ci-platform.sh => ci-playbook.sh} | 69 +++++--- site.yml | 112 +++++++++++- 8 files changed, 395 insertions(+), 103 deletions(-) rename scripts/{ci-platform.sh => ci-playbook.sh} (52%) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index c0185a1..f67f8a0 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -5,20 +5,25 @@ # pinset section the role does not know about. Every one of those has gone # unnoticed here before, which is why this exists. # -# The exception is platform.yml (Argo CD and the monitoring stack's firewall, -# see the README), the only playbook that runs from CI: +# The exception is the playbooks that run from CI: site.yml (the boxes' base: +# packages, sysctls, SSH, firewall, k3s) and platform.yml (Argo CD and the +# monitoring stack's firewall, see the README). For each of them: # -# pull request -> the checks, then a dry run (--check --diff) of -# platform.yml against the boxes if the PR touches it -# merge to main -> the checks, then, if the merge touches platform.yml, a -# dry run of the merged commit, then the real run, once a -# required reviewer has read the dry run and approved it in -# the `production` environment +# pull request -> the checks, then a dry run (--check --diff) against the +# boxes if the PR touches that playbook +# merge to main -> the checks, then, if the merge touches it, a dry run of +# the merged commit, then the real run, once a required +# reviewer has read the dry run and approved it in the +# `production` environment # manual -> the checks, plus optionally a dry run, or a dry run and -# an apply, of platform.yml, limited to some boxes if wanted +# an apply, of one playbook, limited to some boxes if wanted +# +# site.yml goes first: platform.yml relies on what it sets up, so when a merge +# touches both, platform.yml's dry run starts only once site.yml's apply is +# done, and shows what is left to change after it. # # One workflow, so a merge starts one run. Pull requests from forks get no -# secrets: their platform.yml dry run is skipped. +# secrets: their dry runs are skipped. name: ci on: @@ -27,13 +32,18 @@ on: pull_request: workflow_dispatch: inputs: - platform: - description: "platform.yml: none, check (dry run) or apply" + playbook: + description: "Playbook to run against the boxes" + type: choice + options: [site, platform] + default: site + run: + description: "none (checks only), check (dry run) or apply (dry run, then the approved run)" type: choice options: [none, check, apply] default: none limit: - description: "platform.yml: boxes to run on (ansible --limit), empty for all" + description: "Boxes to run on (ansible --limit), empty for all" type: string default: "" @@ -201,14 +211,16 @@ jobs: shellcheck scripts/*.sh # --------------------------------------------------------------------------- - # platform.yml + # Playbooks run from CI: site.yml, then platform.yml # --------------------------------------------------------------------------- - # Whether this run touches what platform.yml applies. On a pull request, + # Whether this run touches what each playbook applies. On a pull request, # against where the branch left main, so other merges to main do not count. + # A manual run picks its playbook itself. changes: runs-on: ubuntu-latest outputs: + site: ${{ steps.diff.outputs.site }} platform: ${{ steps.diff.outputs.platform }} steps: - uses: actions/checkout@v4 @@ -218,46 +230,150 @@ jobs: - id: diff env: EVENT: ${{ github.event_name }} + PLAYBOOK: ${{ inputs.playbook }} RANGE: >- ${{ github.event_name == 'pull_request' && format('{0}...{1}', github.event.pull_request.base.sha, github.event.pull_request.head.sha) || format('{0}..{1}', github.event.before, github.sha) }} run: | set -euo pipefail - if [[ "$EVENT" == workflow_dispatch || "$RANGE" == 0000000000000000000000000000000000000000..* ]]; then - platform=true - elif git diff --name-only "$RANGE" | grep -qE \ - '^(platform\.yml|roles/(argocd|monitoring|ufw_allowlist)/|roles/route_origin/defaults/main\.yml|group_vars/|inventory/|scripts/ci-platform\.sh|\.github/workflows/ci\.yml)'; then - platform=true - else - platform=false + shared='group_vars/|inventory/|ansible\.cfg|scripts/ci-playbook\.sh|\.github/workflows/ci\.yml' + declare -A paths=( + [site]="^(site\.yml|roles/(common|hardening|k3s|storage)/|host_vars/|keys/|$shared)" + [platform]="^(platform\.yml|roles/(argocd|monitoring|ufw_allowlist)/|roles/route_origin/defaults/main\.yml|$shared)" + ) + if [[ "$EVENT" != workflow_dispatch && "$RANGE" != 0000000000000000000000000000000000000000..* ]]; then + changed=$(git diff --name-only "$RANGE") fi - echo "platform=$platform" | tee -a "$GITHUB_OUTPUT" + for playbook in site platform; do + if [[ "$EVENT" == workflow_dispatch ]]; then + hit=$([[ "$PLAYBOOK" == "$playbook" ]] && echo true || echo false) + elif [[ "$RANGE" == 0000000000000000000000000000000000000000..* ]]; then + hit=true + elif grep -qE "${paths[$playbook]}" <<< "$changed"; then + hit=true + else + hit=false + fi + echo "$playbook=$hit" | tee -a "$GITHUB_OUTPUT" + done - # What platform.yml would change, for this exact commit: on a merge, this is - # what the approver reads before approving the apply below. It writes a - # summary on the run's page. Runs only once every check has passed. - platform-dry-run: + # --- site.yml -------------------------------------------------------------- + + # What site.yml would change, for this exact commit: on a merge, this is what + # the approver reads before approving the apply below. It writes a summary on + # the run's page. Runs only once every check has passed. + site-dry-run: needs: [ansible, manifests, generated, shell, changes] if: >- - needs.changes.outputs.platform == 'true' && ( + needs.changes.outputs.site == 'true' && ( (github.event_name == 'pull_request' && github.event.pull_request.head.repo.full_name == github.repository) || github.event_name == 'push' || - (github.event_name == 'workflow_dispatch' && inputs.platform != 'none')) + (github.event_name == 'workflow_dispatch' && inputs.run != 'none')) runs-on: ubuntu-latest # On a pull request, a newer push to the same branch replaces its dry run, # in a group of its own: GitHub keeps one pending run per group, so sharing # the apply's group would let a pull request cancel an apply waiting for # approval. On main (and manual runs) it queues with the apply in - # platform-main, so it never runs while an apply is changing the boxes. + # site-main, so it never runs while an apply is changing the boxes. concurrency: - group: ${{ github.event_name == 'pull_request' && format('platform-check-{0}', github.ref) || 'platform-main' }} + group: ${{ github.event_name == 'pull_request' && format('site-check-{0}', github.ref) || 'site-main' }} cancel-in-progress: ${{ github.event_name == 'pull_request' }} env: # ansible.cfg sets the yaml callback, which no installed collection # provides here (see the manifests job). ANSIBLE_STDOUT_CALLBACK: default ANSIBLE_FORCE_COLOR: "true" + steps: + - uses: actions/checkout@v4 + with: + persist-credentials: false + - uses: actions/setup-python@v5 + with: + python-version: "3.12" + - name: Install Ansible and the collections the roles use + run: | + pip install ansible-core + ansible-galaxy collection install community.general ansible.posix kubernetes.core + - name: Dry run site.yml + env: + ANSIBLE_SSH_PRIVATE_KEY: ${{ secrets.ANSIBLE_SSH_PRIVATE_KEY }} + LIMIT: ${{ inputs.limit }} + run: scripts/ci-playbook.sh site check "$LIMIT" + + # Needs every check and the dry run of this same commit: a commit that fails + # either is never offered for approval. No status function in `if`, so + # GitHub adds success(). + # + # site.yml can lock everyone out (sshd, ufw, the admin keys) or stop k3s, so + # besides the approval it rolls one box at a time (production_rollout), and + # after each box proves a fresh SSH login with sudo still works, every key in + # admin_pubkey_file is still installed and sshd's config is valid. The first + # box that fails any of that ends the run with the others untouched. + site-apply: + needs: [ansible, manifests, generated, shell, changes, site-dry-run] + if: >- + needs.changes.outputs.site == 'true' && ( + github.event_name == 'push' || + (github.event_name == 'workflow_dispatch' && inputs.run == 'apply')) + runs-on: ubuntu-latest + # One at a time with main's dry runs, never cancelled half-way. A newer + # merge waiting behind replaces an older one still waiting for approval: + # the latest main wins, and its own dry run is the one the approver reads. + concurrency: + group: site-main + cancel-in-progress: false + # Required reviewers: nothing reaches the boxes until one approves. + environment: production + env: + ANSIBLE_STDOUT_CALLBACK: default + ANSIBLE_FORCE_COLOR: "true" + steps: + - uses: actions/checkout@v4 + with: + persist-credentials: false + - uses: actions/setup-python@v5 + with: + python-version: "3.12" + - name: Install Ansible and the collections the roles use + run: | + pip install ansible-core + ansible-galaxy collection install community.general ansible.posix kubernetes.core + - name: Run site.yml + env: + ANSIBLE_SSH_PRIVATE_KEY: ${{ secrets.ANSIBLE_SSH_PRIVATE_KEY }} + LIMIT: ${{ inputs.limit }} + run: scripts/ci-playbook.sh site apply "$LIMIT" + + # --- platform.yml ---------------------------------------------------------- + + # As site-dry-run, for platform.yml. It waits for site.yml in the same run: + # a failed or rejected site.yml stops it, since platform.yml relies on what + # site.yml sets up. Skipped is fine (nothing of site.yml's changed), and so + # is cancelled: with the run itself not cancelled (!cancelled()), that means + # a newer merge's site.yml took this one's place in site-main, and this + # merge's platform.yml change would otherwise never be applied. !cancelled() + # also means GitHub no longer adds success(), so every check is named. + platform-dry-run: + needs: [ansible, manifests, generated, shell, changes, site-dry-run, site-apply] + if: >- + !cancelled() && + needs.ansible.result == 'success' && needs.manifests.result == 'success' && + needs.generated.result == 'success' && needs.shell.result == 'success' && + needs.changes.result == 'success' && + contains(fromJSON('["success", "skipped", "cancelled"]'), needs.site-dry-run.result) && + contains(fromJSON('["success", "skipped", "cancelled"]'), needs.site-apply.result) && + needs.changes.outputs.platform == 'true' && ( + (github.event_name == 'pull_request' && github.event.pull_request.head.repo.full_name == github.repository) || + github.event_name == 'push' || + (github.event_name == 'workflow_dispatch' && inputs.run != 'none')) + runs-on: ubuntu-latest + concurrency: + group: ${{ github.event_name == 'pull_request' && format('platform-check-{0}', github.ref) || 'platform-main' }} + cancel-in-progress: ${{ github.event_name == 'pull_request' }} + env: + ANSIBLE_STDOUT_CALLBACK: default + ANSIBLE_FORCE_COLOR: "true" steps: - uses: actions/checkout@v4 with: @@ -276,25 +392,20 @@ jobs: CLOUDFLARE_DNS_TOKEN: ${{ secrets.CLOUDFLARE_DNS_TOKEN }} ARGOCD_REPO_SSH_KEY: ${{ secrets.ARGOCD_REPO_SSH_KEY }} LIMIT: ${{ inputs.limit }} - run: scripts/ci-platform.sh check "$LIMIT" + run: scripts/ci-playbook.sh platform check "$LIMIT" - # Needs every check and the dry run of this same commit: a commit that fails - # either is never offered for approval. No status function in `if`, so - # GitHub adds success(). + # As site-apply, for platform.yml. Its dry run has already waited for + # site.yml, so this needs only that dry run; GitHub adds success(). platform-apply: needs: [ansible, manifests, generated, shell, changes, platform-dry-run] if: >- needs.changes.outputs.platform == 'true' && ( github.event_name == 'push' || - (github.event_name == 'workflow_dispatch' && inputs.platform == 'apply')) + (github.event_name == 'workflow_dispatch' && inputs.run == 'apply')) runs-on: ubuntu-latest - # One at a time with main's dry runs, never cancelled half-way. A newer - # merge waiting behind replaces an older one still waiting for approval: - # the latest main wins, and its own dry run is the one the approver reads. concurrency: group: platform-main cancel-in-progress: false - # Required reviewers: nothing reaches the boxes until one approves. environment: production env: ANSIBLE_STDOUT_CALLBACK: default @@ -317,4 +428,4 @@ jobs: CLOUDFLARE_DNS_TOKEN: ${{ secrets.CLOUDFLARE_DNS_TOKEN }} ARGOCD_REPO_SSH_KEY: ${{ secrets.ARGOCD_REPO_SSH_KEY }} LIMIT: ${{ inputs.limit }} - run: scripts/ci-platform.sh apply "$LIMIT" + run: scripts/ci-playbook.sh platform apply "$LIMIT" diff --git a/AGENTS.md b/AGENTS.md index caa43a8..0138f00 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -120,6 +120,15 @@ ansible-playbook routing.yml -l sing-1 # one box ansible-playbook routing.yml # one box at a time, warmed (default) ``` +`site.yml` and `platform.yml` apply from CI: a merge that touches one gets a +dry run on the run's summary page, then waits for a required reviewer to +approve the real run in the `production` environment (README, "Running +playbooks from CI"). Prefer that to running them from a controller, so what +the boxes run is what is on `main`. site.yml also follows `production_rollout`: +one box at a time, stopping at the first box that fails or loses admin SSH +access. CI refuses to build a box (no root fallback), so a fresh box is still +an operator's `site.yml -l `. + needle stays degraded for tens of minutes after a restart, long after the pod reports Ready — measured at 200 req/s: 11 minutes in, p95 2904 ms with 2937 rejected lookups; by 69 minutes, p95 588–691 ms. That is what diff --git a/README.md b/README.md index 1c3d6a7..1b00eb8 100644 --- a/README.md +++ b/README.md @@ -60,9 +60,10 @@ carries no traffic. With it true, `routing.yml`, `bootstrap.yml`, additionally gates each needle instance on its warm-up: seconds when it restores its address book and routing table snapshots, up to an hour on a cold start (see "Capacity: four needle instances per box"). Every box is a whole site with no failover between them, -so a simultaneous rollout in production is a full outage. `site.yml` is the -exception: it is not serialised, so re-run it against a live fleet one box at -a time with `-l`. +so a simultaneous rollout in production is a full outage. `site.yml` has no +warm-up, but it also goes one box at a time, and stops at the first box that +fails or where the admin user can no longer get in (see "Running playbooks +from CI"). ## Accessing the clusters @@ -1445,21 +1446,9 @@ for itself: ingest) from Cloudflare's edge and the other boxes, and 8000 on sing-1 (`healthchecks_host`, the hub's dead-man's switch) from lith-1. -**It runs from CI**, the only playbook that does; the others still run from an -operator's machine. It is part of the repo's one workflow -(`.github/workflows/ci.yml`), so a merge starts a single run: - -| When | What | -|---|---| -| Pull request touching it | the checks, then `--check --diff` against the boxes (forks get no secrets and are skipped) | -| Merge to `main` touching it | the checks, then a dry run of the merged commit, then the real run, once a required reviewer has read the dry run and approved it in the `production` environment | -| Actions → ci → Run workflow | the checks, plus `platform`: `check` (dry run) or `apply` (dry run, then the approved run), optionally `--limit`ed | - -Each dry run writes what it would change, per box and task, on the run's -summary page; approve only if that is what you expect. - -"Touching it" means `platform.yml`, its roles (`argocd`, `monitoring`, -`ufw_allowlist`), `group_vars/`, `inventory/`, `scripts/ci-platform.sh` or the +**It runs from CI** (see "Running playbooks from CI"). "Touching it" means +`platform.yml`, its roles (`argocd`, `monitoring`, `ufw_allowlist`), +`group_vars/`, `inventory/`, `ansible.cfg`, `scripts/ci-playbook.sh` or the workflow itself. A `group_vars` change for another service therefore also offers an apply; with nothing of its own changed, it changes nothing. @@ -1467,24 +1456,73 @@ It restarts nothing that serves traffic, so it runs every box in one batch: the hub registers the others using what they produced earlier in the same run. CI has no vault files, so `platform.yml`'s vault variables come from -repository secrets (`scripts/ci-platform.sh` maps them), and it signs in with -its own SSH key, `keys/github-actions.pub` in `admin_pubkey_file`. It only -connects to the host keys pinned in `inventory/known_hosts`. +repository secrets (`scripts/ci-playbook.sh` maps them): | Secret | Becomes | |---|---| -| `ANSIBLE_SSH_PRIVATE_KEY` | CI's SSH key as the admin user | | `ARGOCD_AGE_KEY` | `vault_argocd_age_key`: Argo CD's age key (public half in ipni/infra's `.sops.yaml`) | | `CLOUDFLARE_DNS_TOKEN` | `vault_cloudflare_dns_token`: the same token cert-manager uses for WSS | | `ARGOCD_REPO_SSH_KEY` | `vault_argocd_repo_ssh_key`: optional, only while ipni/infra is private | -Copies of all of them are in 1Password (IPFS account, Compass vault): GitHub -cannot show them again. To revoke CI's access to the boxes, remove its line -from `admin_pubkey_file` and re-run the hardening role. - From an operator's machine it still works as usual, with the same variables in the vault: `ansible-playbook platform.yml --check --diff`. +## Running playbooks from CI + +`site.yml` and `platform.yml` run from CI; the others (`routing.yml`, +`bootstrap.yml`, `content.yml`, `k3s-upgrade.yml`) still run from an +operator's machine. Both are part of the repo's one workflow +(`.github/workflows/ci.yml`), so a merge starts a single run: + +| When | What | +|---|---| +| Pull request touching a playbook | the checks, then `--check --diff` of it against the boxes (forks get no secrets and are skipped) | +| Merge to `main` touching a playbook | the checks, then a dry run of the merged commit, then the real run, once a required reviewer has read the dry run and approved it in the `production` environment | +| Actions → ci → Run workflow | the checks, plus for the chosen `playbook`: `check` (dry run) or `apply` (dry run, then the approved run), optionally `--limit`ed | + +Each dry run writes what it would change, per box and task, on the run's +summary page; approve only if that is what you expect. Each playbook needs its +own approval. When a merge touches both, `site.yml` goes first, and +`platform.yml`'s dry run starts only once `site.yml` is applied (and not at +all if it failed or was rejected). + +On `main`, each playbook queues in its own concurrency group (`site-main`, +`platform-main`): never two runs of one playbook at once, and a newer merge +replaces an older one still waiting for approval, so the latest `main` is what +gets applied. The two groups are separate, so a `site.yml` and a +`platform.yml` apply from *different* merges can overlap; approve one at a +time if that matters. + +CI signs in with its own SSH key, `keys/github-actions.pub` in +`admin_pubkey_file` (secret `ANSIBLE_SSH_PRIVATE_KEY`), and only connects to +the host keys pinned in `inventory/known_hosts`. Copies of every CI secret are +in 1Password (IPFS account, Compass vault): GitHub cannot show them again. To +revoke CI's access to the boxes, remove its line from `admin_pubkey_file` and +re-run the hardening role. + +**site.yml.** "Touching it" means `site.yml`, its roles (`common`, +`hardening`, `k3s`, `storage`), `group_vars/`, `host_vars/`, `keys/`, +`inventory/`, `ansible.cfg`, `scripts/ci-playbook.sh` or the workflow. It needs +no secrets beyond CI's SSH key. It can lock everyone out (sshd, ufw, the admin +keys) or restart k3s, so: + +* CI only maintains boxes that are already hardened: it sets + `site_require_hardened`, and a box that does not accept the admin key fails + the run instead of falling back to root. Building a new box (bootstrap mode, + `vault_root_password`) stays an operator's job. +* With `production_rollout` it goes one box at a time. After each box it + proves a fresh SSH login as the admin user with `sudo` still works, that + every key in `admin_pubkey_file` is still in `authorized_keys`, and that + `sshd -t` accepts the configuration sshd would load on its next start. The + first box that fails anything ends the run, leaving the others untouched. + These checks run with any `--tags`, and in dry runs too. +* It does not run `apt upgrade` on a hardened box (`common_apt_upgrade`), so a + merge never pulls in a whole-system upgrade. + +If CI does lock itself out, an operator's key or the break-glass key (AGENTS.md) +still gets in, unless the change removed those too, which the key check above +stops after the first box. + ## Secrets **This repository is public, so no credentials are committed, not even @@ -1698,15 +1736,15 @@ scripts/fetch-upstream-entry.sh fetch one upstream-recovered CID straight fr scripts/recover-upstream-pinset.sh capture the upstream collab cluster's pinset as a follower (fetches no content) dns.txt Cloudflare-importable bootstrap DNS records (generated) dns-libp2p.txt bootstrap.libp2p.io records for the legacy identities (generated) -site.yml base preparation playbook +site.yml base preparation playbook, run from CI k3s-upgrade.yml k3s upgrade, one minor version at a time, backup per step platform.yml Argo CD (first install) and the monitoring stack's prerequisites, run from CI -scripts/ci-platform.sh CI's wrapper for platform.yml: secrets to variables, SSH key, pinned host keys +scripts/ci-playbook.sh CI's wrapper for site.yml and platform.yml: secrets to variables, SSH key, pinned host keys inventory/known_hosts the boxes' SSH host keys, pinned for CI keys/*.pub the SSH keys allowed in as the admin user: takgn, nymd, byo, break-glass, CI (admin_pubkey_file) routing.yml routing service playbook (needle + origin) bootstrap.yml bootstrap node playbook (with cert-manager) content.yml content cluster node playbook (deploy, load the pinset, wait for PINNED, summary) -.github/workflows/ci.yml drift checks (playbook syntax, lint, kustomize build, generated files, shellcheck), then platform.yml: dry run on PRs, approved run on merge +.github/workflows/ci.yml drift checks (playbook syntax, lint, kustomize build, generated files, shellcheck), then site.yml and platform.yml: dry run on PRs, approved run on merge .github/workflows/cloudflare-ranges.yml weekly pinned-vs-published Cloudflare range check ``` diff --git a/group_vars/ipfs_nodes/main.yml b/group_vars/ipfs_nodes/main.yml index 87a6dd8..95a39b7 100644 --- a/group_vars/ipfs_nodes/main.yml +++ b/group_vars/ipfs_nodes/main.yml @@ -56,11 +56,11 @@ firewall_allowed_udp: [] # true (production): one box at a time, and needle's rollout waits for each box # to rebuild the state a restart throws away before the next one is touched. # -# This governs routing.yml, bootstrap.yml, content.yml and k3s-upgrade.yml, via -# the `serial:` expression each of them carries. site.yml is deliberately left -# out: it is the build-the-box playbook, and re-running it against a live fleet -# is an -l-one-box-at-a-time job whatever this is set to. `rollout_serial` -# overrides the batch size on its own if needed. +# This governs site.yml, routing.yml, bootstrap.yml, content.yml and +# k3s-upgrade.yml, via the `serial:` expression each of them carries. site.yml +# has no warm-up to wait for, but it does touch sshd, ufw and k3s, so it goes +# one box at a time and stops at the first box that fails or loses SSH access. +# `rollout_serial` overrides the batch size on its own if needed. # # Every box is a whole site behind its own hostname with no failover between # them, so once these are the only boxes serving traffic, a simultaneous diff --git a/roles/common/tasks/main.yml b/roles/common/tasks/main.yml index ad5c265..da7f28c 100644 --- a/roles/common/tasks/main.yml +++ b/roles/common/tasks/main.yml @@ -23,10 +23,13 @@ when: current_tz.stdout | trim != 'UTC' changed_when: true +# Refreshing the package lists changes nothing installed. Reported as a change, +# it would be one on every run, and in every dry run an approver reads. - name: Update apt cache ansible.builtin.apt: update_cache: true cache_valid_time: 3600 + changed_when: false # Gated: see common_apt_upgrade in defaults/main.yml for why this does not run # unprompted on a box that is already carrying traffic. diff --git a/roles/k3s/tasks/main.yml b/roles/k3s/tasks/main.yml index d25e434..e7bab61 100644 --- a/roles/k3s/tasks/main.yml +++ b/roles/k3s/tasks/main.yml @@ -405,10 +405,14 @@ changed_when: false failed_when: false +# The local kubeconfig is for an operator's tunnel. CI sets +# k3s_write_local_kubeconfig to false: a runner has no use for cluster-admin +# credentials, and a file it never had would show as a change in every dry run. - name: Read the generated kubeconfig ansible.builtin.slurp: src: /etc/rancher/k3s/k3s.yaml register: kubeconfig_raw + when: k3s_write_local_kubeconfig | default(true) | bool - name: Ensure the local kubeconfig directory exists ansible.builtin.file: @@ -418,6 +422,7 @@ delegate_to: localhost become: false run_once: true + when: k3s_write_local_kubeconfig | default(true) | bool # The server URL is rewritten to localhost because access is via SSH tunnel: # ssh -L 6443:127.0.0.1:6443 {{ admin_user }}@{{ ansible_host }} @@ -453,3 +458,4 @@ mode: "0600" delegate_to: localhost become: false + when: k3s_write_local_kubeconfig | default(true) | bool diff --git a/scripts/ci-platform.sh b/scripts/ci-playbook.sh similarity index 52% rename from scripts/ci-platform.sh rename to scripts/ci-playbook.sh index f30de51..cea8aff 100755 --- a/scripts/ci-platform.sh +++ b/scripts/ci-playbook.sh @@ -1,24 +1,35 @@ #!/usr/bin/env bash -# Run platform.yml from CI (the platform-* jobs of .github/workflows/ci.yml). +# Run a playbook from CI (the site-* and platform-* jobs of +# .github/workflows/ci.yml). # -# scripts/ci-platform.sh check [limit] dry run: --check --diff -# scripts/ci-platform.sh apply [limit] the real run +# scripts/ci-playbook.sh site|platform check [limit] dry run: --check --diff +# scripts/ci-playbook.sh site|platform apply [limit] the real run # -# The vault files are gitignored, so CI has none: platform.yml's three vault -# variables come from repository secrets instead, in the environment: -# ARGOCD_AGE_KEY -> vault_argocd_age_key -# CLOUDFLARE_DNS_TOKEN -> vault_cloudflare_dns_token -# ARGOCD_REPO_SSH_KEY -> vault_argocd_repo_ssh_key (optional: only while -# ipni/infra is private) -# and SSH as the admin user from ANSIBLE_SSH_PRIVATE_KEY, whose public half is -# keys/github-actions.pub. Copies of every secret are in 1Password (IPFS -# account, Compass vault). +# SSH as the admin user comes from ANSIBLE_SSH_PRIVATE_KEY, whose public half is +# keys/github-actions.pub. The vault files are gitignored, so CI has none: each +# playbook's vault variables come from repository secrets instead, in the +# environment. Copies of every secret are in 1Password (IPFS account, Compass +# vault). +# +# site.yml none. Its one vault variable, vault_root_password, is only +# for a box that has never been hardened, and CI refuses those +# (site_require_hardened): building a box stays an operator's +# job. Nor does it fetch kubeconfigs (k3s_write_local_kubeconfig). +# platform.yml ARGOCD_AGE_KEY -> vault_argocd_age_key +# CLOUDFLARE_DNS_TOKEN -> vault_cloudflare_dns_token +# ARGOCD_REPO_SSH_KEY -> vault_argocd_repo_ssh_key (optional: +# only while ipni/infra is private) set -euo pipefail -mode=${1:?usage: $0 check|apply [limit]} -limit=${2:-} +playbook=${1:?usage: $0 site|platform check|apply [limit]} +mode=${2:?usage: $0 site|platform check|apply [limit]} +limit=${3:-} +[[ $playbook == site || $playbook == platform ]] || { echo "playbook must be site or platform" >&2; exit 2; } [[ $mode == check || $mode == apply ]] || { echo "mode must be check or apply" >&2; exit 2; } -: "${ANSIBLE_SSH_PRIVATE_KEY:?not set}" "${ARGOCD_AGE_KEY:?not set}" "${CLOUDFLARE_DNS_TOKEN:?not set}" +: "${ANSIBLE_SSH_PRIVATE_KEY:?not set}" +if [[ $playbook == platform ]]; then + : "${ARGOCD_AGE_KEY:?not set}" "${CLOUDFLARE_DNS_TOKEN:?not set}" +fi cd "$(dirname "$0")/.." umask 077 @@ -35,19 +46,23 @@ echo ci > .vault_pass # never appear in a process list or the log. secrets=$(mktemp) trap 'rm -f "$secrets"' EXIT -python3 - "$secrets" <<'PY' +python3 - "$playbook" "$secrets" <<'PY' import json, os, sys -values = { - "vault_argocd_age_key": os.environ["ARGOCD_AGE_KEY"], - "vault_cloudflare_dns_token": os.environ["CLOUDFLARE_DNS_TOKEN"], -} -if os.environ.get("ARGOCD_REPO_SSH_KEY"): - values["vault_argocd_repo_ssh_key"] = os.environ["ARGOCD_REPO_SSH_KEY"] -with open(sys.argv[1], "w") as f: +playbook, path = sys.argv[1], sys.argv[2] +values = {} +if playbook == "site": + values["site_require_hardened"] = True + values["k3s_write_local_kubeconfig"] = False +elif playbook == "platform": + values["vault_argocd_age_key"] = os.environ["ARGOCD_AGE_KEY"] + values["vault_cloudflare_dns_token"] = os.environ["CLOUDFLARE_DNS_TOKEN"] + if os.environ.get("ARGOCD_REPO_SSH_KEY"): + values["vault_argocd_repo_ssh_key"] = os.environ["ARGOCD_REPO_SSH_KEY"] +with open(path, "w") as f: json.dump(values, f) # JSON is YAML, which -e @file reads PY -args=(platform.yml -e "@$secrets") +args=("$playbook.yml" -e "@$secrets") [[ $mode == check ]] && args+=(--check --diff) [[ -n $limit ]] && args+=(--limit "$limit") @@ -64,9 +79,9 @@ set +e ansible-playbook "${args[@]}" 2>&1 | tee "$log" status=${PIPESTATUS[0]} set -e -python3 - "$log" "$status" >> "$GITHUB_STEP_SUMMARY" <<'PY' +python3 - "$playbook" "$log" "$status" >> "$GITHUB_STEP_SUMMARY" <<'PY' import collections, re, sys -log, status = sys.argv[1], int(sys.argv[2]) +playbook, log, status = sys.argv[1], sys.argv[2], int(sys.argv[3]) ansi = re.compile(r"\x1b\[[0-9;]*m") lines = [ansi.sub("", l.rstrip("\n")) for l in open(log, errors="replace")] task, changes, failures, recap, in_recap = "", collections.OrderedDict(), [], [], False @@ -85,7 +100,7 @@ for line in lines: in_recap = False else: recap.append(line) -print("### platform.yml dry run (`--check --diff`)\n") +print(f"### {playbook}.yml dry run (`--check --diff`)\n") if failures: print("**Failed:** " + "; ".join(failures) + "\n") if changes: diff --git a/site.yml b/site.yml index dc6c330..fb3ffe6 100644 --- a/site.yml +++ b/site.yml @@ -7,12 +7,24 @@ # # Safe to re-run: it detects whether hardening has already happened and picks # the right credentials automatically. +# +# With production_rollout (group_vars/ipfs_nodes/main.yml) it prepares one box +# at a time and stops at the first box that fails, so a change that breaks SSH +# or k3s takes down one site, not three. After each box it proves the admin +# user can still get in and every listed key is still installed, before the +# next box is touched. It also runs from CI (.github/workflows/ci.yml), which +# sets site_require_hardened: CI only maintains boxes that are already built. - name: Detect which credentials work for each host hosts: ipfs_nodes gather_facts: false connection: local tags: always + vars: + # Interpreter discovery on this controller-side play would otherwise be + # cached as the BOX's interpreter, and the next play would look for, say, + # a Homebrew python on an Ubuntu box. + ansible_python_interpreter: "{{ ansible_playbook_python }}" tasks: - name: Probe key-based login as the admin user ansible.builtin.command: @@ -42,6 +54,19 @@ bootstrap_mode: false when: admin_probe.rc == 0 + # CI has no root password and must never build a box: a hardened box that + # stops accepting the admin key is an incident to look at, not a reason + # to try root. + - name: Refuse to fall back to root when only hardened boxes are expected + ansible.builtin.fail: + msg: >- + {{ inventory_hostname }}: key login as {{ admin_user }} failed + ({{ admin_probe.stderr | default('') | trim }}), and + site_require_hardened is set, so this run will not try root. + when: + - admin_probe.rc != 0 + - site_require_hardened | default(false) | bool + # no_log: set_fact echoes its arguments, so at -v or through any callback # plugin the vaulted root password would be printed in cleartext. - name: Fall back to root password auth on a freshly provisioned host @@ -62,7 +87,20 @@ - name: Prepare IPFS hosts hosts: ipfs_nodes gather_facts: true - any_errors_fatal: false # one host failing must not abandon the other two + # serial is templated before any host is chosen, so inventory group_vars are + # not visible here: read production_rollout from its group_vars file + # directly. -e production_rollout and -e rollout_serial still override. + # + # One box per batch, and a batch whose only box fails ends the play, so the + # boxes after it are left untouched. With production_rollout false (building + # a fleet that carries no traffic) they all go at once, and one failing must + # not abandon the others. + serial: >- + {{ rollout_serial | default(1 if (production_rollout + | default((lookup('ansible.builtin.file', playbook_dir ~ '/group_vars/ipfs_nodes/main.yml') + | from_yaml).production_rollout | default(false)) | bool) + else (groups['ipfs_nodes'] | length)) }} + any_errors_fatal: false roles: - role: common tags: [common, base] @@ -74,6 +112,78 @@ tags: [k3s, kubernetes] post_tasks: + # The hardening role checks access while it runs; these check it once the + # whole box is done, whatever --tags ran, and before the next box starts. + # A failure here fails this box's batch, which ends the run. + - name: Prove a fresh SSH login as the admin user still works, with sudo + ansible.builtin.command: + argv: + - ssh + - -o + - BatchMode=yes + - -o + - ConnectTimeout=10 + - -o + - StrictHostKeyChecking=accept-new + - -o + - PasswordAuthentication=no + # A new connection, not one multiplexed over a socket opened before + # sshd changed. + - -o + - ControlMaster=no + - -o + - ControlPath=none + - -p + - "{{ ssh_port }}" + - "{{ admin_user }}@{{ hostvars[inventory_hostname].ansible_host }}" + - "sudo -n true && echo SITE_ACCESS_OK" + delegate_to: localhost + become: false + register: site_access + check_mode: false + changed_when: false + retries: 3 + delay: 5 + until: "'SITE_ACCESS_OK' in (site_access.stdout | default(''))" + failed_when: "'SITE_ACCESS_OK' not in (site_access.stdout | default(''))" + tags: always + + # The login above is whoever runs this (CI's key, in CI). Every other key + # in admin_pubkey_file must still be there too, or the run that kept CI in + # has locked an operator - or the break-glass key - out. + - name: Read the admin user's authorized keys + ansible.builtin.slurp: + src: "/home/{{ admin_user }}/.ssh/authorized_keys" + register: site_authorized_keys + check_mode: false + tags: always + + - name: Check every key in admin_pubkey_file is still authorized + vars: + installed: >- + {{ (site_authorized_keys.content | b64decode).splitlines() + | map('split') | select | map(attribute=1) | list }} + pubkey_files: "{{ [admin_pubkey_file] if admin_pubkey_file is string else admin_pubkey_file }}" + expected: >- + {{ query('ansible.builtin.file', *pubkey_files) + | map('split') | map(attribute=1) | list }} + ansible.builtin.assert: + that: expected | difference(installed) | length == 0 + fail_msg: >- + {{ inventory_hostname }}: {{ expected | difference(installed) | length }} + key(s) in admin_pubkey_file are not in authorized_keys. + quiet: true + tags: always + + # sshd only reads its configuration on a restart. A broken one would not + # show until the next reboot, when it would lock everyone out at once. + - name: Check the sshd configuration sshd would load on its next start + ansible.builtin.command: + argv: [/usr/sbin/sshd, -t] + check_mode: false + changed_when: false + tags: always + # Read live state rather than ansible_mounts: facts are cached for an hour # (see ansible.cfg), so a cached fact set gathered before the data volume # existed would report it missing on a later run. From 3eada940e8ed675e63f5a8dbab76d7ab99aa3454 Mon Sep 17 00:00:00 2001 From: Takwa Date: Tue, 13 Oct 2026 19:09:19 +0200 Subject: [PATCH 2/3] README: the other playbooks stay manual on purpose Co-Authored-By: Claude Opus 5.5 --- README.md | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index 1b00eb8..9723d32 100644 --- a/README.md +++ b/README.md @@ -1469,9 +1469,11 @@ in the vault: `ansible-playbook platform.yml --check --diff`. ## Running playbooks from CI -`site.yml` and `platform.yml` run from CI; the others (`routing.yml`, -`bootstrap.yml`, `content.yml`, `k3s-upgrade.yml`) still run from an -operator's machine. Both are part of the repo's one workflow +`site.yml` and `platform.yml` run from CI. The others (`routing.yml`, +`bootstrap.yml`, `content.yml`, `k3s-upgrade.yml`) deliberately stay on an +operator's machine for now: they need the vault (for `bootstrap.yml` and +`content.yml` that includes permanent identity keys), and a `routing.yml` run +takes about an hour per box. Both CI playbooks are part of the repo's one workflow (`.github/workflows/ci.yml`), so a merge starts a single run: | When | What | From a5da5848d63de5b54843c47062d6680e6cc7384e Mon Sep 17 00:00:00 2001 From: Takwa Date: Tue, 13 Oct 2026 19:14:23 +0200 Subject: [PATCH 3/3] README: routing, content and bootstrap are headed for Helm Co-Authored-By: Claude Opus 5.5 --- README.md | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index 9723d32..b33937a 100644 --- a/README.md +++ b/README.md @@ -1473,7 +1473,10 @@ in the vault: `ansible-playbook platform.yml --check --diff`. `bootstrap.yml`, `content.yml`, `k3s-upgrade.yml`) deliberately stay on an operator's machine for now: they need the vault (for `bootstrap.yml` and `content.yml` that includes permanent identity keys), and a `routing.yml` run -takes about an hour per box. Both CI playbooks are part of the repo's one workflow +takes about an hour per box. `routing.yml`, `content.yml` and `bootstrap.yml` +are meant to move to Helm charts later, deployed by Argo CD like the monitoring +stack, so CI is not given their secrets in the meantime. Both CI playbooks are +part of the repo's one workflow (`.github/workflows/ci.yml`), so a merge starts a single run: | When | What |