One workflow: checks, dry run, then the approved apply

.github/workflows/ci.yml

@@ -1,18 +1,41 @@
 ---
-# The repo has no runtime tests - nothing here talks to a box. What it does
-# catch is DRIFT: a generated file that no longer matches its generator, a
-# pinned upstream manifest that changed under its checksum, a playbook that
-# stopped parsing, a pinset section the role does not know about. Every one of
-# those has gone unnoticed here before, which is why this exists.
-name: checks
+# The repo's one workflow. Most of it never talks to a box: it catches DRIFT,
+# a generated file that no longer matches its generator, a pinned upstream
+# manifest that changed under its checksum, a playbook that stopped parsing, a
+# pinset section the role does not know about. Every one of those has gone
+# unnoticed here before, which is why this exists.
+#
+# The exception is platform.yml (Argo CD and the monitoring stack's firewall,
+# see the README), the only playbook that runs from CI:
+#
+#   pull request   -> the checks, then a dry run (--check --diff) of
+#                     platform.yml against the boxes if the PR touches it
+#   merge to main  -> the checks, then, if the merge touches platform.yml, a
+#                     dry run of the merged commit, then the real run, once a
+#                     required reviewer has read the dry run and approved it in
+#                     the `production` environment
+#   manual         -> the checks, plus optionally a dry run, or a dry run and
+#                     an apply, of platform.yml, limited to some boxes if wanted
+#
+# One workflow, so a merge starts one run. Pull requests from forks get no
+# secrets: their platform.yml dry run is skipped.
+name: ci
 
 on:
   push:
     branches: [main]
   pull_request:
   workflow_dispatch:
-  # platform.yml runs these first and only applies if they all pass.
-  workflow_call:
+    inputs:
+      platform:
+        description: "platform.yml: none, check (dry run) or apply"
+        type: choice
+        options: [none, check, apply]
+        default: none
+      limit:
+        description: "platform.yml: boxes to run on (ansible --limit), empty for all"
+        type: string
+        default: ""
 
 permissions:
   contents: read
@@ -176,3 +199,122 @@ jobs:
         run: |
           sudo apt-get update && sudo apt-get install -y shellcheck
           shellcheck scripts/*.sh
+
+  # ---------------------------------------------------------------------------
+  # platform.yml
+  # ---------------------------------------------------------------------------
+
+  # Whether this run touches what platform.yml applies. On a pull request,
+  # against where the branch left main, so other merges to main do not count.
+  changes:
+    runs-on: ubuntu-latest
+    outputs:
+      platform: ${{ steps.diff.outputs.platform }}
+    steps:
+      - uses: actions/checkout@v4
+        with:
+          fetch-depth: 0
+          persist-credentials: false
+      - id: diff
+        env:
+          EVENT: ${{ github.event_name }}
+          RANGE: >-
+            ${{ github.event_name == 'pull_request'
+                && format('{0}...{1}', github.event.pull_request.base.sha, github.event.pull_request.head.sha)
+                || format('{0}..{1}', github.event.before, github.sha) }}
+        run: |
+          set -euo pipefail
+          if [[ "$EVENT" == workflow_dispatch || "$RANGE" == 0000000000000000000000000000000000000000..* ]]; then
+            platform=true
+          elif git diff --name-only "$RANGE" | grep -qE \
+              '^(platform\.yml|roles/(argocd|monitoring|ufw_allowlist)/|roles/route_origin/defaults/main\.yml|group_vars/|inventory/|scripts/ci-platform\.sh|\.github/workflows/ci\.yml)'; then
+            platform=true
+          else
+            platform=false
+          fi
+          echo "platform=$platform" | tee -a "$GITHUB_OUTPUT"
+
+  # What platform.yml would change, for this exact commit: on a merge, this is
+  # what the approver reads before approving the apply below. It writes a
+  # summary on the run's page. Runs only once every check has passed.
+  platform-dry-run:
+    needs: [ansible, manifests, generated, shell, changes]
+    if: >-
+      needs.changes.outputs.platform == 'true' && (
+        (github.event_name == 'pull_request' && github.event.pull_request.head.repo.full_name == github.repository) ||
+        github.event_name == 'push' ||
+        (github.event_name == 'workflow_dispatch' && inputs.platform != 'none'))
+    runs-on: ubuntu-latest
+    # On a pull request, a newer push to the same branch replaces its dry run,
+    # in a group of its own: GitHub keeps one pending run per group, so sharing
+    # the apply's group would let a pull request cancel an apply waiting for
+    # approval. On main (and manual runs) it queues with the apply in
+    # platform-main, so it never runs while an apply is changing the boxes.
+    concurrency:
+      group: ${{ github.event_name == 'pull_request' && format('platform-check-{0}', github.ref) || 'platform-main' }}
+      cancel-in-progress: ${{ github.event_name == 'pull_request' }}
+    env:
+      # ansible.cfg sets the yaml callback, which no installed collection
+      # provides here (see the manifests job).
+      ANSIBLE_STDOUT_CALLBACK: default
+      ANSIBLE_FORCE_COLOR: "true"
+    steps:
+      - uses: actions/checkout@v4
+        with:
+          persist-credentials: false
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+      - name: Install Ansible and the collections the roles use
+        run: |
+          pip install ansible-core
+          ansible-galaxy collection install community.general ansible.posix kubernetes.core
+      - name: Dry run platform.yml
+        env:
+          ANSIBLE_SSH_PRIVATE_KEY: ${{ secrets.ANSIBLE_SSH_PRIVATE_KEY }}
+          ARGOCD_AGE_KEY: ${{ secrets.ARGOCD_AGE_KEY }}
+          CLOUDFLARE_DNS_TOKEN: ${{ secrets.CLOUDFLARE_DNS_TOKEN }}
+          ARGOCD_REPO_SSH_KEY: ${{ secrets.ARGOCD_REPO_SSH_KEY }}
+          LIMIT: ${{ inputs.limit }}
+        run: scripts/ci-platform.sh check "$LIMIT"
+
+  # Needs every check and the dry run of this same commit: a commit that fails
+  # either is never offered for approval. No status function in `if`, so
+  # GitHub adds success().
+  platform-apply:
+    needs: [ansible, manifests, generated, shell, changes, platform-dry-run]
+    if: >-
+      needs.changes.outputs.platform == 'true' && (
+        github.event_name == 'push' ||
+        (github.event_name == 'workflow_dispatch' && inputs.platform == 'apply'))
+    runs-on: ubuntu-latest
+    # One at a time with main's dry runs, never cancelled half-way. A newer
+    # merge waiting behind replaces an older one still waiting for approval:
+    # the latest main wins, and its own dry run is the one the approver reads.
+    concurrency:
+      group: platform-main
+      cancel-in-progress: false
+    # Required reviewers: nothing reaches the boxes until one approves.
+    environment: production
+    env:
+      ANSIBLE_STDOUT_CALLBACK: default
+      ANSIBLE_FORCE_COLOR: "true"
+    steps:
+      - uses: actions/checkout@v4
+        with:
+          persist-credentials: false
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+      - name: Install Ansible and the collections the roles use
+        run: |
+          pip install ansible-core
+          ansible-galaxy collection install community.general ansible.posix kubernetes.core
+      - name: Run platform.yml
+        env:
+          ANSIBLE_SSH_PRIVATE_KEY: ${{ secrets.ANSIBLE_SSH_PRIVATE_KEY }}
+          ARGOCD_AGE_KEY: ${{ secrets.ARGOCD_AGE_KEY }}
+          CLOUDFLARE_DNS_TOKEN: ${{ secrets.CLOUDFLARE_DNS_TOKEN }}
+          ARGOCD_REPO_SSH_KEY: ${{ secrets.ARGOCD_REPO_SSH_KEY }}
+          LIMIT: ${{ inputs.limit }}
+        run: scripts/ci-platform.sh apply "$LIMIT"

README.md

@@ -1445,14 +1445,23 @@ for itself:
   ingest) from Cloudflare's edge and the other boxes, and 8000 on sing-1
   (`healthchecks_host`, the hub's dead-man's switch) from lith-1.
 
-**It runs from CI** (`.github/workflows/platform.yml`), the only playbook that
-does; the others still run from an operator's machine:
+**It runs from CI**, the only playbook that does; the others still run from an
+operator's machine. It is part of the repo's one workflow
+(`.github/workflows/ci.yml`), so a merge starts a single run:
 
 | When | What |
 |---|---|
-| Pull request touching it | `--check --diff` against the boxes (forks get no secrets and are skipped) |
-| Merge to `main` | the repo checks (`ci.yml`) for that commit, then, only if they all pass, the real run, once a required reviewer approves it in the `production` environment |
-| Actions → platform → Run workflow | `check` or `apply` (apply also waits for the checks), optionally `--limit`ed |
+| Pull request touching it | the checks, then `--check --diff` against the boxes (forks get no secrets and are skipped) |
+| Merge to `main` touching it | the checks, then a dry run of the merged commit, then the real run, once a required reviewer has read the dry run and approved it in the `production` environment |
+| Actions → ci → Run workflow | the checks, plus `platform`: `check` (dry run) or `apply` (dry run, then the approved run), optionally `--limit`ed |
+
+Each dry run writes what it would change, per box and task, on the run's
+summary page; approve only if that is what you expect.
+
+"Touching it" means `platform.yml`, its roles (`argocd`, `monitoring`,
+`ufw_allowlist`), `group_vars/`, `inventory/`, `scripts/ci-platform.sh` or the
+workflow itself. A `group_vars` change for another service therefore also
+offers an apply; with nothing of its own changed, it changes nothing.
 
 It restarts nothing that serves traffic, so it runs every box in one batch: the
 hub registers the others using what they produced earlier in the same run.
@@ -1527,9 +1536,9 @@ same path a deploy takes), so what is checked is what would be applied:
 | every `pinset.yml` section is in the role's `known_sections` | a section the content role silently ignores |
 | `shellcheck scripts/*.sh` | the usual shell traps |
 
-`platform.yml` also has its own workflow, which does talk to the boxes (see
-"Argo CD and monitoring"). It calls this one (`ci.yml` is also a reusable
-workflow) and only applies once every job here has passed.
+The same workflow also runs `platform.yml`, the one part that does talk to
+the boxes, and only after every check above has passed (see "Argo CD and
+monitoring").
 
 The Cloudflare range check is a separate weekly workflow rather than part of
 this one, because it reaches `api.cloudflare.com`: an outage there should not
@@ -1692,13 +1701,12 @@ dns-libp2p.txt            bootstrap.libp2p.io records for the legacy identities
 site.yml                  base preparation playbook
 k3s-upgrade.yml           k3s upgrade, one minor version at a time, backup per step
 platform.yml              Argo CD (first install) and the monitoring stack's prerequisites, run from CI
-.github/workflows/platform.yml  platform.yml: dry run on PRs, approved run on merge
 scripts/ci-platform.sh    CI's wrapper for platform.yml: secrets to variables, SSH key, pinned host keys
 inventory/known_hosts     the boxes' SSH host keys, pinned for CI
 keys/*.pub                the SSH keys allowed in as the admin user: takgn, nymd, byo, break-glass, CI (admin_pubkey_file)
 routing.yml               routing service playbook (needle + origin)
 bootstrap.yml             bootstrap node playbook (with cert-manager)
 content.yml               content cluster node playbook (deploy, load the pinset, wait for PINNED, summary)
-.github/workflows/ci.yml  drift checks: playbook syntax, lint, kustomize build, generated files, shellcheck
+.github/workflows/ci.yml  drift checks (playbook syntax, lint, kustomize build, generated files, shellcheck), then platform.yml: dry run on PRs, approved run on merge
 .github/workflows/cloudflare-ranges.yml  weekly pinned-vs-published Cloudflare range check
 ```

group_vars/ipfs_nodes/main.yml

@@ -21,7 +21,7 @@ admin_pubkey_file:
   # only in 1Password (IPFS account, Compass vault); see AGENTS.md. Never
   # remove it without putting another emergency key in its place.
   - "{{ playbook_dir }}/keys/break-glass.pub"
-  # CI (.github/workflows/platform.yml). Its private key is the
+  # CI (.github/workflows/ci.yml, platform.yml). Its private key is the
   # ANSIBLE_SSH_PRIVATE_KEY repository secret.
   - "{{ playbook_dir }}/keys/github-actions.pub"