From c2a953147050115aea64a64e5418166da38463c7 Mon Sep 17 00:00:00 2001 From: "shipmill-romamo[bot]" <338557544+shipmill-romamo[bot]@users.noreply.github.com> Date: Tue, 20 Oct 2026 12:38:39 +0300 Subject: [PATCH] Document deploying from CI with GitHub Actions Every guide ran Cloudfall from an operator's machine. The new CI deploy guide gives one workflow that renders secrets, builds the artifact with cloudfall-engine, deploys with --yes and uploads the receipts; it covers per-environment SSH and age keys, passing the release id between steps, and why required reviewers stand in for approval: manual, which deploy does not enforce. It says plainly that it has not had a live run yet. Fixes #89 Co-Authored-By: Claude Opus 5.5 --- README.md | 1 + changelog.d/89-ci-deploy-guide.md | 3 + docs/ci-deploy-guide.md | 385 ++++++++++++++++++++++++++++++ docs/getting-started.md | 3 + 4 files changed, 392 insertions(+) create mode 100644 changelog.d/89-ci-deploy-guide.md create mode 100644 docs/ci-deploy-guide.md diff --git a/README.md b/README.md index 8031e66..f8c8a13 100644 --- a/README.md +++ b/README.md @@ -238,6 +238,7 @@ per-table row counts and writes a receipt. See the | [Operator](docs/operator-guide.md) | The always-on operator, proposals, approvals, autonomy policy | | [Disk-full incident](examples/disk-full-incident) | `agent investigate` with Nemotron on a real broken host, live or replayed | | [Secrets](docs/secrets-guide.md) | sops/age secrets rendered into per-component env files | +| [CI deploy](docs/ci-deploy-guide.md) | Deploying from GitHub Actions with per-environment keys and required reviewers | | [Logging](docs/logging-service-guide.md) | Loki, Grafana and Alloy behind an mTLS gateway | | [Storage](docs/new-server-storage-guide.md) | Destructive, new-server-only RAID provisioning ([design](docs/hybrid-storage-design.md)) | | [Architecture](ARCHITECTURE.md) | Module boundaries and the long-term fleet vision | diff --git a/changelog.d/89-ci-deploy-guide.md b/changelog.d/89-ci-deploy-guide.md new file mode 100644 index 0000000..ccfd648 --- /dev/null +++ b/changelog.d/89-ci-deploy-guide.md @@ -0,0 +1,3 @@ +### Added + +- A CI deploy guide (`docs/ci-deploy-guide.md`): one GitHub Actions workflow that renders secrets, builds the artifact with `cloudfall-engine artifact build`, deploys with `cloudfall deploy --yes` and uploads the receipts, with one SSH key and one age key per environment and required reviewers standing in for `approval: manual`, which `deploy` does not enforce; not yet run against a real server (#89) diff --git a/docs/ci-deploy-guide.md b/docs/ci-deploy-guide.md new file mode 100644 index 0000000..329c141 --- /dev/null +++ b/docs/ci-deploy-guide.md @@ -0,0 +1,385 @@ +# CI deploy guide + +Every other guide runs Cloudfall from an operator's machine: your age key in +`~/.config/cloudfall/age.key`, your SSH agent, your shell. This guide moves +the same deploy into a GitHub Actions workflow, so a merge to `main` updates +staging and a production deploy waits for a person's approval. + +The record does not change. The runner renders the secrets, builds the +artifact and deploys it with the same commands you run by hand, and the +receipts they write are uploaded with the run so they outlive the runner. + +> **Status:** not proven live. The workflow below is written from the code +> and checked against the flags of Cloudfall 0.7.0 (`--help` and the +> source), but it has not had a run against a real server yet. Treat the +> first run on staging as the proving run, and open an issue for anything +> that differs. + +## What the runner needs + +The runner is a controller like your laptop. It needs: + +- The project checked out: the repository `cloudfall init` created, with its + `pyproject.toml` and `uv.lock` committed +- `uv`, which installs Python 3.14, Cloudfall and `ansible-core` from the + project's lock file +- `ansible-playbook` 2.21.0 or later: it comes with Cloudfall + (`ansible-core>=2.21,<2.22`), so `uv run` finds it in the project's + virtualenv +- `git` 2.24.0 or later, which `artifact build` uses to clone the + component's repository; GitHub's Ubuntu runners ship a newer one +- `sops` 3.8.0 or later, which `secrets render` runs to decrypt fragments; + `age` itself is not needed, because sops decrypts age files on its own +- An SSH key the servers accept, and their host keys in `known_hosts` +- The age key that decrypts this environment's secrets + +These are the versions the commands check before they run (their +`required_tools`); a missing or older tool stops the command with an error +naming it. `uv run cloudfall doctor` checks all of them at once, so the +workflow runs it first. + +## One key per environment + +A runner should hold the keys of one environment and nothing else. Cloudfall +already scopes two of them. + +**SSH keys.** Each `SshPublicKey` names an environment, and the baseline +installs a server's active keys only when their environment matches the +server's. Make a key for the staging runner and declare it for staging: + +```sh +ssh-keygen -t ed25519 -N "" -C "ci staging" -f ci-staging +uv run cloudfall add ssh-key ci-staging.pub --owner ci --id ci-staging --environment staging +uv run cloudfall-engine inventory render --output tmp/ansible-inventory.json +uv run cloudfall-engine playbook run baseline --inventory tmp/ansible-inventory.json +``` + +The baseline writes the key into root's `authorized_keys` on staging servers +only, so the staging runner cannot log in to production. A server whose +`ssh.user` is not `root` needs the key installed for that user by hand. + +**Age keys.** This one needs a change to your project. `cloudfall init` +writes a `.sops.yaml` with one rule that encrypts every fragment under +`secrets/` to one key. A runner holding that key can decrypt everything. +Split the rule by environment directory, and list your own key in each rule +so you can still edit every file: + +```sh +age-keygen -o staging.key +age-keygen -o production.key +``` + +```yaml +creation_rules: + - path_regex: secrets/staging/.*\.env$ + age: age1staging...,age1yourown... + - path_regex: secrets/production/.*\.env$ + age: age1production...,age1yourown... +``` + +sops uses the first rule that matches a path. Files encrypted before the +change keep their old recipients until you re-encrypt them: + +```sh +sops updatekeys secrets/staging/crm.env +``` + +Why this stops a staging runner: a secret reference +`{environment: production, path: /crm}` resolves from +`secrets/production/crm.env`, and `secrets render` decrypts only the files +the component's references name. If a component on the staging runner +references a production fragment, sops has no matching key, and the render +fails with `secrets_decrypt_failed` before anything is written. Cloudfall +does not itself check that a component's references name the environment of +its servers; the key the runner holds is what enforces it. + +Store `staging.key` and `production.key` in a password manager, then delete +the local copies once they are in GitHub. + +## GitHub environments and secrets + +Create one GitHub environment per Cloudfall environment, with the same name +(`staging`, `production`), and put three secrets in each: + +| Secret | Content | +| --- | --- | +| `CLOUDFALL_SSH_KEY` | The private key of that environment's runner key, such as `ci-staging` | +| `CLOUDFALL_KNOWN_HOSTS` | The `known_hosts` lines of that environment's servers | +| `SOPS_AGE_KEY` | The age key file of that environment, such as `staging.key` | + +```sh +gh secret set CLOUDFALL_SSH_KEY --env staging < ci-staging +gh secret set SOPS_AGE_KEY --env staging < staging.key +ssh-keygen -F 203.0.113.20 > staging-known-hosts +gh secret set CLOUDFALL_KNOWN_HOSTS --env staging < staging-known-hosts +``` + +`ssh-keygen -F` reads the host key your own machine already trusts, which is +safer than scanning it from the runner. For a server on another port, look it +up as `[203.0.113.20]:2222`. + +In the repository's settings, under Environments: + +- `production`: add required reviewers and turn on "Prevent self-review" +- Both: limit deployment branches to `main`, so a job on another branch never + sees the secrets + +## The workflow + +Save this as `.github/workflows/deploy.yml` in the project repository. +Replace `crm-backend-staging` with your staging component. + +```yaml +name: deploy + +on: + push: + branches: [main] + workflow_dispatch: + inputs: + environment: + description: Cloudfall environment, and the GitHub environment of the same name + type: choice + options: [staging, production] + default: staging + component: + description: Component to deploy + required: true + default: crm-backend-staging + ref: + description: Git ref of the component's repository to build + required: true + default: main + repository_dispatch: + types: [deploy] + +concurrency: + group: deploy-${{ inputs.environment || github.event.client_payload.environment || 'staging' }} + cancel-in-progress: false + +permissions: + contents: read + +jobs: + deploy: + runs-on: ubuntu-24.04 + environment: ${{ inputs.environment || github.event.client_payload.environment || 'staging' }} + env: + COMPONENT: ${{ inputs.component || github.event.client_payload.component || 'crm-backend-staging' }} + REF: ${{ inputs.ref || github.event.client_payload.ref || 'main' }} + SOPS_VERSION: v3.13.3 + steps: + - uses: actions/checkout@v7 + + - uses: astral-sh/setup-uv@v10.1.0 + + - name: Install Cloudfall from the lock file + run: uv sync --locked + + - name: Install sops + run: | + base="https://github.com/getsops/sops/releases/download/$SOPS_VERSION" + curl -fsSLO "$base/sops-$SOPS_VERSION.linux.amd64" + curl -fsSLO "$base/sops-$SOPS_VERSION.checksums.txt" + sha256sum --check --ignore-missing "sops-$SOPS_VERSION.checksums.txt" + sudo install -m 0755 "sops-$SOPS_VERSION.linux.amd64" /usr/local/bin/sops + + - name: Load the SSH key + env: + CLOUDFALL_SSH_KEY: ${{ secrets.CLOUDFALL_SSH_KEY }} + CLOUDFALL_KNOWN_HOSTS: ${{ secrets.CLOUDFALL_KNOWN_HOSTS }} + run: | + install -m 0700 -d ~/.ssh + printf '%s\n' "$CLOUDFALL_KNOWN_HOSTS" > ~/.ssh/known_hosts + eval "$(ssh-agent -s)" + printf '%s\n' "$CLOUDFALL_SSH_KEY" | ssh-add - + echo "SSH_AUTH_SOCK=$SSH_AUTH_SOCK" >> "$GITHUB_ENV" + + - name: Write the age key + env: + SOPS_AGE_KEY: ${{ secrets.SOPS_AGE_KEY }} + run: | + install -m 0600 /dev/null "$RUNNER_TEMP/age.key" + printf '%s\n' "$SOPS_AGE_KEY" > "$RUNNER_TEMP/age.key" + echo "SOPS_AGE_KEY_FILE=$RUNNER_TEMP/age.key" >> "$GITHUB_ENV" + + - name: Check the toolchain + run: uv run cloudfall doctor + + - name: Validate the config + run: uv run cloudfall config validate + + - name: Render secrets + run: uv run cloudfall secrets render "$COMPONENT" + + - name: Build the artifact + id: build + run: | + uv run cloudfall-engine artifact build "$COMPONENT" --ref "$REF" --format json > "$RUNNER_TEMP/build.json" + echo "release=$(jq -r '.data.release' "$RUNNER_TEMP/build.json")" >> "$GITHUB_OUTPUT" + + - name: Show the plan + env: + RELEASE: ${{ steps.build.outputs.release }} + run: uv run cloudfall deploy "$COMPONENT" --release "$RELEASE" + + - name: Deploy + env: + RELEASE: ${{ steps.build.outputs.release }} + run: uv run cloudfall deploy "$COMPONENT" --release "$RELEASE" --env-file "tmp/env/$COMPONENT.env" --yes + + - name: Remove the decrypted files + if: always() + run: rm -f "tmp/env/$COMPONENT.env" "$RUNNER_TEMP/age.key" + + - name: Upload the receipts + if: always() + uses: actions/upload-artifact@v7 + with: + name: cloudfall-${{ env.COMPONENT }}-${{ github.run_id }} + path: | + tmp/artifacts/**/*.json + tmp/releases/ + tmp/env-receipts/ + retention-days: 90 +``` + +What each step does: + +- **Install Cloudfall from the lock file.** `uv sync --locked` installs the + Cloudfall release the project pins, and fails if `uv.lock` is out of date + rather than resolving a different one +- **Install sops.** The binary is checked against the release's checksum + file before it is installed +- **Load the SSH key.** The key goes into an `ssh-agent`, never onto disk; + Ansible reaches the agent through `SSH_AUTH_SOCK` +- **Write the age key.** The file is created `0600` before the key is + written into it, and `SOPS_AGE_KEY_FILE` points sops at it, as on your own + machine +- **Render secrets.** Writes `tmp/env/.env` (`0600`) and the + receipt `tmp/env-receipts/.json`; the log shows key names and a + hash, never a value +- **Build the artifact.** Clones the component's repository at `REF` and + packages it under `tmp/artifacts//` +- **Show the plan.** `deploy` without `--yes` validates the release and + prints which servers it would change, and changes nothing +- **Deploy.** Ships the release behind the component's health check, and + writes the release receipt `tmp/releases/.json` +- **Remove the decrypted files.** GitHub-hosted runners are thrown away + after the job; on a self-hosted runner this step is what keeps the values + off its disk + +If the project is not at the root of the repository, set `CLOUDFALL_PROJECT` +to its directory in the job's `env`. + +## Passing the release id + +`cloudfall deploy` takes the release to ship as `--release`, and only +`artifact build` produces one. Two things trip people up: + +- `artifact build` is a `cloudfall-engine` command, not a `cloudfall` one. + Through an agent it is the `build_artifact` tool of `cloudfall mcp serve` +- The release id is in the build's JSON answer at `data.release`, such as + `20260101T000000Z-abcdef0` (the build time and the short commit). The + workflow reads it with `jq` and passes it on as a step output + +Off a terminal both CLIs answer in JSON by default; the workflow passes +`--format json` so the build step does not depend on that. + +The artifact is a file on the runner. `deploy` verifies it from +`tmp/artifacts` before shipping it, so build and deploy must run in the same +job. Splitting them across jobs means uploading `tmp/artifacts/` from one and +downloading it into the other. + +`REF` is a ref of the component's repository (its `repository.url`), not of +the project repository. A push to the project's `main` therefore rebuilds the +component's `main`. To ship to production exactly the commit staging ran, +dispatch the production deploy with that commit's full hash as `ref`; the +staging receipt records it as `gitCommit`. The build packages the same +commit again, which gives a new release id and archive. + +## Triggering it from an application repository + +An application repository can start a deploy when its own `main` changes, +with a `repository_dispatch` event on the project repository: + +```sh +gh api repos/acme/fleet/dispatches \ + -f event_type=deploy \ + -f 'client_payload[component]=crm-backend-staging' \ + -f 'client_payload[environment]=staging' \ + -f "client_payload[ref]=$GITHUB_SHA" +``` + +Run it as a step in the application's workflow, authenticated with a token +that may write the project repository's contents (a fine-grained token or a +GitHub App token, stored as a secret of the application repository). The +event runs the workflow from the project's default branch, and a dispatch +naming `production` still waits for its reviewers. + +## Approval: who presses the button + +Every `Application` declares `spec.approval: manual` or `automatic`. +Cloudfall validates the field and writes it into the rendered inventory, but +no command acts on it today: `cloudfall deploy --yes` deploys an application +marked `manual` exactly as it deploys one marked `automatic`. The only gate +inside `deploy` is `--yes`, and a workflow that passes `--yes` has approved +its own deploy. + +So in CI, the approval comes from GitHub, not from Cloudfall: + +- Give every environment that holds an application marked `manual` required + reviewers, and turn on "Prevent self-review", so the person who pushed the + change cannot approve its deploy +- The job waits before its first step until a reviewer approves, so it never + sees the SSH key or the age key of that environment without approval +- Leave `staging` without reviewers if its applications are `automatic` + +Read `approval: manual` as your declaration that such a gate must exist; the +environment's reviewers are what enforce it. The approval itself lands in +GitHub's deployment history, not in Cloudfall's record. + +## Keeping the receipts + +`tmp/` is ignored by git, and the runner disappears with the job. The upload +step keeps, for 90 days: + +- `tmp/artifacts//.json`: the artifact metadata, with the + commit and the archive's SHA-256 +- `tmp/releases/.json`: the release receipt from `deploy` +- `tmp/env-receipts/.json`: the key names and hash of the + rendered environment file, never the values + +It never uploads `tmp/env/`, which holds the decrypted values. Add +`tmp/artifacts/` whole if you want the archives kept as well. + +If you want the receipts in git, copy them to a tracked directory and commit +them back from the job. Give the job `contents: write` and add a step after +the deploy: + +```yaml + - name: Commit the receipt + env: + RELEASE: ${{ steps.build.outputs.release }} + run: | + mkdir -p receipts/releases + cp "tmp/releases/$COMPONENT.json" "receipts/releases/$COMPONENT-$RELEASE.json" + git config user.name "github-actions[bot]" + git config user.email "41898282+github-actions[bot]@users.noreply.github.com" + git add receipts/releases + git commit -m "Record $COMPONENT $RELEASE" + git push +``` + +Only the kind directories are read as resources, so `receipts/` is safe in +the project. A push made with the workflow's own token does not start +another workflow run, so this does not loop. A branch protection rule on +`main` will refuse the push unless it allows the workflow to bypass it. + +## What this guide does not cover + +- Moving `artifact build` into the `cloudfall` CLI, so CI installs and pins + one tool +- A published reusable action +- Converging the baseline, services or domains from CI; the same pattern + applies, with the matching `cloudfall-engine playbook run` command diff --git a/docs/getting-started.md b/docs/getting-started.md index 32220f9..f0f5f5b 100644 --- a/docs/getting-started.md +++ b/docs/getting-started.md @@ -71,6 +71,9 @@ shows them); project playbooks by path, with `--roles` directories searched before the bundled roles. To move a project to a newer Cloudfall commit, bump `rev` and run `uv sync`. +To deploy from a pipeline instead of your own machine, so a merge to `main` +updates staging, see the [CI deploy guide](ci-deploy-guide.md). + ## The agent contract Humans and AI agents both edit the resources and both run the CLI, with a