Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@ ansible-playbook -i inventory.yml exadev.github_runner.site --limit node1 --chec
ansible-playbook -i inventory.yml exadev.github_runner.site --limit node1 # real run
```

The playbook runs up to seven roles, each with its own README: `github_runner_host_tools` installs Homebrew and mise on every macOS and Debian-family Linux host first, keeps both at their latest release and puts them in the connecting user's shell (it uses `become` and Homebrew's installer, so run the play with passwordless sudo or `--ask-become-pass`, and `bootstrap.sh` turns it off), `github_runner_macos_host` sets up a Mac's host baseline next (AC power settings and `disablesleep` that stop it sleeping, lid closed included, and its own Tailscale client as the `tailscaled` system daemon; it uses `become`, as do the Linux host role and a native node, so run the play with passwordless sudo or `--ask-become-pass`, and `bootstrap.sh` turns it off), `github_runner_linux_host` sets up a Linux host's baseline (keeping it from suspending and its Wi-Fi from power saving; both off unless a host's variables turn them on), `github_runner_cluster` brings the host up as a k3s node, in a container under Docker Compose or, with `github_runner_cluster_node_mode: native`, directly under systemd on a Linux host, `github_runner_arc` installs ARC and the fleet-health platform, `github_runner_k8s_client` (included by the other two) gives a host the kubernetes Python client, and `github_runner_secrets_onepassword` reads the secrets from 1Password when `github_runner_secrets_source` is `onepassword`. The cluster and ARC roles read every secret (`K3S_TOKEN`, the mesh keys, GHCR credentials, the heartbeat PAT, each org's App private key) as a plain variable, so any source Ansible reads works: ansible-vault, SOPS, an env or file lookup, HashiCorp Vault or AWS SSM (see `examples/ansible-vault/`, `examples/env/` and `examples/onepassword/`). With `github_runner_secrets_source: onepassword`, every secret is read live from one 1Password item at task time; `roles/github_runner_secrets_onepassword/README.md` lists the item's fields and the `op://` references. `bootstrap.sh` sets the same variables directly from `.env.bootstrap`. The cluster role ships the node's `docker-compose.yml` and k3s image build context in `roles/github_runner_cluster/files/`, copies them on every run into each host's `github_runner_cluster_dir` (`~/.github-runner` by default), and templates the `.env` for their `k3s`/`k3s-agent` services there, so a host needs no checkout of this repository. The Compose project name is pinned to `github-runner` (`github_runner_cluster_compose_project`) rather than taken from the directory's name, because the named volumes holding the k3s datastore and each node's mesh identity are prefixed with it; the directory can move without losing them.
The playbook runs up to seven roles, each with its own README: `github_runner_host_tools` installs Homebrew and mise on every macOS and Debian-family Linux host first, keeps both at their latest release and puts them in the connecting user's shell, and writes npm and pnpm package-install security settings and shell environment variables from the variables a host gives (it uses `become` and Homebrew's installer, so run the play with passwordless sudo or `--ask-become-pass`, and `bootstrap.sh` turns it off), `github_runner_macos_host` sets up a Mac's host baseline next (AC power settings and `disablesleep` that stop it sleeping, lid closed included, and its own Tailscale client as the `tailscaled` system daemon; it uses `become`, as do the Linux host role and a native node, so run the play with passwordless sudo or `--ask-become-pass`, and `bootstrap.sh` turns it off), `github_runner_linux_host` sets up a Linux host's baseline (keeping it from suspending and its Wi-Fi from power saving; both off unless a host's variables turn them on), `github_runner_cluster` brings the host up as a k3s node, in a container under Docker Compose or, with `github_runner_cluster_node_mode: native`, directly under systemd on a Linux host, `github_runner_arc` installs ARC and the fleet-health platform, `github_runner_k8s_client` (included by the other two) gives a host the kubernetes Python client, and `github_runner_secrets_onepassword` reads the secrets from 1Password when `github_runner_secrets_source` is `onepassword`. The cluster and ARC roles read every secret (`K3S_TOKEN`, the mesh keys, GHCR credentials, the heartbeat PAT, each org's App private key) as a plain variable, so any source Ansible reads works: ansible-vault, SOPS, an env or file lookup, HashiCorp Vault or AWS SSM (see `examples/ansible-vault/`, `examples/env/` and `examples/onepassword/`). With `github_runner_secrets_source: onepassword`, every secret is read live from one 1Password item at task time; `roles/github_runner_secrets_onepassword/README.md` lists the item's fields and the `op://` references. `bootstrap.sh` sets the same variables directly from `.env.bootstrap`. The cluster role ships the node's `docker-compose.yml` and k3s image build context in `roles/github_runner_cluster/files/`, copies them on every run into each host's `github_runner_cluster_dir` (`~/.github-runner` by default), and templates the `.env` for their `k3s`/`k3s-agent` services there, so a host needs no checkout of this repository. The Compose project name is pinned to `github-runner` (`github_runner_cluster_compose_project`) rather than taken from the directory's name, because the named volumes holding the k3s datastore and each node's mesh identity are prefixed with it; the directory can move without losing them.

Which hosts are k3s servers is worked out from the inventory, not set per host. The `github_runner_cluster` inventory group lists the cluster's hosts in order: the first N become servers (control-plane/etcd members running `k3s server`, genuine voting members of the raft quorum), where N is the largest odd number no greater than both the number of hosts and `github_runner_cluster_max_servers` (5 by default), so 1 or 2 hosts give 1 server, 3 or 4 give 3, and 5 or more give 5. The first server bootstraps the cluster (`--cluster-init`, embedded etcd) and every other host joins it, the rest as agents (`k3s agent`, schedulable capacity with no control-plane role). Join URLs and every server's `--tls-san` list are derived from each server's node name and `github_runner_cluster_tailnet_domain`. A host can still set `github_runner_cluster_node_role`, `github_runner_cluster_bootstrap`, `github_runner_cluster_server_url` or `github_runner_cluster_tls_sans` to override its part, and the run fails with an explanation if the result has an even number of servers, none at all, or more than one bootstrap host; `roles/github_runner_cluster/defaults/main.yml` describes each. Installing the ARC controller, each org's runner scale set, and the fleet-health platform is not tied to a node role at all: each is driven purely by whether the installing host's own `github_runner_arc_orgs`/`github_runner_arc_heartbeat_gist_id` is set, so any server host can be the one Ansible installs against, and moving which host does the installing is a config change (which host's `host_vars` sets it), not a role change. See the [Architecture](#architecture) section for why 3 servers is the minimum sane count for genuine HA (2 is worse than 1 for quorum math, not better). To add a server or agent, add the host to the `github_runner_cluster` group in the fleet's inventory and give it its own `host_vars/<host>.yml`; keep the bootstrap host first.

Expand Down
Loading
Loading