From 6e0b90c96864eadb3a802cd01aa9bcd251ccc6c6 Mon Sep 17 00:00:00 2001 From: Joseph Mearman Date: Mon, 2 Nov 2026 23:42:12 +0000 Subject: [PATCH 1/3] feat(cluster): start the node's Colima profile from a launchd daemon at boot The login agent only runs once someone logs in, so a Mac that reboots or loses power sits at the login window with the node's Colima profile stopped, and its k3s container never comes back. The profile named by github_runner_cluster_runtime_boot_profile (node) is now started by a launchd daemon, a property list in /Library/LaunchDaemons owned by root:wheel with mode 0644, loaded into the system domain and running as the connecting user (UserName) with HOME, PATH and COLIMA_HOME in its environment. Writing and loading it use become. Any other profile keeps its login agent, rendered and installed exactly as before. github_runner_cluster_runtime_autostart_at (auto, login or boot) overrides the choice per host. A profile has one kind of job at a time. Installing the daemon unloads and deletes the profile's agent from an earlier run, whether or not the user has a login session, and installing the agent does the same to a daemon, so the profile is not started twice. Neither unload stops the VM. --- README.md | 2 +- roles/github_runner_cluster/README.md | 20 +- roles/github_runner_cluster/defaults/main.yml | 13 +- .../meta/argument_specs.yml | 7 +- roles/github_runner_cluster/tasks/runtime.yml | 73 +---- .../tasks/runtime_agent.yml | 104 +++++++ .../tasks/runtime_daemon.yml | 91 ++++++ .../templates/colima-launchd.plist.j2 | 12 + tests/runtime_autostart/render_plist.yml | 3 +- tests/runtime_autostart/run.yml | 3 +- tests/unit/test_runtime_autostart.py | 269 +++++++++++++++++- 11 files changed, 508 insertions(+), 89 deletions(-) create mode 100644 roles/github_runner_cluster/tasks/runtime_agent.yml create mode 100644 roles/github_runner_cluster/tasks/runtime_daemon.yml diff --git a/README.md b/README.md index e7faa3b2..c9a8b904 100644 --- a/README.md +++ b/README.md @@ -8,7 +8,7 @@ There are two ways to deploy. A fleet's hosts are managed from a control machine with Ansible, from the fleet's own inventory (see [Ansible](#ansible) below). `bootstrap.sh`, described here, brings up a single machine from a checkout of this repository without a control node or inventory. It runs the same Ansible role against the machine itself, so both paths deploy identically. -Prerequisites on the host: Docker (or Colima) with the Compose plugin, any release but 2.37.1 to 2.38.x (the cluster role refuses those; see Docker Compose versions in `roles/github_runner_cluster/README.md`), `python3`, `jq`, `openssl`, and `gh`. On macOS, the playbook installs `helm`, `kubectl` and Docker through Homebrew itself, and the cluster role starts a Colima host's profile at every login and fails unless the runtime is up (Docker Desktop needs its own start-at-login setting; see Container runtime at login in `roles/github_runner_cluster/README.md`). Elsewhere, install `helm` and `kubectl` from the host's package manager first. +Prerequisites on the host: Docker (or Colima) with the Compose plugin, any release but 2.37.1 to 2.38.x (the cluster role refuses those; see Docker Compose versions in `roles/github_runner_cluster/README.md`), `python3`, `jq`, `openssl`, and `gh`. On macOS, the playbook installs `helm`, `kubectl` and Docker through Homebrew itself, and the cluster role starts a Colima host's profile again after a reboot (the node's profile at boot, with nobody logged in, any other at login) and fails unless the runtime is up (Docker Desktop needs its own start-at-login setting; see Container runtime after a reboot in `roles/github_runner_cluster/README.md`). Elsewhere, install `helm` and `kubectl` from the host's package manager first. 1. **Create a GitHub App** on the organisation with **Self-hosted runners: read & write** permission, and install it on the organisation. `playbooks/github_app_setup.yml` does this through GitHub's manifest flow (see `roles/github_runner_arc/README.md`). For one repository's runners instead, which is how a repository of a personal account uses the fleet, the App needs repository **Administration: read & write** and **Metadata: read**, installed where it reaches that repository, and `RUNNER_REPOSITORY` names it (see Repository scope in `roles/github_runner_arc/README.md`). 2. **Configure the environment.** Copy `.env.bootstrap.example` to `.env.bootstrap` and fill in the following (the file's own comments cover the rest, such as joining an existing cluster or running as an agent): diff --git a/roles/github_runner_cluster/README.md b/roles/github_runner_cluster/README.md index 35102e4e..85662f2b 100644 --- a/roles/github_runner_cluster/README.md +++ b/roles/github_runner_cluster/README.md @@ -136,21 +136,26 @@ By default the role drives whichever Docker daemon the host's current Docker con Set `github_runner_cluster_docker_context` in the host's variables to pin the context by name (`colima` in that example) on any host with more than one engine. Every `community.docker` module call in the role then passes it as `cli_context`, and every `docker` command the role runs (Compose, `exec`, `inspect`, `volume`, image pulls, and the commands in `playbooks/recover_in_cluster_mesh.yml` and `playbooks/promote_headscale_standby.yml`) runs with `DOCKER_CONTEXT` set to it, so the modules and the CLI always reach the same daemon. The role checks the pin before it touches the host and fails if the context does not exist there, or if `DOCKER_HOST` is also set, since the two would name the daemon in conflicting ways. It never creates a context and never changes the host's current context; `docker context show` on the host reads the same before and after a run. The Headscale tasks that act on another host (the `headscale_hosted` server and standby, and the promotion playbook) use that host's own value. Run `docker context show` on each host to see what it currently points at before choosing a value. -## Container runtime at login (macOS) +## Container runtime after a reboot (macOS) On a Mac the container runtime is a Linux VM (Colima, or Docker Desktop) that stays stopped after a reboot until something starts it. The node's k3s container has `restart: unless-stopped`, so it comes back by itself once the runtime does, but a host whose runtime never starts never rejoins the cluster. With three servers, two such hosts leave the cluster without quorum and no runner pods anywhere. Only macOS hosts are affected by any of this: the role does nothing about the runtime on a Linux host, where Docker is a system service. Which runtime a Mac runs is taken from the Docker context the role drives (see [Docker contexts](#docker-contexts)): `colima`, or `colima-` for another profile, is Colima, and `desktop-linux` is Docker Desktop. `github_runner_cluster_runtime` says so explicitly for a host whose context does not tell, for example one that uses `DOCKER_HOST`, and `github_runner_cluster_colima_profile` names the profile to start when the context does not. -**Colima.** With `github_runner_cluster_runtime_autostart` on (the default, which acts on Colima hosts only), the role: +**Colima.** With `github_runner_cluster_runtime_autostart` on (the default, which acts on Colima hosts only), the role starts the profile if it is stopped, with `colima start --profile --activate=false`, so the profile does not make its context the host's current one, and installs the launchd job that starts it again after a reboot. There are two kinds, chosen by `github_runner_cluster_runtime_autostart_at` (`login`, `boot` or `auto`, the default). `auto` is `boot` for the profile named by `github_runner_cluster_runtime_boot_profile` (default `node`, the profile the bridged k3s node runs in) and `login` for any other, such as a `default` profile a host keeps for other workloads. A profile has one kind at a time: installing either retires the other's property list, unloading it first, so the profile is not started twice. + +**At login (an agent).** The role: -- starts the profile if it is stopped, with `colima start --profile --activate=false`, so the profile does not make its context the host's current one; - installs a launchd agent, `com.exadev.github-runner.colima.`, as `~/Library/LaunchAgents/