feat(cluster): start the node Colima profile from a launchd daemon at boot

README.md

@@ -8,7 +8,7 @@
 
 There are two ways to deploy. A fleet's hosts are managed from a control machine with Ansible, from the fleet's own inventory (see [Ansible](#ansible) below). `bootstrap.sh`, described here, brings up a single machine from a checkout of this repository without a control node or inventory. It runs the same Ansible role against the machine itself, so both paths deploy identically.
 
-Prerequisites on the host: Docker (or Colima) with the Compose plugin, any release but 2.37.1 to 2.38.x (the cluster role refuses those; see Docker Compose versions in `roles/github_runner_cluster/README.md`), `python3`, `jq`, `openssl`, and `gh`. On macOS, the playbook installs `helm`, `kubectl` and Docker through Homebrew itself, and the cluster role starts a Colima host's profile at every login and fails unless the runtime is up (Docker Desktop needs its own start-at-login setting; see Container runtime at login in `roles/github_runner_cluster/README.md`). Elsewhere, install `helm` and `kubectl` from the host's package manager first.
+Prerequisites on the host: Docker (or Colima) with the Compose plugin, any release but 2.37.1 to 2.38.x (the cluster role refuses those; see Docker Compose versions in `roles/github_runner_cluster/README.md`), `python3`, `jq`, `openssl`, and `gh`. On macOS, the playbook installs `helm`, `kubectl` and Docker through Homebrew itself, and the cluster role starts a Colima host's profile again after a reboot (the node's profile at boot, with nobody logged in, any other at login) and fails unless the runtime is up (Docker Desktop needs its own start-at-login setting; see Container runtime after a reboot in `roles/github_runner_cluster/README.md`). Elsewhere, install `helm` and `kubectl` from the host's package manager first.
 
 1. **Create a GitHub App** on the organisation with **Self-hosted runners: read & write** permission, and install it on the organisation. `playbooks/github_app_setup.yml` does this through GitHub's manifest flow (see `roles/github_runner_arc/README.md`). For one repository's runners instead, which is how a repository of a personal account uses the fleet, the App needs repository **Administration: read & write** and **Metadata: read**, installed where it reaches that repository, and `RUNNER_REPOSITORY` names it (see Repository scope in `roles/github_runner_arc/README.md`).
 2. **Configure the environment.** Copy `.env.bootstrap.example` to `.env.bootstrap` and fill in the following (the file's own comments cover the rest, such as joining an existing cluster or running as an agent):

roles/github_runner_cluster/defaults/main.yml

@@ -174,8 +174,12 @@ github_runner_cluster_node_password_volume: "{{ github_runner_cluster_compose_pr
 # The most seconds the role waits, after it has recreated a server's container, for that server's node to report Ready again before it recreates the next server's. A recreated server restarts k3s, rejoins etcd, and registers its node and starts its kubelet, which can take minutes on a busy host; the wait ends as soon as the node is Ready on a heartbeat from the new container, so this only bounds a server that does not come back, which then fails the run with the other servers still untouched.
 github_runner_cluster_server_ready_timeout: 600
 
-# macOS hosts only (Linux hosts are never touched): whether the role makes the container runtime come back after a reboot. On a Colima host it installs a per-user launchd agent that starts the Colima profile at login, loads it, and starts the profile at once if it is stopped. true (the default) does that for Colima hosts and nothing for the others: the role cannot start Docker Desktop, whose start-at-login setting is documented in the README. false installs nothing and leaves an agent from an earlier run as it is. Either way the end checks fail unless the runtime is up and the node's container is running.
+# macOS hosts only (Linux hosts are never touched): whether the role makes the container runtime come back after a reboot. On a Colima host it installs a launchd job that starts the Colima profile, loads it, and starts the profile at once if it is stopped. Which kind of job is github_runner_cluster_runtime_autostart_at. true (the default) does that for Colima hosts and nothing for the others: the role cannot start Docker Desktop, whose start-at-login setting is documented in the README. false installs nothing and leaves an agent or daemon from an earlier run as it is. Either way the end checks fail unless the runtime is up and the node's container is running.
 github_runner_cluster_runtime_autostart: true
+# When the role starts a Colima profile: login installs a per-user launchd agent (a property list in the connecting user's ~/Library/LaunchAgents, loaded into the user's login session), which runs only once that user has logged in; boot installs a launchd daemon (/Library/LaunchDaemons, owned by root, loaded into the system domain, run as the connecting user through UserName), which runs when the Mac starts, with nobody logged in, so a node recovers from a power loss or a reboot at the login window; auto is boot for the profile named by github_runner_cluster_runtime_boot_profile and login for any other. Only one of the two exists for a profile at a time: installing either removes the other's property list, and unloads it, so the profile is not started twice. A daemon takes root (become) to write and to load, which the agent never does.
+github_runner_cluster_runtime_autostart_at: auto
+# The Colima profile that runs this collection's k3s node, which must come back with no one logged in: the one github_runner_cluster_runtime_autostart_at auto starts at boot. Any other profile on the host (the mini's default profile for its own workloads, say) keeps the login agent.
+github_runner_cluster_runtime_boot_profile: node
 # Which container runtime a macOS host runs: auto takes it from the Docker context the role drives (colima or colima-<profile> is Colima, desktop-linux is Docker Desktop, anything else is other), colima, docker_desktop and other say so for a host whose context does not tell (DOCKER_HOST set, a context with another name).
 github_runner_cluster_runtime: auto
 # The Colima profile to start. Empty takes it from the Docker context (colima is the default profile, colima-<profile> another) and falls back to the default profile when the context names none.
@@ -189,6 +193,13 @@ github_runner_cluster_runtime_path: "/opt/homebrew/bin:/opt/homebrew/sbin:/usr/l
 github_runner_cluster_runtime_launchd_label: "com.exadev.github-runner.colima.{{ github_runner_cluster_effective_runtime.colima_profile }}"
 # Derived, never set directly: where the agent's property list lives, in the connecting user's own LaunchAgents directory.
 github_runner_cluster_runtime_agent_plist: "{{ ansible_facts['env']['HOME'] }}/Library/LaunchAgents/{{ github_runner_cluster_runtime_launchd_label }}.plist"
+# Derived, never set directly: whether the profile is started by a launchd daemon at boot (true) or a launchd agent at login (false); see github_runner_cluster_runtime_autostart_at.
+github_runner_cluster_runtime_at_boot: "{{ github_runner_cluster_runtime_autostart_at == 'boot' or (github_runner_cluster_runtime_autostart_at == 'auto' and github_runner_cluster_effective_runtime.colima_profile == github_runner_cluster_runtime_boot_profile) }}"
+# Derived, never set directly: where the daemon's property list lives, which launchd requires to be owned by root and writable by no one else.
+github_runner_cluster_runtime_daemon_plist: "/Library/LaunchDaemons/{{ github_runner_cluster_runtime_launchd_label }}.plist"
+# Constants, not settings: the owner and group launchd requires of a daemon's property list (root, with the group of /Library/LaunchDaemons); variables only so the tests, which cannot be root, can narrow them.
+github_runner_cluster_runtime_daemon_owner: root
+github_runner_cluster_runtime_daemon_group: wheel
 # What the Colima VM's kernel needs for the node, loaded and set now and at every VM boot on every run (tasks/colima_kernel.yml). A pod that talks to a Service whose only backend is on the same node gets no reply unless bridged traffic passes through iptables, and that needs the br_netfilter module and the two sysctls below. Constants, not settings; variables only so the tests can narrow them.
 github_runner_cluster_colima_kernel_module: br_netfilter
 github_runner_cluster_colima_sysctl_settings:

roles/github_runner_cluster/tasks/runtime.yml

@@ -1,7 +1,7 @@
 ---
 # macOS hosts only (main.yml includes it nowhere else). Makes the container runtime come back after a reboot, and stops the run, with a message that says what to do, when the runtime is not up. The node's own container already restarts with the runtime (restart: unless-stopped in docker-compose.yml), so the runtime is the only thing a reboot leaves stopped.
 #
-# On a Colima host with github_runner_cluster_runtime_autostart on, it starts the profile if it is stopped, installs a per-user launchd agent that starts it at every login (a property list in the connecting user's ~/Library/LaunchAgents) and loads that agent into the user's login session. Docker Desktop cannot be started by the role, so there it only checks. The agent runs as the connecting user and the role never uses become for it, since a LaunchAgent belongs to the user whose login session loads it.
+# On a Colima host with github_runner_cluster_runtime_autostart on, it starts the profile if it is stopped and installs the launchd job that starts it again after a reboot, either a per-user agent that starts it at every login (runtime_agent.yml: a property list in the connecting user's ~/Library/LaunchAgents, loaded into the user's login session, no become since a LaunchAgent belongs to the user whose login session loads it) or, for the profile that runs the node (github_runner_cluster_runtime_at_boot), a daemon that starts it when the Mac starts with nobody logged in (runtime_daemon.yml: a property list in /Library/LaunchDaemons, loaded into the system domain with become, running as the connecting user), which retires the agent of an earlier run. Docker Desktop cannot be started by the role, so there it only checks.
 #
 # On a Colima host it also makes the VM load br_netfilter and set the bridge sysctls now and at every VM boot (tasks/colima_kernel.yml), whatever the autostart setting says.
 #
@@ -17,6 +17,13 @@
       - env
   when: "'user_uid' not in ansible_facts or 'env' not in ansible_facts"
 
+- name: Check when the Colima profile is to be started
+  # Before anything changes. Not in meta/argument_specs.yml, which declares the native node's variables only.
+  ansible.builtin.assert:
+    that: github_runner_cluster_runtime_autostart_at in ['auto', 'login', 'boot']
+    fail_msg: "github_runner_cluster_runtime_autostart_at must be auto, login or boot, not {{ github_runner_cluster_runtime_autostart_at | quote }}."
+    quiet: true
+
 - name: Work out which container runtime this host runs
   ansible.builtin.set_fact:
     github_runner_cluster_effective_runtime: >-
@@ -73,72 +80,13 @@
       changed_when: true
       when: not github_runner_cluster_runtime_up | bool
 
-    - name: Create the connecting user's LaunchAgents directory
-      ansible.builtin.file:
-        path: "{{ ansible_facts['env']['HOME'] }}/Library/LaunchAgents"
-        state: directory
-        mode: "0755"
-
-    - name: Install the launchd agent that starts the Colima profile at login
-      # The label is per profile, so two profiles on one host get two agents. plutil rejects a malformed property list before it replaces the installed one, since launchd would otherwise fail to load the agent at the next login with nobody watching. The task's environment gives plutil the same PATH as the agent.
-      ansible.builtin.template:
-        src: colima-launchd.plist.j2
-        dest: "{{ github_runner_cluster_runtime_agent_plist }}"
-        mode: "0644"
-        validate: plutil -lint %s
-      environment: "{{ github_runner_cluster_runtime_environment }}"
-      register: github_runner_cluster_runtime_agent
-
-    - name: Read whether the user has a login session to load the agent into
-      # gui/<uid> exists only while that user is logged in at the console. Without it the agent cannot be loaded now, and launchd loads it at the next login, which is all it is for.
-      ansible.builtin.command:
-        argv: [launchctl, print, "gui/{{ ansible_facts['user_uid'] }}"]
-      environment: "{{ github_runner_cluster_runtime_environment }}"
-      register: github_runner_cluster_runtime_session
-      changed_when: false
-      failed_when: false
-      check_mode: false
-
-    - name: Read whether the agent is loaded
-      ansible.builtin.command:
-        argv: [launchctl, print, "gui/{{ ansible_facts['user_uid'] }}/{{ github_runner_cluster_runtime_launchd_label }}"]
-      environment: "{{ github_runner_cluster_runtime_environment }}"
-      register: github_runner_cluster_runtime_loaded
-      changed_when: false
-      failed_when: false
-      check_mode: false
-
-    - name: Unload the agent so that its new definition is read
-      # launchd keeps the definition it loaded, whatever the file says now.
-      ansible.builtin.command:
-        argv: [launchctl, bootout, "gui/{{ ansible_facts['user_uid'] }}/{{ github_runner_cluster_runtime_launchd_label }}"]
-      environment: "{{ github_runner_cluster_runtime_environment }}"
-      changed_when: true
-      when:
-        - github_runner_cluster_runtime_agent is changed
-        - github_runner_cluster_runtime_session.rc == 0
-        - github_runner_cluster_runtime_loaded.rc == 0
-
-    - name: Load the agent into the user's login session
-      # Loading runs it once (RunAtLoad), which finds the profile already started above and does nothing.
-      ansible.builtin.command:
-        argv:
-          - launchctl
-          - bootstrap
-          - "gui/{{ ansible_facts['user_uid'] }}"
-          - "{{ github_runner_cluster_runtime_agent_plist }}"
-      environment: "{{ github_runner_cluster_runtime_environment }}"
-      changed_when: true
-      when:
-        - github_runner_cluster_runtime_session.rc == 0
-        - github_runner_cluster_runtime_loaded.rc != 0 or github_runner_cluster_runtime_agent is changed
+    - name: Keep the Colima profile starting at login, with a launchd agent
+      ansible.builtin.include_tasks: runtime_agent.yml
+      when: not github_runner_cluster_runtime_at_boot | bool
 
-    - name: Say that the agent waits for the next login
-      ansible.builtin.debug:
-        msg: >-
-          The connecting user has no login session on {{ inventory_hostname }}, so launchd cannot load
-          {{ github_runner_cluster_runtime_launchd_label }} now. It is installed and loads at the next login.
-      when: github_runner_cluster_runtime_session.rc != 0
+    - name: Keep the Colima profile starting at boot, with a launchd daemon
+      ansible.builtin.include_tasks: runtime_daemon.yml
+      when: github_runner_cluster_runtime_at_boot | bool
 
     - name: Read whether the container runtime is up now
       ansible.builtin.include_tasks: runtime_probe.yml