feat: keep Tailscale current in the host baseline roles

README.md

@@ -37,7 +37,7 @@ ansible-playbook -i inventory.yml exadev.github_runner.site --limit node1 --chec
 ansible-playbook -i inventory.yml exadev.github_runner.site --limit node1           # real run
 ```
 
-The playbook runs up to seven roles, each with its own README: `github_runner_host_tools` installs Homebrew and mise on every macOS and Debian-family Linux host first, keeps both at their latest release and puts them in the connecting user's shell, and writes npm and pnpm package-install security settings and shell environment variables from the variables a host gives (it uses `become` and Homebrew's installer, so run the play with passwordless sudo or `--ask-become-pass`, and `bootstrap.sh` turns it off), `github_runner_macos_host` sets up a Mac's host baseline next (AC power settings and `disablesleep` that stop it sleeping, lid closed included, and its own Tailscale client as the `tailscaled` system daemon; it uses `become`, as do the Linux host role and a native node, so run the play with passwordless sudo or `--ask-become-pass`, and `bootstrap.sh` turns it off), `github_runner_linux_host` sets up a Linux host's baseline (keeping it from suspending and its Wi-Fi from power saving; both off unless a host's variables turn them on), `github_runner_cluster` brings the host up as a k3s node, in a container under Docker Compose or, with `github_runner_cluster_node_mode: native`, directly under systemd on a Linux host, `github_runner_arc` installs ARC and the fleet-health platform, `github_runner_k8s_client` (included by the other two) gives a host the kubernetes Python client, and `github_runner_secrets_onepassword` reads the secrets from 1Password when `github_runner_secrets_source` is `onepassword`. The cluster and ARC roles read every secret (`K3S_TOKEN`, the mesh keys, GHCR credentials, the heartbeat PAT, each org's App private key) as a plain variable, so any source Ansible reads works: ansible-vault, SOPS, an env or file lookup, HashiCorp Vault or AWS SSM (see `examples/ansible-vault/`, `examples/env/` and `examples/onepassword/`). With `github_runner_secrets_source: onepassword`, every secret is read live from one 1Password item at task time; `roles/github_runner_secrets_onepassword/README.md` lists the item's fields and the `op://` references. `bootstrap.sh` sets the same variables directly from `.env.bootstrap`. The cluster role ships the node's `docker-compose.yml` and k3s image build context in `roles/github_runner_cluster/files/`, copies them on every run into each host's `github_runner_cluster_dir` (`~/.github-runner` by default), and templates the `.env` for their `k3s`/`k3s-agent` services there, so a host needs no checkout of this repository. The Compose project name is pinned to `github-runner` (`github_runner_cluster_compose_project`) rather than taken from the directory's name, because the named volumes holding the k3s datastore and each node's mesh identity are prefixed with it; the directory can move without losing them.
+The playbook runs up to seven roles, each with its own README: `github_runner_host_tools` installs Homebrew and mise on every macOS and Debian-family Linux host first, keeps both at their latest release and puts them in the connecting user's shell, and writes npm and pnpm package-install security settings and shell environment variables from the variables a host gives (it uses `become` and Homebrew's installer, so run the play with passwordless sudo or `--ask-become-pass`, and `bootstrap.sh` turns it off), `github_runner_macos_host` sets up a Mac's host baseline next (AC power settings and `disablesleep` that stop it sleeping, lid closed included, and its own Tailscale client as the `tailscaled` system daemon, which it keeps at the latest release with `github_runner_macos_host_tailscale_update`; it uses `become`, as do the Linux host role and a native node, so run the play with passwordless sudo or `--ask-become-pass`, and `bootstrap.sh` turns it off), `github_runner_linux_host` sets up a Linux host's baseline (keeping it from suspending and its Wi-Fi from power saving, both off unless a host's variables turn them on, and an installed Tailscale package current with `github_runner_linux_host_tailscale_update`, on by default), `github_runner_cluster` brings the host up as a k3s node, in a container under Docker Compose or, with `github_runner_cluster_node_mode: native`, directly under systemd on a Linux host, `github_runner_arc` installs ARC and the fleet-health platform, `github_runner_k8s_client` (included by the other two) gives a host the kubernetes Python client, and `github_runner_secrets_onepassword` reads the secrets from 1Password when `github_runner_secrets_source` is `onepassword`. The cluster and ARC roles read every secret (`K3S_TOKEN`, the mesh keys, GHCR credentials, the heartbeat PAT, each org's App private key) as a plain variable, so any source Ansible reads works: ansible-vault, SOPS, an env or file lookup, HashiCorp Vault or AWS SSM (see `examples/ansible-vault/`, `examples/env/` and `examples/onepassword/`). With `github_runner_secrets_source: onepassword`, every secret is read live from one 1Password item at task time; `roles/github_runner_secrets_onepassword/README.md` lists the item's fields and the `op://` references. `bootstrap.sh` sets the same variables directly from `.env.bootstrap`. The cluster role ships the node's `docker-compose.yml` and k3s image build context in `roles/github_runner_cluster/files/`, copies them on every run into each host's `github_runner_cluster_dir` (`~/.github-runner` by default), and templates the `.env` for their `k3s`/`k3s-agent` services there, so a host needs no checkout of this repository. The Compose project name is pinned to `github-runner` (`github_runner_cluster_compose_project`) rather than taken from the directory's name, because the named volumes holding the k3s datastore and each node's mesh identity are prefixed with it; the directory can move without losing them.
 
 Which hosts are k3s servers is worked out from the inventory, not set per host. The `github_runner_cluster` inventory group lists the cluster's hosts in order: the first N become servers (control-plane/etcd members running `k3s server`, genuine voting members of the raft quorum), where N is the largest odd number no greater than both the number of hosts and `github_runner_cluster_max_servers` (5 by default), so 1 or 2 hosts give 1 server, 3 or 4 give 3, and 5 or more give 5. The first server bootstraps the cluster (`--cluster-init`, embedded etcd) and every other host joins it, the rest as agents (`k3s agent`, schedulable capacity with no control-plane role). Join URLs and every server's `--tls-san` list are derived from each server's node name and `github_runner_cluster_tailnet_domain`. A host can still set `github_runner_cluster_node_role`, `github_runner_cluster_bootstrap`, `github_runner_cluster_server_url` or `github_runner_cluster_tls_sans` to override its part, and the run fails with an explanation if the result has an even number of servers, none at all, or more than one bootstrap host; `roles/github_runner_cluster/defaults/main.yml` describes each. Installing the ARC controller, each org's runner scale set, and the fleet-health platform is not tied to a node role at all: each is driven purely by whether the installing host's own `github_runner_arc_orgs`/`github_runner_arc_heartbeat_gist_id` is set, so any server host can be the one Ansible installs against, and moving which host does the installing is a config change (which host's `host_vars` sets it), not a role change. See the [Architecture](#architecture) section for why 3 servers is the minimum sane count for genuine HA (2 is worse than 1 for quorum math, not better). To add a server or agent, add the host to the `github_runner_cluster` group in the fleet's inventory and give it its own `host_vars/<host>.yml`; keep the bootstrap host first.
 

roles/github_runner_linux_host/README.md

@@ -1,6 +1,6 @@
 # github_runner_linux_host
 
-The host-level baseline of a Linux host that carries a k3s node, for a machine that is dedicated to the cluster. It does nothing on any other operating system, so it can be applied to a mixed inventory, and every step is off by default and behind a variable of its own, because each changes how the host behaves for whoever else uses it. `playbooks/site.yml` runs it before the fleet play, on every host of the cluster group; a fleet turns the steps on per host in `host_vars`. It touches neither the node nor the host's Tailscale client, which `github_runner_cluster` owns (see its [Native nodes](../github_runner_cluster/README.md#native-nodes)).
+The host-level baseline of a Linux host that carries a k3s node, for a machine that is dedicated to the cluster. It does nothing on any other operating system, so it can be applied to a mixed inventory, and every step but the Tailscale update is off by default and behind a variable of its own, because each changes how the host behaves for whoever else uses it. `playbooks/site.yml` runs it before the fleet play, on every host of the cluster group; a fleet turns the steps on per host in `host_vars`. It touches neither the node nor the host's Tailscale client, which `github_runner_cluster` owns (see its [Native nodes](../github_runner_cluster/README.md#native-nodes)).
 
 Its tasks that change the host use `become` on their own, so the connecting user needs passwordless sudo, or the play needs `--ask-become-pass`. Everything the role only reads runs as the connecting user, and under `--check` nothing is changed and each change a real run would make is reported. Every variable is checked against `meta/argument_specs.yml` when the role runs.
 
@@ -51,6 +51,12 @@ The display manager is stopped before it is disabled, so a name that is an alias
 
 The reboot is guarded. A host on which any unit of `github_runner_linux_host_node_services` (`k3s-agent.service` and `k3s.service` by default; a Docker node would list `docker.service`) is active is a running node of the cluster, and rebooting it takes the node off the cluster and fails the jobs its runner pods are running. The role refuses: it fails, after the packages are upgraded and before any reboot or autoremove, naming the active unit. Draining the node and stopping its service is the operator's, and a run with `github_runner_linux_host_upgrade_reboot_running_node: true` reboots it anyway. `github_runner_linux_host_upgrade_reboot: false` installs the upgrade, says a reboot is due, and reboots nothing. A release upgrade (`do-release-upgrade`) is not part of this.
 
+## Keeping Tailscale current
+
+`github_runner_linux_host_tailscale_update` (default `true`) is the one step that is on by default, because it installs nothing and changes only what the host already runs. When the package `tailscale` is installed and apt has a newer release of it, the role upgrades that package alone (`apt-get install --only-upgrade tailscale`, never a blanket upgrade; the index is refreshed first with `apt-get update`), and it enables and starts the `tailscaled` service if either is missing (`systemctl enable --now tailscaled`). Tailscale's own package scripts restart a running `tailscaled` on upgrade and keep its login, and its apt repository is where newer releases come from ([Install Tailscale on Linux](https://tailscale.com/kb/1031/install-linux)); the restart drops the host\'s tailnet connections, an SSH session over it included, for a moment. The role never installs the package, adds the repository, joins a tailnet or logs in.
+
+A host that is not Debian or Ubuntu, a host without the package, and a package no repository carries a newer release of are skipped, with a message and without failing the run. A native node of the cluster (`github_runner_cluster_node_mode: native` while `github_runner_cluster_native_tailscale_install` is true, its default) is skipped too: the cluster role installs the release it pins there, `github_runner_cluster_tailscale_version`, and puts a host back to it on every run, so a newer release put here would be replaced on the same run. Under `--check` each change is reported and none is made.
+
 ## Variables
 
 - `github_runner_linux_host_keep_awake` (default `false`): mask the sleep targets and have logind ignore the lid switch.
@@ -65,7 +71,8 @@ The reboot is guarded. A host on which any unit of `github_runner_linux_host_nod
 - `github_runner_linux_host_upgrade_reboot` (default `true`), `github_runner_linux_host_upgrade_reboot_running_node` (default `false`), `github_runner_linux_host_node_services` and `github_runner_linux_host_reboot_timeout` (default 600): whether the upgrade reboots, whether it may reboot a running node, which units make a host a running node, and how long it waits for the host to come back.
 - `github_runner_linux_host_sleep_targets`, `github_runner_linux_host_lid_switch_settings`, `github_runner_linux_host_kernel_modules`, `github_runner_linux_host_sysctl_settings` and `github_runner_linux_host_default_target`: the targets masked, the logind settings set to `ignore`, the modules loaded, the sysctls set and the target booted to. Constants, not settings; variables only so the tests can narrow them.
 - `github_runner_linux_host_logind_dropin`, `github_runner_linux_host_networkmanager_dropin`, `github_runner_linux_host_modules_load_file`, `github_runner_linux_host_sysctl_file`, `github_runner_linux_host_sys_module_dir`, `github_runner_linux_host_power_supply_dir`, `github_runner_linux_host_battery_unit`, `github_runner_linux_host_unattended_upgrades_file`, `github_runner_linux_host_reboot_required_file`: where the role's files go and what it reads and runs. Not settings; variables only so the tests can point them at a stand-in.
+- `github_runner_linux_host_tailscale_update` (default `true`): keep the installed Tailscale package current and its service running, as under Keeping Tailscale current above.
 
 ## Tests
 
-`tests/unit/test_linux_host.py` runs the role (`tests/linux_host/run.yml`) against stand-ins for `systemctl`, `nmcli`, `iw`, `modprobe`, `sysctl`, `apt-get`, `dpkg-query`, `uname` and `shutdown` on `PATH`, with every file the role writes or reads redirected and the host's facts stood in for. It checks for each step that it is off by default, does what it says, changes nothing on a second run, and under `--check` reports each change and makes none; and in particular the order of the modules and the sysctls, a battery without the attribute, a limit outside 1 to 100, a display manager worked out from the alias, from its own unit when the alias is not loaded and from a unit the host names, a host with none, services the host does not have, the apt steps refused on a host without apt, the order of the upgrade, a running node's reboot refused with the packages already upgraded, and the defaults, argument specs and this README agreeing.
+`tests/unit/test_linux_host.py` runs the role (`tests/linux_host/run.yml`) against stand-ins for `systemctl`, `nmcli`, `iw`, `modprobe`, `sysctl`, `apt-get`, `dpkg-query`, `uname` and `shutdown` on `PATH`, with every file the role writes or reads redirected and the host's facts stood in for. It checks for each step but the Tailscale update that it is off by default (and that the update is on, and leaves the host alone without the package), does what it says, changes nothing on a second run, and under `--check` reports each change and makes none; and in particular the order of the modules and the sysctls, a battery without the attribute, a limit outside 1 to 100, a display manager worked out from the alias, from its own unit when the alias is not loaded and from a unit the host names, a host with none, services the host does not have, the apt steps refused on a host without apt, the order of the upgrade, a running node's reboot refused with the packages already upgraded, and the defaults, argument specs and this README agreeing.

roles/github_runner_linux_host/defaults/main.yml

@@ -1,5 +1,5 @@
 ---
-# The host-level baseline of a Linux host that carries a k3s node: nothing here touches the node itself, or the Tailscale client. Every variable is checked against meta/argument_specs.yml when the role runs. Every step is off by default, each behind a variable of its own, because each changes how the host behaves for whoever else uses it: a fleet turns them on per host, for a machine that is dedicated to the cluster.
+# The host-level baseline of a Linux host that carries a k3s node: nothing here touches the node itself, or the Tailscale client. Every variable is checked against meta/argument_specs.yml when the role runs. Every step but the last, the Tailscale update, is off by default, each behind a variable of its own, because each changes how the host behaves for whoever else uses it: a fleet turns them on per host, for a machine that is dedicated to the cluster.
 
 # Whether the host is kept from suspending: sleep.target, suspend.target, hibernate.target and hybrid-sleep.target are masked, so nothing can put it to sleep (a desktop's idle timer, a power button, `systemctl suspend`), and systemd-logind is told to ignore the lid switch, which a laptop would otherwise suspend on whenever its lid closes, with or without an external monitor or power. A host that sleeps drops off the network, so its node leaves the cluster and its runner pods stop.
 github_runner_linux_host_keep_awake: false
@@ -92,3 +92,6 @@ github_runner_linux_host_upgrade_reboot_running_node: false
 github_runner_linux_host_reboot_timeout: 600
 # Not a setting: the file Debian and Ubuntu create when an upgrade needs a reboot. A variable only so the tests can point it at a stand-in.
 github_runner_linux_host_reboot_required_file: /var/run/reboot-required
+
+# Whether the host's Tailscale client is kept current, the one step that is on by default because it installs nothing and changes only what the host already runs: when the package tailscale is installed (from Tailscale's apt repository, https://tailscale.com/kb/1031/install-linux) and apt has a newer release of it, only that package is upgraded (apt-get install --only-upgrade tailscale, never a blanket upgrade), and the tailscaled service is enabled and started if it is not. A host without the package, and one that is not Debian or Ubuntu, is skipped with a message, and the role never installs the package, joins a tailnet or logs in. A native node of the cluster is skipped too, because the cluster role installs the release it pins there (github_runner_cluster_tailscale_version) and would put it back on the same run. The upgrade restarts a running tailscaled, which keeps its login and drops the host's tailnet connections for a moment. Under --check each change is reported and none is made.
+github_runner_linux_host_tailscale_update: true

roles/github_runner_linux_host/meta/argument_specs.yml

@@ -4,7 +4,7 @@ argument_specs:
   main:
     short_description: 'Prepares a Linux host that carries a k3s node: keeps it awake, headless and up to date, and ready for k3s.'
     description:
-      - Runs on Linux hosts only and does nothing on any other. Every step is off by default and behind a variable of its own. Its tasks that change the host use become (sudo), so run the play with passwordless sudo for the connecting user or with --ask-become-pass.
+      - Runs on Linux hosts only and does nothing on any other. Every step but the Tailscale update is off by default, and each is behind a variable of its own. Its tasks that change the host use become (sudo), so run the play with passwordless sudo for the connecting user or with --ask-become-pass.
     options:
       github_runner_linux_host_keep_awake:
         type: bool
@@ -161,3 +161,7 @@ argument_specs:
         type: path
         default: /var/run/reboot-required
         description: The file that says an upgrade needs a reboot. Not a setting, and a variable only so the tests can point it at a stand-in.
+      github_runner_linux_host_tailscale_update:
+        type: bool
+        default: true
+        description: "Whether the host's Tailscale client is kept current, the one step that is on by default: when the package tailscale is installed and apt has a newer release of it, only that package is upgraded (apt-get install --only-upgrade tailscale), and the tailscaled service is enabled and started if it is not. Nothing is installed and no tailnet is joined. A host without the package, a host that is not Debian or Ubuntu, and a native node of the cluster (whose cluster role installs the release it pins) are skipped with a message."