fix(cluster): load br_netfilter in the Colima VM now and at every boot

README.md

@@ -214,7 +214,7 @@ Commits follow Conventional Commits (`fix:`, `feat:`, `tune:`, and so on).
 
 - **A scale-set's listener pod can crash-loop forever with `ephemeralrunnersets... not found` after a Helm upgrade, even though the release itself succeeds.** ARC names the `EphemeralRunnerSet` it creates for a release with a spec hash suffix, and gives it a new name whenever the pod template changes (a `helm upgrade` that touches resource limits, `nodeSelector`, or similar). The `AutoscalingListener` is supposed to track this rename, but can be left pointing at the old, now-deleted name - it then restarts every second or two, each time failing at startup with `could not patch ephemeral runner set <old-name>... not found`, and the scale set's own job queue never gets serviced even though every other release's listener stays healthy. Diagnose with `kubectl get autoscalinglisteners -n actions-runner-controller -o custom-columns='NAME:.metadata.name,ERS:.spec.ephemeralRunnerSetName'` against `kubectl get ephemeralrunnersets -A` - a mismatch confirms it. Fix: `kubectl delete autoscalinglistener <name> -n actions-runner-controller`; the controller recreates it immediately with the current `EphemeralRunnerSet` name.
 - **k3s cannot overwrite an existing kubeconfig through Colima's macOS share.** The container-root-to-Mac-user file mapping leaves `/output/kubeconfig.yaml` owner-read-only, so k3s crash-loops with "permission denied" on every restart (for example after a Mac reboot) unless the file is removed first. `k3s/node-entrypoint.sh` deletes it before starting k3s each time — do not remove that step.
-- **After a Colima VM restart (a host reboot, for example) pods can lose every cluster service IP, cluster DNS included, until the Linux VM loads `br_netfilter`.** Without the module (`/proc/sys/net/bridge` is missing in the VM) packets bridged between pods never traverse iptables, so the service DNAT never applies: a pod reaches another pod's IP and the node's resolver directly but times out on `10.43.0.10`. Nothing looks wrong at first, because ARC keeps working from cached state; it fails only when it next needs a fresh GitHub token, when the controller logs `lookup api.github.com: i/o timeout`, a scale set's listener is not recreated, its `AutoscalingRunnerSet` stays `Pending` and the heartbeat reports `has no running listener`. Load it with `colima ssh -- sudo modprobe br_netfilter` and keep it across restarts with `echo br_netfilter | colima ssh -- sudo tee /etc/modules-load.d/br_netfilter.conf`; check with `colima ssh -- ls /proc/sys/net/bridge`.
+- **Pods can lose every cluster service IP, cluster DNS included, whenever the Colima Linux VM has not loaded `br_netfilter` (a restarted VM, after a host reboot for example).** Without the module (`/proc/sys/net/bridge` is missing in the VM) packets bridged between pods never traverse iptables, so the service DNAT never applies: a pod reaches another pod's IP and the node's resolver directly but times out on `10.43.0.10` when the Service's only backend pod (CoreDNS, say) is on the same node as the caller, since a bridged hairpin reply skips the reverse NAT; backends on other nodes still answer, which is why it can stay latent until such a pod is rescheduled onto the node. Nothing looks wrong at first, because ARC keeps working from cached state; it fails only when it next needs a fresh GitHub token, when the controller logs `lookup api.github.com: i/o timeout`, a scale set's listener is not recreated, its `AutoscalingRunnerSet` stays `Pending` and the heartbeat reports `has no running listener`. The cluster role does this on every run for a Colima host (`tasks/colima_kernel.yml`, see its README's Colima kernel paragraph): it loads the module, writes `/etc/modules-load.d/br_netfilter.conf` and a sysctl file in the VM and sets the bridge sysctls. To repair a host before the playbook has run, load it with `colima ssh -- sudo modprobe br_netfilter`; check with `colima ssh -- ls /proc/sys/net/bridge`.
 - **The k3s node name must stay stable across container recreations.** Without a pinned `hostname`, k3s derives the node name from the container's hostname (its ID), so every recreation (for example on an image pull) registers a new node and leaves the old one stale in etcd. The cluster role's `docker-compose.yml` pins every server host's own hostname to `${K3S_NODE_NAME:-k3s-server}`, Ansible-templated per host (defaulting to that host's own inventory name; `github_runner_cluster_node_name` overrides it, for example to keep a pre-existing device identity rather than forcing a disruptive rename).
 - **`kubectl get nodes` can succeed with zero nodes.** It returns an empty, still-successful list before the node object has registered. The role polls for at least one node before calling `kubectl wait`, which fails outright ("no matching resources found") against zero nodes.
 - **Zero runner pods in an `arc-runners-<org>` namespace (or its per-profile `-<suffix>` variant) when idle is expected, not a bug** — `minRunners: 0` means pods exist only while a job is running.

roles/github_runner_cluster/README.md

@@ -156,7 +156,7 @@ A Mac still has to log in for the agent to run: a launch agent starts with the u
 
 **End checks.** On every macOS host the role checks, before anything reads the daemon and again after it has started the node's Compose project, that the runtime answers (`docker info` through the host's Docker context), and after starting the project that the node's k3s container (`github-runner-k3s`, or `github-runner-k3s-agent` on an agent) is running. A converge of a host that rebooted therefore repairs a stopped Colima or fails with a message that says what to do, and never reports success over a node that is not up. A `--check` run does not fail on a stopped runtime or container it has not started.
 
-A Colima VM that starts after a reboot is a restarted VM, so the `br_netfilter` note under Non-obvious behaviour in the repository README applies to it: pods can lose cluster service IPs until that module is loaded in the VM.
+**Colima kernel.** On every Colima host, whether or not `github_runner_cluster_runtime_autostart` is on, once the VM answers (`tasks/colima_kernel.yml`), the role makes the VM's kernel pass bridged pod traffic through iptables, now and at every VM boot: it writes `/etc/modules-load.d/br_netfilter.conf` and `/etc/sysctl.d/90-github-runner-br-netfilter.conf` in the VM, runs `modprobe br_netfilter` when the VM has no `/proc/sys/net/bridge`, and sets `net.bridge.bridge-nf-call-iptables` and `net.bridge.bridge-nf-call-ip6tables` to 1 when they differ. Without the module a pod that talks to a Service whose only backend pod is on the same node gets no reply, because the Service's reverse NAT never applies to the bridged hairpin traffic; pods on other nodes keep working, so it stays latent until a single-backend Service such as CoreDNS lands on the node (the `br_netfilter` note under Non-obvious behaviour in the repository README has the symptoms). Everything runs in the VM through `colima ssh --profile <profile>`, as the VM's user with `sudo`, and no `become` on the host. A file is written only when its content differs, so a rerun changes nothing, and the module file takes the name the README's note told people to create by hand, so an existing one is replaced by the managed one. Under `--check` each change is reported and none is made, and nothing is read from a VM that a check run would only have started. Docker Desktop's VM is not reachable by the role and other runtimes are not touched.
 
 This is the role's whole scope: it never moves a host between Colima and Docker Desktop.
 

roles/github_runner_cluster/defaults/main.yml

@@ -188,6 +188,17 @@ github_runner_cluster_runtime_path: "/opt/homebrew/bin:/opt/homebrew/sbin:/usr/l
 github_runner_cluster_runtime_launchd_label: "com.exadev.github-runner.colima.{{ github_runner_cluster_effective_runtime.colima_profile }}"
 # Derived, never set directly: where the agent's property list lives, in the connecting user's own LaunchAgents directory.
 github_runner_cluster_runtime_agent_plist: "{{ ansible_facts['env']['HOME'] }}/Library/LaunchAgents/{{ github_runner_cluster_runtime_launchd_label }}.plist"
+# What the Colima VM's kernel needs for the node, loaded and set now and at every VM boot on every run (tasks/colima_kernel.yml). A pod that talks to a Service whose only backend is on the same node gets no reply unless bridged traffic passes through iptables, and that needs the br_netfilter module and the two sysctls below. Constants, not settings; variables only so the tests can narrow them.
+github_runner_cluster_colima_kernel_module: br_netfilter
+github_runner_cluster_colima_sysctl_settings:
+  net.bridge.bridge-nf-call-iptables: 1
+  net.bridge.bridge-nf-call-ip6tables: 1
+# Not settings: the directory that exists in the VM exactly while the module is loaded (the bridge sysctls live under it), and the files in the VM that load the module and set the sysctls at boot. The module file keeps the name the repository README's note has long told people to create by hand, so a host that has one is adopted, not duplicated.
+github_runner_cluster_colima_bridge_proc_dir: /proc/sys/net/bridge
+github_runner_cluster_colima_modules_load_file: /etc/modules-load.d/br_netfilter.conf
+github_runner_cluster_colima_sysctl_file: /etc/sysctl.d/90-github-runner-br-netfilter.conf
+# Derived, never set directly: the command that runs another command in the profile's VM as the VM's user, which has passwordless sudo.
+github_runner_cluster_colima_vm_command: "{{ ['colima', 'ssh', '--profile', github_runner_cluster_effective_runtime.colima_profile, '--'] }}"
 
 # The directory on the target host the role manages as this host's Compose project: it copies the collection's docker-compose.yml and k3s/ build context there, templates .env there, and k3s writes the kubeconfig to its kubeconfig/ subdirectory. Needs no checkout of anything on the host. The default reads HOME from the target's gathered facts, so it is the connecting user's home on that host, not the control node's.
 github_runner_cluster_dir: "{{ ansible_facts['env']['HOME'] }}/{{ github_runner_cluster_dir_name }}"

roles/github_runner_cluster/tasks/colima_kernel.yml

@@ -0,0 +1,116 @@
+---
+# Colima hosts only (runtime.yml includes it once the runtime answers). Makes the Linux VM the node runs in load br_netfilter and set the bridge sysctls, now and at every VM boot: a modules-load.d file and a sysctl.d file in the VM for boot, and modprobe and sysctl -w for now. Without the module, bridged pod traffic skips iptables, so a pod that talks to a Service whose only backend pod is on the same node gets no reply (the Service's reverse NAT never applies); pods on other nodes keep working, which hides it until a single-backend Service such as CoreDNS is rescheduled onto this node. See the repository README's note on br_netfilter. Idempotent: a file is written only when its content differs, the module is loaded only when the VM has no bridge sysctl directory, and a sysctl is set only when its value differs. Under --check each change is reported and none is made. Every command runs in the VM through colima ssh, as the VM's user with sudo, and needs no become on the host.
+
+- name: Work out the files the VM keeps for the module and the sysctls
+  ansible.builtin.set_fact:
+    github_runner_cluster_colima_vm_files:
+      - path: "{{ github_runner_cluster_colima_modules_load_file }}"
+        content: |-
+          # Managed by the exadev.github_runner collection's github_runner_cluster role. Do not edit by hand.
+          {{ github_runner_cluster_colima_kernel_module }}
+      - path: "{{ github_runner_cluster_colima_sysctl_file }}"
+        content: |-
+          # Managed by the exadev.github_runner collection's github_runner_cluster role. Do not edit by hand.
+          {% for key, value in github_runner_cluster_colima_sysctl_settings.items() %}
+          {{ key }} = {{ value }}
+          {% endfor %}
+
+- name: Read the files in the VM
+  # A file the VM does not have reads as no content, which differs from the wanted one.
+  ansible.builtin.command:
+    argv: "{{ github_runner_cluster_colima_vm_command + ['cat', item.path] }}"
+  environment: "{{ github_runner_cluster_runtime_environment }}"
+  loop: "{{ github_runner_cluster_colima_vm_files }}"
+  loop_control:
+    label: "{{ item.path }}"
+  register: github_runner_cluster_colima_vm_file_state
+  changed_when: false
+  failed_when: false
+  check_mode: false
+
+- name: Write the files in the VM whose content differs
+  ansible.builtin.command:
+    argv: "{{ github_runner_cluster_colima_vm_command + ['sudo', 'tee', item.item.path] }}"
+    stdin: "{{ item.item.content | trim }}"
+  environment: "{{ github_runner_cluster_runtime_environment }}"
+  loop: "{{ github_runner_cluster_colima_vm_file_state.results }}"
+  loop_control:
+    label: "{{ item.item.path }}"
+  changed_when: true
+  when:
+    - item.stdout | trim != item.item.content | trim
+    - not ansible_check_mode
+
+- name: Report the files in the VM a real run would write
+  ansible.builtin.debug:
+    msg: "write {{ item.item.path }} in the VM"
+  loop: "{{ github_runner_cluster_colima_vm_file_state.results }}"
+  loop_control:
+    label: "{{ item.item.path }}"
+  changed_when: true
+  when:
+    - item.stdout | trim != item.item.content | trim
+    - ansible_check_mode
+
+- name: Read whether the VM has loaded the module
+  # The bridge sysctl directory is what the sysctls need and what the module's absence takes away, so it is the one thing read: it also holds for a kernel with the module built in, which modprobe would report as loaded every run.
+  ansible.builtin.command:
+    argv: "{{ github_runner_cluster_colima_vm_command + ['test', '-d', github_runner_cluster_colima_bridge_proc_dir] }}"
+  environment: "{{ github_runner_cluster_runtime_environment }}"
+  register: github_runner_cluster_colima_module
+  changed_when: false
+  failed_when: false
+  check_mode: false
+
+- name: Load the module in the VM
+  ansible.builtin.command:
+    argv: "{{ github_runner_cluster_colima_vm_command + ['sudo', 'modprobe', '--', github_runner_cluster_colima_kernel_module] }}"
+  environment: "{{ github_runner_cluster_runtime_environment }}"
+  changed_when: true
+  when:
+    - github_runner_cluster_colima_module.rc != 0
+    - not ansible_check_mode
+
+- name: Report the module a real run would load in the VM
+  ansible.builtin.debug:
+    msg: "modprobe {{ github_runner_cluster_colima_kernel_module }} in the VM"
+  changed_when: true
+  when:
+    - github_runner_cluster_colima_module.rc != 0
+    - ansible_check_mode
+
+- name: Read the sysctls in the VM
+  # A key that does not exist yet (a bridge key before the module is loaded, which a real run has done by now) reads as no value, which differs from the wanted one.
+  ansible.builtin.command:
+    argv: "{{ github_runner_cluster_colima_vm_command + ['sysctl', '-n', item.key] }}"
+  environment: "{{ github_runner_cluster_runtime_environment }}"
+  loop: "{{ github_runner_cluster_colima_sysctl_settings | dict2items }}"
+  loop_control:
+    label: "{{ item.key }}"
+  register: github_runner_cluster_colima_sysctls
+  changed_when: false
+  failed_when: false
+  check_mode: false
+
+- name: Set the sysctls in the VM whose value differs
+  ansible.builtin.command:
+    argv: "{{ github_runner_cluster_colima_vm_command + ['sudo', 'sysctl', '-w', item.item.key ~ '=' ~ item.item.value] }}"
+  environment: "{{ github_runner_cluster_runtime_environment }}"
+  loop: "{{ github_runner_cluster_colima_sysctls.results }}"
+  loop_control:
+    label: "{{ item.item.key }}"
+  changed_when: true
+  when:
+    - item.stdout | trim != item.item.value | string
+    - not ansible_check_mode
+
+- name: Report the sysctls a real run would set in the VM
+  ansible.builtin.debug:
+    msg: "sysctl -w {{ item.item.key }}={{ item.item.value }} in the VM"
+  loop: "{{ github_runner_cluster_colima_sysctls.results }}"
+  loop_control:
+    label: "{{ item.item.key }}"
+  changed_when: true
+  when:
+    - item.stdout | trim != item.item.value | string
+    - ansible_check_mode

roles/github_runner_cluster/tasks/runtime.yml

@@ -3,6 +3,8 @@
 #
 # On a Colima host with github_runner_cluster_runtime_autostart on, it starts the profile if it is stopped, installs a per-user launchd agent that starts it at every login (a property list in the connecting user's ~/Library/LaunchAgents) and loads that agent into the user's login session. Docker Desktop cannot be started by the role, so there it only checks. The agent runs as the connecting user and the role never uses become for it, since a LaunchAgent belongs to the user whose login session loads it.
 #
+# On a Colima host it also makes the VM load br_netfilter and set the bridge sysctls now and at every VM boot (tasks/colima_kernel.yml), whatever the autostart setting says.
+#
 # Publishes github_runner_cluster_effective_runtime ({kind, colima_profile}), github_runner_cluster_runtime_environment (what docker and colima run with), github_runner_cluster_runtime_down_message and github_runner_cluster_runtime_up, for end_checks.yml.
 
 - name: Read the connecting user and its environment
@@ -128,3 +130,10 @@
   when:
     - not github_runner_cluster_runtime_up | bool
     - not ansible_check_mode
+
+- name: Make the Colima VM load br_netfilter and set the bridge sysctls
+  # After the failure above, so the VM answers (or this is a check run that would have started it, which has no VM to read yet). Whatever the autostart setting says: the node needs the module in the VM however the VM was started. See tasks/colima_kernel.yml.
+  ansible.builtin.include_tasks: colima_kernel.yml
+  when:
+    - github_runner_cluster_effective_runtime.kind == 'colima'
+    - github_runner_cluster_runtime_up | bool