diff --git a/README.md b/README.md index f363c1dc..52f3e987 100644 --- a/README.md +++ b/README.md @@ -214,7 +214,7 @@ Commits follow Conventional Commits (`fix:`, `feat:`, `tune:`, and so on). - **A scale-set's listener pod can crash-loop forever with `ephemeralrunnersets... not found` after a Helm upgrade, even though the release itself succeeds.** ARC names the `EphemeralRunnerSet` it creates for a release with a spec hash suffix, and gives it a new name whenever the pod template changes (a `helm upgrade` that touches resource limits, `nodeSelector`, or similar). The `AutoscalingListener` is supposed to track this rename, but can be left pointing at the old, now-deleted name - it then restarts every second or two, each time failing at startup with `could not patch ephemeral runner set ... not found`, and the scale set's own job queue never gets serviced even though every other release's listener stays healthy. Diagnose with `kubectl get autoscalinglisteners -n actions-runner-controller -o custom-columns='NAME:.metadata.name,ERS:.spec.ephemeralRunnerSetName'` against `kubectl get ephemeralrunnersets -A` - a mismatch confirms it. Fix: `kubectl delete autoscalinglistener -n actions-runner-controller`; the controller recreates it immediately with the current `EphemeralRunnerSet` name. - **k3s cannot overwrite an existing kubeconfig through Colima's macOS share.** The container-root-to-Mac-user file mapping leaves `/output/kubeconfig.yaml` owner-read-only, so k3s crash-loops with "permission denied" on every restart (for example after a Mac reboot) unless the file is removed first. `k3s/node-entrypoint.sh` deletes it before starting k3s each time — do not remove that step. -- **After a Colima VM restart (a host reboot, for example) pods can lose every cluster service IP, cluster DNS included, until the Linux VM loads `br_netfilter`.** Without the module (`/proc/sys/net/bridge` is missing in the VM) packets bridged between pods never traverse iptables, so the service DNAT never applies: a pod reaches another pod's IP and the node's resolver directly but times out on `10.43.0.10`. Nothing looks wrong at first, because ARC keeps working from cached state; it fails only when it next needs a fresh GitHub token, when the controller logs `lookup api.github.com: i/o timeout`, a scale set's listener is not recreated, its `AutoscalingRunnerSet` stays `Pending` and the heartbeat reports `has no running listener`. Load it with `colima ssh -- sudo modprobe br_netfilter` and keep it across restarts with `echo br_netfilter | colima ssh -- sudo tee /etc/modules-load.d/br_netfilter.conf`; check with `colima ssh -- ls /proc/sys/net/bridge`. +- **Pods can lose every cluster service IP, cluster DNS included, whenever the Colima Linux VM has not loaded `br_netfilter` (a restarted VM, after a host reboot for example).** Without the module (`/proc/sys/net/bridge` is missing in the VM) packets bridged between pods never traverse iptables, so the service DNAT never applies: a pod reaches another pod's IP and the node's resolver directly but times out on `10.43.0.10` when the Service's only backend pod (CoreDNS, say) is on the same node as the caller, since a bridged hairpin reply skips the reverse NAT; backends on other nodes still answer, which is why it can stay latent until such a pod is rescheduled onto the node. Nothing looks wrong at first, because ARC keeps working from cached state; it fails only when it next needs a fresh GitHub token, when the controller logs `lookup api.github.com: i/o timeout`, a scale set's listener is not recreated, its `AutoscalingRunnerSet` stays `Pending` and the heartbeat reports `has no running listener`. The cluster role does this on every run for a Colima host (`tasks/colima_kernel.yml`, see its README's Colima kernel paragraph): it loads the module, writes `/etc/modules-load.d/br_netfilter.conf` and a sysctl file in the VM and sets the bridge sysctls. To repair a host before the playbook has run, load it with `colima ssh -- sudo modprobe br_netfilter`; check with `colima ssh -- ls /proc/sys/net/bridge`. - **The k3s node name must stay stable across container recreations.** Without a pinned `hostname`, k3s derives the node name from the container's hostname (its ID), so every recreation (for example on an image pull) registers a new node and leaves the old one stale in etcd. The cluster role's `docker-compose.yml` pins every server host's own hostname to `${K3S_NODE_NAME:-k3s-server}`, Ansible-templated per host (defaulting to that host's own inventory name; `github_runner_cluster_node_name` overrides it, for example to keep a pre-existing device identity rather than forcing a disruptive rename). - **`kubectl get nodes` can succeed with zero nodes.** It returns an empty, still-successful list before the node object has registered. The role polls for at least one node before calling `kubectl wait`, which fails outright ("no matching resources found") against zero nodes. - **Zero runner pods in an `arc-runners-` namespace (or its per-profile `-` variant) when idle is expected, not a bug** — `minRunners: 0` means pods exist only while a job is running. diff --git a/roles/github_runner_cluster/README.md b/roles/github_runner_cluster/README.md index 0f4c614a..8885a014 100644 --- a/roles/github_runner_cluster/README.md +++ b/roles/github_runner_cluster/README.md @@ -156,7 +156,7 @@ A Mac still has to log in for the agent to run: a launch agent starts with the u **End checks.** On every macOS host the role checks, before anything reads the daemon and again after it has started the node's Compose project, that the runtime answers (`docker info` through the host's Docker context), and after starting the project that the node's k3s container (`github-runner-k3s`, or `github-runner-k3s-agent` on an agent) is running. A converge of a host that rebooted therefore repairs a stopped Colima or fails with a message that says what to do, and never reports success over a node that is not up. A `--check` run does not fail on a stopped runtime or container it has not started. -A Colima VM that starts after a reboot is a restarted VM, so the `br_netfilter` note under Non-obvious behaviour in the repository README applies to it: pods can lose cluster service IPs until that module is loaded in the VM. +**Colima kernel.** On every Colima host, whether or not `github_runner_cluster_runtime_autostart` is on, once the VM answers (`tasks/colima_kernel.yml`), the role makes the VM's kernel pass bridged pod traffic through iptables, now and at every VM boot: it writes `/etc/modules-load.d/br_netfilter.conf` and `/etc/sysctl.d/90-github-runner-br-netfilter.conf` in the VM, runs `modprobe br_netfilter` when the VM has no `/proc/sys/net/bridge`, and sets `net.bridge.bridge-nf-call-iptables` and `net.bridge.bridge-nf-call-ip6tables` to 1 when they differ. Without the module a pod that talks to a Service whose only backend pod is on the same node gets no reply, because the Service's reverse NAT never applies to the bridged hairpin traffic; pods on other nodes keep working, so it stays latent until a single-backend Service such as CoreDNS lands on the node (the `br_netfilter` note under Non-obvious behaviour in the repository README has the symptoms). Everything runs in the VM through `colima ssh --profile `, as the VM's user with `sudo`, and no `become` on the host. A file is written only when its content differs, so a rerun changes nothing, and the module file takes the name the README's note told people to create by hand, so an existing one is replaced by the managed one. Under `--check` each change is reported and none is made, and nothing is read from a VM that a check run would only have started. Docker Desktop's VM is not reachable by the role and other runtimes are not touched. This is the role's whole scope: it never moves a host between Colima and Docker Desktop. diff --git a/roles/github_runner_cluster/defaults/main.yml b/roles/github_runner_cluster/defaults/main.yml index 35b03d97..f155ee3c 100644 --- a/roles/github_runner_cluster/defaults/main.yml +++ b/roles/github_runner_cluster/defaults/main.yml @@ -188,6 +188,17 @@ github_runner_cluster_runtime_path: "/opt/homebrew/bin:/opt/homebrew/sbin:/usr/l github_runner_cluster_runtime_launchd_label: "com.exadev.github-runner.colima.{{ github_runner_cluster_effective_runtime.colima_profile }}" # Derived, never set directly: where the agent's property list lives, in the connecting user's own LaunchAgents directory. github_runner_cluster_runtime_agent_plist: "{{ ansible_facts['env']['HOME'] }}/Library/LaunchAgents/{{ github_runner_cluster_runtime_launchd_label }}.plist" +# What the Colima VM's kernel needs for the node, loaded and set now and at every VM boot on every run (tasks/colima_kernel.yml). A pod that talks to a Service whose only backend is on the same node gets no reply unless bridged traffic passes through iptables, and that needs the br_netfilter module and the two sysctls below. Constants, not settings; variables only so the tests can narrow them. +github_runner_cluster_colima_kernel_module: br_netfilter +github_runner_cluster_colima_sysctl_settings: + net.bridge.bridge-nf-call-iptables: 1 + net.bridge.bridge-nf-call-ip6tables: 1 +# Not settings: the directory that exists in the VM exactly while the module is loaded (the bridge sysctls live under it), and the files in the VM that load the module and set the sysctls at boot. The module file keeps the name the repository README's note has long told people to create by hand, so a host that has one is adopted, not duplicated. +github_runner_cluster_colima_bridge_proc_dir: /proc/sys/net/bridge +github_runner_cluster_colima_modules_load_file: /etc/modules-load.d/br_netfilter.conf +github_runner_cluster_colima_sysctl_file: /etc/sysctl.d/90-github-runner-br-netfilter.conf +# Derived, never set directly: the command that runs another command in the profile's VM as the VM's user, which has passwordless sudo. +github_runner_cluster_colima_vm_command: "{{ ['colima', 'ssh', '--profile', github_runner_cluster_effective_runtime.colima_profile, '--'] }}" # The directory on the target host the role manages as this host's Compose project: it copies the collection's docker-compose.yml and k3s/ build context there, templates .env there, and k3s writes the kubeconfig to its kubeconfig/ subdirectory. Needs no checkout of anything on the host. The default reads HOME from the target's gathered facts, so it is the connecting user's home on that host, not the control node's. github_runner_cluster_dir: "{{ ansible_facts['env']['HOME'] }}/{{ github_runner_cluster_dir_name }}" diff --git a/roles/github_runner_cluster/tasks/colima_kernel.yml b/roles/github_runner_cluster/tasks/colima_kernel.yml new file mode 100644 index 00000000..f237e537 --- /dev/null +++ b/roles/github_runner_cluster/tasks/colima_kernel.yml @@ -0,0 +1,116 @@ +--- +# Colima hosts only (runtime.yml includes it once the runtime answers). Makes the Linux VM the node runs in load br_netfilter and set the bridge sysctls, now and at every VM boot: a modules-load.d file and a sysctl.d file in the VM for boot, and modprobe and sysctl -w for now. Without the module, bridged pod traffic skips iptables, so a pod that talks to a Service whose only backend pod is on the same node gets no reply (the Service's reverse NAT never applies); pods on other nodes keep working, which hides it until a single-backend Service such as CoreDNS is rescheduled onto this node. See the repository README's note on br_netfilter. Idempotent: a file is written only when its content differs, the module is loaded only when the VM has no bridge sysctl directory, and a sysctl is set only when its value differs. Under --check each change is reported and none is made. Every command runs in the VM through colima ssh, as the VM's user with sudo, and needs no become on the host. + +- name: Work out the files the VM keeps for the module and the sysctls + ansible.builtin.set_fact: + github_runner_cluster_colima_vm_files: + - path: "{{ github_runner_cluster_colima_modules_load_file }}" + content: |- + # Managed by the exadev.github_runner collection's github_runner_cluster role. Do not edit by hand. + {{ github_runner_cluster_colima_kernel_module }} + - path: "{{ github_runner_cluster_colima_sysctl_file }}" + content: |- + # Managed by the exadev.github_runner collection's github_runner_cluster role. Do not edit by hand. + {% for key, value in github_runner_cluster_colima_sysctl_settings.items() %} + {{ key }} = {{ value }} + {% endfor %} + +- name: Read the files in the VM + # A file the VM does not have reads as no content, which differs from the wanted one. + ansible.builtin.command: + argv: "{{ github_runner_cluster_colima_vm_command + ['cat', item.path] }}" + environment: "{{ github_runner_cluster_runtime_environment }}" + loop: "{{ github_runner_cluster_colima_vm_files }}" + loop_control: + label: "{{ item.path }}" + register: github_runner_cluster_colima_vm_file_state + changed_when: false + failed_when: false + check_mode: false + +- name: Write the files in the VM whose content differs + ansible.builtin.command: + argv: "{{ github_runner_cluster_colima_vm_command + ['sudo', 'tee', item.item.path] }}" + stdin: "{{ item.item.content | trim }}" + environment: "{{ github_runner_cluster_runtime_environment }}" + loop: "{{ github_runner_cluster_colima_vm_file_state.results }}" + loop_control: + label: "{{ item.item.path }}" + changed_when: true + when: + - item.stdout | trim != item.item.content | trim + - not ansible_check_mode + +- name: Report the files in the VM a real run would write + ansible.builtin.debug: + msg: "write {{ item.item.path }} in the VM" + loop: "{{ github_runner_cluster_colima_vm_file_state.results }}" + loop_control: + label: "{{ item.item.path }}" + changed_when: true + when: + - item.stdout | trim != item.item.content | trim + - ansible_check_mode + +- name: Read whether the VM has loaded the module + # The bridge sysctl directory is what the sysctls need and what the module's absence takes away, so it is the one thing read: it also holds for a kernel with the module built in, which modprobe would report as loaded every run. + ansible.builtin.command: + argv: "{{ github_runner_cluster_colima_vm_command + ['test', '-d', github_runner_cluster_colima_bridge_proc_dir] }}" + environment: "{{ github_runner_cluster_runtime_environment }}" + register: github_runner_cluster_colima_module + changed_when: false + failed_when: false + check_mode: false + +- name: Load the module in the VM + ansible.builtin.command: + argv: "{{ github_runner_cluster_colima_vm_command + ['sudo', 'modprobe', '--', github_runner_cluster_colima_kernel_module] }}" + environment: "{{ github_runner_cluster_runtime_environment }}" + changed_when: true + when: + - github_runner_cluster_colima_module.rc != 0 + - not ansible_check_mode + +- name: Report the module a real run would load in the VM + ansible.builtin.debug: + msg: "modprobe {{ github_runner_cluster_colima_kernel_module }} in the VM" + changed_when: true + when: + - github_runner_cluster_colima_module.rc != 0 + - ansible_check_mode + +- name: Read the sysctls in the VM + # A key that does not exist yet (a bridge key before the module is loaded, which a real run has done by now) reads as no value, which differs from the wanted one. + ansible.builtin.command: + argv: "{{ github_runner_cluster_colima_vm_command + ['sysctl', '-n', item.key] }}" + environment: "{{ github_runner_cluster_runtime_environment }}" + loop: "{{ github_runner_cluster_colima_sysctl_settings | dict2items }}" + loop_control: + label: "{{ item.key }}" + register: github_runner_cluster_colima_sysctls + changed_when: false + failed_when: false + check_mode: false + +- name: Set the sysctls in the VM whose value differs + ansible.builtin.command: + argv: "{{ github_runner_cluster_colima_vm_command + ['sudo', 'sysctl', '-w', item.item.key ~ '=' ~ item.item.value] }}" + environment: "{{ github_runner_cluster_runtime_environment }}" + loop: "{{ github_runner_cluster_colima_sysctls.results }}" + loop_control: + label: "{{ item.item.key }}" + changed_when: true + when: + - item.stdout | trim != item.item.value | string + - not ansible_check_mode + +- name: Report the sysctls a real run would set in the VM + ansible.builtin.debug: + msg: "sysctl -w {{ item.item.key }}={{ item.item.value }} in the VM" + loop: "{{ github_runner_cluster_colima_sysctls.results }}" + loop_control: + label: "{{ item.item.key }}" + changed_when: true + when: + - item.stdout | trim != item.item.value | string + - ansible_check_mode diff --git a/roles/github_runner_cluster/tasks/runtime.yml b/roles/github_runner_cluster/tasks/runtime.yml index 5e17f90c..b0e787f6 100644 --- a/roles/github_runner_cluster/tasks/runtime.yml +++ b/roles/github_runner_cluster/tasks/runtime.yml @@ -3,6 +3,8 @@ # # On a Colima host with github_runner_cluster_runtime_autostart on, it starts the profile if it is stopped, installs a per-user launchd agent that starts it at every login (a property list in the connecting user's ~/Library/LaunchAgents) and loads that agent into the user's login session. Docker Desktop cannot be started by the role, so there it only checks. The agent runs as the connecting user and the role never uses become for it, since a LaunchAgent belongs to the user whose login session loads it. # +# On a Colima host it also makes the VM load br_netfilter and set the bridge sysctls now and at every VM boot (tasks/colima_kernel.yml), whatever the autostart setting says. +# # Publishes github_runner_cluster_effective_runtime ({kind, colima_profile}), github_runner_cluster_runtime_environment (what docker and colima run with), github_runner_cluster_runtime_down_message and github_runner_cluster_runtime_up, for end_checks.yml. - name: Read the connecting user and its environment @@ -128,3 +130,10 @@ when: - not github_runner_cluster_runtime_up | bool - not ansible_check_mode + +- name: Make the Colima VM load br_netfilter and set the bridge sysctls + # After the failure above, so the VM answers (or this is a check run that would have started it, which has no VM to read yet). Whatever the autostart setting says: the node needs the module in the VM however the VM was started. See tasks/colima_kernel.yml. + ansible.builtin.include_tasks: colima_kernel.yml + when: + - github_runner_cluster_effective_runtime.kind == 'colima' + - github_runner_cluster_runtime_up | bool diff --git a/tests/unit/test_runtime_autostart.py b/tests/unit/test_runtime_autostart.py index 54a75322..5b384105 100644 --- a/tests/unit/test_runtime_autostart.py +++ b/tests/unit/test_runtime_autostart.py @@ -27,6 +27,8 @@ UID = 501 HOME = "/Users/example" DARWIN_ONLY = "ansible_facts['system'] == 'Darwin'" +MODULES_FILE = "/etc/modules-load.d/br_netfilter.conf" +SYSCTL_FILE = "/etc/sysctl.d/90-github-runner-br-netfilter.conf" def ansible_env() -> dict[str, str]: @@ -76,6 +78,27 @@ def test_every_new_variable_is_in_the_readme(self) -> None: class GuardTest(unittest.TestCase): + def test_the_vm_needs_the_module_and_both_bridge_sysctls_on(self) -> None: + self.assertEqual(DEFAULTS["github_runner_cluster_colima_kernel_module"], "br_netfilter") + self.assertEqual(DEFAULTS["github_runner_cluster_colima_sysctl_settings"], {"net.bridge.bridge-nf-call-iptables": 1, "net.bridge.bridge-nf-call-ip6tables": 1}) + self.assertEqual(DEFAULTS["github_runner_cluster_colima_bridge_proc_dir"], "/proc/sys/net/bridge") + self.assertEqual(DEFAULTS["github_runner_cluster_colima_modules_load_file"], MODULES_FILE) + self.assertEqual(DEFAULTS["github_runner_cluster_colima_sysctl_file"], SYSCTL_FILE) + + def test_the_vm_is_prepared_last_in_the_runtime_tasks_and_only_for_colima_once_it_answers(self) -> None: + tasks = yaml.safe_load((ROLE / "tasks" / "runtime.yml").read_text()) + names = [task.get("ansible.builtin.include_tasks") or ("fail" if "ansible.builtin.fail" in task else None) for task in tasks] + self.assertEqual(names[-2:], ["fail", "colima_kernel.yml"]) + self.assertEqual(tasks[-1]["when"], ["github_runner_cluster_effective_runtime.kind == 'colima'", "github_runner_cluster_runtime_up | bool"]) + self.assertNotIn("github_runner_cluster_runtime_autostart", json.dumps(tasks[-1])) + + def test_the_readmes_say_what_the_role_does_to_the_vm(self) -> None: + readme = (ROLE / "README.md").read_text() + self.assertIn("**Colima kernel.**", readme) + self.assertIn(MODULES_FILE, readme) + self.assertIn(SYSCTL_FILE, readme) + self.assertIn("colima_kernel.yml", (ROOT / "README.md").read_text()) + def test_both_includes_run_on_macos_only(self) -> None: tasks = yaml.safe_load((ROLE / "tasks" / "main.yml").read_text()) blocks = [task for task in tasks if "block" in task] @@ -201,7 +224,7 @@ def test_the_comment_cannot_end_the_document_early(self) -> None: self.assertNotIn(b"--", comment.group(1)) -# Stands in for docker, colima, launchctl and plutil, which are all this one script under four names. State is files in $RUNTIME_STUB: runtime-up (the Docker daemon answers), session (the user has a login session), loaded (the agent is loaded), .state (what docker ps prints), colima-start-fails; every call is appended to calls.log with the Docker context it ran with. +# Stands in for docker, colima, launchctl and plutil, which are all this one script under four names. State is files in $RUNTIME_STUB: runtime-up (the Docker daemon answers), session (the user has a login session), loaded (the agent is loaded), .state (what docker ps prints), colima-start-fails; every call is appended to calls.log with the Docker context it ran with. The Colima VM is the directory vm in $RUNTIME_STUB: its files are under vm/files, /proc/sys/net/bridge exists under vm/files only while br_netfilter is loaded, and a sysctl is the file vm/sysctl/ (a bridge key reads 1 once the module is loaded, as the kernel's default, unless the file says otherwise). STUB = """#!{python} import os, sys stub = os.environ["RUNTIME_STUB"] @@ -224,6 +247,45 @@ def touch(file): print(open(os.path.join(stub, container + ".state")).read(), end="") else: sys.exit("unexpected docker call: " + " ".join(args)) +elif name == "colima" and args[0] == "ssh": + if not exists("runtime-up"): + sys.exit("colima: the VM is not running") + command = args[args.index("--") + 1:] + sudo = command[0] == "sudo" + if sudo: + command = command[1:] + vm = os.path.join(stub, "vm") + def vm_file(path): + return os.path.join(vm, "files", path.lstrip("/")) + def sysctl_file(key): + return os.path.join(vm, "sysctl", key) + if command[0] == "cat": + if not os.path.exists(vm_file(command[1])): + sys.exit("cat: " + command[1] + ": No such file or directory") + print(open(vm_file(command[1])).read(), end="") + elif command[0] == "test" and command[1] == "-d": + sys.exit(0 if os.path.isdir(vm_file(command[2])) else 1) + elif command[0] == "sysctl" and command[1] == "-n": + if command[2].startswith("net.bridge.") and not os.path.isdir(vm_file("/proc/sys/net/bridge")): + sys.exit("sysctl: cannot stat /proc/sys/" + command[2].replace(".", "/") + ": No such file or directory") + print(open(sysctl_file(command[2])).read().strip() if os.path.exists(sysctl_file(command[2])) else "1") + elif sudo and command[0] == "sysctl" and command[1] == "-w": + key, value = command[2].split("=") + if key.startswith("net.bridge.") and not os.path.isdir(vm_file("/proc/sys/net/bridge")): + sys.exit("sysctl: cannot stat /proc/sys/" + key.replace(".", "/") + ": No such file or directory") + os.makedirs(os.path.dirname(sysctl_file(key)), exist_ok=True) + open(sysctl_file(key), "w").write(value) + elif sudo and command[0] == "modprobe": + if command[-1] != "br_netfilter": + sys.exit("modprobe: FATAL: Module " + command[-1] + " not found") + os.makedirs(vm_file("/proc/sys/net/bridge"), exist_ok=True) + elif sudo and command[0] == "tee": + os.makedirs(os.path.dirname(vm_file(command[1])), exist_ok=True) + content = sys.stdin.read() + open(vm_file(command[1]), "w").write(content) + print(content, end="") + else: + sys.exit("unexpected colima ssh call: " + " ".join(command)) elif name == "colima": if args[0] != "start" or args[-1] != "--activate=false": sys.exit("unexpected colima call: " + " ".join(args)) @@ -307,7 +369,14 @@ def calls(self) -> list[str]: return log.read_text().splitlines() if log.exists() else [] def changes(self) -> list[str]: - return [call for call in self.calls() if re.match(r"\S+ (colima|launchctl (bootstrap|bootout))", call)] + return [call for call in self.calls() if re.match(r"\S+ (colima start|launchctl (bootstrap|bootout))", call)] + + def vm_changes(self) -> list[str]: + """The commands run as root in the Colima VM, in order, without the colima ssh prefix.""" + return [match.group(1) for call in self.calls() if (match := re.match(r"\S+ colima ssh --profile \S+ -- (sudo .*)", call))] + + def vm_file(self, path: str) -> Path: + return self.stub / "vm" / "files" / path.lstrip("/") def recap(self, result: subprocess.CompletedProcess[str]) -> dict[str, int]: match = re.search(r"node1\s+: ok=(\d+)\s+changed=(\d+)\s+unreachable=(\d+)\s+failed=(\d+)", result.stdout) @@ -338,7 +407,7 @@ def test_a_second_run_changes_nothing(self) -> None: result = self.run_role() self.assertEqual(result.returncode, 0, result.stdout + result.stderr) self.assertEqual(self.recap(result)["changed"], 0, result.stdout) - self.assertEqual([call for call in self.calls()[len(before) :] if not re.match(r"\S+ (docker (info|ps)|launchctl print)", call)], []) + self.assertEqual([call for call in self.calls()[len(before) :] if not re.match(r"\S+ (docker (info|ps)|launchctl print|colima ssh --profile \S+ -- (cat|test|sysctl -n) )", call)], []) self.assertNotIn("bootstrap", " ".join(self.calls()[len(before) :])) def test_a_running_runtime_is_not_started_again_but_still_gets_the_agent(self) -> None: @@ -441,6 +510,99 @@ def test_check_mode_reports_a_changed_definition(self) -> None: self.assertGreater(self.recap(result)["changed"], 0) self.assertIn("stop", self.plist.read_text()) + def test_a_colima_vm_without_the_module_gets_the_files_that_load_it_and_the_module(self) -> None: + self.state("runtime-up") + result = self.run_role(github_runner_cluster_runtime_autostart=False) + self.assertEqual(result.returncode, 0, result.stdout + result.stderr) + self.assertEqual(self.vm_changes(), [f"sudo tee {MODULES_FILE}", f"sudo tee {SYSCTL_FILE}", "sudo modprobe -- br_netfilter"]) + self.assertEqual(self.vm_file(MODULES_FILE).read_text().splitlines()[-1], "br_netfilter") + self.assertEqual(self.vm_file(SYSCTL_FILE).read_text().splitlines()[1:], ["net.bridge.bridge-nf-call-iptables = 1", "net.bridge.bridge-nf-call-ip6tables = 1"]) + self.assertTrue(self.vm_file("/proc/sys/net/bridge").is_dir()) + + def test_a_second_run_changes_nothing_in_the_vm(self) -> None: + self.state("runtime-up") + self.run_role(github_runner_cluster_runtime_autostart=False) + before = self.calls() + result = self.run_role(github_runner_cluster_runtime_autostart=False) + self.assertEqual(result.returncode, 0, result.stdout + result.stderr) + self.assertEqual(self.recap(result)["changed"], 0, result.stdout) + self.assertEqual([call for call in self.calls()[len(before) :] if " sudo " in call], []) + + def test_the_module_is_loaded_again_when_a_restarted_vm_has_lost_it_and_the_files_are_left_alone(self) -> None: + self.state("runtime-up") + self.run_role(github_runner_cluster_runtime_autostart=False) + shutil.rmtree(self.vm_file("/proc")) + before = len(self.vm_changes()) + result = self.run_role(github_runner_cluster_runtime_autostart=False) + self.assertEqual(result.returncode, 0, result.stdout + result.stderr) + self.assertEqual(self.vm_changes()[before:], ["sudo modprobe -- br_netfilter"]) + + def test_a_loaded_module_is_not_loaded_again_but_the_files_are_still_written(self) -> None: + self.state("runtime-up") + self.vm_file("/proc/sys/net/bridge").mkdir(parents=True) + result = self.run_role(github_runner_cluster_runtime_autostart=False) + self.assertEqual(result.returncode, 0, result.stdout + result.stderr) + self.assertEqual(self.vm_changes(), [f"sudo tee {MODULES_FILE}", f"sudo tee {SYSCTL_FILE}"]) + + def test_the_file_a_person_wrote_by_hand_is_replaced_by_the_managed_one(self) -> None: + self.state("runtime-up") + self.vm_file(MODULES_FILE).parent.mkdir(parents=True) + self.vm_file(MODULES_FILE).write_text("br_netfilter\n") + self.run_role(github_runner_cluster_runtime_autostart=False) + self.assertIn(f"sudo tee {MODULES_FILE}", self.vm_changes()) + self.assertIn("Managed by the exadev.github_runner collection", self.vm_file(MODULES_FILE).read_text()) + + def test_a_bridge_sysctl_that_is_off_is_set_after_the_module_is_loaded(self) -> None: + self.state("runtime-up") + (self.stub / "vm" / "sysctl").mkdir(parents=True) + (self.stub / "vm" / "sysctl" / "net.bridge.bridge-nf-call-iptables").write_text("0") + result = self.run_role(github_runner_cluster_runtime_autostart=False) + self.assertEqual(result.returncode, 0, result.stdout + result.stderr) + self.assertEqual(self.vm_changes()[2:], ["sudo modprobe -- br_netfilter", "sudo sysctl -w net.bridge.bridge-nf-call-iptables=1"]) + + def test_the_vm_of_the_hosts_profile_is_the_one_prepared(self) -> None: + self.state("runtime-up") + self.run_role(context="colima-example", github_runner_cluster_runtime_autostart=False) + ssh_calls = [call for call in self.calls() if " colima ssh " in call] + self.assertTrue(ssh_calls) + self.assertTrue(all(call.startswith("colima-example colima ssh --profile example -- ") for call in ssh_calls), ssh_calls) + + def test_the_vm_is_prepared_with_the_agent_installed_as_well(self) -> None: + self.state("session") + result = self.run_role() + self.assertEqual(result.returncode, 0, result.stdout + result.stderr) + self.assertEqual(self.changes(), ["colima colima start --profile default --activate=false", f"colima launchctl bootstrap gui/{UID} {self.plist}"]) + self.assertEqual(self.vm_changes()[-1], "sudo modprobe -- br_netfilter") + + def test_check_mode_reports_what_the_vm_needs_and_changes_none_of_it(self) -> None: + self.state("runtime-up") + result = self.run_role("--check", github_runner_cluster_runtime_autostart=False) + self.assertEqual(result.returncode, 0, result.stdout + result.stderr) + self.assertEqual(self.vm_changes(), []) + self.assertFalse((self.stub / "vm").exists()) + for reported in (f"write {MODULES_FILE} in the VM", f"write {SYSCTL_FILE} in the VM", "modprobe br_netfilter in the VM"): + self.assertIn(reported, result.stdout) + + def test_a_vm_that_would_only_be_started_by_a_check_run_is_not_read(self) -> None: + self.state("session") + result = self.run_role("--check") + self.assertEqual(result.returncode, 0, result.stdout + result.stderr) + self.assertEqual([call for call in self.calls() if " colima ssh " in call], []) + + def test_a_runtime_that_is_not_colima_leaves_the_kernel_alone(self) -> None: + self.state("runtime-up") + for context in ("desktop-linux", "orbstack"): + with self.subTest(context=context): + result = self.run_role(context=context, github_runner_cluster_runtime_autostart=False) + self.assertEqual(result.returncode, 0, result.stdout + result.stderr) + self.assertEqual([call for call in self.calls() if " colima " in call], []) + + def test_a_stopped_runtime_that_is_not_started_fails_before_the_vm_is_touched(self) -> None: + self.state("session") + self.run_role(github_runner_cluster_runtime_autostart=False) + self.assertEqual(self.vm_changes(), []) + self.assertEqual([call for call in self.calls() if " colima " in call], []) + def test_the_end_checks_pass_with_the_runtime_up_and_the_container_running(self) -> None: self.state("runtime-up") result = self.run_role(github_runner_cluster_runtime_autostart=False)