From 0d2839c199aa1ee9b9fe556eb717dbecce20bd28 Mon Sep 17 00:00:00 2001 From: Joseph Mearman Date: Tue, 3 Nov 2026 14:02:55 +0000 Subject: [PATCH 1/3] fix(cluster): name only the Compose files the host has when checking a deploy Under --check the role writes nothing, so a Compose file it would add to a host that had run it before was not there, and the read-only compose command and the Compose module that named it failed the check: docker-compose.node-labels.yml, for a host turning the role controller on. A check run now leaves out of every Compose command the files the role writes and the host lacks, a server that would gain one is reported as recreated by name, and a Compose file of the user's own that is missing is still an error. A real run names all of them, having written them first. The agent's start moves to a task file of its own so that it can be run as a server's is. The test stand-ins now fail on a missing Compose file as Compose and its module do. --- roles/github_runner_cluster/README.md | 2 +- roles/github_runner_cluster/defaults/main.yml | 11 ++ .../tasks/compose_files_in_use.yml | 18 ++ roles/github_runner_cluster/tasks/main.yml | 32 +--- .../tasks/mesh/headscale_in_cluster.yml | 2 +- .../tasks/regrade_service.yml | 2 +- .../tasks/server_recreate_check.yml | 7 +- .../tasks/start_agent.yml | 29 +++ .../tasks/start_server.yml | 2 +- tests/unit/cluster_stand_ins.py | 18 ++ .../unit/test_cluster_compose_files_in_use.py | 166 ++++++++++++++++++ tests/unit/test_server_recreate.py | 9 +- 12 files changed, 262 insertions(+), 36 deletions(-) create mode 100644 roles/github_runner_cluster/tasks/compose_files_in_use.yml create mode 100644 roles/github_runner_cluster/tasks/start_agent.yml create mode 100644 tests/unit/test_cluster_compose_files_in_use.py diff --git a/roles/github_runner_cluster/README.md b/roles/github_runner_cluster/README.md index b82e22d3..f15ca8a8 100644 --- a/roles/github_runner_cluster/README.md +++ b/roles/github_runner_cluster/README.md @@ -66,7 +66,7 @@ The volume is named from the Compose project, so changing `github_runner_cluster Compose recreates a container when the hash it stored on it (the `com.docker.compose.config-hash` label) differs from the hash of the configuration it renders now. Before it starts a server, the role makes the same comparison, between the container's label and `docker compose config --hash '*'` for the files and `.env` on the host: -- With `--check`, a server that would be recreated is reported as a change, with the reason (the two hashes, shortened). A hash is read from the files already on the host, so a difference that only this run's own copy of the Compose file or `.env` would create is visible once they are written, not under `--check`. +- With `--check`, a server that would be recreated is reported as a change, with the reason (the two hashes, shortened). A hash is read from the files already on the host, so a difference that only this run's own copy of the Compose file or `.env` would create is visible once they are written, not under `--check`. A Compose file the role would add to a host that has run it before, such as `docker-compose.node-labels.yml` for a host that turns the role controller on, is not there under `--check`, so every Compose command and the Compose modules are given only the files the host has (`github_runner_cluster_compose_files_in_use`), a server that would gain a file is reported as recreated by name, and a Compose file of your own that is missing is still an error. In a real run the role writes its files first and names all of them. - A running server is recreated only while every other server reports Ready, read through the server's own container. Recreating it stops its etcd member, and the cluster must keep its quorum meanwhile. When another server is not Ready, is not registered or cannot be read, the run fails for that host with the server named and nothing changed, under `--check` too. While none of the other servers is registered yet (the bootstrap server of a cluster that is still being brought up, restarted before anything has joined it) there is no quorum to keep, so that recreate goes ahead and is not waited for. To recreate a server anyway when the others cannot come back, run `docker compose up -d` for it by hand, as in the upgrade steps under [Versions](#versions). A server whose container is stopped is not a member of the quorum, so it is recreated without the check. Agents are exempt from it. - The servers take turns, one at a time and in no fixed order: each turn checks, starts the server and, when it recreated a running container, waits for that server's node to report Ready on a heartbeat from the new container, meaning its lease (`kube-node-lease/`) was renewed after the container started, since the API keeps the old container's Ready condition for a while (for as long as `github_runner_cluster_server_ready_timeout` allows) before the next server's turn begins, so no two servers are ever away together and a server that does not return stops the run with the others untouched. This relies on Ansible's linear strategy, the default; under `strategy: free` hosts do not wait for each other and nothing keeps the turns apart. diff --git a/roles/github_runner_cluster/defaults/main.yml b/roles/github_runner_cluster/defaults/main.yml index da3e3a49..00ccd7ea 100644 --- a/roles/github_runner_cluster/defaults/main.yml +++ b/roles/github_runner_cluster/defaults/main.yml @@ -178,6 +178,17 @@ github_runner_cluster_compose_file_list: >- + (['docker-compose.node-labels.yml'] if github_runner_cluster_node_labels | length > 0 else []) + github_runner_cluster_compose_files + (['docker-compose.host-network.yml'] if github_runner_cluster_network_mode == 'host' else []) }} +# Derived, never set directly: the Compose files this role writes into github_runner_cluster_dir itself, by the name the list above gives them. A user's own entry in github_runner_cluster_compose_files is not among them, and a missing one of those is an error in a check run as in a real one. +github_runner_cluster_compose_files_written: + - docker-compose.yml + - docker-compose.api-port.yml + - docker-compose.node-labels.yml + - docker-compose.host-network.yml + - docker-compose.mesh.yml +# Derived, never set directly: the Compose files a Compose command run now can name. Every file in the list above in a real run, since the run has written the ones it owns before any Compose command; under --check, which writes nothing, tasks/compose_files_in_use.yml leaves out the ones this role would write and the host does not have yet. Compose refuses a command that names a file that is not there, and a check run that fails on one fails the README's promise that --check runs before every deploy. +github_runner_cluster_compose_files_in_use: "{{ github_runner_cluster_compose_file_list }}" +# Derived, never set directly: the files this role writes that a check run left out of the list above because the host does not have them yet; empty in a real run. +github_runner_cluster_compose_files_unwritten: [] # The Docker context this host's node runs in, pinned by name (for example "colima"). Empty, the default, follows the host's current Docker context, or DOCKER_HOST when that is set. When set, every community.docker module call in the role passes it as cli_context and every docker CLI command runs with DOCKER_CONTEXT set to it, so both always reach the same daemon, whatever the host's current context is at the time. The role fails before touching the host if the context does not exist there, or if DOCKER_HOST is set as well, and it never changes the host's current context. Set it on any host with more than one Docker engine installed: on a Mac running its node in Colima with Docker Desktop also installed, merely launching Docker Desktop makes desktop-linux the current context, and an unpinned run would then build a second, empty node in Docker Desktop instead of managing the real one. Per host, like every value here that names something on the host; the Headscale tasks read it from the inventory of the host they act on. github_runner_cluster_docker_context: "" # The names of this host's k3s server container and k3s agent container, which docker-compose.yml takes from .env. Container names are unique on a Docker daemon, so a second cluster on the same Docker hosts needs both set to names of its own (see github_runner_cluster_api_port); the defaults are the names every existing node runs under, so leaving them unset changes nothing. Changing one on a running host makes Compose recreate that container under the new name, on the same volumes, which a server's turn treats as any other recreate. diff --git a/roles/github_runner_cluster/tasks/compose_files_in_use.yml b/roles/github_runner_cluster/tasks/compose_files_in_use.yml new file mode 100644 index 00000000..466688ec --- /dev/null +++ b/roles/github_runner_cluster/tasks/compose_files_in_use.yml @@ -0,0 +1,18 @@ +--- +# Works out which of the Compose files a Compose command can name now (github_runner_cluster_compose_files_in_use) and, under --check, which the role would write and the host lacks (github_runner_cluster_compose_files_unwritten). Run right after the task that places the role's Compose files, and before any Compose command. In a real run that task has written every file the list names, so the defaults already hold the full list and nothing here changes it. Under --check nothing is written, so a file the role would add to a host that has run it before (a new override, as docker-compose.node-labels.yml is for a host that turns the role controller on) is not there, and Compose, asked for it, fails the read-only commands that run under --check and the Compose modules' own dry runs. Only files the role writes are left out: a user's own Compose file that is missing is an error in both kinds of run. + +- name: Read which of the Compose files this role writes are on the host + ansible.builtin.stat: + path: "{{ github_runner_cluster_dir }}/{{ item }}" + loop: "{{ github_runner_cluster_compose_file_list | select('in', github_runner_cluster_compose_files_written) | list }}" + register: github_runner_cluster_compose_files_stat + check_mode: false + when: ansible_check_mode + +- name: Leave out of the Compose commands the files this check run would write + ansible.builtin.set_fact: + github_runner_cluster_compose_files_unwritten: >- + {{ github_runner_cluster_compose_files_stat.results | rejectattr('stat.exists') | map(attribute='item') | list }} + github_runner_cluster_compose_files_in_use: >- + {{ github_runner_cluster_compose_file_list | reject('in', github_runner_cluster_compose_files_stat.results | rejectattr('stat.exists') | map(attribute='item') | list) | list }} + when: ansible_check_mode diff --git a/roles/github_runner_cluster/tasks/main.yml b/roles/github_runner_cluster/tasks/main.yml index 02f79ec2..003ce8e9 100644 --- a/roles/github_runner_cluster/tasks/main.yml +++ b/roles/github_runner_cluster/tasks/main.yml @@ -108,6 +108,10 @@ - {src: k3s/Dockerfile, mode: "0644"} - {src: k3s/node-entrypoint.sh, mode: "0755"} +- name: Work out which Compose files a Compose command can name + # Right after the files are placed and before any Compose command: under --check the placement above writes nothing. + ansible.builtin.include_tasks: compose_files_in_use.yml + - name: Keep this node's password across container recreation # Before the mesh step, which can start the bootstrap server's Compose project, and before any Compose task below: the first recreate that mounts the node's password volume must find the password in it. See tasks/node_password.yml. ansible.builtin.include_tasks: node_password.yml @@ -163,32 +167,8 @@ mode: "0600" no_log: true - - name: Read whether the agent's Compose project directory exists - # Under --check on a first run the directory was not created above, and Compose cannot read a project directory that does not exist. - ansible.builtin.stat: - path: "{{ github_runner_cluster_dir }}" - register: github_runner_cluster_dir_state - check_mode: false - - - name: Report that the agent would be started in a project directory this run creates - ansible.builtin.debug: - msg: >- - Would create {{ github_runner_cluster_dir }} and start the k3s agent there, as {{ github_runner_cluster_node_container }} - in the Compose project {{ github_runner_cluster_compose_project }}. - changed_when: true - when: - - ansible_check_mode - - not github_runner_cluster_dir_state.stat.exists - - - name: Join the cluster as an agent - when: not (ansible_check_mode and not github_runner_cluster_dir_state.stat.exists) - community.docker.docker_compose_v2: - project_src: "{{ github_runner_cluster_dir }}" - project_name: "{{ github_runner_cluster_compose_project }}" - profiles: ["agent"] - cli_context: "{{ github_runner_cluster_docker_cli_context or omit }}" - files: "{{ github_runner_cluster_compose_file_list if github_runner_cluster_compose_file_list | length > 1 else omit }}" - state: present + - name: Start this agent with Docker Compose + ansible.builtin.include_tasks: start_agent.yml - name: Check the runtime and this node's container are up ansible.builtin.include_tasks: end_checks.yml diff --git a/roles/github_runner_cluster/tasks/mesh/headscale_in_cluster.yml b/roles/github_runner_cluster/tasks/mesh/headscale_in_cluster.yml index 4ef1a76f..9b3c8eb1 100644 --- a/roles/github_runner_cluster/tasks/mesh/headscale_in_cluster.yml +++ b/roles/github_runner_cluster/tasks/mesh/headscale_in_cluster.yml @@ -137,7 +137,7 @@ project_name: "{{ github_runner_cluster_compose_project }}" profiles: ["server"] cli_context: "{{ github_runner_cluster_docker_cli_context or omit }}" - files: "{{ github_runner_cluster_compose_file_list }}" + files: "{{ github_runner_cluster_compose_files_in_use }}" state: present - name: Wait for Headscale to answer inside the bootstrap server's k3s container diff --git a/roles/github_runner_cluster/tasks/regrade_service.yml b/roles/github_runner_cluster/tasks/regrade_service.yml index 269efd02..4620598e 100644 --- a/roles/github_runner_cluster/tasks/regrade_service.yml +++ b/roles/github_runner_cluster/tasks/regrade_service.yml @@ -38,7 +38,7 @@ profiles: ["regrade"] services: ["regrade"] cli_context: "{{ github_runner_cluster_docker_cli_context or omit }}" - files: "{{ github_runner_cluster_compose_file_list if github_runner_cluster_compose_file_list | length > 1 else omit }}" + files: "{{ github_runner_cluster_compose_files_in_use if github_runner_cluster_compose_files_in_use | length > 1 else omit }}" recreate: "{{ 'always' if github_runner_cluster_regrade_scripts.changed or github_runner_cluster_regrade_settings.changed else 'auto' }}" state: present diff --git a/roles/github_runner_cluster/tasks/server_recreate_check.yml b/roles/github_runner_cluster/tasks/server_recreate_check.yml index 4392ee5d..db6362a9 100644 --- a/roles/github_runner_cluster/tasks/server_recreate_check.yml +++ b/roles/github_runner_cluster/tasks/server_recreate_check.yml @@ -1,5 +1,5 @@ --- -# Decides, before Docker Compose touches this server's container, whether that run would recreate it, and refuses when the recreate would take quorum away. Compose recreates a container whose configuration hash (the com.docker.compose.config-hash label it put on the container) differs from the hash of the configuration it renders now, so the role compares the two the same way, from the files on the host. A server's recreate stops its etcd member, so it goes ahead only while every other server reports Ready; the caller (tasks/start_server.yml) takes one server's turn at a time, so two servers are never away together. Publishes github_runner_cluster_recreate_reason (empty when Compose would keep the container, or there is none), github_runner_cluster_recreate_running (whether the container that would be recreated is running) and github_runner_cluster_recreate_held (whether other servers are registered and so hold quorum for it). Every task only reads; under --check the same answers are reported, as a change when a container would be recreated and as a failure when the run would refuse. Compares what the files on the host hold now, so a difference this same run would only write (the Compose file or .env a task before this one has just changed, which --check does not write) shows once they are written. +# Decides, before Docker Compose touches this server's container, whether that run would recreate it, and refuses when the recreate would take quorum away. Compose recreates a container whose configuration hash (the com.docker.compose.config-hash label it put on the container) differs from the hash of the configuration it renders now, so the role compares the two the same way, from the files on the host. A server's recreate stops its etcd member, so it goes ahead only while every other server reports Ready; the caller (tasks/start_server.yml) takes one server's turn at a time, so two servers are never away together. Publishes github_runner_cluster_recreate_reason (empty when Compose would keep the container, or there is none), github_runner_cluster_recreate_running (whether the container that would be recreated is running) and github_runner_cluster_recreate_held (whether other servers are registered and so hold quorum for it). Every task only reads; under --check the same answers are reported, as a change when a container would be recreated and as a failure when the run would refuse. Compares what the files on the host hold now, so a difference this same run would only write (the .env or a Compose file a task before this one has just changed, which --check does not write) shows once they are written. A Compose file the role would add to the host is named in Compose's rendering by leaving it out (github_runner_cluster_compose_files_in_use) and in the reason by name, since adding an override file always changes the configuration. - name: Read the state and Compose configuration hash of this server's container # docker ps rather than docker inspect, so that a container that does not exist reads as absent while a daemon that cannot be reached fails the run. The label is empty on a container Compose did not make. @@ -20,7 +20,7 @@ ansible.builtin.command: argv: >- {{ ['docker', 'compose', '--project-name', github_runner_cluster_compose_project, '--project-directory', github_runner_cluster_dir] - + (github_runner_cluster_compose_file_list | map('regex_replace', '^(?!/)', github_runner_cluster_dir ~ '/') | map('regex_replace', '^', '--file=') | list) + + (github_runner_cluster_compose_files_in_use | map('regex_replace', '^(?!/)', github_runner_cluster_dir ~ '/') | map('regex_replace', '^', '--file=') | list) + ['--profile', github_runner_cluster_effective_node_role, 'config', '--hash', '*'] }} environment: "{{ github_runner_cluster_docker_cli_environment }}" register: github_runner_cluster_recreate_rendered @@ -36,7 +36,8 @@ github_runner_cluster_recreate_reason: >- {{ ((github_runner_cluster_recreate_lines | select('match', '^hash=') | first | regex_replace('^hash=', '')) | exadev.github_runner.cluster_recreate_reason(github_runner_cluster_recreate_rendered.stdout - | exadev.github_runner.cluster_rendered_config_hash(github_runner_cluster_node_service))) + | exadev.github_runner.cluster_rendered_config_hash(github_runner_cluster_node_service)) + or ('the configuration gains ' ~ (github_runner_cluster_compose_files_unwritten | join(', ')) ~ ', which a real run writes' if github_runner_cluster_compose_files_unwritten | length > 0 else '')) if 'state=absent' not in github_runner_cluster_recreate_lines else '' }} - name: List the other servers of the cluster diff --git a/roles/github_runner_cluster/tasks/start_agent.yml b/roles/github_runner_cluster/tasks/start_agent.yml new file mode 100644 index 00000000..6f344175 --- /dev/null +++ b/roles/github_runner_cluster/tasks/start_agent.yml @@ -0,0 +1,29 @@ +--- +# One agent's start: reads whether its Compose project directory exists, and joins the cluster with Docker Compose (tasks/start_server.yml is a server's). Under --check on a first run the role has created no directory, and Compose cannot read one that does not exist, so the start is skipped and reported instead. Compose is given the files github_runner_cluster_compose_files_in_use names (tasks/compose_files_in_use.yml), which under --check leaves out the ones the role would write and the host lacks. + +- name: Read whether the agent's Compose project directory exists + # Under --check on a first run the directory was not created above, and Compose cannot read a project directory that does not exist. + ansible.builtin.stat: + path: "{{ github_runner_cluster_dir }}" + register: github_runner_cluster_dir_state + check_mode: false + +- name: Report that the agent would be started in a project directory this run creates + ansible.builtin.debug: + msg: >- + Would create {{ github_runner_cluster_dir }} and start the k3s agent there, as {{ github_runner_cluster_node_container }} + in the Compose project {{ github_runner_cluster_compose_project }}. + changed_when: true + when: + - ansible_check_mode + - not github_runner_cluster_dir_state.stat.exists + +- name: Join the cluster as an agent + when: not (ansible_check_mode and not github_runner_cluster_dir_state.stat.exists) + community.docker.docker_compose_v2: + project_src: "{{ github_runner_cluster_dir }}" + project_name: "{{ github_runner_cluster_compose_project }}" + profiles: ["agent"] + cli_context: "{{ github_runner_cluster_docker_cli_context or omit }}" + files: "{{ github_runner_cluster_compose_files_in_use if github_runner_cluster_compose_files_in_use | length > 1 else omit }}" + state: present diff --git a/roles/github_runner_cluster/tasks/start_server.yml b/roles/github_runner_cluster/tasks/start_server.yml index 4e43d4ee..da79db36 100644 --- a/roles/github_runner_cluster/tasks/start_server.yml +++ b/roles/github_runner_cluster/tasks/start_server.yml @@ -28,7 +28,7 @@ project_name: "{{ github_runner_cluster_compose_project }}" profiles: ["server"] cli_context: "{{ github_runner_cluster_docker_cli_context or omit }}" - files: "{{ github_runner_cluster_compose_file_list if github_runner_cluster_compose_file_list | length > 1 else omit }}" + files: "{{ github_runner_cluster_compose_files_in_use if github_runner_cluster_compose_files_in_use | length > 1 else omit }}" state: present - name: Wait for this server's node to report Ready again after its container was recreated diff --git a/tests/unit/cluster_stand_ins.py b/tests/unit/cluster_stand_ins.py index aa195865..bcece27b 100644 --- a/tests/unit/cluster_stand_ins.py +++ b/tests/unit/cluster_stand_ins.py @@ -160,6 +160,11 @@ def set_node(node, status): die("Error response from daemon: No such container: " + container) absorb(filesystem(container)) elif command == "compose" and "config" in args: + # Real Compose reads every file it is given, and fails on one that is not there. + for argument in args: + if argument.startswith("--file=") and not os.path.exists(argument.removeprefix("--file=")): + die("open " + argument.removeprefix("--file=") + ": no such file or directory") + write("config-files.log", text=(read("config-files.log") or "") + args[args.index("--project-name") + 1] + ": " + ",".join(os.path.basename(a.removeprefix("--file=")) for a in args if a.startswith("--file=")) + "\\n") text = read("rendered", args[args.index("--project-name") + 1] + ".txt") if text is None: die("no configuration to render") @@ -212,6 +217,13 @@ def main(): ) stub = os.environ["DOCKER_STUB"] project = module.params["project_name"] + # The module, like Compose itself, reads the files it is named (docker-compose.yml when none) from the project directory, and fails on one that is not there, in check mode as in a real run. + named = module.params["files"] or ["docker-compose.yml"] + for name in named: + if not os.path.exists(os.path.join(module.params["project_src"], name)): + module.fail_json(msg='Cannot find Compose file "%s" relative to project directory "%s"' % (name, module.params["project_src"])) + with open(os.path.join(stub, "compose-files.log"), "a") as log: + log.write("%s: %s\\n" % (project, ",".join(named))) with open(os.path.join(stub, "calls.log"), "a") as log: log.write("module up %s profiles=%s check=%s\\n" % (project, ",".join(module.params["profiles"] or []), module.check_mode)) changed = False @@ -284,6 +296,7 @@ def hosts(self, servers: tuple[str, ...], agents: tuple[str, ...] = (), **extra: server = name in servers # Every earlier run of the role created the node's Compose project directory, which the role reads before starting Compose under --check. (self.work / "nodes" / name).mkdir(parents=True, exist_ok=True) + (self.work / "nodes" / name / "docker-compose.yml").write_text("services: {}\n") hosts[name] = { "github_runner_cluster_effective_node_role": "server" if server else "agent", "github_runner_cluster_effective_node_name": name, @@ -331,6 +344,11 @@ def rendered(self, project: str, config_hash: str) -> None: def nodes(self, **status: str) -> None: self.write("cluster.nodes", text="".join(f"{name} {value}\n" for name, value in status.items())) + def compose_files(self, log: str) -> list[str]: + """The lines of a log of the Compose files each call named, as ``project: file,file`` (log is ``config-files.log`` for the commands, ``compose-files.log`` for the module).""" + path = self.stub / log + return path.read_text().splitlines() if path.exists() else [] + def calls(self) -> list[str]: log = self.stub / "calls.log" return log.read_text().splitlines() if log.exists() else [] diff --git a/tests/unit/test_cluster_compose_files_in_use.py b/tests/unit/test_cluster_compose_files_in_use.py new file mode 100644 index 00000000..1a81d94e --- /dev/null +++ b/tests/unit/test_cluster_compose_files_in_use.py @@ -0,0 +1,166 @@ +"""Tests for the Compose files the cluster role names to Compose (tasks/compose_files_in_use.yml): under --check the role writes nothing, so a file it would add to a host that has run it before (docker-compose.node-labels.yml, for a host that turns the role controller on) is not there, and a read-only Compose command or the Compose module that names it fails, which broke `--check` before a deploy on every host that had run the role. The stand-ins fail as Compose does on a file that is not there (tests/unit/cluster_stand_ins.py), and the tests run the role's own task files in check mode and in a real run, on a server and on an agent.""" + +from __future__ import annotations + +import re +import subprocess +import unittest +from pathlib import Path +from typing import Any + +import yaml +from cluster_stand_ins import HASH_NEW, HASH_OLD, ROOT, RoleTestCase + +ROLE = ROOT / "roles" / "github_runner_cluster" +LABELS = "docker-compose.node-labels.yml" +API_PORT = "docker-compose.api-port.yml" +WITH_LABELS = ["docker-compose.yml", LABELS] + + +class FilesInUseTest(RoleTestCase): + def on_host(self, node: str, *files: str) -> None: + """Puts these Compose files in the node's project directory, as a host that has run the role before has.""" + (self.work / "nodes" / node).mkdir(parents=True, exist_ok=True) + for name in files: + (self.work / "nodes" / node / name).write_text("services: {}\n") + + def server(self, *extra: str, check: bool, tasks: str = "compose_files_in_use,start_servers", **hostvars: Any) -> subprocess.CompletedProcess[str]: + hosts, group_vars = self.hosts(("node1", "node2", "node3"), **hostvars) + return self.run_tasks(tasks, hosts, group_vars, *extra, check=check, limit="node1") + + def agent(self, *extra: str, check: bool, **hostvars: Any) -> subprocess.CompletedProcess[str]: + hosts, group_vars = self.hosts(("node1", "node2", "node3"), ("agent1",), **hostvars) + return self.run_tasks("compose_files_in_use,start_agent", hosts, group_vars, *extra, check=check, limit="agent1") + + def running_server(self) -> None: + self.container("node1-k3s", config_hash=HASH_OLD) + self.rendered("node1", HASH_NEW) + self.nodes(node1="True", node2="True", node3="True") + + +class CheckModeTest(FilesInUseTest): + def test_a_server_check_run_names_only_the_files_the_host_has_and_says_it_would_gain_the_rest(self) -> None: + self.running_server() + self.on_host("node1") + result = self.server(check=True, github_runner_cluster_compose_file_list=WITH_LABELS) + self.assertEqual(result.returncode, 0, result.stdout + result.stderr) + self.assertEqual(self.compose_files("config-files.log"), ["node1: docker-compose.yml"]) + self.assertIn("Compose would recreate node1-k3s on node1: the running container's Compose configuration hash", result.stdout.replace("\n", " ")) + + def test_the_reason_names_the_file_when_the_rest_of_the_configuration_is_unchanged(self) -> None: + # The hash Compose renders from the files the host has equals the container's, but the run writes a file that adds to the configuration, so the container would be recreated all the same. + self.container("node1-k3s", config_hash=HASH_NEW) + self.rendered("node1", HASH_NEW) + self.nodes(node1="True", node2="True", node3="True") + result = self.server(check=True, github_runner_cluster_compose_file_list=WITH_LABELS) + self.assertEqual(result.returncode, 0, result.stdout + result.stderr) + self.assertIn(f"Compose would recreate node1-k3s on node1: the configuration gains {LABELS}, which a real run writes", result.stdout.replace("\n", " ")) + + def test_nothing_is_reported_when_every_file_is_there_and_the_hashes_agree(self) -> None: + self.container("node1-k3s", config_hash=HASH_NEW) + self.rendered("node1", HASH_NEW) + self.nodes(node1="True", node2="True", node3="True") + self.on_host("node1", LABELS) + result = self.server(check=True, github_runner_cluster_compose_file_list=WITH_LABELS) + self.assertEqual(result.returncode, 0, result.stdout + result.stderr) + self.assertNotIn("Compose would recreate node1-k3s", result.stdout) + self.assertEqual(self.compose_files("config-files.log"), [f"node1: {','.join(WITH_LABELS)}"]) + + def test_the_compose_module_is_given_the_files_the_host_has(self) -> None: + self.running_server() + self.on_host("node1", API_PORT) + files = ["docker-compose.yml", API_PORT, LABELS] + result = self.server(check=True, github_runner_cluster_compose_file_list=files) + self.assertEqual(result.returncode, 0, result.stdout + result.stderr) + self.assertEqual(self.compose_files("compose-files.log"), [f"node1: docker-compose.yml,{API_PORT}"]) + + def test_with_only_the_base_file_left_the_module_is_given_none_and_reads_the_default(self) -> None: + self.running_server() + result = self.server(check=True, github_runner_cluster_compose_file_list=WITH_LABELS) + self.assertEqual(result.returncode, 0, result.stdout + result.stderr) + self.assertEqual(self.compose_files("compose-files.log"), ["node1: docker-compose.yml"]) + + def test_an_agent_check_run_names_only_the_files_the_host_has(self) -> None: + self.on_host("agent1") + result = self.agent(check=True, github_runner_cluster_compose_file_list=["docker-compose.yml", API_PORT, LABELS]) + self.assertEqual(result.returncode, 0, result.stdout + result.stderr) + self.assertEqual(self.compose_files("compose-files.log"), ["agent1: docker-compose.yml"]) + + def test_an_agent_with_the_file_already_there_is_given_it(self) -> None: + self.on_host("agent1", LABELS) + result = self.agent(check=True, github_runner_cluster_compose_file_list=WITH_LABELS) + self.assertEqual(result.returncode, 0, result.stdout + result.stderr) + self.assertEqual(self.compose_files("compose-files.log"), [f"agent1: {','.join(WITH_LABELS)}"]) + + def test_a_users_own_missing_compose_file_is_still_an_error_in_a_check_run(self) -> None: + self.running_server() + result = self.server(check=True, github_runner_cluster_compose_file_list=["docker-compose.yml", "compose.mine.yml"]) + self.assertNotEqual(result.returncode, 0) + self.assertIn("compose.mine.yml", result.stdout) + + def test_a_host_whose_directory_does_not_exist_yet_still_skips_the_start_and_reports_it(self) -> None: + self.running_server() + hosts, group_vars = self.hosts(("node1", "node2", "node3"), github_runner_cluster_compose_file_list=WITH_LABELS) + hosts["node1"]["github_runner_cluster_dir"] = str(self.work / "nodes" / "not-yet") + result = self.run_tasks("compose_files_in_use,start_servers", hosts, group_vars, check=True, limit="node1") + self.assertIn("Would create", result.stdout) + + +class RealRunTest(FilesInUseTest): + def test_a_real_run_names_the_whole_list_once_the_role_has_written_it(self) -> None: + self.running_server() + self.on_host("node1", *WITH_LABELS) + result = self.server(check=False, github_runner_cluster_compose_file_list=WITH_LABELS) + self.assertEqual(result.returncode, 0, result.stdout + result.stderr) + self.assertEqual(self.compose_files("config-files.log"), [f"node1: {','.join(WITH_LABELS)}"]) + self.assertEqual(self.compose_files("compose-files.log"), [f"node1: {','.join(WITH_LABELS)}"]) + + def test_a_real_run_does_not_leave_a_file_out_that_is_missing_because_that_is_an_error(self) -> None: + # The role writes its files before any Compose command, so one that is missing at a real run's Compose command is a failure to see, not one to hide. + self.running_server() + result = self.server(check=False, github_runner_cluster_compose_file_list=WITH_LABELS) + self.assertNotEqual(result.returncode, 0) + self.assertIn(LABELS, result.stdout) + + +class OrderTest(unittest.TestCase): + """The first real run writes a file before the first Compose command that names it, and no Compose command names the list directly.""" + + TASKS = ROLE / "tasks" + + def main_tasks(self) -> list[dict[str, Any]]: + return yaml.safe_load((self.TASKS / "main.yml").read_text()) + + def test_the_files_are_placed_then_the_list_is_worked_out_then_the_mesh_and_every_start_run(self) -> None: + names = [task["name"] for task in self.main_tasks()] + placed = names.index("Place the collection's Compose file and k3s build context in it") + worked_out = names.index("Work out which Compose files a Compose command can name") + self.assertEqual(worked_out, placed + 1) + for later in ("Set up the mesh the nodes join, and publish its join key and control server", "Set up and start this host as a k3s server, bootstrapping or joining", "Join as a k3s agent"): + matching = [index for index, name in enumerate(names) if name.startswith(later[:40])] + self.assertTrue(matching, later) + self.assertGreater(matching[0], worked_out, later) + + def test_no_task_names_the_whole_list_to_a_compose_command_except_where_it_is_defined_or_checked(self) -> None: + allowed = {"defaults/main.yml", "tasks/compose_files.yml", "tasks/compose_files_in_use.yml", "templates/env.j2", "templates/env-agent.j2"} + offenders = [] + for path in sorted(ROLE.rglob("*")): + relative = str(path.relative_to(ROLE)) + if path.is_file() and relative not in allowed and path.suffix in {".yml", ".j2"} and not relative.startswith(("README", "meta/", "files/")): + if re.search(r"compose_file_list", path.read_text()): + offenders.append(relative) + self.assertEqual(offenders, [], "these use the whole list where a Compose command needs github_runner_cluster_compose_files_in_use") + + def test_the_defaults_give_the_whole_list_when_nothing_has_been_worked_out(self) -> None: + defaults = yaml.safe_load((ROLE / "defaults" / "main.yml").read_text()) + self.assertEqual(defaults["github_runner_cluster_compose_files_unwritten"], []) + self.assertIn("github_runner_cluster_compose_file_list", defaults["github_runner_cluster_compose_files_in_use"]) + + def test_the_files_the_role_writes_are_the_ones_it_copies_and_the_mesh_provider_writes(self) -> None: + copied = [item["src"] for task in self.main_tasks() if task["name"].startswith("Place the collection's Compose file") for item in task["loop"] if item["src"].startswith("docker-compose.")] + defaults = yaml.safe_load((ROLE / "defaults" / "main.yml").read_text()) + self.assertCountEqual(copied + ["docker-compose.mesh.yml"], defaults["github_runner_cluster_compose_files_written"]) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/unit/test_server_recreate.py b/tests/unit/test_server_recreate.py index b1b21348..6281d918 100644 --- a/tests/unit/test_server_recreate.py +++ b/tests/unit/test_server_recreate.py @@ -5,6 +5,7 @@ import importlib.util import itertools import re +import shutil import subprocess import tempfile import unittest @@ -436,7 +437,7 @@ def test_check_mode_on_a_first_run_reports_the_start_it_cannot_ask_compose_about # A first run's dry run creates no project directory, and Compose cannot read a project directory that does not exist. hosts, group_vars = self.hosts(("node1", "node2", "node3")) for name in hosts: - (self.work / "nodes" / name).rmdir() + shutil.rmtree(self.work / "nodes" / name) result = self.run_tasks("start_servers", hosts, group_vars, check=True) self.assertEqual(result.returncode, 0, result.stdout + result.stderr) for n in (1, 2, 3): @@ -487,8 +488,10 @@ def test_a_volume_added_to_a_service_changes_only_that_service_s_hash(self) -> N class TaskStructureTest(unittest.TestCase): def test_servers_start_through_their_turns_and_agents_do_not(self) -> None: tasks = flatten(role_tasks("main.yml")) - starts = [task for task in tasks if "community.docker.docker_compose_v2" in task] - self.assertEqual([task["community.docker.docker_compose_v2"]["profiles"] for task in starts], [["agent"]]) + self.assertEqual([task for task in tasks if "community.docker.docker_compose_v2" in task], [], "a Compose start is in a task file of its own, which --check and the role's tests can run") + self.assertIn("start_agent.yml", [task.get("ansible.builtin.include_tasks") for task in tasks]) + (start,) = [task for task in flatten(role_tasks("start_agent.yml")) if "community.docker.docker_compose_v2" in task] + self.assertEqual(start["community.docker.docker_compose_v2"]["profiles"], ["agent"]) self.assertIn("start_servers.yml", [task.get("ansible.builtin.include_tasks") for task in tasks]) self.assertNotIn("server_recreate_check.yml", [task.get("ansible.builtin.include_tasks") for task in tasks], "agents are exempt from the quorum guard") From bbde1ba8cd30baeee4d6f0b8fa96710c08354f30 Mon Sep 17 00:00:00 2001 From: Joseph Mearman Date: Tue, 3 Nov 2026 14:33:10 +0000 Subject: [PATCH 2/3] test(cluster): follow the agent's Compose start into its own task file The guard on the order of the runtime's end checks looked for the agent's Compose task in the role's main task file, where it was before the agent's start became a task file of its own. --- tests/unit/test_runtime_autostart.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/tests/unit/test_runtime_autostart.py b/tests/unit/test_runtime_autostart.py index d80c6564..1b109484 100644 --- a/tests/unit/test_runtime_autostart.py +++ b/tests/unit/test_runtime_autostart.py @@ -146,8 +146,8 @@ def test_the_end_checks_follow_the_compose_start_on_servers_and_agents(self) -> tasks = yaml.safe_load((ROLE / "tasks" / "main.yml").read_text()) for role in ("server", "agent"): [block] = [task for task in tasks if "block" in task and task.get("when") == f'github_runner_cluster_effective_node_role == "{role}"'] - # A server starts through its turn (tasks/start_servers.yml), which runs the Compose start; an agent starts with the Compose task itself. - steps = [("compose" if step.get("ansible.builtin.include_tasks") == "start_servers.yml" else step.get("ansible.builtin.include_tasks")) or ("compose" if "community.docker.docker_compose_v2" in step else None) for step in block["block"]] + # A server starts through its turn (tasks/start_servers.yml) and an agent through tasks/start_agent.yml, each of which runs the Compose start. + steps = [("compose" if step.get("ansible.builtin.include_tasks") in ("start_servers.yml", "start_agent.yml") else step.get("ansible.builtin.include_tasks")) or ("compose" if "community.docker.docker_compose_v2" in step else None) for step in block["block"]] with self.subTest(role=role): self.assertLess(steps.index("compose"), steps.index("end_checks.yml")) From 05cc54bf975180d47da8a311dda02aa72cdd8734 Mon Sep 17 00:00:00 2001 From: Joseph Mearman Date: Tue, 3 Nov 2026 14:35:21 +0000 Subject: [PATCH 3/3] test(role): check a deploy on hosts that lack the node-labels Compose file The role_controller mesh scenario removes docker-compose.node-labels.yml from every node's project directory as a host that ran the role before the controller was on does, and runs the role with --check against real Docker: it must pass, write nothing and restart nothing. The real run after it writes the file first and recreates no container. --- tests/mesh/run.sh | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/tests/mesh/run.sh b/tests/mesh/run.sh index d34abec7..8a1df673 100755 --- a/tests/mesh/run.sh +++ b/tests/mesh/run.sh @@ -1148,6 +1148,22 @@ scenario_role_controller() { if kubectl_on 1 get node "$(node_name 4)" -o 'jsonpath={.metadata.labels}' | grep -F 'node-role.kubernetes.io/etcd' >/dev/null; then fail "the fourth node is a server"; fi ids=$(etcd_member_ids 1) + log "A check run on hosts that lack the node-labels Compose file, as a host that ran the role before the controller was turned on does, passes and writes nothing" + local before_started + before_started="$(started_at)" + for index in $(seq 1 "$node_count"); do rm -f "${work}/$(node_name "$index")/docker-compose.node-labels.yml"; done + run_role --check || fail "the role's check run failed on hosts without docker-compose.node-labels.yml" + for index in $(seq 1 "$node_count"); do + [ ! -e "${work}/$(node_name "$index")/docker-compose.node-labels.yml" ] || fail "the check run wrote docker-compose.node-labels.yml on $(node_name "$index")" + done + [ "$before_started" = "$(started_at)" ] || fail "the check run restarted a node" + log "The real run after it writes the file first, then names it, and recreates no container" + run_role + for index in $(seq 1 "$node_count"); do + [ -e "${work}/$(node_name "$index")/docker-compose.node-labels.yml" ] || fail "the real run did not write docker-compose.node-labels.yml on $(node_name "$index")" + done + [ "$before_started" = "$(started_at)" ] || fail "the real run restarted a node whose configuration it left as it was" + log "Building the role controller image from this checkout and loading it into every node" docker build --quiet --file "${repo_root}/fleet-role/Dockerfile" --tag "$role_controller_image" "$repo_root" >/dev/null image_file="${work}/fleet-role-image.tar"