fix(cluster): name only the Compose files the host has when checking a deploy

roles/github_runner_cluster/README.md

@@ -66,7 +66,7 @@ The volume is named from the Compose project, so changing `github_runner_cluster
 
 Compose recreates a container when the hash it stored on it (the `com.docker.compose.config-hash` label) differs from the hash of the configuration it renders now. Before it starts a server, the role makes the same comparison, between the container's label and `docker compose config --hash '*'` for the files and `.env` on the host:
 
-- With `--check`, a server that would be recreated is reported as a change, with the reason (the two hashes, shortened). A hash is read from the files already on the host, so a difference that only this run's own copy of the Compose file or `.env` would create is visible once they are written, not under `--check`.
+- With `--check`, a server that would be recreated is reported as a change, with the reason (the two hashes, shortened). A hash is read from the files already on the host, so a difference that only this run's own copy of the Compose file or `.env` would create is visible once they are written, not under `--check`. A Compose file the role would add to a host that has run it before, such as `docker-compose.node-labels.yml` for a host that turns the role controller on, is not there under `--check`, so every Compose command and the Compose modules are given only the files the host has (`github_runner_cluster_compose_files_in_use`), a server that would gain a file is reported as recreated by name, and a Compose file of your own that is missing is still an error. In a real run the role writes its files first and names all of them.
 - A running server is recreated only while every other server reports Ready, read through the server's own container. Recreating it stops its etcd member, and the cluster must keep its quorum meanwhile. When another server is not Ready, is not registered or cannot be read, the run fails for that host with the server named and nothing changed, under `--check` too. While none of the other servers is registered yet (the bootstrap server of a cluster that is still being brought up, restarted before anything has joined it) there is no quorum to keep, so that recreate goes ahead and is not waited for. To recreate a server anyway when the others cannot come back, run `docker compose up -d` for it by hand, as in the upgrade steps under [Versions](#versions). A server whose container is stopped is not a member of the quorum, so it is recreated without the check. Agents are exempt from it.
 - The servers take turns, one at a time and in no fixed order: each turn checks, starts the server and, when it recreated a running container, waits for that server's node to report Ready on a heartbeat from the new container, meaning its lease (`kube-node-lease/<node>`) was renewed after the container started, since the API keeps the old container's Ready condition for a while (for as long as `github_runner_cluster_server_ready_timeout` allows) before the next server's turn begins, so no two servers are ever away together and a server that does not return stops the run with the others untouched. This relies on Ansible's linear strategy, the default; under `strategy: free` hosts do not wait for each other and nothing keeps the turns apart.
 

roles/github_runner_cluster/defaults/main.yml

@@ -178,6 +178,17 @@ github_runner_cluster_compose_file_list: >-
      + (['docker-compose.node-labels.yml'] if github_runner_cluster_node_labels | length > 0 else [])
      + github_runner_cluster_compose_files
      + (['docker-compose.host-network.yml'] if github_runner_cluster_network_mode == 'host' else []) }}
+# Derived, never set directly: the Compose files this role writes into github_runner_cluster_dir itself, by the name the list above gives them. A user's own entry in github_runner_cluster_compose_files is not among them, and a missing one of those is an error in a check run as in a real one.
+github_runner_cluster_compose_files_written:
+  - docker-compose.yml
+  - docker-compose.api-port.yml
+  - docker-compose.node-labels.yml
+  - docker-compose.host-network.yml
+  - docker-compose.mesh.yml
+# Derived, never set directly: the Compose files a Compose command run now can name. Every file in the list above in a real run, since the run has written the ones it owns before any Compose command; under --check, which writes nothing, tasks/compose_files_in_use.yml leaves out the ones this role would write and the host does not have yet. Compose refuses a command that names a file that is not there, and a check run that fails on one fails the README's promise that --check runs before every deploy.
+github_runner_cluster_compose_files_in_use: "{{ github_runner_cluster_compose_file_list }}"
+# Derived, never set directly: the files this role writes that a check run left out of the list above because the host does not have them yet; empty in a real run.
+github_runner_cluster_compose_files_unwritten: []
 # The Docker context this host's node runs in, pinned by name (for example "colima"). Empty, the default, follows the host's current Docker context, or DOCKER_HOST when that is set. When set, every community.docker module call in the role passes it as cli_context and every docker CLI command runs with DOCKER_CONTEXT set to it, so both always reach the same daemon, whatever the host's current context is at the time. The role fails before touching the host if the context does not exist there, or if DOCKER_HOST is set as well, and it never changes the host's current context. Set it on any host with more than one Docker engine installed: on a Mac running its node in Colima with Docker Desktop also installed, merely launching Docker Desktop makes desktop-linux the current context, and an unpinned run would then build a second, empty node in Docker Desktop instead of managing the real one. Per host, like every value here that names something on the host; the Headscale tasks read it from the inventory of the host they act on.
 github_runner_cluster_docker_context: ""
 # The names of this host's k3s server container and k3s agent container, which docker-compose.yml takes from .env. Container names are unique on a Docker daemon, so a second cluster on the same Docker hosts needs both set to names of its own (see github_runner_cluster_api_port); the defaults are the names every existing node runs under, so leaving them unset changes nothing. Changing one on a running host makes Compose recreate that container under the new name, on the same volumes, which a server's turn treats as any other recreate.

roles/github_runner_cluster/tasks/compose_files_in_use.yml

@@ -0,0 +1,18 @@
+---
+# Works out which of the Compose files a Compose command can name now (github_runner_cluster_compose_files_in_use) and, under --check, which the role would write and the host lacks (github_runner_cluster_compose_files_unwritten). Run right after the task that places the role's Compose files, and before any Compose command. In a real run that task has written every file the list names, so the defaults already hold the full list and nothing here changes it. Under --check nothing is written, so a file the role would add to a host that has run it before (a new override, as docker-compose.node-labels.yml is for a host that turns the role controller on) is not there, and Compose, asked for it, fails the read-only commands that run under --check and the Compose modules' own dry runs. Only files the role writes are left out: a user's own Compose file that is missing is an error in both kinds of run.
+
+- name: Read which of the Compose files this role writes are on the host
+  ansible.builtin.stat:
+    path: "{{ github_runner_cluster_dir }}/{{ item }}"
+  loop: "{{ github_runner_cluster_compose_file_list | select('in', github_runner_cluster_compose_files_written) | list }}"
+  register: github_runner_cluster_compose_files_stat
+  check_mode: false
+  when: ansible_check_mode
+
+- name: Leave out of the Compose commands the files this check run would write
+  ansible.builtin.set_fact:
+    github_runner_cluster_compose_files_unwritten: >-
+      {{ github_runner_cluster_compose_files_stat.results | rejectattr('stat.exists') | map(attribute='item') | list }}
+    github_runner_cluster_compose_files_in_use: >-
+      {{ github_runner_cluster_compose_file_list | reject('in', github_runner_cluster_compose_files_stat.results | rejectattr('stat.exists') | map(attribute='item') | list) | list }}
+  when: ansible_check_mode

roles/github_runner_cluster/tasks/main.yml

@@ -108,6 +108,10 @@
     - {src: k3s/Dockerfile, mode: "0644"}
     - {src: k3s/node-entrypoint.sh, mode: "0755"}
 
+- name: Work out which Compose files a Compose command can name
+  # Right after the files are placed and before any Compose command: under --check the placement above writes nothing.
+  ansible.builtin.include_tasks: compose_files_in_use.yml
+
 - name: Keep this node's password across container recreation
   # Before the mesh step, which can start the bootstrap server's Compose project, and before any Compose task below: the first recreate that mounts the node's password volume must find the password in it. See tasks/node_password.yml.
   ansible.builtin.include_tasks: node_password.yml
@@ -163,32 +167,8 @@
         mode: "0600"
       no_log: true
 
-    - name: Read whether the agent's Compose project directory exists
-      # Under --check on a first run the directory was not created above, and Compose cannot read a project directory that does not exist.
-      ansible.builtin.stat:
-        path: "{{ github_runner_cluster_dir }}"
-      register: github_runner_cluster_dir_state
-      check_mode: false
-
-    - name: Report that the agent would be started in a project directory this run creates
-      ansible.builtin.debug:
-        msg: >-
-          Would create {{ github_runner_cluster_dir }} and start the k3s agent there, as {{ github_runner_cluster_node_container }}
-          in the Compose project {{ github_runner_cluster_compose_project }}.
-      changed_when: true
-      when:
-        - ansible_check_mode
-        - not github_runner_cluster_dir_state.stat.exists
-
-    - name: Join the cluster as an agent
-      when: not (ansible_check_mode and not github_runner_cluster_dir_state.stat.exists)
-      community.docker.docker_compose_v2:
-        project_src: "{{ github_runner_cluster_dir }}"
-        project_name: "{{ github_runner_cluster_compose_project }}"
-        profiles: ["agent"]
-        cli_context: "{{ github_runner_cluster_docker_cli_context or omit }}"
-        files: "{{ github_runner_cluster_compose_file_list if github_runner_cluster_compose_file_list | length > 1 else omit }}"
-        state: present
+    - name: Start this agent with Docker Compose
+      ansible.builtin.include_tasks: start_agent.yml
 
     - name: Check the runtime and this node's container are up
       ansible.builtin.include_tasks: end_checks.yml

roles/github_runner_cluster/tasks/mesh/headscale_in_cluster.yml

@@ -137,7 +137,7 @@
             project_name: "{{ github_runner_cluster_compose_project }}"
             profiles: ["server"]
             cli_context: "{{ github_runner_cluster_docker_cli_context or omit }}"
-            files: "{{ github_runner_cluster_compose_file_list }}"
+            files: "{{ github_runner_cluster_compose_files_in_use }}"
             state: present
 
     - name: Wait for Headscale to answer inside the bootstrap server's k3s container

roles/github_runner_cluster/tasks/regrade_service.yml

@@ -38,7 +38,7 @@
         profiles: ["regrade"]
         services: ["regrade"]
         cli_context: "{{ github_runner_cluster_docker_cli_context or omit }}"
-        files: "{{ github_runner_cluster_compose_file_list if github_runner_cluster_compose_file_list | length > 1 else omit }}"
+        files: "{{ github_runner_cluster_compose_files_in_use if github_runner_cluster_compose_files_in_use | length > 1 else omit }}"
         recreate: "{{ 'always' if github_runner_cluster_regrade_scripts.changed or github_runner_cluster_regrade_settings.changed else 'auto' }}"
         state: present
 

roles/github_runner_cluster/tasks/server_recreate_check.yml

@@ -1,5 +1,5 @@
 ---
-# Decides, before Docker Compose touches this server's container, whether that run would recreate it, and refuses when the recreate would take quorum away. Compose recreates a container whose configuration hash (the com.docker.compose.config-hash label it put on the container) differs from the hash of the configuration it renders now, so the role compares the two the same way, from the files on the host. A server's recreate stops its etcd member, so it goes ahead only while every other server reports Ready; the caller (tasks/start_server.yml) takes one server's turn at a time, so two servers are never away together. Publishes github_runner_cluster_recreate_reason (empty when Compose would keep the container, or there is none), github_runner_cluster_recreate_running (whether the container that would be recreated is running) and github_runner_cluster_recreate_held (whether other servers are registered and so hold quorum for it). Every task only reads; under --check the same answers are reported, as a change when a container would be recreated and as a failure when the run would refuse. Compares what the files on the host hold now, so a difference this same run would only write (the Compose file or .env a task before this one has just changed, which --check does not write) shows once they are written.
+# Decides, before Docker Compose touches this server's container, whether that run would recreate it, and refuses when the recreate would take quorum away. Compose recreates a container whose configuration hash (the com.docker.compose.config-hash label it put on the container) differs from the hash of the configuration it renders now, so the role compares the two the same way, from the files on the host. A server's recreate stops its etcd member, so it goes ahead only while every other server reports Ready; the caller (tasks/start_server.yml) takes one server's turn at a time, so two servers are never away together. Publishes github_runner_cluster_recreate_reason (empty when Compose would keep the container, or there is none), github_runner_cluster_recreate_running (whether the container that would be recreated is running) and github_runner_cluster_recreate_held (whether other servers are registered and so hold quorum for it). Every task only reads; under --check the same answers are reported, as a change when a container would be recreated and as a failure when the run would refuse. Compares what the files on the host hold now, so a difference this same run would only write (the .env or a Compose file a task before this one has just changed, which --check does not write) shows once they are written. A Compose file the role would add to the host is named in Compose's rendering by leaving it out (github_runner_cluster_compose_files_in_use) and in the reason by name, since adding an override file always changes the configuration.
 
 - name: Read the state and Compose configuration hash of this server's container
   # docker ps rather than docker inspect, so that a container that does not exist reads as absent while a daemon that cannot be reached fails the run. The label is empty on a container Compose did not make.
@@ -20,7 +20,7 @@
   ansible.builtin.command:
     argv: >-
       {{ ['docker', 'compose', '--project-name', github_runner_cluster_compose_project, '--project-directory', github_runner_cluster_dir]
-         + (github_runner_cluster_compose_file_list | map('regex_replace', '^(?!/)', github_runner_cluster_dir ~ '/') | map('regex_replace', '^', '--file=') | list)
+         + (github_runner_cluster_compose_files_in_use | map('regex_replace', '^(?!/)', github_runner_cluster_dir ~ '/') | map('regex_replace', '^', '--file=') | list)
          + ['--profile', github_runner_cluster_effective_node_role, 'config', '--hash', '*'] }}
   environment: "{{ github_runner_cluster_docker_cli_environment }}"
   register: github_runner_cluster_recreate_rendered
@@ -36,7 +36,8 @@
     github_runner_cluster_recreate_reason: >-
       {{ ((github_runner_cluster_recreate_lines | select('match', '^hash=') | first | regex_replace('^hash=', ''))
           | exadev.github_runner.cluster_recreate_reason(github_runner_cluster_recreate_rendered.stdout
-                                                         | exadev.github_runner.cluster_rendered_config_hash(github_runner_cluster_node_service)))
+                                                         | exadev.github_runner.cluster_rendered_config_hash(github_runner_cluster_node_service))
+          or ('the configuration gains ' ~ (github_runner_cluster_compose_files_unwritten | join(', ')) ~ ', which a real run writes' if github_runner_cluster_compose_files_unwritten | length > 0 else ''))
          if 'state=absent' not in github_runner_cluster_recreate_lines else '' }}
 
 - name: List the other servers of the cluster

roles/github_runner_cluster/tasks/start_agent.yml

@@ -0,0 +1,29 @@
+---
+# One agent's start: reads whether its Compose project directory exists, and joins the cluster with Docker Compose (tasks/start_server.yml is a server's). Under --check on a first run the role has created no directory, and Compose cannot read one that does not exist, so the start is skipped and reported instead. Compose is given the files github_runner_cluster_compose_files_in_use names (tasks/compose_files_in_use.yml), which under --check leaves out the ones the role would write and the host lacks.
+
+- name: Read whether the agent's Compose project directory exists
+  # Under --check on a first run the directory was not created above, and Compose cannot read a project directory that does not exist.
+  ansible.builtin.stat:
+    path: "{{ github_runner_cluster_dir }}"
+  register: github_runner_cluster_dir_state
+  check_mode: false
+
+- name: Report that the agent would be started in a project directory this run creates
+  ansible.builtin.debug:
+    msg: >-
+      Would create {{ github_runner_cluster_dir }} and start the k3s agent there, as {{ github_runner_cluster_node_container }}
+      in the Compose project {{ github_runner_cluster_compose_project }}.
+  changed_when: true
+  when:
+    - ansible_check_mode
+    - not github_runner_cluster_dir_state.stat.exists
+
+- name: Join the cluster as an agent
+  when: not (ansible_check_mode and not github_runner_cluster_dir_state.stat.exists)
+  community.docker.docker_compose_v2:
+    project_src: "{{ github_runner_cluster_dir }}"
+    project_name: "{{ github_runner_cluster_compose_project }}"
+    profiles: ["agent"]
+    cli_context: "{{ github_runner_cluster_docker_cli_context or omit }}"
+    files: "{{ github_runner_cluster_compose_files_in_use if github_runner_cluster_compose_files_in_use | length > 1 else omit }}"
+    state: present

roles/github_runner_cluster/tasks/start_server.yml

@@ -28,7 +28,7 @@
     project_name: "{{ github_runner_cluster_compose_project }}"
     profiles: ["server"]
     cli_context: "{{ github_runner_cluster_docker_cli_context or omit }}"
-    files: "{{ github_runner_cluster_compose_file_list if github_runner_cluster_compose_file_list | length > 1 else omit }}"
+    files: "{{ github_runner_cluster_compose_files_in_use if github_runner_cluster_compose_files_in_use | length > 1 else omit }}"
     state: present
 
 - name: Wait for this server's node to report Ready again after its container was recreated