Skip to content

feat(cluster): record a Docker host's role on the host and keep it across runs - #214

Merged
Mearman merged 7 commits into
mainfrom
feat/docker-role-recorded-on-host
Nov 4, 2026
Merged

Mearman merged 7 commits into
mainfrom
feat/docker-role-recorded-on-host

Conversation

@Mearman

@Mearman Mearman commented Nov 4, 2026 •

Copy link
Copy Markdown
Member

PR C of the decided plan in #183: a Docker host's role is recorded on the host and kept across runs, and no run changes a node while the role controller holds the regrade Lease for a voter swap.

  • One .env (templates/env.j2) holds both node services' variables; templates/env-agent.j2 is gone. Each service interpolates only its own variables, so docker compose config --hash is the same for k3s and k3s-agent as from the old per-role files and no running container is recreated.
  • An eligible Docker host's role lives in fleet-role.d/role, written from the inventory only when absent (force: false), as the native role file is. cluster_host_role in the new plugins/filter/cluster_role.py replaces cluster_native_role and decides both paths.
  • tasks/main.yml picks the profile, the container and the regrade service from the recorded role; tasks/start_servers.yml gives a turn to every play host whose recorded role is server.
  • A host whose running container's role differs from the role it is to run, or that runs both containers, or that is not eligible and has a role recorded, is refused before anything changes, with a message saying what to stop or set.
  • Every run, Docker or native, refuses while the fleet-regrade Lease is held unexpired by fleet-role (tasks/role_lease_guard.yml), and also when no server answered and one could not be read. --check reports the same refusals.
  • The regrade service runs beside a server only, so an eligible host keeps github_runner_cluster_regrade: true across a swap.

Decisions where the plan and the code needed one:

  • The Lease guard filter, cluster_role_lease_guard, sits in plugins/filter/cluster_guard.py beside cluster_lease_claim rather than in cluster_role.py. Both read the same Lease, so they now share one expiry rule (renew time, or acquire time, plus duration, as the regrade core judges it) instead of two copies; cluster_role.py holds only the role decision. The guard uses the existing github_runner_cluster_regrade_lease_name and _namespace constants rather than a second pair naming the same Lease; only the controller's holder and the swap annotation are new.
  • The plan lists start_servers.yml and regrade_service.yml, but every Docker task that chose by the inventory's role has to follow the recorded one, or an eligible host recorded as the other role is read as the wrong node. That covers instance_check.yml, running_version.yml, server_recreate_check.yml, host_network.yml, and, after the rebase onto the Colima VM size work, the shared quorum guard (restart_guard.yml) and the resize turn (colima_size_turn.yml). The quorum guard counts this host as a server only when its recorded role is server; other hosts stay the inventory's servers, and a server the inventory does not name still counts against a restart. The mesh scenarios' recreate report (tests/mesh/recreate_check.yml) runs docker_role.yml before the check, as main.yml does.
  • regrade.yml lets an eligible Docker host keep regrading on whatever role it runs, since the service follows the recorded role; a host that is not eligible still has to be an inventory server.

Discrimination proof, run in a throwaway clone of this branch: each rule's production code broken minimally, its test run (red), the bytes restored with git diff empty, the test run again (green).

Rule Break Test Red Green
An eligible host keeps its recorded role over the inventory cluster_host_role returns the inventory's role test_an_eligible_host_keeps_the_role_it_records_over_the_inventory fail pass
The role is written only when absent force: true in docker_role_record.yml test_the_role_is_recorded_for_an_eligible_host_only_and_only_when_absent fail pass
The role is recorded for an eligible host only the record's when made true same fail pass
The servers' turns follow the recorded role turn list selects on the inventory's role test_the_servers_turns_follow_the_recorded_role fail pass
A role mismatch is refused mismatch condition made false test_a_host_that_is_not_eligible_and_runs_the_other_role_is_refused fail pass
Both containers running is refused threshold raised past two test_both_containers_running_is_refused fail pass
A run is refused while fleet-role holds the Lease (Docker) the guard reports a held Lease as free test_every_run_is_refused_while_the_role_controller_holds_the_lease fail pass
The same refusal on a native host the guard's include skipped in native/main.yml test_a_run_is_refused_before_anything_is_changed_while_the_role_controller_holds_the_lease fail pass
An expired Lease does not refuse expiry never reached test_a_lease_regrading_holds_or_one_that_has_run_out_lets_the_run_go_on fail pass
A cluster that cannot be read refuses refusal only on held test_a_cluster_that_cannot_be_read_refuses_and_one_that_is_not_running_does_not fail pass
--check reports the refusals check_mode: false removed from the Docker Lease read test_check_mode_reports_the_refusals fail pass
The regrade service runs beside a server only role condition dropped from regrade_service.yml test_the_regrade_service_is_removed_from_a_host_recorded_as_an_agent_and_never_started fail pass
One .env holds both services' variables the agent's node name rendered for an agent only test_there_is_one_template_and_it_is_the_same_for_either_role fail pass
The merged .env keeps each service's configuration hash the agent's server URL changed test_each_service_renders_as_it_did_from_the_env_its_role_got_before fail pass
The quorum guard counts this host by its recorded role this host always counted as a server test_an_eligible_host_recorded_as_an_agent_is_read_through_the_other_servers_and_not_itself fail pass

The first run of the proof found the write-only-when-absent test green with force: true, because the recorded file already held exactly what the run would write. The test now starts from a record with different bytes, and the row above is from the rerun.

Refs #183

…ross runs

One env file replaces the per-role pair, the recorded role picks the Compose profile, and a role
run refuses to change a unit or container while the controller holds the fleet-regrade Lease.
Part of the voter swap work: a role chosen by the controller survives the next role run.
The stand-ins, the env rendering tests and the role tests follow the single recorded role file.
… guard's expiry rule

The guard that refuses a run during a voter swap reads the same fleet-regrade Lease
that a Colima VM resize claims, so it now sits beside cluster_lease_claim in cluster_guard.py
and both take the expiry from one helper.
The Lease's name and namespace come from the regrade constants instead of a second copy,
and cluster_role.py keeps only the role decision.
…ima resize

The restart guard and the Colima resize turn chose the server reads and the readiness wait
by the inventory's role, so an eligible host recorded as the other role was read as the wrong node.
They now take the role the host runs, as every other Docker task does.
@Mearman
Mearman force-pushed the feat/docker-role-recorded-on-host branch from 6fbc35e to b95d31c Compare November 4, 2026 05:45
…s quorum

The quorum guard took this host's server status from the inventory,
so an eligible host recorded as an agent was counted as a voter it no longer is
and a move between networks or a Colima resize was refused for missing etcd reads.
The record test now starts from a role file without the trailing newline a run writes,
so a record that is replaced changes its bytes and the test fails.
…its container

The mesh scenarios run the role's recreate check on its own, and that check now names the profile
and container of the role tasks/docker_role.yml decides, so the playbook runs that step first.
@Mearman
Mearman marked this pull request as ready for review November 4, 2026 06:53
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 4, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
🔒 Security Review ✅ Completed 2026-11-04T06:57:44.067353Z 9be3e6a Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@Mearman
Mearman merged commit 8746be2 into main Nov 4, 2026
33 checks passed
@Mearman
Mearman deleted the feat/docker-role-recorded-on-host branch November 4, 2026 06:58
@github-actions

github-actions Bot commented Nov 4, 2026

Copy link
Copy Markdown

🎉 This PR is included in version 2.36.0 🎉

The release is available on:

Your semantic-release bot 📦🚀

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant