Repository navigation
feat(cluster): record a Docker host's role on the host and keep it across runs - #214
Merged
Merged
Conversation
…ross runs One env file replaces the per-role pair, the recorded role picks the Compose profile, and a role run refuses to change a unit or container while the controller holds the fleet-regrade Lease. Part of the voter swap work: a role chosen by the controller survives the next role run.
The stand-ins, the env rendering tests and the role tests follow the single recorded role file.
… guard's expiry rule The guard that refuses a run during a voter swap reads the same fleet-regrade Lease that a Colima VM resize claims, so it now sits beside cluster_lease_claim in cluster_guard.py and both take the expiry from one helper. The Lease's name and namespace come from the regrade constants instead of a second copy, and cluster_role.py keeps only the role decision.
…ima resize The restart guard and the Colima resize turn chose the server reads and the readiness wait by the inventory's role, so an eligible host recorded as the other role was read as the wrong node. They now take the role the host runs, as every other Docker task does.
Mearman
force-pushed
the
feat/docker-role-recorded-on-host
branch
from
November 4, 2026 05:45
6fbc35e to
b95d31c
Compare
…s quorum The quorum guard took this host's server status from the inventory, so an eligible host recorded as an agent was counted as a voter it no longer is and a move between networks or a Colima resize was refused for missing etcd reads.
The record test now starts from a role file without the trailing newline a run writes, so a record that is replaced changes its bytes and the test fails.
…its container The mesh scenarios run the role's recreate check on its own, and that check now names the profile and container of the role tasks/docker_role.yml decides, so the playbook runs that step first.
Mearman
marked this pull request as ready for review
November 4, 2026 06:53
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
|
🎉 This PR is included in version 2.36.0 🎉 The release is available on:
Your semantic-release bot 📦🚀 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
PR C of the decided plan in #183: a Docker host's role is recorded on the host and kept across runs, and no run changes a node while the role controller holds the regrade Lease for a voter swap.
.env(templates/env.j2) holds both node services' variables;templates/env-agent.j2is gone. Each service interpolates only its own variables, sodocker compose config --hashis the same fork3sandk3s-agentas from the old per-role files and no running container is recreated.fleet-role.d/role, written from the inventory only when absent (force: false), as the native role file is.cluster_host_rolein the newplugins/filter/cluster_role.pyreplacescluster_native_roleand decides both paths.tasks/main.ymlpicks the profile, the container and the regrade service from the recorded role;tasks/start_servers.ymlgives a turn to every play host whose recorded role isserver.fleet-regradeLease is held unexpired byfleet-role(tasks/role_lease_guard.yml), and also when no server answered and one could not be read.--checkreports the same refusals.github_runner_cluster_regrade: trueacross a swap.Decisions where the plan and the code needed one:
cluster_role_lease_guard, sits inplugins/filter/cluster_guard.pybesidecluster_lease_claimrather than incluster_role.py. Both read the same Lease, so they now share one expiry rule (renew time, or acquire time, plus duration, as the regrade core judges it) instead of two copies;cluster_role.pyholds only the role decision. The guard uses the existinggithub_runner_cluster_regrade_lease_nameand_namespaceconstants rather than a second pair naming the same Lease; only the controller's holder and the swap annotation are new.start_servers.ymlandregrade_service.yml, but every Docker task that chose by the inventory's role has to follow the recorded one, or an eligible host recorded as the other role is read as the wrong node. That coversinstance_check.yml,running_version.yml,server_recreate_check.yml,host_network.yml, and, after the rebase onto the Colima VM size work, the shared quorum guard (restart_guard.yml) and the resize turn (colima_size_turn.yml). The quorum guard counts this host as a server only when its recorded role is server; other hosts stay the inventory's servers, and a server the inventory does not name still counts against a restart. The mesh scenarios' recreate report (tests/mesh/recreate_check.yml) runsdocker_role.ymlbefore the check, asmain.ymldoes.regrade.ymllets an eligible Docker host keep regrading on whatever role it runs, since the service follows the recorded role; a host that is not eligible still has to be an inventory server.Discrimination proof, run in a throwaway clone of this branch: each rule's production code broken minimally, its test run (red), the bytes restored with
git diffempty, the test run again (green).cluster_host_rolereturns the inventory's roletest_an_eligible_host_keeps_the_role_it_records_over_the_inventoryforce: trueindocker_role_record.ymltest_the_role_is_recorded_for_an_eligible_host_only_and_only_when_absentwhenmadetruetest_the_servers_turns_follow_the_recorded_rolefalsetest_a_host_that_is_not_eligible_and_runs_the_other_role_is_refusedtest_both_containers_running_is_refusedfleet-roleholds the Lease (Docker)test_every_run_is_refused_while_the_role_controller_holds_the_leasenative/main.ymltest_a_run_is_refused_before_anything_is_changed_while_the_role_controller_holds_the_leasetest_a_lease_regrading_holds_or_one_that_has_run_out_lets_the_run_go_onheldtest_a_cluster_that_cannot_be_read_refuses_and_one_that_is_not_running_does_not--checkreports the refusalscheck_mode: falseremoved from the Docker Lease readtest_check_mode_reports_the_refusalsregrade_service.ymltest_the_regrade_service_is_removed_from_a_host_recorded_as_an_agent_and_never_started.envholds both services' variablestest_there_is_one_template_and_it_is_the_same_for_either_role.envkeeps each service's configuration hashtest_each_service_renders_as_it_did_from_the_env_its_role_got_beforetest_an_eligible_host_recorded_as_an_agent_is_read_through_the_other_servers_and_not_itselfThe first run of the proof found the write-only-when-absent test green with
force: true, because the recorded file already held exactly what the run would write. The test now starts from a record with different bytes, and the row above is from the rerun.Refs #183