Promote a fit standby to an etcd voter and demote an unfit voter, one swap at a time #183

With native servers, the systemd regrade actuator and the dry-run controller in place, the controller gets the power to act: promote an eligible agent to a voter and demote an unfit voter to an agent, as one swap, when FLEET_ROLE_DRY_RUN=false. Part of the dynamic regrading epic, whose body has the reasoning and sources for each step.

A swap, step by step, all under the fleet-regrade Lease, which the controller renews for as long as the swap runs:

1. Check again that every voter is at full, Ready, and that etcd has a leader and no learner or unstarted member. Take an on-demand snapshot on a voter (k3s etcd-snapshot save) and record its name.
2. Set github-runner.exadev.com/desired-role=server on the candidate's Node. The host's actuator cordons it, waits for its runner pods to finish (a planned change never cuts a job; past the operation timeout the swap is abandoned and the node uncordoned), drains it, writes the role file and a join URL naming a current voter, stops the agent unit and starts the server unit. k3s joins it as a learner and the etcd leader promotes it once it has caught up. The step succeeds when it is a voter and its Node is Ready on a heartbeat after the start.
3. Check health again, now with four voters.
4. Set desired-role=agent on the outgoing voter. Its actuator steps it to etcd-only through regrading (drained, no API server, so k3s's leader-elected controllers run elsewhere), then the controller sets etcd.k3s.cattle.io/remove=true on its Node. k3s's member controller on another server removes the member; the outgoing server's etcd logs that it was removed and writes its tombstone. The actuator then stops the server unit, moves server/db aside (keeping only the newest such copy), clears the regrade level, removes the control-plane, etcd and master role labels and k3s's etcd annotations from the Node, writes the role file and starts the agent unit, and uncordons the node once it is Ready.
5. Release the Lease and start the cooldown.

Failure at each step. A candidate that does not become a voter (k3s removes a learner stalled for five minutes, or the step times out) is rolled back: the controller removes a learner that is still listed, which never touches quorum, and the actuator stops the server unit, moves its server/db aside and starts the agent again, so membership is as it was. A failure between steps 3 and 4 leaves four healthy voters, with the same tolerance as three; the controller's next action, before any other, is to finish the removal, and it alerts through the heartbeat while the count is even. A removal the member controller rejects is retried only while the remaining voters are all healthy. If the outgoing server's agent does not come back, its member is already gone and quorum is untouched; node-recovery handles the node as any agent.

Docker hosts. Today's voters run under Compose, so the Docker side needs the same role switch: demotion stops the server service and starts the agent profile with the server's node password copied into the agent's node volume (the two services use different volumes, and a new password is rejected), and promotion is the reverse. The Compose project records the role on the host so a role run does not undo it, and the entrypoint never passes --cluster-init once the host has joined a cluster, so a former bootstrap host that is promoted again joins the existing cluster.

Interactions. The regrade core's VOTER_LABEL selection stays right because a demoted node loses the etcd label. node-recovery leaves alone the node the Lease holder names, so a node restarting into its new role is not tainted out of service mid-swap, and its prune pass sees a demoted node as an agent. The heartbeat and scripts/fleet-alert.sh report a swap in progress rather than an outage.

What stays manual. Removing a voter that is down and not coming back, restoring after quorum loss, and changing the target count. Promotion of a host the role has not pre-provisioned is refused.

Tests: the mesh harness proves every step with real k3s and real etcd: a native standby promoted and a Docker server demoted in one swap, with writes succeeding throughout and three voters at the end; the same swap with the outgoing voter paused after step 3 (the removal still completes); a candidate whose peer port is blocked (k3s removes the learner after five minutes and the rollback leaves membership unchanged); a swap refused while a voter is paused before step 1; the demoted node's labels and annotations gone, its server/db moved aside, and a later promotion of the former bootstrap host joining the existing cluster (one cluster ID before and after). Unit tests cover the controller's state machine for every failure branch. On the live fleet the first swap is run by hand with the controller's command while the owner watches, before the flag is turned off for good.

Implementation plan (decided)

Read from origin/main at 1b781aa (v2.31.0). Paths are relative to roles/github_runner_cluster/ unless they start with scripts/, tests/, plugins/ or roles/.

(a) What already exists on main, and what does not

Exists:

• Native role switching through Ansible.
  • The role file and unit guard: files/native/fleet-role-is.sh, used by ExecCondition in templates/native/k3s.service.j2:33.
  • An eligible host keeps the role its file records: cluster_native_role, plugins/filter/cluster_native.py:94-114.
  • Eligible hosts are staged as standbys: tasks/native/role.yml:46.
  • Promotion in place: the promote turn in tasks/native/server_turn.yml:65-162.
  • Demotion: tasks/native/demote.yml. It refuses unless three other Ready voters remain (line 30), writes the role file before asking for removal (54), annotates etcd.k3s.cattle.io/remove (61), waits for removed-node-name (72), moves server/db aside keeping one copy (108), and strips the role labels and etcd annotations (120).
  • Draining with a runner wait, uncordoning if it gives up: tasks/native/drain.yml.
  • A former member never bootstraps: templates/native/join.yaml.j2 and tasks/native/role.yml:61-92.
• Regrading:
  • The levels and records: files/regrade/fleet-regrade-k3s.sh, fleet-regrade-systemd.sh and fleet-regrade-docker.sh.
  • The Lease: acquire_lease, release_lease and lease_state in files/regrade/fleet-regrade-core.sh:184-250.
  • select_api_server over KUBE_SERVERS, which already lists eligible hosts (templates/regrade.env.j2:28-32).
• The dry-run controller: scripts/fleet-role.sh, scripts/fleet-role.jq and scripts/fleet-role-probe.sh. The settings checks are tasks/voters.yml. Eligible hosts' TLS SANs come from plugins/filter/cluster_topology.py:123. The heartbeat republishes fleet-role-status (scripts/heartbeat.sh:132). scripts/fleet-alert.sh:89 already ignores a server carrying a regrade level.

Does not exist:

• desired-role appears nowhere in scripts, roles or tests.
• No host-side actuator, and no process on an agent host that could act on the API at all.
• The controller cannot write. scripts/fleet-role.sh:43 refuses FLEET_ROLE_DRY_RUN other than true, and its ClusterRole is get, list on nodes (roles/github_runner_arc/templates/fleet-role.yaml.j2, node_rules).
• Nothing renews a Lease: the core acquires once, sized at three operation timeouts (defaults/main.yml:50).
• Nothing triggers a snapshot.
• scripts/node-recovery.sh never reads a Lease.
• The heartbeat has no swap fields.
• Docker hosts have no role switch or recorded role. tasks/main.yml:131-182 picks the profile and .env template from the inventory role.
• The Docker entrypoint passes --cluster-init whenever K3S_JOIN_SERVER_URL is empty (files/k3s/node-entrypoint.sh:177-182). A demoted bootstrap host with its server/db moved aside would therefore start a second cluster on its next promotion.
• No test removes a member with real k3s. The native_server mesh scenario only promotes, and demote.yml is tested only against stand-ins (tests/unit/test_native_server.py:161).

Where issue 183 is wrong or inconsistent with the code (each decided)

1. Who cordons and drains. The issue gives this to the host actuator. An agent host has no API credential that can evict pods, and its own k3s is down during the switch.

   • Decided: the controller does everything on the API: cordon, the runner wait, drain, the taint, the level label, member removal, label clean-up and uncordon.
   • The actuator does only host-local work. Its credential can read Nodes and patch one ConfigMap, nothing more.

2. The snapshot. "k3s etcd-snapshot save on a voter" cannot run from the controller pod, which has neither the k3s binary nor the server token.

   • Decided: the controller asks one remaining voter's actuator through a Node annotation.
   • The controller then checks that an ETCDSnapshotFile with that name reports readyToUse. k3s v1.34 publishes these objects.

3. Renewing the Lease. The epic says the holder renews it; the core never does. Decided: add renew_lease to fleet-regrade-core.sh, which the controller calls on every pass. A regrade step keeps its sized duration.

4. node-recovery and the Lease holder. "node-recovery leaves alone the node the Lease holder names" has nothing to build on, and a Lease held by the controller names no node.

   • Decided: a Lease annotation lists the swap's nodes.
   • node-recovery also stops tainting a server held at etcd-only by regrading. Today it applies out-of-service to one after NODE_RECOVERY_AFTER_SECONDS, and a swap's outgoing voter passes through etcd-only.

5. Copying only the node password on a Docker switch is not enough. The agent service keeps its own Tailscale state (files/docker-compose.yml:62 against :106). It would come up as a new device under the same hostname, the tailnet would give it a suffixed name, and every join URL and SAN by name would break.

   • Decided: copy the node password and the Tailscale state between the two services' volumes, with both containers stopped.

6. "VOTER_LABEL stays right because a demoted node loses the etcd label." k3s does not remove that label (demote.yml says so).

   • A learner also carries node-role.kubernetes.io/etcd before it votes, so anything that reads a voter from that label (scripts/fleet-role.jq:66, the core's VOTER_LABEL, node-recovery) counts a learner.
   • Decided: the controller removes the label in step 4, and judges a voter by EtcdIsVoter=True as well as the label.

7. Step 4 writes the role file after the removal; that is unsafe.

   • Once removed, k3s writes server/db/etcd/tombstone. On its next start, Register backs the datastore up and rejoins as a learner (pkg/etcd/etcd.go:682-689 at v1.34.1+k3s1). A unit restarted by Restart=always, or a container restarted by restart: unless-stopped, would therefore come back as a member.
   • Decided: switch the role file to agent after the etcd-only restart and before the removal, as demote.yml:54 already does.
   • Also refuse any server start while a tombstone exists. This is the only guard that holds when the outgoing voter was down during the removal.

8. The join URL alone does not change what a native server waits for. The unit's GITHUB_RUNNER_JOIN_HOST is fixed when Ansible writes it (templates/native/k3s.service.j2:16), and files/native/tailscale-up-wait.sh waits for that host without a bound. A join drop-in alone would still wait on the inventory's bootstrap host, which may by then be an agent or down. Decided: the actuator writes a persistent unit drop-in naming the same host as the join drop-in.

9. The wait for runner pods. The issue bounds it by the operation timeout; the role bounds it by github_runner_cluster_native_runner_wait_timeout (defaults/main.yml:303, tasks/native/drain.yml:29). Decided: the controller uses the runner wait timeout, so both paths agree.

10. Issue 184's claim that regrading is a prerequisite on every eligible server. The swap uses the level mechanism, which works whether or not regrading is on: the Docker entrypoint's set_regrade_flags (files/k3s/node-entrypoint.sh:57-66) and the native ExecStartPre (pinned by tests/unit/test_native_regrade.py:73). It does not use the regrade watcher. Decided: the regrade service is not a prerequisite, and Decide whether memory-pressure regrading becomes the default for servers #140 stays separate.

The PRs, in dependency order

A and B can land in any order. C needs A. D1 needs C. D2 needs D1. E1 is independent. E2 needs B, D2 and E1. F needs E2.

PR A: fix(cluster): never start a second cluster from a former member, and keep a retired server from starting

Files: files/k3s/node-entrypoint.sh, files/docker-compose.yml, files/native/fleet-role-is.sh, tasks/main.yml, the role README.md, tests/unit/test_node_entrypoint.py (new), tests/unit/test_native_server.py, tests/mesh/run.sh.

Interfaces:

• A role directory. {{ github_runner_cluster_dir }}/fleet-role.d/ is bind-mounted read-only at /etc/github-runner into both k3s services. It is a directory, not two files, because an atomic mv gives a file a new inode, which a single-file bind mount does not see.
  • It holds role (server or agent) and join-url.
  • The role creates the directory empty.
  • An absent role file behaves exactly as today.
• The entrypoint's guards. When role names the other role, or (on a server) /var/lib/rancher/k3s/server/db/etcd/tombstone exists, the entrypoint logs why and runs exec sleep infinity. It never exits, because unless-stopped would restart a container that exits.
• The join URL is the join-url file when it is not empty, otherwise K3S_JOIN_SERVER_URL.
• --cluster-init:
  • With no join URL and server/db present: passed as today (k3s ignores it).
  • With no join URL and server/db.demoted present: k3s is not started, and the container exits 1 with a message.
  • With neither: passed (a first bootstrap).
• Native: fleet-role-is.sh server also exits 1 (start skipped) while a tombstone exists.

Reuses: the existing mesh_server_args structure, and join_host_of and wait_for_mesh_host with the chosen URL.

Tests:

• Unit:
  • the join URL's precedence;
  • --cluster-init in each of the three datastore states;
  • the idle guard for the wrong role;
  • the idle guard on a tombstone;
  • no change with no role file;
  • the native condition on a tombstone.
• Mesh scenario rejoin, with real k3s and etcd:
  • stop the bootstrap Docker server;
  • move server/db to db.demoted in its volume;
  • start it with no join-url: it refuses;
  • write a join-url naming node 2: it joins as a learner and is promoted;
  • the etcd cluster ID (etcdctl endpoint status -w json) is the same before and after;
  • a role file naming agent leaves the server container idle with no k3s process.

Size: 8 files, 10 tests (9 unit, 1 mesh scenario).

PR B: feat(arc): leave alone the nodes of a role swap and servers stepped down by regrading in node-recovery

Files: scripts/node-recovery.sh, roles/github_runner_arc/templates/node-recovery.yaml.j2, roles/github_runner_arc/defaults/main.yml, tests/unit/test_node_recovery.py, tests/unit/test_node_recovery_role.py, the README.

Interfaces:

• node-recovery reads kube-system/fleet-regrade through a new Role with get on that name only.
• While the Lease is unexpired (the core's lease_state rule, reimplemented in node-recovery's jq), node-recovery leaves alone:
  • the nodes listed in its annotation github-runner.exadev.com/swap-nodes (comma-separated);
  • its holderIdentity when that is a node name.
• It never taints a node carrying github-runner.exadev.com/regrade-level.

Tests:

• exempt while held;
• acted on once expired;
• a regrade holder exempted;
• the level label exempted;
• the taint cleared as before;
• the prune pass unchanged for a node with no server labels;
• the RBAC renders scoped to the Lease name.

Size: 6 files, 7 tests.

PR C: feat(cluster): record a Docker host's role on the host and keep it across runs

Files:

• templates/env.j2, which becomes one template holding both services' variables; templates/env-agent.j2 is removed;
• plugins/filter/cluster_role.py, new: cluster_host_role, moved from cluster_native_role, which is removed with every caller updated;
• tasks/native/role.yml, tasks/docker_role.yml (new), tasks/main.yml, tasks/start_servers.yml, tasks/regrade_service.yml;
• tasks/role_lease_guard.yml, new;
• tests/unit/test_docker_role.py (new), tests/unit/test_native_node.py, the README.

Interfaces:

• An eligible Docker host's role lives in fleet-role.d/role. It is written from the inventory only when absent (force: false), as the native role file is.
• main.yml chooses the profile, the container to start and the regrade service from the recorded role.
• start_servers.yml gives a turn to every play host whose recorded role is server, not to topology.servers.
• A host that is not eligible, and whose running container's role differs from its inventory role, is refused with a message. Today the second profile would start beside the first.
• Every run refuses to change a unit or container while the fleet-regrade Lease is held by fleet-role.

Tests:

• the recorded role kept over the inventory;
• written only when absent;
• the merged .env renders the same configuration for the k3s and k3s-agent services as before (docker compose config hashes are equal, so no container is recreated);
• the turn list follows the recorded role;
• the refusal on a role mismatch;
• the refusal while the Lease is held;
• the regrade service only on a server;
• --check reports the refusals.

Size: 12 files, 12 tests.

PR D1: feat(cluster): switch a native host's role from a host-side actuator driven by its Node

Files:

• New: files/role/fleet-role-actuator.sh (the command and platform-neutral logic), files/role/fleet-role-actuator-systemd.sh, templates/native/fleet-role-actuator.service.j2, templates/role-actuator.env.j2, tasks/role_actuator.yml, tests/role_actuator/ (fakes reusing tests/regrade/fake-*.sh), tests/unit/test_role_actuator.py.
• Changed: templates/native/fleet-regrade.service.j2 (gains ExecCondition=... fleet-role-is server), tasks/native/demote.yml and tasks/native/server_turn.yml (refactored), defaults/main.yml, meta/argument_specs.yml, the README.

Interfaces fixed here:

• Node label github-runner.exadev.com/desired-role (server or agent). Only the controller sets it. It is persistent, and never a kubelet registration label.
• Node annotations, all under github-runner.exadev.com/:
  • role-swap: the swap id, <UTC %Y%m%dT%H%M%SZ>-<out>-<in>;
  • role-step: promote, step-down or rollback;
  • role-join-node: the node name of the voter to join;
  • snapshot-request: a swap id.
• The Lease, kube-system/fleet-regrade:
  • holderIdentity: fleet-role;
  • annotations github-runner.exadev.com/swap-id and github-runner.exadev.com/swap-nodes.
• Actuator state. The ConfigMap fleet-role-actuators in the platform namespace holds one key per node: {"swap","step","phase":"running|done|failed|refused","role","started_at","at","message","snapshot"}.
• Credential.
  • ServiceAccount fleet-role-actuator: a ClusterRole with get, list, watch on nodes, and a Role with get, patch on that ConfigMap by name.
  • A token Secret, created through tasks/native/api.yml's API host.
  • A kubeconfig, mode 0600, at /etc/rancher/k3s/fleet-role-actuator.kubeconfig.
  • KUBE_SERVERS comes from the same expression templates/regrade.env.j2 uses.
  • A file mapping each server and eligible host to its join URL is written from the topology's peer_server_urls and address.
• Commands and exit statuses. The commands are fleet-role-actuator watch|status|promote ID|step-down ID|finish-demote|rollback ID|snapshot ID.
  • 0: done.
  • 1: failed. The host is left with one role's unit at most, and a role file naming it.
  • 2: refused. That means one of:
    • the Lease is not held by fleet-role with this swap id;
    • the host is not provisioned (native: no server.yaml);
    • the role named is not a role.
  • 3: not yet (finish-demote before the member is gone).
• Authority to act. Every step that could add a member needs the Lease with a matching swap-id. finish-demote needs only role file agent together with a tombstone, removed-node-name naming its member, or no etcd label. Completing the demotion of a removed member can never cost quorum, so a host that was down during its swap finishes on return.
• Promote:
  • role file server;
  • server.yaml.d/20-fleet-role-join.yaml with server:;
  • a persistent unit drop-in setting GITHUB_RUNNER_JOIN_HOST;
  • systemctl disable --now k3s-agent, then enable --now k3s;
  • report started_at.
• Step-down:
  • write_level "$LEVEL_FILE" etcd-only, restart_into etcd-only and stop_pod_shims (reused from fleet-regrade-k3s.sh and fleet-regrade-systemd.sh);
  • then role file agent;
  • and systemctl stop fleet-regrade.
• Finish-demote and rollback:
  • disable --now k3s;
  • move server/db to db.demoted, keeping one copy;
  • remove the level file and the join drop-ins;
  • enable --now k3s-agent.
• Snapshot: k3s etcd-snapshot save --name fleet-role-<id>.
• Refactor. demote.yml and server_turn.yml call promote-local and demote-local for their host-local part. Ansible and the actuator then share one implementation.

Reuses: kubectl_, select_api_server, lease_state, micro_time and now_epoch (core); write_level, restart_into and etcd_serving (k3s); stop_pod_shims, on_server and restart_server (systemd); fleet-role-is.sh.

Tests:

• Unit:
  • each command's success, failure, refusal and status;
  • refusal without the Lease or with a stale swap id;
  • finish-demote without the Lease;
  • a tombstone treated as removed;
  • one db.demoted copy kept;
  • the drop-ins written and removed;
  • the regrade unit's condition;
  • the RBAC renders scoped;
  • the token stays out of the output;
  • demote.yml and server_turn.yml unchanged in behaviour against the existing harness.
• Mesh scenario role_actuator, native part, with the test acting as the controller (setting the Lease and annotations by hand) and real k3s:
  • the native standby is promoted: learner, then voter, writes succeed;
  • stepped down: etcd serves and no shim is left;
  • removed through the annotation;
  • finished to an agent: the same password hash and Tailscale IP, db.demoted present, both units never active together.

Size: 16 files, 18 tests (17 unit, 1 mesh scenario).

PR D2: feat(cluster): switch a Docker host's role from the same actuator

Files: files/role/fleet-role-actuator-docker.sh (new), files/role/Dockerfile (new: Alpine with bash jq docker-cli docker-cli-compose kubectl, built locally as the regrade image is, scripts mounted), files/node-identity.sh (new; the copy pipe from tasks/node_password.yml, generalised), files/docker-compose.yml, files/regrade/fleet-regrade.sh, tasks/role_actuator.yml, tests/unit/test_role_actuator.py, tests/mesh/run.sh, the README.

Interfaces:

• A Compose service role-actuator with profile role-actuator, added for every server and eligible host while the controller is on. It mounts the Docker socket, the project directory at its own absolute path, and the kubeconfig.
• Promote:
  • write fleet-role.d/role=server and join-url;
  • docker compose --profile agent stop k3s-agent;
  • copy the identity: ${project}_k3s-agent-node to _k3s-node, and _k3s-agent-tailscale-data to _k3s-tailscale-data, through a throwaway container from the node image, logging nothing;
  • --profile server up -d k3s.
• Step-down reuses fleet-regrade-docker.sh (write_level, restart_into) and then writes the role file.
• Finish-demote:
  • stop the server;
  • in its volume, move server/db aside and remove the level file;
  • copy the identity back;
  • up -d k3s-agent;
  • stop the regrade service.
• The regrade wrapper idles on a non-server host.

Reuses: on_server, restart_server and etcd_serving (docker actuator), and the seeding pipe from tasks/node_password.yml.

Tests:

• Unit:
  • each command against the fake docker and kubectl;
  • the identity copy in both directions, never with either container running;
  • the volume names derived from the project;
  • no secret in the output;
  • the regrade wrapper idling.
• Mesh scenario role_actuator, Docker part:
  • a Docker server is demoted by hand-set annotations;
  • the same Tailscale IP and password hash;
  • one Node;
  • db.demoted in k3s-data;
  • the agent container Ready;
  • then promoted again, joining the same cluster ID.

Size: 9 files, 12 tests (11 unit, 1 mesh scenario).

PR E1: feat(role): judge voters by etcd's own vote and let a holder renew the regrade Lease

Files: files/regrade/fleet-regrade-core.sh, scripts/fleet-role.jq, tests/regrade/regrade-lease.sh, tests/unit/test_fleet_role.py, tests/unit/fleet_role_support.py.

Interfaces:

• renew_lease HOLDER: replaces the Lease with a new renewTime, keeping acquireTime and annotations; fails if the holder changed.
• fleet-role.jq gains $config.expected_voters: the target, or the target plus one while a swap holds a new voter.
• A voter needs the etcd label and EtcdIsVoter=True. A node with the label but not the condition is a block, "member but not a voter".

Tests:

• renew keeps the holder;
• renew refused for another holder;
• annotations survive a renewal;
• a learner by label is not a voter and blocks;
• the expected count of target plus one;
• the existing gate tests unchanged.

Size: 5 files, 8 tests.

PR E2: feat(role): carry out a voter swap, by decision or by command

Files: scripts/fleet-role.sh (subcommands, resumable passes), scripts/fleet-role-swap.jq (new; pure next-action function), roles/github_runner_arc/templates/fleet-role.yaml.j2, roles/github_runner_arc/defaults/main.yml, roles/github_runner_arc/meta/argument_specs.yml, roles/github_runner_arc/README.md, the repository README (Role controller section), fleet-role/Dockerfile, tests/unit/test_fleet_role_swap.py (new), tests/mesh/run.sh.

Command line and modes:

• fleet-role.sh with no arguments runs one pass.
• fleet-role.sh swap OUT IN records a manual request in fleet-role-status (request: {out, in, at}). Exit statuses:
  • 0: accepted, every gate holding now;
  • 1: refused, with each reason;
  • 2: usage error, or an unknown or ineligible node.
• fleet-role.sh status prints the swap and its step.
• Run as kubectl -n github-runner-platform exec deploy/fleet-role -- /app/scripts/fleet-role.sh swap <out> <in>.
• FLEET_ROLE_DRY_RUN=true reports automatic swaps only. A manual request is carried out.
• false carries out automatic decisions as well.
• A swap already in progress (a role-swap annotation on a Node) is always driven to completion or rolled back, whatever the setting, so an even count is never left behind.
• A manual swap honours every gate but the fitness verdict and the cooldown. It refuses a pinned, ineligible or unprovisioned node.

New settings: FLEET_ROLE_OPERATION_TIMEOUT_SECONDS, FLEET_ROLE_RUNNER_WAIT_SECONDS, FLEET_ROLE_LEASE_SECONDS (the regrade Lease duration) and FLEET_ROLE_ACTUATOR_CONFIGMAP, each from the cluster role's values.

RBAC (the controller only; the probe keeps read-only):

• nodes: patch;
• pods: list in all namespaces;
• pods/eviction: create;
• apps/daemonsets: get;
• coordination.k8s.io/leases: create, plus get, update on fleet-regrade;
• k3s.cattle.io/etcdsnapshotfiles: get, list;
• the actuator ConfigMap: get.

The state machine. Each pass renews the Lease and takes at most one action, so a rescheduled controller pod resumes from the Node annotations and the status.

1. The gates (blocks empty, the Lease free). Take the Lease. Request a snapshot from a remaining voter and wait for readyToUse.
2. Cordon the candidate and wait for runner pods. Past the runner wait timeout, abandon: uncordon, release the Lease, no cooldown. Otherwise drain, set desired-role=server with role-step=promote and role-join-node. Wait for EtcdIsVoter=True and Ready on a heartbeat after started_at (the shell form of tasks/native/server_ready.yml), then uncordon. The deadline is k3s's learner stall time plus the operation timeout.
3. The gates again, with expected_voters at the target plus one.
4. Taint, cordon and drain the outgoing node; label it regrade-level=etcd-only; set desired-role=agent with role-step=step-down. When the actuator reports done, annotate remove=true and wait for removed-node-name.
   • If the actuator does not report and the node is not Ready, remove anyway while the other voters are healthy, as the epic allows.
   • If k3s clears the annotation (ErrMemberNotEnoughStarted), set it again only while the others are healthy.
   • Then wait for the actuator's finish-demote. Remove the control-plane, etcd and master labels, the etcd.k3s.cattle.io/* annotations, the level label and the taint. Wait for Ready, then uncordon. A host that does not come back is left to node-recovery, with outgoing_pending recorded.
5. Clear the swap annotations, release the Lease, and record the change for the cooldown.

Rollback. A candidate that is not a voter by the deadline is rolled back:

• if its Node has etcd annotations, annotate remove;
• desired-role=agent with role-step=rollback;
• clear the server labels and uncordon;
• done once every voter's probe reports known_members equal to the target.

Status. fleet-role-status gains swap (id, out, in, step, started_at, deadline, manual, snapshot), last_swap (outcome: completed|rolled-back|abandoned, with reasons), request and the real dry_run.

Tests:

• Unit, one per branch:
  • each step's success;
  • abandon on the runner wait;
  • rollback on the deadline, with and without etcd annotations;
  • a rollback that waits on known_members;
  • removal rejected, then retried;
  • removal held while a remaining voter is unhealthy;
  • removal of an outgoing voter that is down;
  • resume after a restart at each step;
  • the Lease taken by regrade mid-swap (wait, never act);
  • an even count finished before any new decision;
  • dry-run carrying out a manual request only;
  • a manual request refused for a pinned or unprovisioned node, and with a paused voter;
  • no second swap in the cooldown;
  • Lease renewal on every pass;
  • snapshot wait and timeout;
  • RBAC renders;
  • the probe keeps read-only rules.
• Mesh, real k3s and etcd:
  • role_swap: three Docker servers and a native standby. A manual swap brings the native host in and the Docker bootstrap host out, with writes succeeding throughout, three EtcdIsVoter voters at the end, the demoted node's labels and annotations gone, db.demoted in its volume, its identity kept and the Lease released. A second swap then brings the former bootstrap host back in, joining the existing cluster: one cluster ID before and after.
  • role_swap_paused: the outgoing k3s container is paused after step 3. The removal still completes, with three voters. On unpause, the tombstone guard keeps it out (the member count stays three through a learner stall time) and the actuator finishes it as an agent.
  • role_swap_blocked_peer: TCP 2380 is dropped in the candidate's namespace. k3s removes the learner after five minutes, and the rollback leaves the member list and the voters as they were.
  • role_swap_refused: a voter is paused before step 1. swap exits 1 naming it, and no label or Lease is set.
  • role_controller is extended: with dry-run off after the voter slowed by tc netem is reported unfit, the controller swaps it for the Docker standby. This proves a Docker promotion of a host that was never a server.

Size: 11 files, 26 tests (22 unit, 4 new mesh scenarios and 1 extended).

PR F: feat(heartbeat): report a voter swap in progress and alert on an even voter count left behind

Files: scripts/heartbeat.sh, scripts/fleet-alert.sh, tests/unit/test_heartbeat.py, tests/unit/test_fleet_alert.py, the README.

Interfaces. fleet-health.json gains:

• role_swap: fleet-role-status's swap, or null. It is published whatever the health, unlike the healthy-only status file.
• etcd_voters: {count, target, even_since}, counted by EtcdIsVoter.
• not_ready_nodes[].role_swap: a boolean.

fleet-alert.sh changes:

• It skips server_not_ready for a swap node until role_swap.deadline.
• New kind role_swap_overdue: now is past the deadline plus FLEET_ALERT_AFTER_SECONDS.
• New kind voters_even: even_since older than the window, with no swap inside its deadline.

Tests:

• the fields with and without a swap;
• not-ready entries flagged;
• even_since carried from the previous tick;
• no alert for a swap node within the deadline;
• an overdue swap alerts;
• an even count alerts after the window;
• not while a swap is within its deadline;
• recovery clears both.

Size: 5 files, 9 tests.

(b) The Docker role switch, in one place

• Profiles. Compose keeps the server and agent profiles. One .env holds both services' variables, and the role recorded at fleet-role.d/role decides which profile Ansible and the actuator start (PR C).
• Never both. The entrypoint idles when the role file names the other role, so the two services never run together.
• Identity. The node password and the Tailscale state are copied between the two services' volumes while both containers are stopped (PR D2). The host keeps its node name, password Secret, Tailscale device and mesh IP, so etcd's peer URL is unchanged.
• --cluster-init. It is never passed once the host holds server/db.demoted or a join-url. A former member with neither refuses to start (PR A).
• Ansible never undoes the controller. An eligible host's role file is written only when absent. Every run refuses to change a container while the Lease is held by fleet-role (PR C).

(c) Interactions

• node-recovery: the swap nodes listed on the Lease and any regrade-level node are exempt (PR B). Once its labels are gone, a demoted node is an agent to the prune pass. Its existing server refusal (scripts/node-recovery.sh:68) reads those labels, so it is right only after step 4's clean-up. That is why the clean-up comes before the agent's Ready wait ends the swap.
• Regrade core: a Lease held by fleet-role refuses every regrade step (acquire_lease, fleet-regrade-core.sh:231). release_lease never frees it (fleet-regrade-core.sh:248). If the controller is gone long enough for the Lease to expire, regrade may take it, and the controller waits before acting again. The swap's record lives on the Nodes and in the status, not on the Lease, so nothing is lost.
• Heartbeat: it publishes the swap and the voter count (PR F). An etcd-only outgoing voter is already excluded by its regrade_level.
• fleet-alert: it holds off on swap nodes until the deadline, and alerts on an overdue swap or an even count left behind (PR F).
• The probe: its DaemonSet follows the etcd label and the eligible label, so a demoted node keeps its pod while it stays eligible. A restart in the swap window marks the swapped hosts unfit for one window, and the cooldown already outlasts that window.

What stays manual, and a No

The manual tasks stay as issue 183 lists them: removing a voter that is down and not coming back, restoring after quorum loss, changing the target count, and promoting a host that has not been provisioned (refused in D1 and E2).

No: an Ansible-driven role switch for a Docker host. For an eligible host, the controller's swap command covers it, and PR C refuses the mismatch that would otherwise start a second container silently. Revisit when an inventory needs to move a non-eligible Docker host between roles by hand.