Move etcd votes between hosts by health, and let native nodes be servers #179

Memory-pressure regrading (#112, #114) lets a server step down and back up while it keeps its etcd vote. This epic adds the rest of the dynamic behaviour: native nodes that can be servers and regrade like the Docker ones, and a controller that moves etcd votes between hosts by health, so the control plane runs on the best hosts without inventory edits. The motivating case is a fleet whose voters are laptops on Wi-Fi or relayed mesh paths, which stall etcd, while its native Linux hosts that could hold a vote are agent-only.

The sub-issues are in dependency order. Native server mode and the controller's dry-run signal collection have no dependency on each other and touch no membership on a live cluster; actuation needs both, and the fleet adopts it last.

Design

Membership changes are make-before-break, through k3s's own paths. k3s joins every new server as an etcd learner (MemberAddAsLearner in join, pkg/etcd/etcd.go at v1.34.1+k3s1), and the etcd leader's manageLearners loop promotes it only once etcd accepts MemberPromote, which it refuses until the learner's log has caught up; a learner that makes no progress for learnerMaxStallTime (5 minutes) is removed. A learner does not count towards quorum (etcd learner design, archived), so adding one to three healthy voters cannot cost quorum, and a misconfigured one never becomes a voter. A swap therefore adds the new voter first (3 voters, then 4 with quorum 3, the same tolerance of one failure as 3), then removes the old one (back to 3, all healthy). Removing first would leave two voters, which etcd calls unsafe (runtime reconfiguration, archived). The add is started only while every voter is healthy, which is the case etcd's FAQ warns about (FAQ, archived); the remove may then proceed even if the outgoing voter is down, since that is etcd's own order for an unhealthy member and the remaining members are all started. etcd's strict-reconfig-check stays on (k3s does not set it).

A voter is removed through k3s's member removal controller: the annotation etcd.k3s.cattle.io/remove=true on its Node (pkg/etcd/member_controller.go), which a leader-elected controller on a server with an API server acts on, refusing to remove the last voter. The outgoing server is first stepped to etcd-only through regrading, so it is drained and runs none of the leader-elected controllers itself. Its member is removed while its k3s is still running; its etcd then sees ErrMemberRemoved and writes a tombstone. Only then is its server unit stopped, so the cluster never waits on a stopped member, and the unit is guarded so that Restart=always cannot start it again as a server and rejoin it as a learner. Its server/db is moved aside, because k3s reuses any datastore it finds on disk and ignores --server and --cluster-init (HA embedded etcd, archived). A Node is never deleted to remove a member.

A host keeps its identity across a role change. Same node name, same /etc/rancher/node/password, same Tailscale device and mesh address, so the node password Secret stays valid and etcd's peer URL is the mesh address before and after. k3s builds etcd's peer URL from --node-ip before --vpn-auth rebuilds the node IP (the reason the Docker servers pass --node-ip), so a native server passes the host's own Tailscale address as --node-ip too, never a LAN address, which breaks whenever a laptop changes network.

One Lease serialises every transition. Role changes take the fleet-regrade Lease the regrade core already uses, so a regrade step and a role change can never overlap, and the holder renews it for as long as a step runs. No role change starts while any voter is below full, is not Ready, or etcd reports a learner, an unstarted member or no leader.

The controller only chooses among hosts the owner marks eligible, each pre-provisioned by the role as a standby (server binary, units, configuration, certificate SANs) before it can be promoted. Unrestricted promotion of any agent is out of scope: a host the role has not prepared has no server configuration to switch to. Hosts can be pinned so they are never demoted automatically. The controller holds the voter count at a target that is odd, at least 3, and at most github_runner_cluster_max_servers minus one, since a swap needs one place of headroom. It ships in dry-run, as the autoscaler does.

Fitness thresholds come from etcd's guidance and the cluster's own timing, not chosen numbers (tuning, archived; FAQ above). The election timeout must be at least ten times the round trip and the heartbeat about 0.5 to 1.5 times it, so a voter is unfit when its p99 round trip to any other voter exceeds a tenth of the configured election timeout, and a candidate is fit only at or below a twentieth of it (the 0.5 end of the band), the gap between the two being the hysteresis. Disk: WAL fsync p99 under 10 ms and backend commit p99 under 25 ms. Memory: never over the regrade limit and MemAvailable at least etcd's storage quota (2 GiB unless set), which the FAQ says the machine must cover. Stability: any NotReady spell or k3s restart in the window. Every figure is read over a window of 100 samples per peer pair, the fewest that give a p99 one sample of its own. Link type (wired over Wi-Fi) ranks hosts that pass, and is never a gate on its own, since a Wi-Fi host with a good path can be fitter than a wired one on a relay. A swap happens only when an unpinned voter has been unfit for a whole window and an eligible candidate has been fit for the same window and ranks above it, one swap at a time, with a cooldown after each of k3s's learner stall time plus one window, so every reading after a change comes from the new membership.

What the inventory decides and what the cluster decides. For an eligible host the inventory gives only the role a host starts with; afterwards the live role, persisted on the host, is the truth, and an Ansible run never undoes the controller's choice (as it cannot undo a regrade step). --cluster-init is written only when bootstrapping a cluster that does not exist yet, so a former bootstrap host that is demoted and later promoted joins the existing cluster and can never start a second one.

What is left to people. Removing a voter that is unhealthy and not coming back, recovering from quorum loss, and growing or shrinking the target count are owner actions. The controller never removes a member to repair the cluster.

Risk

The worst failure is quorum loss: the API stops, every runner stops picking up jobs, and recovery is a restore. It is prevented by learner-first adds, one membership change at a time under the Lease, never starting an add with any voter unhealthy, never going below 3 or above github_runner_cluster_max_servers, removing before stopping, and an on-demand k3s etcd-snapshot save on a voter before each swap. The next worst is a second cluster from a stray --cluster-init, prevented as above and tested. If quorum is lost anyway, the owner restores from a snapshot on one server with k3s server --cluster-reset --cluster-reset-restore-path=<snapshot> (the server's token file is needed), then moves server/db aside on every other server and rejoins them (etcd-snapshot, archived). Every server takes scheduled snapshots at 00:00 and 12:00 and keeps five locally, so a native server keeps its own, and a demoted host's moved-aside server/db keeps the snapshots it took.

Test plan

The mesh harness (tests/mesh/run.sh, .github/workflows/mesh-integration.yml) runs real k3s and real etcd, so membership changes are proven there: a native node as a privileged systemd container beside the Docker nodes, a real learner add and promotion, a real removal through the annotation, a voter made slow with tc netem on its mesh interface and one made unhealthy with docker pause, a learner whose peer port is blocked until k3s removes it, and ConfigMap writes throughout to show quorum held. What only a real host proves (link type from the real sysfs, fsync on the real disk, round trips over real Wi-Fi and DERP paths, the host's own Tailscale client as a server, persistence across a reboot) is run against each eligible host in dry-run before any actuation. A stand-in that lacks the real structure hides bugs (a find -L over sysfs passed stand-ins and failed on a real host), so every probe runs against the real thing at least once before its result is acted on.