Repository navigation
Description
The role controller of the dynamic regrading epic decides which eligible hosts hold the etcd votes. This sub-issue builds what it decides from, and runs the decision in dry-run only: it reads, computes and reports, and never changes a role. It needs no other sub-issue and changes no membership, so it can run on a live cluster as soon as it ships.
Probe. A fleet-role-probe DaemonSet on every node except etcd-only servers (which have no kubelet; their side is read from their peers), with host networking so it measures the path the node's own etcd and kubelet use (on a Docker node that is the k3s container's network, where the node's Tailscale client lives). Every sample period it records, per node:
- the TCP round trip to every voter's mesh address on the etcd peer port, so loss shows as tail latency rather than being lost;
- on a voter, etcd's own
etcd_network_peer_round_trip_time_seconds,etcd_disk_wal_fsync_duration_seconds,etcd_disk_backend_commit_duration_secondsandetcd_server_leader_changes_seen_totalfrom the metrics listener k3s keeps on127.0.0.1:2381; - on a non-voter, an fdatasync probe in the k3s data directory with the write size etcd's own fio check uses, once per window, never on a voter (whose etcd is measured directly);
- memory pressure and MemAvailable, read with the regrade core's own functions;
- the link the default route leaves by: wired or Wi-Fi from
/sys/class/net/<iface>,unknownwhere the node cannot see the host's interfaces (a Docker VM).
Results go to a ConfigMap per node, in a status shape the controller and the heartbeat read.
Controller. A fleet-role Deployment in the platform namespace, built and released like node-recovery, with FLEET_ROLE_DRY_RUN=true by default (the autoscaler's convention) and a fleet-role-status ConfigMap the heartbeat republishes beside autoscaler-status. Each pass it computes, for every eligible host, fit or unfit and why, and the swap it would make, logging the decision and the reason. Thresholds, all derived:
- round trip: unfit when the p99 to any voter exceeds the election timeout divided by ten (etcd's rule that the election timeout is at least ten round trips); fit for promotion only at or below the election timeout divided by twenty (the 0.5 end of etcd's heartbeat band); the election timeout is the cluster's configured value, or k3s's 1000 ms when unset;
- disk: WAL fsync p99 under 10 ms and backend commit p99 under 25 ms (etcd's FAQ);
- memory: never over the regrade limit in the window, and MemAvailable at least etcd's storage quota (2 GiB unless set);
- stability: no NotReady spell and no k3s restart in the window;
- window: 100 samples per peer pair, the fewest that give a p99 a sample of its own, at
github_runner_cluster_role_sample_seconds(15 by default, node-recovery's poll); - rank among the fit: wired before Wi-Fi before unknown, then p99 round trip, then fsync.
The decision rules are the ones the epic states: a swap only when an unpinned voter has been unfit for a whole window and a fit eligible candidate ranks above it, never while any voter is below full, not Ready, or etcd reports a learner, an unstarted member or no leader, a target voter count that is odd, at least 3 and at most github_runner_cluster_max_servers minus one, and a cooldown of k3s's learner stall time plus one window after any membership change.
Inventory. github_runner_cluster_voter_eligible and github_runner_cluster_voter_pinned per host and github_runner_cluster_voters for the group, published as Node labels by the role; the role refuses an even target, a target under 3 or without a place of headroom below max_servers, and fewer eligible hosts than the target.
Tests: tests/unit for every threshold and decision rule against recorded probe output, including the hysteresis band, the window boundary, ties and each refusal. A mesh scenario runs the probe and the controller in dry-run on the three-server cluster, makes one voter slow with tc netem on its mesh interface, and checks the controller reports it unfit and names the swap, then pauses a voter and checks it reports that it would not act. Every probe is then run against each real host the fleet marks eligible, in dry-run, and its readings compared with a reading taken by hand on that host (ip route, the sysfs entry, ping, the etcd metrics), because a stand-in that lacks the real structure hides bugs: a find -L over sysfs once passed stand-ins and failed on a real host.
Activity