Skip to content

Measure each host's fitness to hold an etcd vote and report a role controller's decisions in dry-run #182

Description

@Mearman

The role controller of the dynamic regrading epic decides which eligible hosts hold the etcd votes. This sub-issue builds what it decides from, and runs the decision in dry-run only: it reads, computes and reports, and never changes a role. It needs no other sub-issue and changes no membership, so it can run on a live cluster as soon as it ships.

Probe. A fleet-role-probe DaemonSet on every node except etcd-only servers (which have no kubelet; their side is read from their peers), with host networking so it measures the path the node's own etcd and kubelet use (on a Docker node that is the k3s container's network, where the node's Tailscale client lives). Every sample period it records, per node:

  • the TCP round trip to every voter's mesh address on the etcd peer port, so loss shows as tail latency rather than being lost;
  • on a voter, etcd's own etcd_network_peer_round_trip_time_seconds, etcd_disk_wal_fsync_duration_seconds, etcd_disk_backend_commit_duration_seconds and etcd_server_leader_changes_seen_total from the metrics listener k3s keeps on 127.0.0.1:2381;
  • on a non-voter, an fdatasync probe in the k3s data directory with the write size etcd's own fio check uses, once per window, never on a voter (whose etcd is measured directly);
  • memory pressure and MemAvailable, read with the regrade core's own functions;
  • the link the default route leaves by: wired or Wi-Fi from /sys/class/net/<iface>, unknown where the node cannot see the host's interfaces (a Docker VM).

Results go to a ConfigMap per node, in a status shape the controller and the heartbeat read.

Controller. A fleet-role Deployment in the platform namespace, built and released like node-recovery, with FLEET_ROLE_DRY_RUN=true by default (the autoscaler's convention) and a fleet-role-status ConfigMap the heartbeat republishes beside autoscaler-status. Each pass it computes, for every eligible host, fit or unfit and why, and the swap it would make, logging the decision and the reason. Thresholds, all derived:

  • round trip: unfit when the p99 to any voter exceeds the election timeout divided by ten (etcd's rule that the election timeout is at least ten round trips); fit for promotion only at or below the election timeout divided by twenty (the 0.5 end of etcd's heartbeat band); the election timeout is the cluster's configured value, or k3s's 1000 ms when unset;
  • disk: WAL fsync p99 under 10 ms and backend commit p99 under 25 ms (etcd's FAQ);
  • memory: never over the regrade limit in the window, and MemAvailable at least etcd's storage quota (2 GiB unless set);
  • stability: no NotReady spell and no k3s restart in the window;
  • window: 100 samples per peer pair, the fewest that give a p99 a sample of its own, at github_runner_cluster_role_sample_seconds (15 by default, node-recovery's poll);
  • rank among the fit: wired before Wi-Fi before unknown, then p99 round trip, then fsync.

The decision rules are the ones the epic states: a swap only when an unpinned voter has been unfit for a whole window and a fit eligible candidate ranks above it, never while any voter is below full, not Ready, or etcd reports a learner, an unstarted member or no leader, a target voter count that is odd, at least 3 and at most github_runner_cluster_max_servers minus one, and a cooldown of k3s's learner stall time plus one window after any membership change.

Inventory. github_runner_cluster_voter_eligible and github_runner_cluster_voter_pinned per host and github_runner_cluster_voters for the group, published as Node labels by the role; the role refuses an even target, a target under 3 or without a place of headroom below max_servers, and fewer eligible hosts than the target.

Tests: tests/unit for every threshold and decision rule against recorded probe output, including the hysteresis band, the window boundary, ties and each refusal. A mesh scenario runs the probe and the controller in dry-run on the three-server cluster, makes one voter slow with tc netem on its mesh interface, and checks the controller reports it unfit and names the swap, then pauses a voter and checks it reports that it would not act. Every probe is then run against each real host the fleet marks eligible, in dry-run, and its readings compared with a reading taken by hand on that host (ip route, the sysfs entry, ping, the etcd metrics), because a stand-in that lacks the real structure hides bugs: a find -L over sysfs once passed stand-ins and failed on a real host.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions