Repository navigation
feat(role): measure etcd vote fitness and report the role controller's decisions in dry-run - #185
Merged
Merged
Conversation
Mearman
changed the title
Mearman
force-pushed
the
feat/role-controller-signals
branch
2 times, most recently
from
November 3, 2026 11:40
84b1a3d to
96d83fc
Compare
Reads each node's probe entry and the Nodes, and reports per host whether it is fit, unfit or unknown with the numbers behind it, and the swap of an unfit voter for a fit standby that it would make. It changes nothing in the cluster: its only write is its own status ConfigMap. Every limit derives from the election timeout, the tail percentile or a named fact about etcd or k3s. The unit tests hold each to its boundary, and the memory limit to the regrade core's own is_over.
…lth for the controller A per-node probe appends one sample per period to its key of a shared ConfigMap and keeps the last window of them: the TCP connect time to every voter's peer port, what a voter's etcd metrics gained since the last sample, an fdatasync reading on a node that is not a voter, the memory signals the regrade core reads, and whether the default route leaves by wired, Wi-Fi or an interface whose link cannot be seen. The tests read the link from a sysfs tree whose links loop as the kernel's do and from the host's own sysfs where there is one, the connect times from real listeners, etcd's metrics from pages a real three-member cluster served (and a live local cluster when an etcd binary is available), and the disk reading from the real fio.
One image holds the controller, which its entrypoint runs on an interval, and the probe, which the DaemonSet starts by name. It carries fio, the disk check etcd's own documentation uses, and the regrade core whose functions the probe reads memory with. The unit job installs fio so that the probe's fdatasync reading is tested with the real tool.
…n by hand `fleet-role-probe.sh check link|connect|memory|etcd|fsync` prints one reading and exits without touching the cluster, so that every probe can be run on a real host and its output set beside a reading taken there by hand.
A fleet-role Deployment and a fleet-role-probe DaemonSet in the platform namespace, each with a ServiceAccount that can get and list nodes and reach only its own ConfigMaps by name, and neither able to change a node. The controller's settings follow the cluster role's own where that role defines them, with fallbacks the tests hold to its defaults, and the target is refused unless it is odd, at least three and a place below the maximum number of servers.
…toscaler's The heartbeat reads the controller's status ConfigMap, and is allowed that one by name, only when the role controller is installed, and publishes it to the gist while the fleet is vouched for, on the same terms as the autoscaler's status.
…he controller The role works out the cluster's election timeout once and gives the same figure to the controller and the probe, so neither script holds a default of its own that could differ from the other.
A probe on any other node recorded a window of samples nobody reads, and the shared ConfigMap, which holds at most a mebibyte, would have filled with the size of the fleet. Such a node now records nothing and loses an entry it had, so the ConfigMap grows with the hosts that decide the control plane.
…to act A voter that is paused or unreachable stops reporting while its last sample still looks healthy, so the gate on etcd's state read stale gauges. It now says the probe has not reported, as the reason nothing would change.
Linux limits one argument to a process to 128 KiB, and the entries of a few voters together pass it, so the controller failed with an argument list too long on a runner while it passed on macOS. The tests now run with a jq that refuses such an argument, and hold both scripts to it with inputs the size of a full ConfigMap.
…runs The role controller's probe runs from its own image, whose Dockerfile contents are held by the role controller's tests instead.
…er_role_controller The opt-in is the cluster role's variable, off by default, which the ARC role takes as the default of its own switch, so one inventory setting turns on the probe and the controller for the group. The tests hold that switch and the names the status is published under.
…on pull requests A build-only job builds fleet-role/Dockerfile as the release does and runs what the pods run from the image as built. The unit job installs the etcd v3.6.15 release, checked against the SHA-256 its SHA256SUMS asset publishes before it is extracted, for the probe's live test.
…the voter it replaces A slow link makes both its ends unfit, so ranking unfit voters alone could send a healthy peer out in place of the one on the poor path, and a standby that measures the slow voter slow was never fit. The voter on the most slow links is now the one to replace, nothing is swapped when it is pinned, and a candidate is judged against the voters that would remain.
…o it is up etcd keeps one gauge per peer that counts its active streams, up to two, so a follower with both streams to the leader and one to the other follower read as connected to one peer and the gate held every swap back. The fixture is a page a follower of a real three-server k3s cluster served.
…refuses one A probe on a poor path made four sequential calls a sample and went stale, and its process exited on the first refusal and came back as a restart. It now reads its own entry once, reads the nodes once a sample, and a patch the API will not take is tried again next period with the first-sample restart mark kept until a sample is recorded.
… a paused one A mesh scenario with three servers and a standby agent runs the probe and controller from the image built from the checkout, checks their permissions, waits for a full window of real samples with every host fit, puts tc netem on a voter's mesh interface and checks the swap is named, pauses a voter and checks the controller says it would not act, and that nothing in the cluster changed. etcd's data is on memory so that the machine's disk does not decide the verdict.
…or the role controller While github_runner_cluster_role_controller is on, the role refuses a voter count that is even, under three or without a place of headroom below the most servers allowed, a datastore with no etcd, and fewer eligible hosts than the target once any host is marked. With none marked the controller only reports. The settings default off, to three voters and to no host eligible or pinned.
…bels Every runtime passes the two labels to k3s as --node-label: a Docker server or agent through the entrypoint, and a native node through its configuration. Both labels are written with true or false on every host that uses the controller, since a kubelet never removes a label and a setting turned off is written over. A host that uses none of it gets no label and no extra Compose file, so no container is recreated by an upgrade. The ARC role's controller takes its target and label names from these settings.
…ntory The role_controller mesh scenario marks every host eligible and pins the first in its inventory, as an owner would, in place of labelling the nodes with kubectl, and checks that the labels reach the nodes of the servers and of the agent and that the controller reads them.
Mearman
force-pushed
the
feat/role-controller-signals
branch
from
November 3, 2026 12:55
7a67b0c to
93b6340
Compare
Mearman
marked this pull request as ready for review
November 3, 2026 13:20
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
|
This security review couldn’t start because the paying account has exhausted its Codex allowance and has insufficient available credits. The payer can check their balance and add credits in the Codex usage dashboard, then try again. If you do not manage the paying account, contact this repository's admins. |
|
🎉 This PR is included in version 2.26.0 🎉 The release is available on:
Your semantic-release bot 📦🚀 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Refs #182
The role controller of #179, in dry-run: it collects the fitness signals and reports what it would do, and changes nothing in the cluster.
fleet-role-probe(a DaemonSet on every Linux node with a kubelet, host network) appends one sample every period to its key of thefleet-role-probeConfigMap: TCP connect time to each voter's peer port, what a voter's etcd metrics gained since the last sample (WAL fsync, backend commit, per-peer round trip, leader changes, leader/learner/known-member gauges), a fio fdatasync reading once per window on a non-voter, the regrade core's own memory readings, and whether the default route leaves by wired, Wi-Fi or an interface whose link can't be seen. A node that is neither a voter nor marked eligible records nothing, so the ConfigMap grows with the hosts that decide the control plane.fleet-role(Deployment) reads those and the Nodes and publishesfleet-role-statusingithub-runner-platform, which the heartbeat republishes beside the autoscaler's. Per host it gives fit, unfit or unknown with every reason and its numbers, and the swap it would make or why it would not.Same form as
node-recoveryandpsi-reporter: bash, jq and kubectl in an Alpine image built by the release workflow, installed by the ARC role, ClusterRoles that can only get and list nodes, ConfigMap access by name. It is off unlessgithub_runner_cluster_role_controlleris true (the name #184 fixes;github_runner_arc_role_controller_enabledtakes it as its default), and the controller refuses to run with anything butFLEET_ROLE_DRY_RUN=true.Thresholds are derived as the epic states them (a tenth and a twentieth of the election timeout, 10 ms and 25 ms disk, memory against the regrade limit and the storage quota, a window of 100 samples = the fewest that give a p99 its own sample, cooldown = k3s's learner stall time plus a window). Two decisions the epic leaves implicit, both found by running it against real k3s: a slow link makes both its ends unfit, so the voter to replace is the one on the most slow links (and nothing is swapped when that voter is pinned), and a standby is judged against the voters that would remain, since it measures the slow one slow too.
Tests
is_over. The probes are tested against the real thing: a sysfs tree whose links loop like the kernel's (and the host's own/syswhere there is one), real listeners and curl, pages from real etcd 3.6.15 and from a follower of a real k3s cluster, a live local three-member etcd (pinned release, SHA-256 checked in CI), the real fio. A jq stand-in refuses an argument over 128 KiB as Linux does, since passing the probe data as arguments failed on a runner and passed on macOS.role_controller(tests/mesh/run.sh): three servers and a standby agent on real k3s and etcd, the image built from this checkout, the ARC role's rendered manifests. It checks the RBAC withkubectl auth can-i, waits for a full window of real samples with every host fit, makes a voter slow withtc netemon its mesh interface and checks it is reported unfit with the swap named, pauses a voter and checks the controller says it would not act, and that the etcd members and nodes are unchanged. etcd's data (and the standby's fdatasync directory) is on tmpfs in this scenario, since a virtual disk does not reliably meet etcd's 10 ms fsync limit (a laptop's measured 15 to 30 ms) and the scenario tests the pipeline, not the machine's disk. Run locally on a headless Colima profile raised to 4 CPUs and 10 GiB (the default 2 and 4 are too little for four k3s nodes and Headscale); it also passes on the runner. On a laptop under heavy load the baseline can read the voters unfit from real 1 s round trips, which is the controller working, not a flake in it.tests/fleet_role/image-smoke.sh).Inventory and labels
The cluster role holds the controller's inventory:
github_runner_cluster_role_controller(off),github_runner_cluster_voters(3), and per hostgithub_runner_cluster_voter_eligibleandgithub_runner_cluster_voter_pinned(both false). While the controller is on, the role refuses an even target, one under 3, one without a place of headroom belowgithub_runner_cluster_max_servers, a datastore that is not etcd, and fewer eligible hosts than the target once any host is marked (with none marked the controller only reports). The two host settings reach the nodes as--node-labelon every runtime (Docker server, Docker agent, native), written astrueorfalseon every host that uses the controller because a kubelet never removes a label; a host that uses none of it gets no label and no extra Compose file, so no container is recreated by an upgrade. The ARC role's target and label names default to these. The mesh scenario sets them in its inventory and checks they reach the servers and the agent.After the first release publishes
ghcr.io/exadev/github-runner-fleet-role, the package has to be made public (GitHub's REST API has no endpoint for package visibility, only list, get, delete and restore, so this is the web UI): the organisation's Packages tab, the package, Package settings, Danger Zone, Change visibility, Public (docs, archived). It cannot be undone, as the page says.Real-host checks (read-only)
Each reading alone, from inside the probe pod on the node, with nothing written to the cluster (
fsyncwrites and unlinks a small file in the probe's own directory). Compare each with the hand reading beside it. Pod for a node:kubectl -n github-runner-platform get pod -l app.kubernetes.io/name=fleet-role-probe --field-selector spec.nodeName=NODE -o name.Then, with the role controller on for at least one window,
kubectl -n github-runner-platform get cm fleet-role-status -o jsonpath='{.data.status\.json}' | jq .against the hosts, andkubectl -n github-runner-platform logs deploy/fleet-role.