Repository navigation
feat(cluster): run a native host as a k3s server and etcd voter - #186
Merged
Merged
Conversation
A native host was forced to an agent and refused the server role and the bootstrap. It is now a server when it sets the server role or the bootstrap (github_runner_cluster_node_role or github_runner_cluster_bootstrap), and stays an agent otherwise. It still takes no automatic server place and does not count towards the automatic number, so an inventory whose native hosts set neither keeps them all agents whatever max_servers allows. A host marked github_runner_cluster_voter_eligible is named on every server's TLS SAN list, so a standby is covered before it is promoted, and the topology publishes each host's mode, eligibility and the join URLs of the other servers. A native bootstrap host is refused under headscale_in_cluster, whose bootstrap needs the Docker node's supervisor.
The version check took this node's role from the inventory. A native host can now switch between its server and agent units at run time, so where the cluster lists this node, its control-plane label decides which rules apply to it, as it already did for every other node.
A native host now has two units, k3s-agent and k3s, each with its own configuration file passed as --config, and a role file under k3s's data directory that both units' ExecCondition reads, so the two never run together and Restart=always never starts the unit of a role the host no longer has. The server's configuration carries the flags every Docker server passes (the critical ones k3s requires to match), and etcd's peer URL is the host's Tailscale address, written as node-ip into a drop-in before each start. cluster-init is written only for the bootstrap host with no datastore, none moved aside, and no other server answering, so a former member always joins the cluster that exists. A server's restart, first start beside a cluster and promotion go ahead only while every other etcd voter, read from the cluster's labels, is Ready, one native server at a time. A switch between the units keeps the node's name, password and device: it empties the node first, and a demotion leaves at least three Ready voters, removes the member through k3s, and moves the datastore aside. The version checks read native servers through k3s kubectl, and regrading is refused on a native host.
The native_server scenario runs a native host as a privileged systemd container that the role manages over Ansible's docker connection. It joins three Docker servers as an agent, then the inventory makes it a server: its k3s must join etcd as a learner and be promoted to a voter with its mesh address as its peer URL, while writes through another server keep succeeding, and a rerun must not restart it.
…them A play runs a native host's tasks before any Docker host's, since the role's first task includes them and the linear strategy runs an include for its hosts while the others wait, so the native agent waited on a bootstrap server that did not exist yet. The scenario now starts the Docker servers in a run of their own, the role README says to do the same, and a failure prints the native host's units, role file, Tailscale status and journal.
node_status builds a Docker node's name from its index, so the scenario looked for a node that does not exist and timed out with the native agent Ready.
CI deals whole test modules out to its shards, and the native node module had grown past one shard's time limit once it ran the role for servers too. The harness the two share moves to native_stand_ins.py, the server tests to test_native_server.py, and the agent tests stay where they were.
The promotion and demotion tests ran the role once only to leave the host running the role they then switch from. They now write the role file and the unit's state directly, as a run leaves them, since the role writes every file it reads afresh, so each spends one run of the role on its subject.
… guard The native server tests made the heaviest shard run past its timeout. The modules are dealt out by size, a rough stand-in for their work, so a fifth shard spreads them further, and the timeout is set above the heaviest shard's run as a guard against a hung test rather than a budget the shards are fitted to.
… its error The writes during the promotion reused write_marker's random suffix, which repeats often enough over that many writes that a refusal for a name already taken would count as a lost write, and the error was discarded. Each write is now named by its sequence number, and a failed one records what kubectl said.
Mearman
marked this pull request as ready for review
November 3, 2026 11:26
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
|
This security review couldn’t start because the paying account has exhausted its Codex allowance and has insufficient available credits. The payer can check their balance and add credits in the Codex usage dashboard, then try again. If you do not manage the paying account, contact this repository's admins. |
|
🎉 This PR is included in version 2.24.0 🎉 The release is available on:
Your semantic-release bot 📦🚀 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #180
A native host (
github_runner_cluster_node_mode: native) can now be a k3s server and etcd voter beside the Docker servers. It is a server only when it setsgithub_runner_cluster_node_role: serverorgithub_runner_cluster_bootstrap: true; it still never takes an automatic server place, so an inventory whose native hosts set neither keeps them agents whatevermax_serversallows.github_runner_cluster_voter_eligible: truestages a standby: server config written, name on every server's TLS SAN list, role unchanged.Every native host gets both units,
k3s-agent.serviceandk3s.service, each with its own--configfile (agent.yaml,server.yamlplusserver.yaml.d/). Both haveExecCondition=fleet-role-is <role> /var/lib/rancher/k3s/fleet-role, so they never run together andRestart=alwaysnever starts the wrong one (systemd'sservice_shall_restartreturns false forSERVICE_SKIP_CONDITION). A host from before this change runs the agent unit and is recorded as an agent; its old/etc/rancher/k3s/config.yamlis removed.server.yamlcarries the same flags as the Docker servers'mesh_server_args(traefik and network policy disabled, TLS SANs, etcd timing, API port).node-ipis the host's own Tailscale address, written intoserver.yaml.d/00-node-ip.yamlby anExecStartPrebefore every start, so etcd's peer URL is never a LAN address.10-join.yamlgetscluster-initonly for the bootstrap host with noserver/db, noserver/db.demoted, and no other server answering; an unreadable server refuses the run.Native servers take turns like recreated Docker servers. A restart, a first start beside a cluster or a promotion is refused while any other voter is not Ready, reading voters from each node's live
node-role.kubernetes.io/etcdlabel. A restart that is refused stays pending (a marker file) and the next run makes it. Switching role keeps the node name, node password and Tailscale device: cordon, wait for runner pods, drain (an abandoned switch uncordons and changes nothing). A demotion needs at least three other Ready voters. It writes the role file first, removes the member through k3s'setcd.k3s.cattle.io/removeannotation, then stops the server, movesserver/dbtoserver/db.demotedand strips the server labels. The version checks read native servers throughk3s kubectland judge this node by its live role.github_runner_cluster_regradeis refused on native hosts (the actuator is Docker-only), and a native bootstrap host is refused underheadscale_in_cluster.Not here: the controller (#182), the regrade actuator (#181) and automated promotion/demotion (#183). The firewall rule is documented, not applied: the role opens and closes nothing on the host.
Tests:
tests/unit/test_native_node.py(agent) andtests/unit/test_native_server.py(server), sharingtests/unit/native_stand_ins.py, run the role against stand-ins on PATH and a server data directory laid out as k3s leaves it (bootstrap, join, voter guard, pending restart, promotion, abandoned promotion, demotion refused and made, a former member never starting a second cluster, legacy agent upgrade). The newnative_servermesh scenario runs a real native host as a privileged systemd container managed over Ansible's docker connection. It joins three Docker servers as an agent, then is promoted in place with node 3 left outside the inventory. It must join etcd as a learner (Adding memberin its journal,Promoted learnerin a server's log) and end as a voter with a100.xpeer URL, with no failed writes through node 1 during the promotion, and a rerun must not restart it. Its kubelet resolver file points at public servers through a test drop-in, because the container has only Docker's127.0.0.11. The Docker servers come up in a run of their own first: a play runs a native host's tasks before any Docker host's (the role's first task includes them), so a native node joining servers that the same run starts would wait on nothing. The README now says so.The server tests pushed the heaviest unit shard past its 20 minute timeout, so CI now runs five shards with a 30 minute hang guard. The host_network mesh scenario failed once on a direct path through the runner's public endpoint and passed on rerun; that is #189 and is unrelated to this diff.
Only a real host shows that a native server on its own Tailscale client survives a reboot. Check mode on asus or i5 (already native agents), with
github_runner_cluster_node_role: serverset for that host. The inventory's server count must stay odd: with three Docker servers, either leave one of them out of the inventory for this run (as the mesh test does) or set both asus and i5 as servers.It should report
promote k3s on asuswith no refusal. If another voter is not Ready it should refuse instead, naming that voter. Once a real promotion has run: