Repository navigation
feat(regrade): regrade a native server under memory pressure with a systemd actuator - #194
Merged
Merged
Conversation
Mearman
force-pushed
the
feat/systemd-regrade-actuator
branch
from
November 3, 2026 14:07
d3872a8 to
e716942
Compare
…hat answers The regrade service was given one API server, the first other server of the inventory, fixed when the role ran. Once the servers change at run time that host may be an agent, and a step would be refused for want of an API. The settings now list every other server and then every host marked github_runner_cluster_voter_eligible, by the addresses the servers' certificates carry and never the server's own, as KUBE_SERVERS. The core asks each for /readyz in turn and goes through the first that answers, before every step, again for the Lease's release and whenever a step waits for the API. A step none of them answers for is refused and names them.
What makes a k3s server regradable is the same whatever runs it: the three levels, a requested level on the data directory and a started level on a tmpfs, the Node label, and a restart into etcd-only and out of it that waits for etcd and then for a heartbeat from the new kubelet. That moves out of the Docker actuator into fleet-regrade-k3s.sh, leaving a platform to say where k3s runs, how it restarts and how its log since the restart is read. The wrapper now runs the actuator FLEET_REGRADE_ACTUATOR names in the settings, which the role sets to docker for the Compose service, and its refusal of a kernel without memory pressure no longer assumes a Docker host.
…stemd actuator A native server can now step down to etcd-only and back up as the Docker servers do. The systemd actuator records the requested level on k3s's data directory, and the k3s unit's ExecStartPre renders the server.yaml.d drop-in from it before every start (the four disable keys at etcd-only, none otherwise) and records the level k3s starts at in the unit's runtime directory, which the actuator reads. Every file is written atomically. A step into or out of etcd-only restarts the unit and returns once the new run's journal says etcd is serving and, for a level that runs the agent, once the Node is Ready on a heartbeat later than the restart. Entering etcd-only stops the containerd shims that KillMode=process left running, as k3s-killall.sh does, so no pod keeps its memory without a kubelet. The role runs the fleet-regrade unit with the host's own memory pressure, k3s kubectl and the admin kubeconfig, installs jq and procps where they are missing, and keeps the refusals the Docker service has, plus a kernel with no memory pressure, judging a native host by the role it is to run. A server turn at etcd-only waits for etcd instead of a Node that has no kubelet, and a demotion forgets the level so a later promotion starts the server whole. Refs #181
…ressure The native_server scenario ends by turning regrading on for the native server and driving its fleet-regrade unit with simulated memory pressure through a drop-in, against the real k3s unit and its ExecStartPre. It checks the step through control-plane to etcd-only and back up, the membership unchanged and the native member a started voter, a write with a Docker server stopped, the level kept across systemctl restart and across the host container stopping and starting, and the shims of a DaemonSet's pod, present at full, gone at etcd-only and back once the agent runs again. Turning regrading off then removes the unit without restarting the server. Refs #181
systemd removes a unit's runtime directory whenever the unit leaves the active state, including a start that its ExecCondition skips. With both units sharing /run/github-runner-k3s, a start of k3s-agent on a server removed the running server's resolver file, so the kubelet could not build a pod sandbox, and with it the record of the regrading level k3s started at. Each unit now has its own directory (github-runner-k3s-agent and github-runner-k3s-server under run_dir), derived in the defaults and used for its resolver file and, on a server, the started level.
Both regrade test modules sourced the settings file with their own copy of the shell call; regrade_settings.sourced is now the one that does it, exporting every assignment so variables the shell sets itself are left out. The check that the core and an actuator are given every setting they read matched names of letters and underscores only, so it never saw K3S_SERVICE or K3S_DATA_DIR; it now matches every shell variable name and asserts that it finds K3S_SERVICE. The native role tests compare the files a check run or a refused run leaves on the stand-in host with those the host held before, which now include its memory pressure file, and list the regrade level writer and the fleet-regrade unit template among the role's files.
The shard loop ran under set -e, so the first failing module ended the shard and every module after it went unrun and unreported. Each module now runs whatever the earlier ones did, and the shard fails afterwards naming each module that failed.
Mearman
force-pushed
the
feat/systemd-regrade-actuator
branch
from
November 3, 2026 16:01
e716942 to
d7795aa
Compare
Mearman
marked this pull request as ready for review
November 3, 2026 16:26
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
|
This security review couldn’t start because the paying account has exhausted its Codex allowance and has insufficient available credits. The payer can check their balance and add credits in the Codex usage dashboard, then try again. If you do not manage the paying account, contact this repository's admins. |
|
🎉 This PR is included in version 2.28.0 🎉 The release is available on:
Your semantic-release bot 📦🚀 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds the systemd actuator for memory-pressure regrading, so a native server (
github_runner_cluster_node_mode: native) steps down to etcd-only and back as the Docker servers do, and makes the core reach the cluster through the first candidate server whose/readyzanswers instead of a server fixed when the role ran.Closes #181
The core's one change: the settings list every other server and then every host marked
github_runner_cluster_voter_eligible, by the addresses the servers' certificates carry and never the server's own (KUBE_SERVERS), and the core goes through the first that answers, chosen before each step, again for the Lease's release and whenever a step waits for the API. The Docker actuator takes it too.The k3s parts both actuators need (levels, the requested and started level files, the Node label, the restart into and out of etcd-only with its waits) moved from the Docker actuator into
files/regrade/fleet-regrade-k3s.sh;fleet-regrade.shruns the actuatorFLEET_REGRADE_ACTUATORnames.The systemd actuator (
files/regrade/fleet-regrade-systemd.sh) writes the level to/var/lib/rancher/k3s/regrade-level; thek3sunit'sExecStartPre(files/native/write-regrade-level.sh) rendersserver.yaml.d/50-regrade.yamlfrom it and records the started level in/run/github-runner-k3s-server/regrade-level. After a restart it waits forETCD server is now runningin the new run's journal (by invocation id) and, for a level with the agent, a Node heartbeat later than the restart. Entering etcd-only stops the containerd shimsKillMode=processleft, ask3s-killall.shfinds them. The role runs it asfleet-regrade.service(host PSI and meminfo,k3s kubectl, admin kubeconfig), installsjqandprocpswhere missing, keeps the Docker refusals plus a no-PSI refusal, judges a native host by the role it is to run, waits for etcd rather than Ready in a server turn at etcd-only, and forgets the level on demotion.Tests:
tests/regrade/regrade-systemd.sh(fakesystemctl,journalctl,date,k3swith the existing fakekubectl; real processes stand in for the shims) for every level change in both directions and each wait;tests/unit/test_native_regrade.pyfor the role; and the meshnative_serverscenario now steps the native server to etcd-only and back with its real unit.Each k3s unit now has a runtime directory of its own (
github-runner-k3s-serverandgithub-runner-k3s-agent). systemd removes a unit's runtime directory when itsExecConditionskips a start, so with one shared directory a start ofk3s-agenton a server removed the server's resolver file, after which its kubelet could not create a pod sandbox, and the started level with it.The unit test shards now run every module before failing, so one failing module no longer hides the rest; that had hidden a settings check that could not see
K3S_*names, now fixed.