Skip to content

Regrade a native server under memory pressure with a systemd actuator #181

Description

@Mearman

Memory-pressure regrading (github_runner_cluster_regrade) has a Docker actuator only. A native server (the native server mode sub-issue) needs a systemd actuator implementing the core's three functions, levels, current_level and apply_level NAME, with files/regrade/fleet-regrade-core.sh used as it is. Part of the dynamic regrading epic.

Levels are the same three: full, control-plane and etcd-only. Between full and control-plane nothing restarts; the core's taint and cordon are the difference. etcd-only is reached by configuration and a restart of k3s.service: the actuator writes the level to /var/lib/rancher/k3s/regrade-level (persistent, so a stepped-down server stays down across a restart or a reboot, as the Docker actuator's does), and the unit's ExecStartPre renders /etc/rancher/k3s/server.yaml.d/50-regrade.yaml from it (disable-apiserver, disable-controller-manager, disable-scheduler, disable-agent at etcd-only, empty otherwise) and records the level k3s starts at in /run/github-runner-k3s/regrade-level, which current_level reads, since k3s hides its arguments from /proc. Both files are written atomically.

Waiting for the step. After a restart apply_level returns only when etcd serves again (k3s's ETCD server is now running in the journal since the restart, the line the Docker actuator waits for) and, for a level that runs the agent, when the Node is Ready on a heartbeat later than the restart, because the core releases the Lease as soon as it returns.

Pods at etcd-only. The unit's KillMode=process leaves pod shims running when k3s restarts without its agent, so pods that survived the drain (DaemonSet pods) would keep running with no kubelet and keep their memory, which defeats the step. Entering etcd-only stops the remaining shims, as k3s-killall.sh does for containers; leaving it starts the agent, which recreates them.

The service. fleet-regrade.service runs the core and this actuator with the host's own /proc/pressure/memory and /proc/meminfo, KUBECTL="k3s kubectl" and the admin kubeconfig, and the environment file the role already templates. The role's refusals carry over unchanged (not on an agent, not on the only server, not on a headscale_in_cluster bootstrap server, not turned off below full), plus a kernel without PSI is refused before anything changes, as the Docker wrapper stops for it.

Reaching the API. KUBE_SERVER is today the first other server in inventory order, fixed when the role runs. Once roles change at run time that server may be an agent, so the core takes a list of candidate servers (every other server and every eligible host, by their certificate names) and uses the first whose /readyz answers, never its own. This is the one change to the core, and the Docker actuator takes it too.

Tests: tests/regrade gains the systemd actuator against the existing fake kubectl and a fake systemctl and journal, for every level change in both directions and the waits. The mesh scenario with the native server steps it down to etcd-only and back under simulated pressure with the real unit, real k3s and a real drop-in: membership unchanged, the vote kept (a write succeeds with one other server stopped), the level kept across systemctl restart and across stopping and starting the systemd container, and no shim left at etcd-only. On a real native server, fleet-regrade status runs before regrading is turned on.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions