Skip to content

feat(arc): leave a role swap's nodes and regraded servers alone in node-recovery - #209

Merged
Mearman merged 2 commits into
mainfrom
feat/node-recovery-swap-aware
Nov 3, 2026
Merged

Mearman merged 2 commits into
mainfrom
feat/node-recovery-swap-aware

Conversation

@Mearman

@Mearman Mearman commented Nov 3, 2026

Copy link
Copy Markdown
Member

node-recovery gave a silent node the out-of-service taint after the threshold, whatever the reason it was silent. A voter swap and a regrade step both make a node silent on purpose, so the taint would evict pods the controller is draining deliberately.

While the kube-system/fleet-regrade Lease is unexpired, node-recovery now neither taints nor prunes a node its github-runner.exadev.com/swap-nodes annotation lists or its holder names. A node carrying the regrade level label is never tainted. A taint applied earlier is still cleared once the node is Ready. The Lease is read through a Role with get on that one name; a Lease that cannot be read fails the pass before any change.

The expiry rule is lease_state's from the regrade core, reimplemented in the script's jq. The level label follows github_runner_cluster_regrade_level_label, as the role controller's does. The Lease name, namespace and annotation key are fixed names in vars/main.yml.

Refs #183

…ade level label

A Role in the Lease's namespace allows get on the one Lease that serialises voter swaps and
regrading, and nothing else there. The watcher is told the Lease, the annotation that lists a
swap's nodes and the level label through its environment; the level label follows the cluster
role's setting.
…de-recovery

While the fleet-regrade Lease is unexpired, node-recovery neither taints nor prunes a node its
swap-nodes annotation lists or its holder names, and it never taints a node carrying the regrade
level label. A swap or a regrade step makes a node silent on purpose, and an out-of-service taint
would evict pods the controller is draining deliberately. A taint applied earlier is still
cleared once the node is Ready, and a Lease that cannot be read fails the pass before any change.

Refs #183
@Mearman
Mearman marked this pull request as ready for review November 3, 2026 21:43
@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
🔒 Security Review ⚠️ Failed 2026-11-03T21:43:11.178885Z
We couldn’t complete this review. Comment @codex security review to try again.
643de6b Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector

Copy link
Copy Markdown

This security review couldn’t start because the paying account has exhausted its Codex allowance and has insufficient available credits. The payer can check their balance and add credits in the Codex usage dashboard, then try again. If you do not manage the paying account, contact this repository's admins.

@Mearman
Mearman merged commit d2ce821 into main Nov 3, 2026
22 checks passed
@Mearman
Mearman deleted the feat/node-recovery-swap-aware branch November 3, 2026 21:43
@github-actions

github-actions Bot commented Nov 3, 2026

Copy link
Copy Markdown

🎉 This PR is included in version 2.32.0 🎉

The release is available on:

Your semantic-release bot 📦🚀

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant