CKS: change the node settings of a cluster and restart its nodes with them, a few at a time by bhouse-nexthop · Pull Request #14616 · apache/cloudstack

bhouse-nexthop · 2026-11-08T03:33:34Z

Description

Lets updateKubernetesCluster change the service offering, root disk size and affinity groups of each node type of a CKS cluster. With rollout=true, it then restarts the existing nodes that differ with the new settings, a few at a time, cordoning and draining each one first.

Fixes #14615

Depends on #14609 and #14611. Their commits are the first two on this branch, and they drop out once 4.22 is merged forward into main. Please review only the last commit, "CKS: restart the nodes of a cluster with new node settings, a few at a time". It uses #14609's compare-and-set state changes (transitToOperationState) and #14611's drain helper (drainKubernetesClusterNode).

This targets main only, because it adds three columns.

Nodes are restarted, not changed live

Each node is restarted. This PR doesn't scale VMs live, migrate them live, or resize disks online:

Live scaling of a KVM VM depends on the offering being dynamic and on the guest hot-plugging the CPUs and memory. kubelet doesn't pick up the new capacity until it restarts anyway.
An affinity group only takes effect when the deployment planner places the VM, which happens on start.
Growing the root partition and file system is done by cloud-init at boot.

A stop and start applies all three changes the same way on every hypervisor, and the node is drained first, so the pods move before it goes down.

API

updateKubernetesCluster gets these parameters:

nodeofferings, noderootdisksizes (GB) and nodeaffinitygroups, set per node type (control, worker, etcd).
rollout (default false).
maxunavailable: the number of workers restarted at a time, default 1, only with rollout=true.
cancelrollout.

With rollout=false, the settings are only stored, and nodes created later get them. This works in any state except while another operation is running on the cluster or it is being destroyed.

A cluster created before its node types had settings of their own first gets the cluster's service offering copied to each type, so changing one type leaves the others as they are.
If all nodes of a type share the same affinity groups, those are recorded for the type.
Templates are not copied, and this API doesn't change them.

A root disk size is refused with rollout=true if it would shrink an existing node's disk, or if the disk's offering has a strictly fixed size. With rollout=false it is accepted, as createKubernetesCluster accepts it.

The response adds:

the root disk size of each type;
controlnodesoutofdate, workernodesoutofdate and etcdnodesoutofdate, the number of nodes of each type that differ from its settings;
lasterror: why the last rollout stopped, or which control or etcd nodes it could not restart.

Rollout

rollout=true moves the cluster from Running or Alert to a new transient state, Updating, then stores the settings. It restarts the out-of-date nodes in this order:

the external etcd nodes;
the control nodes, one at a time;
the workers, maxunavailable at a time. This is clamped to min(maxunavailable, workers - 1), and is always at least 1.

A node is out of date when its service offering differs, its root disk is smaller than the type's size, or its affinity groups differ from those recorded for the type. Explicit dedication groups, which CKS adds by itself, are not compared and are kept. A type with no recorded groups leaves the nodes' groups as they are. nodeaffinitygroups refuses explicit dedication groups. Stopped nodes, and nodes still carrying the annotation from a rollout that stopped, go first.

Each batch goes through these steps:

Preflight: every other VM must be Running and every other Kubernetes node Ready.
- For a control or etcd node, a quorum of the other etcd members must also be healthy, and etcd must report ready.
- With external etcd, another control node must be serving the API.
- Some types can't lose a node at all, for example a single control node or a 2-node stacked control plane. A rollout=true that would change such a type's settings is refused at the API. If such a type is out of date for another reason, its nodes are skipped and reported in lasterror, and the other types are still rolled.
Cordon and drain: each node is annotated with cloudstack.apache.org/rollout-boot-id=<bootID>, then cordoned. A node someone else has already cordoned is left without the annotation, and stays cordoned. Each node is then drained with CKS: fix node drain on scale-down and upgrade, make a failed upgrade resumable #14611's helper.
Change: the VM is stopped, then upgradeVirtualMachine sets the offering, resizeVolume grows the root disk (offline, grow only), and the affinity groups are updated. The VM is then started, and the deployment planner places it by its new groups.
Verify: the node must come back Ready with a new boot ID. A Ready condition left over from before the stop is not accepted. The VM must also no longer differ from the settings. Once both hold, the node is uncordoned and the annotation removed.
- Each batch has cloud.kubernetes.cluster.update.node.timeout seconds (default 1800).

cancelrollout=true stops the rollout between two batches.

However a rollout ends, a single idempotent release() runs. It:

uncordons the annotated nodes that are Ready with a new boot ID, and leaves the rest cordoned for someone to look at;
recounts the cluster's CPU cores and memory from the VMs' actual offerings;
moves the cluster to Running if it is healthy, or to Alert if not;
records or clears lasterror.

Nothing else is stored. The nodes that still differ, are stopped or still carry the annotation are the work left to do. A rollout that stopped is resumed by calling rollout=true again, which also uncordons any node that came back after the last run.

While a cluster is Updating, the other mutating APIs are refused. If the management server running a rollout stops, the state scanner moves the cluster from Updating to Alert and recounts its capacity. It doesn't reach into the nodes. The next rollout picks them up.

Data model

The schema change adds control_node_root_disk_size, worker_node_root_disk_size and etcd_node_root_disk_size to kubernetes_cluster, idempotently, in schema-42300to2400.sql. Nodes created later use these sizes. Cancel requests and last errors are cluster details (rollout.cancel, rollout.lasterror). There are no new tables.

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)
build/CI
test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Major
Minor

How Has This Been Tested?

Unit tests only. Nothing here has been run against a live cluster yet.

mvn -o test -pl plugins/integrations/kubernetes-service passes (233 tests). engine/schema compiles. The new tests are:

KubernetesClusterRollingUpdateWorkerTest runs the worker against a simulated cluster (VMs, kubectl output, boot IDs). It covers:
- node selection;
- worker batching and the maxunavailable clamp;
- control nodes one at a time on a 3-node stacked control plane;
- quorum refusals: a 2-node stacked control plane, an unhealthy etcd member, a single control node whose workers still roll, and etcd and control both skipped;
- replacing the affinity groups while keeping the dedication group;
- the annotate → cordon → drain → stop → change → start → verify order;
- rejecting a stale Ready with the same boot ID;
- a node cordoned by someone else;
- release() after success, a drain failure, a start failure, a cancel and an early exception, each run twice;
- a cancel asked for while the settings are being stored;
- a failure reported together with the skipped nodes;
- rerunning after a failure, which takes the stopped node first and uncordons a node that came back without restarting it.
KubernetesClusterManagerImplUpdateTest covers:
- rollout=false;
- gen 1 to gen 2;
- the API validation (maxunavailable, disk sizes, shrink and strict offerings, node types, dedication groups, control settings that can't be rolled);
- refusing an update while another operation runs;
- Running or Alert for a rollout;
- Updating before the settings are stored, and going back to the previous state, with the last error kept, if storing fails or the settings are refused;
- refusing settings that would make a node of a 1-control-node or 2-etcd-node cluster out of date, while accepting no-op settings, nodes that already differed and stopped nodes;
- cancel;
- the out-of-date reasons, including explicit dedication groups;
- the capacity recount;
- the Updating guards on the other APIs.
UpdateKubernetesClusterCmdTest covers parsing of the per-type maps.

Not tested yet, and the main risks in a live test:

Root partition growth: this relies on the CKS node templates' cloud-init running growpart and resizefs at boot. If a template doesn't, the disk grows but the file system doesn't. Nothing here grows it over SSH.
Affinity on start: the deployment planner's affinity processors should honour the new groups when the stopped VM starts, including the last-host handling. I've read the code but haven't tried it.
etcd health checks: each check runs over SSH on the node itself: etcdctl endpoint health on an external etcd node, and curl http://127.0.0.1:2381/health, the metrics address kubeadm sets up, on a stacked control node. The cluster-wide check is kubectl get --raw=/readyz/etcd. All of these were checked against the CKS scripts, not against a running cluster.
Quorum on a 3-node stacked control plane: in particular, kubectl has to go through another control node while the first is down. On a shared network, admin.conf points at the first control node.
Drain timeouts with PodDisruptionBudgets: a PDB that never allows an eviction fails the batch at the drain. The node is uncordoned, as it never went down.

…tates A Kubernetes cluster's state was written by id alone, so two operations started on the same cluster at once both got past their first transition. Each then ran its steps against VMs the other was changing, and whichever finished last decided the final state. The status scanner could also overwrite the state of an operation that started after it read the cluster. The state is now compared and set in one update, as the VM and volume states are. Each operation moves the cluster into its own state before it changes anything, and fails without touching the cluster when another operation got there first. From then on it only moves the cluster while the cluster is in a state the operation moved it to. Starting a cluster in Alert now moves it to Starting like any other start. A start from Error, a stop from Created or Error, and a retried destroy still go on without a state change, as they did before; the start and stop then leave the state alone when they end. From any other state with no transition for it, an operation is now refused instead of running alongside the one that holds the cluster. The scanner only completes a stop or a recovery that it started itself. A management server that stopped during an operation left the cluster in that operation's state. Nothing moved it on, and every later operation on it was refused until the database was edited by hand. The scanner now fails the operation of a cluster in an operation's state when no job is in progress for it on any management server. The existing Alert reconciliation then returns a healthy cluster to Running. To find those jobs, the start, stop, scale and upgrade jobs now record their cluster, as the create, delete and node jobs already did. A cluster named by a job in progress on a management server of an older version, which does not record it, is left alone. This replaces the scanner's first-run handling of Starting clusters, which tried a transition that does not exist. No schema or API change. Signed-off-by: Brad House <bhouse@nexthop.ai> (cherry picked from commit fee9834)

…resumable Scale-down and upgrade both drain a node before touching it, each with its own copy of the drain loop, and neither works well: - Scale-down gives up at once when kubectl drain exits non-zero (a PodDisruptionBudget that does not allow an eviction yet, say), and when it does retry after an exception it counts the attempt twice. The node is left cordoned, and the cluster goes to Alert although nothing has been removed and every node still works. - Upgrade retries straight away, with no wait for the eviction to become possible, and on the last failure leaves the node cordoned. - In both, kubectl drain has no --timeout, so it is cut off by a fixed 60s SSH timeout rather than a deadline of its own. Both workers now use one drain helper in KubernetesClusterActionWorker: one count per attempt, a retry (after a wait) on a non-zero exit or an exception, kubectl drain --timeout from the new setting cloud.kubernetes.cluster.node.drain.timeout (300s), cut down so that no attempt runs past the operation's deadline, with the SSH wait derived from it, and the end of the drain output in the error. The caller uncordons the node, and only if it cordoned it. Scale-down cordons every node it is going to remove (leaving alone the ones that were cordoned already), drains them, and if one cannot be drained uncordons them all and puts the cluster back in the state it was in, with an error that names the node, instead of Alert; the same happens if the scale times out before every node is drained. Once they are drained, the removal is not stopped part way by the timeout any more. If removing a node fails, the nodes still in the cluster are uncordoned. The upgrade had two more problems: - It only waited for the first control node to be Ready, so the next node could be drained while the previous one was still coming back. It now waits for every node, after it has been uncordoned. - It reached node i on sshPort + i. The port-forwarding rules are not created in the order of the upgrade's VM list (manually upgraded nodes are left out of that list, and nodes added later get the next free port), so the script could run on the wrong node. It now looks up the node's own SSH port-forwarding rule, as the remove worker already does. A failed upgrade left the cluster in Alert with no way to finish it, as upgradeKubernetesCluster only accepts a Running cluster. An upgrade now records the version it is upgrading to as a cluster detail (an existing key/value table) and removes it once the cluster is Running or Stopped again. upgradeKubernetesCluster accepts a cluster in Alert (Alert -> Upgrading) when that detail is the requested version (or another one with the same semantic version), which is a real upgrade of the cluster's own version, and all of the cluster's VMs are running, and refuses any other Alert cluster with a clear message. For an upgrade that failed before this change, a cluster still at the old version with nodes recorded at the target version (kubernetesnodeversion, which is set as each node is done) is accepted too. Nodes that are already at the target version, both as recorded and as reported by kubectl get node, and that are Ready, are skipped. The first control node is still upgraded first; if it is already done, kubeadm upgrade apply is not run again. The node version check now compares versions exactly, so 1.28.1 no longer matches 1.28.10, line by line, as the SSH output can hold stderr too, and ignoring semver build metadata. Failure messages name the node the upgrade stopped on. The upgrade stops if it can't move the cluster to Upgrading, and the Alert state scanner no longer marks a cluster Running when it couldn't move it to Recovering, and a failed upgrade detaches its ISO before going to Alert, so a retry and the scanner or the failing job don't undo each other's work. No schema change. Signed-off-by: Brad House <bhouse@nexthop.ai> (cherry picked from commit 0dcaa8f)

…a time A CloudManaged Kubernetes cluster could get a new service offering only through a scale, which changes every node at once with no drain, its affinity groups only while stopped, and its root disk size not at all. updateKubernetesCluster sets, for each node type (control, worker, etcd), the service offering, the root disk size and the affinity groups. The nodes created from then on get them. A cluster created before its node types had settings of their own first gets the service offering of the cluster copied to them; templates are left as they are. With rollout=true, the nodes that differ from the settings of their type are restarted with them: the external etcd nodes, then the control nodes, one at a time, then the worker nodes, maxunavailable at a time (1 by default, and always leaving a worker running). For each batch: - every other node must be running and Ready, and a control or etcd node is only taken down if the others keep a quorum - each Kubernetes node is annotated with its boot ID, unless someone else cordoned it, then cordoned and drained - its VM is stopped, given the offering, a larger root disk and the affinity groups, and started, which places it by its new groups; cloud-init grows the root partition and file system at boot - once it is Ready with another boot ID, and etcd is healthy, it is uncordoned and the annotation removed A control or etcd node type with too few nodes to lose one (a single control node, a 2-node stacked control plane, 2 etcd nodes) can not be rolled: a rollout=true that would make one of its nodes differ from the settings is refused, and its nodes that differ anyway are left out and reported in lasterror, while the other types are still rolled. Nodes are restarted rather than changed live, which CloudStack can not do reliably for a KVM VM, and which would leave the guest and kubelet unaware of the new resources. Nothing else is recorded: the nodes that still differ, are stopped or carry the annotation are the work left, and go first when the rollout is run again, which is how one that failed is resumed. However a rollout ends, it uncordons the nodes that are back, leaving the others cordoned, counts the CPU cores and memory of the cluster from its VMs, and moves the cluster from the new Updating state to Running, or to Alert if it is not healthy. Why it stopped is shown as lasterror. cancelrollout=true stops a rollout between two batches. Other operations are refused while a cluster is Updating, and the state scanner moves a cluster left Updating by a management server that stopped to Alert. The response also shows the number of nodes of each type that differ from the settings of their type. Signed-off-by: Brad House <bhouse@nexthop.ai>

codecov · 2026-11-08T03:39:56Z

Codecov Report

❌ Patch coverage is 62.76518% with 509 lines in your changes missing coverage. Please review.
✅ Project coverage is 20.91%. Comparing base (fb60564) to head (b682b1c).

Files with missing lines	Patch %	Lines
...nworkers/KubernetesClusterRollingUpdateWorker.java	68.81%	101 Missing and 44 partials ⚠️
...bernetes/cluster/KubernetesClusterManagerImpl.java	74.86%	54 Missing and 40 partials ⚠️
...er/actionworkers/KubernetesClusterScaleWorker.java	37.07%	56 Missing ⚠️
...r/actionworkers/KubernetesClusterActionWorker.java	67.60%	31 Missing and 15 partials ⚠️
...kubernetes/cluster/UpdateKubernetesClusterCmd.java	39.68%	38 Missing ⚠️
.../actionworkers/KubernetesClusterUpgradeWorker.java	41.17%	28 Missing and 2 partials ⚠️
...dstack/api/response/KubernetesClusterResponse.java	0.00%	21 Missing ⚠️
...KubernetesClusterResourceModifierActionWorker.java	36.66%	17 Missing and 2 partials ⚠️
...er/actionworkers/KubernetesClusterStartWorker.java	0.00%	15 Missing ⚠️
...bernetes/cluster/dao/KubernetesClusterDaoImpl.java	62.06%	10 Missing and 1 partial ⚠️
... and 11 more

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #14616      +/-   ##
============================================
+ Coverage     20.78%   20.91%   +0.12%     
- Complexity    21931    22269     +338     
============================================
  Files          6496     6498       +2     
  Lines        591657   592878    +1221     
  Branches      72592    72837     +245     
============================================
+ Hits         122966   123981    +1015     
- Misses       455346   455438      +92     
- Partials      13345    13459     +114

Flag	Coverage Δ
uitests	`3.88% <ø> (ø)`
unittests	`22.22% <62.76%> (+0.13%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

bhouse-nexthop added 3 commits November 8, 2026 02:36

boring-cyborg Bot added component:api component:kubernetes component:UI labels Nov 8, 2026

Navigation Menu

Repository navigation

CKS: change the node settings of a cluster and restart its nodes with them, a few at a time - #14616

CKS: change the node settings of a cluster and restart its nodes with them, a few at a time#14616
bhouse-nexthop wants to merge 3 commits into
apache:mainfrom
bhouse-nexthop:cks-rolling-update

bhouse-nexthop commented Nov 8, 2026

Uh oh!

codecov Bot commented Nov 8, 2026 •

edited

Loading

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

bhouse-nexthop commented Nov 8, 2026

Description

Nodes are restarted, not changed live

API

Rollout

Data model

Types of changes

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

How Has This Been Tested?

Uh oh!

codecov Bot commented Nov 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

codecov Bot commented Nov 8, 2026 •

edited

Loading