Repository navigation
Coroot's hcloud volumes can still strand on an autoscaler node #3287
Description
🤖 Generated by the Agentic Engineer
Evidence
Measured live on prod 2026-08-21. Every hcloud PVC in the cluster and the node its pod runs on:
| PVC | Pod | Node |
|---|---|---|
observability/data-coroot-clickhouse-keeper-0 |
coroot-clickhouse-keeper-0 |
autoscale-cx43-6406b46e4820452e ← ephemeral |
openbao/data-openbao-0, audit-openbao-0 |
openbao-0 |
autoscale-cx43-… ← handled by #3286 |
the other 13 hcloud PVCs |
— | prod-worker-{1,2,3} |
hcloud volumes attach to exactly one node and are not force-detached when that node departs, so a volume on an autoscaler node is stranded the moment the autoscaler reclaims it.
Why the existing guards do not cover this
The cluster already makes a deliberate split, and block storage falls through it:
| Storage | Guard | Strength |
|---|---|---|
| Longhorn replicas | restrict-storage-to-baseline-workers.yaml |
hard (allowScheduling: false + eviction) |
| General pods | prefer-baseline-nodes.yaml |
soft — deliberately, so burst still works (FinOps) |
hcloud block volumes |
none | — falls through to the soft bias only |
The soft bias is the right default for stateless workloads and is documented as such ("a hard nodeAffinity would defeat burst"). It is measurably insufficient for single-attach block storage: openbao-0 carried that weight-100 preference and was on the autoscaler node anyway.
Note the failure mode is worse here than for Longhorn, which is hard-guarded: Longhorn replicates across nodes, an hcloud volume does not.
Impact
The realised cost of one stranded hcloud volume is on record (#2363): on 2026-07-01 data-openbao-0 stayed attached to a departed node for ~9h and the prod merge queue was wedged for the duration. Coroot is observability rather than delivery-path, so the blast radius is smaller — but "we lose monitoring while the cluster is unhealthy" is a bad time to lose monitoring.
Expected behaviour
No hcloud-backed pod is schedulable onto a node the autoscaler may reclaim.
Smallest useful first step
Extend the same required node affinity #3286 applies to OpenBao to the Coroot StatefulSets (coroot-clickhouse-keeper, coroot-clickhouse-shard, coroot-coroot, coroot-prometheus).
Worth deciding along the way
Per-workload patches do not generalise — the next chart that requests an hcloud PVC is exposed again with nothing to catch it. A durable alternative is a Kyverno policy that adds the required affinity to any pod mounting a PVC whose storageClassName is hcloud, mirroring restrict-storage-to-baseline-workers for the other storage class. That needs an admission-time apiCall to resolve the PVC's class, so it is worth designing rather than assuming.
Acceptance criteria
- No pod mounting an
hcloudPVC can schedule onto a node labelledksail.io/autoscaled. - The guard is verified by a negative control against the live node set, not by inspection.
- The general FinOps burst behaviour for non-
hcloudworkloads is unchanged.
Rough size
S–M. Per-workload patches are S; the generalised policy is M.
Part of #2363.
Metadata
Metadata
Assignees
Labels
Type
Fields
Priority
Effort
Projects
- Status
Activity
devantler commentedon Aug 21, 2026
Adding a fail-closed requirement to this issue's scope, from CodeRabbit's review of #3286.
The affinity #3286 ships uses a negative discriminator —
ksail.io/autoscaled DoesNotExist— which fails open for a node that carries no label at all. A positive match (Inon a baseline-pool label) would fail closed. That is the better shape and this issue should deliver it.It could not be done in #3286, and the reason constrains the design here:
talos/workers/label-nodes.yaml, a single shared file applied to every Talos worker. Autoscaler nodes boot from that same config — verified live:autoscale-cx43-…and all threeprod-worker-*nodes all carry itsnode.longhorn.io/create-default-disk=true. Apermanent-worker=trueadded there would land on the autoscaler node too, so the positive match would match everything — worse than the negative one.ksail.io/autoscaledis stamped by KSail at node creation (ksail#5113), so its inverse cannot be minted from this repo.autoscale-*name discriminator the sibling Longhorn policies use is not expressible in an affinity — documented verbatim inprefer-baseline-nodes.yaml.So a fail-closed guard needs a real label source, and there are two routes:
ksail.io/autoscaled), which any affinity can then match withIn.NodeCRs matchingnames: ["prod-worker-*"]applies the label — the same name-based mechanismrestrict-storage-to-baseline-workers.yamlalready uses for exactly this reason.Route 2 is self-contained here; route 1 is cleaner but crosses a repo boundary. Worth deciding explicitly, because it also fixes the unlabelled-legacy-node hole that #3178 tracks — the same root gap, reached from a different direction.
Added acceptance criteria
devantler commentedon Sep 13, 2026
Live re-measurement 2026-09-13: the negative guards are inert on EVERY current autoscaler node
Read-only against prod, 2026-09-13 ~04:30Z.
1. No node carries
ksail.io/autoscaled— including both autoscaler nodes.ksail.io/autoscalednode.longhorn.io/create-default-diskautoscale-cx43-59ee3c84…trueautoscale-cx43-55e35623…trueprod-worker-{1,2,3}trueApart from the hostname, the label sets on the autoscaler nodes and the baseline workers are identical. These are not legacy nodes: both were created long after ksail#5113 shipped. KSail's
GenerateAutoscalerWorkerConfigstamps the marker and strips the Longhorn disk label, so a node carrying the disk label and no marker did not boot from that stripped config. Why is a separate question; I have not chased it here.The consequence is what matters for this issue. Every guard that keys on
ksail.io/autoscaled DoesNotExistcurrently admits every node:openbao/patches/keep-off-autoscale-nodes.yaml(fix(openbao): keep the hcloud volumes off autoscaler nodes #3286) — its required affinity excludes nothing. OpenBao is onprod-worker-{1,2,3}today by placement, not by the guard.prefer-baseline-nodes.yaml— the FinOps soft bias is a no-op.2. hcloud volumes are attached to autoscaler nodes right now. 9
VolumeAttachments point at the two autoscaler nodes, including:observability/coroot-clickhouse-shard-0-1→autoscale-cx43-59ee3c84…(running)openbao/vault-snapshot-*CronJob pods (Completed) →autoscale-cx43-55e35623…, so the snapshot job is also an hcloud consumer this issue's scope did not list.So this issue's premise stands and is broader than filed. It also confirms the added criterion: only a positive match fails closed.
Plan (claimed; branch
claude/baseline-worker-node-label-3287)Route 2 from the comment above, split so no scheduling changes until the label source is proven live:
Nodes namedprod-worker-*with a positive baseline-worker label (admission, plus a retrofit rule for existing nodes). The mechanism is the same name-based matchrestrict-storage-to-baseline-workersalready uses, and[Node,*,*]is already dropped from Kyverno's resource filters. Nothing reads the label yet, so merging cannot move a pod. After merge, verify against the live node set that exactly the three baseline workers carry it.DoesNotExistwith a positiveInon that label for every hcloud consumer (OpenBao, the Coroot StatefulSets, vault-snapshot). Add the expression inside each existingnodeSelectorTerm, because terms are ORed and a separate term would fail open.chore: agent-claim #3287 nonce=23c76060aa29062d2d2adb06f0e76d71
devantler commentedon Sep 13, 2026
Slice 1 is live and verified: #3758 merged 05:28:32Z
Read-only against prod right after the merge-queue deploy:
platform.devantler.tech/baseline-workerprod-worker-1,prod-worker-2,prod-worker-3trueautoscale-cx43-55e35623…,autoscale-cx43-59ee3c84…prod-control-plane-{2,4,5}label-baseline-workersandlabel-existing-baseline-workersare bothReady=True.infrastructure-controllersandinfrastructureare Ready on the latest revision.The label source is therefore proven against the live node set, as this issue's added acceptance criterion requires. The existing nodes were labelled by the retrofit policy, and a node that registers later is labelled at admission.
Slice 2 (next): switch the hcloud guards to the positive label
Nothing requires the label yet. The remaining work is to make every consumer of
hcloudblock storage require it, replacing the inertksail.io/autoscaled DoesNotExistcheck:openbao/patches/keep-off-autoscale-nodes.yaml, the required affinity.coroot-clickhouse-keeper,coroot-clickhouse-shard,coroot-corootandcoroot-prometheusStatefulSets, via the Coroot CR.vault-snapshotCronJob andvault-snapshot-initJob, which mount thevault-snapshotshcloud PVC.Add
platform.devantler.tech/baseline-worker In ["true"]inside each existingnodeSelectorTerm. Terms are ORed, so a separate term would fail open.At merge,
coroot-clickhouse-shard-0-1was still running onautoscale-cx43-59ee3c84…. A required affinity does not move a running pod, so the switch needs a planned reschedule of that pod, and a negative control against the live node set.The unlabelled-marker root cause is tracked separately in devantler-tech/ksail#7013.
chore: agent-claim #3287 nonce=f534b55d6cc26f776ce9d97dff4428d1
devantler commentedon Sep 13, 2026
Slice 2 in review: #3761
Every hcloud consumer in the infrastructure layers now requires
platform.devantler.tech/baseline-worker In ["true"]in each requirednodeSelectorTerm:ksail.io/autoscaled DoesNotExist(inert)vault-snapshotCronJob,vault-snapshot-initJobProof (static):
scripts/tests/test-hcloud-baseline-worker-affinity.shrenders both hetzner overlays. It finds Coroot's hcloud components by walking the CR, so a component that gains hcloud storage later fails until it is guarded. Onmainit fails all 7 consumers; on the branch it passes all 7. Three ablations each failed for the right reason: an extra unlabelled OR term on the CronJob (1 of 2), the Keeper affinity removed, and OpenBao's value set to"false".Capacity, read-only 2026-09-13 ~06:05Z: memory requests on
prod-worker-{1,2,3}are 70% / 70% / 82%.coroot-clickhouse-shard-0-1(512Mi request) is the only Coroot pod on an autoscaler node, and its anti-affinity peer is onprod-worker-3, so it lands on worker 1 or 2.Still open after this slice:
storageClass: hcloudwith no node rule. It needs its own hetzner apps overlay patch, so it is the next slice.prod-worker-*, and that the rendered StatefulSets carry the rule.10 remaining items
chore: agent-claim #3287 nonce=244dee5378f2239d6653169cc2c5ec39
fix(openbao): re-approve the EKS authorization surface for the hostna…
devantler commentedon Sep 13, 2026
OpenBao slice is live: #3780 merged and verified in production
#3780 merged at 22:11:40Z at its evaluated head
17f45c4c. In its merge-queue run (34785385939),🚀 Deploy to Prodsucceeded at 22:11:07Z and the heal job was skipped. Every check below is a read-only production read taken after that.openbao/openbaoHelmRelease is Ready (Helm upgrade succeeded … openbao.v40).openbaoStatefulSet's required node affinity is nowkubernetes.io/hostname In [prod-worker-1, prod-worker-2, prod-worker-3], replacingksail.io/autoscaled DoesNotExist. A node label change can no longer place OpenBao, or itshcloudvolumes, on an autoscaler node.openbao-0on worker-2,openbao-1on worker-3 andopenbao-2on worker-1. That is the same placement as before, so nothing was rescheduled.This issue stays open for the Coroot
clickhouse-keepervolume it also tracks.chore: agent-claim #3287 nonce=5dc645798b770667b34109143357ee55
devantler commentedon Sep 14, 2026
Remaining work, re-measured 2026-09-14 (read-only)
1. OpenBao's running Pods do not carry the #3780 guard yet. The StatefulSet template requires
kubernetes.io/hostname In [prod-worker-1..3]. The update strategy isOnDelete, andcurrentRevisionstill differs fromupdateRevision. All three running Pods (openbao-0created 2026-08-26) still carry the earlierksail.io/autoscaled DoesNotExistrule, which matches every node today. OpenBao is on the static workers by placement, not by the guard. The guard takes effect only when those Pods are replaced. Replacing them is a deliberate OpenBao rolling step and is not part of this issue's code work.2. The keeper volume this issue still mentioned is covered.
coroot-clickhouse-keeperrequiresplatform.devantler.tech/baseline-worker In ["true"], and all three keepers run onprod-worker-{1,2,3}. Everycsi.hetzner.cloudVolumeAttachment is on aprod-worker-*node (3 / 6 / 5), and none is on an autoscaler node.3. What is left is the general guard. The first acceptance criterion says no Pod mounting an hcloud claim can schedule on an autoscaler node. The per-workload rules meet that only for the consumers that exist today and are known by name. #3794 adds the admission-time guard for any future consumer.
fix(kyverno): accept nodeSelector pinning and report failed claim loo…
fix(kyverno): judge a Pod that sets nodeName by that node, not its se…
devantler commentedon Sep 14, 2026
The general guard is live and verified: #3794 merged and deployed
#3794 merged at 05:28:02Z at its evaluated head
c81b7e81. Its merge-queue CI run, including the production deploy, succeeded. Read-only checks against production at about 06:00Z:1. No pod mounting an
hcloudclaim can schedule onto an autoscaler node. Therequire-baseline-workers-for-hcloud-volumesClusterPolicy isReady. Itsrequire-baseline-worker-affinityrule runs inEnforceon PodCREATE, so a future consumer without the baseline-worker affinity is refused at admission rather than scheduled.2. Negative control against the live node set. The background scan fails exactly 2 Pods. Both are completed
vault-snapshotPods from 2026-09-12 and 2026-09-13 that ran on an autoscaler node before the snapshot CronJob gained its required affinity. Today's 03:30Z snapshot ran on a baseline worker, and the CronJob template now requiresplatform.devantler.tech/baseline-worker In [true]. Results: 54 pass, 2 fail, 0 error, 0 claim-lookup failures. The 8PolicyViolationevents since the deploy are all background-scan reports on those 2 Pods. None is an admission refusal.3. Burst for other workloads is unchanged. 456 results are
skip(the Pod mounts nohcloudclaim). Avault-configJob Pod was scheduled onto an autoscaler node at 05:56Z and is reportedskip, so non-hcloudPods still burst onto autoscaler nodes.All three acceptance criteria hold, so I am closing this issue. The running OpenBao Pods still carry the older placement rule until they are replaced. That is an OpenBao rolling step, not code work here, and they already run on baseline workers.