Decide how to split the Longhorn V2 preparation change

Problem

Draft #5596 did not converge. CodeRabbit's full review of 5064b6b raised three new major findings, two fix cycles after the reshape recorded at 2d33504, so the PR is parked and takes no more fix rounds. These findings have not been checked against the code beyond reading the lines they name:

The benchmark's completion wait uses the general readiness timeout, which may not cover the fio trials and their overhead.
The replica-exclusion test's target inventory may leave out the Node that triggers the policy, so the mutation it means to prove may not run.
The shared worker disk annotation may give the fixed migration nodes a disk before capacity activation, when their disk maps should stay empty.

Earlier cycles found problems in Pod placement, cordon release and retries. The one change holds several concerns: moving Pods, benchmarking, disk configuration, the workflow and the policy test. A finding in any of them restarts review of all of them.

Decision needed

Choose how to split or narrow #5596, so each part can reach a green review alone. Record which parts move to linked follow-up issues and the order they land in. The disk configuration comes first to decide, because it reaches every worker through shared Talos config.

Acceptance criteria

The split or narrowed scope is recorded here, with the head or PRs it produced.
Each of the three findings above is confirmed or refuted against the code in whichever part owns it.
feat(storage): add guarded preparation for Longhorn V2 #5596 is closed in favour of the split, or unparked with its narrower scope.

Part of #5582

Decision: split #5596 by the order its operations run, keeping every slice dormant until the last. A review finding in one operation should only restart review of that operation.

#	Slice	What it holds	Live effect when merged
1	Shared tool and `refresh-clients`	Kubernetes and journal helpers, pool verification, worker client refresh, the dispatch workflow	None: dispatch only, and it holds until later slices supply the V2 pool and host settings
2	Benchmark	`benchmark-v1`, `benchmark-v2`, fio assets	None until dispatched
3	Temporary capacity	Pool activation, admission and client-maintenance checks, the replica exclusion test, the V2 pool, the autoscaler eviction allowances	Admission policy and one extra fixed worker
4	Worker preparation	Evacuation, reset, Talos partition layout and default disk settings	New layout loaded; existing partitions do not change until a dispatch
5	Host settings	Huge pages and kernel modules	Worker kernel settings; last, because kubelet then needs the memory decided on #5582

Why this order. The benchmark requires huge pages to be ready on all three workers, so it cannot come before the refresh code. Capacity and preparation both change live objects, so they follow the parts that change nothing. A compile of slice 1 confirmed it needs the journal, node and pool helpers, which is why they sit there and not in a later slice.

The three findings, read against the code at 5064b6b.

Benchmark wait: confirmed. waitReady has a fixed 15 minute limit, and the fio trials alone take at least 12 minutes before preparation and status delay. Fixed in slice 2.
Worker disk defaults: looks valid, not yet proven. The shared worker config gives every worker the label and the default disk annotation, and capacity activation needs an empty disk map. What Longhorn does on a node that admission has fenced still needs testing. Owned by slice 4.
Replica exclusion test: not confirmed. The test ran in CI at this head with no failure, which argues against the finding, and it asserts the output is non-empty. I will re-run it natively in slice 3 before accepting or refuting it.

#5596 stays parked as the source for the later slices, and closes once the last slice is cut from it, with its history linked. Nothing in slice 1 has run on the real cluster.

chore: agent-claim #5655 nonce=7f9cb3dc1602346a045f2a88be89fe9f

Slice 1 merged at 07:19Z as #5657. Reviewing it found two serious gaps and four medium ones in code carried over from #5596, all now fixed there. Slice 2 is open as #5663 and holds the fix for the confirmed benchmark wait finding. #5596 stays parked as the source for slices 3 to 5.

Slice status, 2026-11-01.

Slice 1 (shared code and refresh clients): merged as feat(storage): add the guarded worker client refresh for Longhorn V2 #5657.
Slice 2 (benchmark): feat(storage): add the Longhorn V2 storage benchmark #5663 is in the merge queue at position 6, pinned to ce0343da3, with CI green and a local review round green at that head. It has not merged yet. CodeRabbit gave a full review one commit earlier and was rate limited at this head.
Slices 3 to 5 (temporary capacity, worker preparation, host settings): not cut. feat(storage): add guarded preparation for Longhorn V2 #5596 stays parked as their source and closes once the last slice is cut.

Not proven yet: nothing from these slices has run on the real cluster, and no test drives a whole benchmark. The Velero exclusion label on the scratch objects rests on documentation, not on a restore.

Follow-ups filed under #5582: #5665 (benchmark recovery) and #5666 (checking for a newer main during long runs).

chore: agent-claim #5655 nonce=b3b881f16fdab4523509dbf4ba19eb37

Slice 3 is now two changes, and the worker moves after slice 4. Reading the slice 3 files against the decision above found an ordering problem: the V2 pool's own comment says its worker is created after the worker layout changes, and slice 4 owns that layout. A worker created in slice 3 would come up with the old disk layout.

Slice	Holds	Live effect
3a, #5680	Admission and client-maintenance recognition of armed migration storage, the replica exclusion test	None. Read on 2026-11-01: 0 of 6 Longhorn nodes carry the enabled label, and no replica is on an autoscale node
4	Worker preparation, as decided	As decided
3b, after 4	The V2 pool, buffer changes, autoscaler eviction allowances, pool activation, the real-config pool assertion	One extra fixed worker, one warm spare released

Replica exclusion test finding: refuted. I ran it natively. The retrofit step produced the fence on the trigger with the trigger absent from the target inventory, and the stale-trigger control fails when its condition reads the trigger and not the target. Mutating the pool pattern or the target condition fails the test.

Before 3b: it needs the ninth node slot checked against whatever compute overflow is running at the time. Nothing here has run on the cluster with a label set.

Slice 4 is now two changes, and the disk-defaults finding is refuted.

Slice	Holds	Live effect
4a, #5686	The prepare-worker step, its journal, and the deploy gate's allowance for a cordon that journal holds	None. A dispatch stops before any write: the reviewed config does not resolve both pools until 3b
4b	Talos partition layout, the worker label moving to `config`, the default-disk annotation	New layout loaded on every worker; existing partitions do not change until a dispatch

The layout reaches every worker through shared Talos config, so it should not share a review with about 800 lines of tool code.

Worker disk defaults finding: refuted, read against the code. The finding said the shared annotation could give a fixed migration node a disk before activation. Longhorn only reads that annotation on a node with its default-disk label while create-default-disk-labeled-nodes is on, and it is on live. KSail's autoscaler worker config removes that label key from both machine.nodeLabels and the kubelet arguments, whatever its value (autoscaler_worker_config.go, #7013 there). Read live today: all three autoscale nodes carry no such label and an empty disk map, and the three permanent workers carry it. Not proven: what Longhorn does with the annotation on a real node, because 4b has not deployed. 4b should read this back on the first autoscale node after it deploys.

Test gaps found in the carried code. Running mutants against the destructive paths, four of nine survived: the budget check at start, the loaded-layout and partition-identity checks before reset, and the post-reset disk check. The V2-engine test was passing for the wrong reason, stopping at an earlier check. All four are now killed by tests in #5686.

For 4b:

It should land after feat(storage): recognise armed Longhorn migration storage #5680, which teaches client maintenance to accept the config label value. Before that, client maintenance would find no baseline storage workers.
A permanent worker replaced after 4b gets the raw-disk default rather than a V1 disk, while V1 is still in use. Decide this before merging.
The partition checks match the field names in Talos machinery v1.14.4, but none of them has read a real node. This host's Talos credential was refused the volume read on an autoscale node, so that stays unproven until a dispatch.

Slice 4b is open as draft #5689, now that #5680 has merged. It should merge after #5686. Its description lists what changes when it deploys and what is not proven, including a smaller temporary partition on new overflow nodes. The V2 pool (3b) and host settings (5) are still to cut.

Slice 3b is now two changes, because production has no free node slot. Read on 2026-11-01: nine nodes against a nine-node ceiling, including one ordinary overflow node (autoscale-cx43) kept alive by the warm spare.

Slice	Holds	Live effect
3b-1, #5698	Warm spare released; eviction allowances for Crossplane, the Hubble relay and the Coroot agent	Overflow node retires, eight nodes, one server less
3b-2, after it	The V2 pool, its buffer and pod template, the longer readback, the real-config pool assertion, `verifyV2EngineEnabled`	One fixed worker added, nine nodes

Why split it. In one change, the deploy gate would wait for a worker that can only start after another node retires. A pod that blocked the retirement would fail the gate, evict the PR and trigger a heal. Separately, the retirement is read back first, and only then does the worker depend on it.

Live read of the overflow node. Crossplane and the Coroot agent are the two pods holding emptyDir volumes without an allowance. Its requests are 21% CPU and 30% memory, and each migration worker read about 20%, so its pods fit there. The permanent workers are at 87-99% of memory requested, so that spare room is what makes this safe. The relay is not on the node today, and its allowance covers it landing there later.

Not proven: that the node retires, and that the Coroot operator passes the annotation to its pod. Both get read back after #5698 deploys.

docs(storage): name the buffer the migration note refers to

chore: agent-claim #5655 nonce=2e0a743ac0b34765d6ab6ad367897575

The overflow node has retired, and slice 3b-2 is open as draft #5726. #5719 deployed, and the autoscaler stopped refusing the node: it was marked unneeded from 07:38Z and removed about 07:49Z. Read at 07:53Z on 2 November: eight nodes, no Pending pods. The per-workload allowances were enough this time, but #5726 needs the slot to still be free when it is queued, so its description asks for a fresh node count first.

Change to the slice list. verifyV2EngineEnabled moves out of 3b-2. Its only caller is the capacity activation step, which is not on main, so it goes with the slice that adds enable-capacity. Still to cut after #5726:

Slice	Holds
Capacity activation	`bootstrap.go`, the `enable-capacity` dispatch and `verifyV2EngineEnabled`
5	Host settings: huge pages and kernel modules

chore: agent-claim #5655 nonce=11ce4f048f2a5d779b2b499a9044f000

chore: agent-claim #5655 nonce=cb24766a7b51b738fe5932ce7238790d

chore: agent-claim #5655 nonce=a1aed779d14e4f2a6a4321c72ce26709

#5726 was evicted at 09:25Z on 2 November, and the cause is a limit the plan missed. The namespace's generated quota, 8 CPU and 16 GiB, also caps virtual buffer demand, and the V1 buffer uses all of it. The V2 buffer was limited to zero, so no worker was requested. No node was created, and the eviction started the normal heal.

Order	PR	Holds	State
1	#5744	Exempts the namespace from the quota	Queued. After it deploys, read that the quota object is gone
2	#5726	The V2 pool	Re-queue after that read, with a fresh node count
3	#5747	Capacity activation, no live effect	Draft. Rebase after #5726, then review
4	Not cut	Slice 5, host settings	After those

The 3b-2 readback worked as designed: it failed the deploy when the worker did not appear, rather than accepting one less node.

fix(storage): place the V2 migration buffer outside the generic quota

The order needs one change: host settings come before #5747, not after it. #5726 merged at 17:12 UTC. #5747 now carries it and has a clean Codex review, but it cannot meet the "tried as a user" rule, because a real activation needs the benchmark baseline and the baseline needs the host settings.

Order	Item	State
1	Slice 5, host settings	Not cut. Next
2	V1 benchmark baseline	Needs slice 5
3	#5747, capacity activation	Draft, parked until a baseline exists

I have not read the deployed V2 worker yet, so this does not confirm #5726's rollout.

Host settings are cut as draft #5774, and they found a fault that would have stopped the baseline. The refresh step looked for nvme_tcp in /proc/modules. That module is built into the Talos kernel on the running workers, so it is never listed there and refresh-clients would have held on every worker. #5774 reads /sys/module instead and leaves nvme_tcp out of the load list.

Order	Item	State
1	#5774, host settings	Draft, in review
2	Dispatch `refresh-clients`	Promptly after #5774 deploys: until then the scheduler does not count the 2 GiB, and permanent workers are 93-99% requested
3	V1 benchmark baseline	Needs step 2
4	#5747, capacity activation	Draft, parked until a baseline exists

#5726's rollout, read at 18:03 UTC on 2 November: nine nodes, autoscale-longhorn-v2 Ready, no Pending pods. I have not read the buffer replicas.

chore: agent-claim #5655 nonce=4fe010799e4dd306234a3604b81fa578

chore: agent-claim #5655 nonce=7924a2296f5cadd2f5f06ad6ab59f7f4

The split decision is complete. #5596 is now closed in favor of its replacements; its retirement receipt records the complete slice map and the three review dispositions.

The last uncut pieces are now retained in #5747 (capacity activation, still draft) and #5774 (host settings, queued). Earlier slices are merged. A fresh file audit against main 6a88aff432b72debd2a9312b04f8271ea563aaa4 confirms the remaining bootstrap files belong to #5747, the host-settings file belongs to #5774, and the old wait/workflow-path tests have maintained successors. The benchmark finding was confirmed and repaired; the inventory and disk-default claims have their recorded refutations on this issue.

Closing the research issue as completed does not close #5582, #5665 or #5666, admit capacity activation, accept the benchmark baseline, or certify live disk behavior. Those implementation and runtime gates remain with the existing work.

chore(storage): integrate current inventory workflow controls

Navigation Menu

Repository navigation

Decide how to split the Longhorn V2 preparation change #5655

Problem

Decision needed

Acceptance criteria

3 remaining items

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

Decide how to split the Longhorn V2 preparation change #5655

Description

Problem

Decision needed

Acceptance criteria

Activity

devantler commented on Nov 1, 2026

devantler commented on Nov 1, 2026

devantler commented on Nov 1, 2026

devantler commented on Nov 1, 2026

devantler commented on Nov 1, 2026

devantler commented on Nov 1, 2026

devantler commented on Nov 1, 2026

3 remaining items

devantler commented on Nov 2, 2026

devantler commented on Nov 2, 2026

devantler commented on Nov 2, 2026

devantler commented on Nov 2, 2026

devantler commented on Nov 2, 2026

Metadata

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

Issue actions