Resolve the native huge-page allocation hold before V2 rollout

Description

The guarded client refresh stopped because a worker had not allocated the reviewed huge-page reservation after configuration deployment. This blocks the V1 baseline and V2 rollout. Configuration delivery alone did not establish the required host state.

Steps to Reproduce

Deploy the reviewed host prerequisites.
Run the confirmed protected client refresh.
Observe the allocation guard stop before moving Pods or restarting the affected worker's kubelet.

Expected Behavior

Every storage worker proves the required native allocation and recovers with matching advertised capacity before storage activation.

Actual Behavior

The allocation guard holds. No disk reset or V2 activation occurred.

Acceptance Criteria

Obtain bounded read-only native allocation and memory counters, bound to the observed worker identity.
Identify the remedy from actual evidence and implement it through reviewed protected production automation.
Verify every worker's allocation, advertised capacity, workload recovery and storage redundancy.
Complete the clean V1 baseline before V2 activation.

Do not lower the reservation, bypass the guard or assume that a failed read proves zero allocated pages. Any reboot must preserve data and obey disruption budgets.

Part of #5582

🔗 Related PRs

#5584 - chore(storage): prepare worker capacity for Longhorn V2 [merged]

📝 Issue Planner

_{Check the box below or use the @coderabbitai plan command to generate an implementation plan and prompts that you can use with your favorite coding assistant.}

Create Plan

🧪 Issue enrichment is currently in open beta.

You can configure auto-planning by selecting labels in the issue_enrichment configuration.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord!

chore: agent-claim #5791 nonce=bac0c4528e445f4959cd8b641b451119

feat(storage): inspect bounded native worker block layouts

chore: agent-claim #5791 nonce=84fa622c29a40705a8a4ef952c698a91

fix(storage): honor the host recovery deadline

docs(storage): explain the host reboot safety boundaries

A read-only cross-check found a production-queue dependency to account for in this recovery. Preparation run 41618559536 stopped at 10:28:50 UTC with the native host-memory observation incomplete or changed, and explicitly retained its journal and owned maintenance state.

At the subsequent live read, production deployment 41617563309 was still executing while infrastructure health checks were progressing and dependent reconciliation was blocked. The guarded follow-up in run 41619913150 had a pending job with no runner assigned. Both reviewed workflows use the same production concurrency group. This supports a possible serialization dependency while recovery waits; it does not establish a permanent deadlock.

The recovery checkpoint needs complete native allocation evidence tied to the existing journal, followed by workload recovery and a successful platform-health read. Preserve the existing journal, ownership and no-replay safeguards while the current owner resolves that checkpoint.

chore: agent-claim #5791 nonce=187b8166857b91ee666439e281882903

Read-only evidence for the serialization question above, gathered between 10:50 and 11:05 UTC. Nothing was changed.

The worker held by the preparation journal has been cordoned since 10:13:52 UTC. One secrets-store replica was evicted from it and has been Pending for about 45 minutes. The scheduler reports that no other node fits: the remaining workers are short on memory or ruled out by anti-affinity, and the autoscaler reports that its node groups are at maximum size.
The CI-runner namespace reaches its secret store only through a service that targets that one replica. Since 10:14 UTC that store has reported InvalidProviderConfig (connection refused). Every other tenant store is still Valid.
The infrastructure Kustomization timed out after 20 minutes on exactly that store (SecretStore … status: InProgress) and then started a new health-check window at 10:52 UTC. apps and the KHoist documentation Kustomization report DependencyNotReady behind it.

So the production deployment cannot become healthy while the cordon holds, and the guarded follow-up run is queued behind that deployment in the same concurrency group. That makes this a real wait cycle, which ends only when the deployment times out or the hold is released. It does not resolve by waiting alone. The decision to release or complete the hold stays with the journal's owner. This lane is not touching the node, the journal or either run.

The follow-up is no longer waiting for a runner. Preparation run 41619913150, attempt 1 at reviewed main 7d38b1bce1f0612f84a9df1d1b115e7abab3aa3e, completed with failure at 11:03 UTC. The preparation step again stopped on an incomplete or changed native host-memory observation and explicitly retained its journal and owned maintenance state.

The earlier production-deployment job was cancelled; its separate main-recovery job is now executing. These observations narrow the checkpoint: queue progress alone has not supplied allocation or workload-recovery evidence. Preserve the journal, ownership and no-replay safeguards while the existing recovery owner resolves the native observation. This run performed read-only verification and did not retry or alter production.

chore: agent-claim #5791 nonce=04c5559b59324817c850d56051d5d537

The main-recovery attempt is now a confirmed failure. Heal job 125877080168 finished at 11:33:52 UTC: “Re-deploy main to Production” failed, and the check for resources left behind by the unsuccessful deployment was skipped. A fresh platform-health read still reports failure. The earlier recovery-in-progress observation is therefore superseded; restoration and cleanup remain unverified.

Draft #5828 is the current host-memory observation repair, with a recent commit and validation still pending. Recovery acceptance still requires complete journal-bound native allocation evidence, successful main redeployment with its leftover-resource check, and workload/Flux recovery. This was an independent read-only verification; no production action or retry was performed.

The review checkpoint for #5828 has advanced. Codex completed a clean review of current head a6c044d506386035640895ec7c2d3247a28741cc. Its 51 reported checks are settled: 28 successful, 22 skipped and one neutral; the inline finding is resolved. This supersedes the review-pending observation, and establishes source-level validation only.

A fresh platform-health check still fails. Recovery acceptance remains open: the existing owner must complete the protected repair, obtain complete journal-bound native allocation evidence, and verify successful main redeployment, its leftover-resource check and workload/Flux recovery. This run performed independent read-only verification and did not retry or change production.

The source repair #5828 is now protected-merged at bcb59909aa5922828a3a93d67437414f39ab1b83 (12:42:42 UTC), retaining reviewed head a6c044d506386035640895ec7c2d3247a28741cc. Merge-group CI completed successfully. Its production-deployment and main-healing jobs were skipped, so this run establishes source delivery only.

The latest manual preparation remains the failed run 41619913150, and a fresh platform-health check still fails. Recovery acceptance therefore remains open: resume the retained journal through the protected current-main workflow, prove native allocation, and verify main redeployment, leftover-resource checks and workload/Flux recovery. Do not infer recovery from the merged repair or the successful merge-group checks. This was independent read-only verification; no production retry or mutation was performed.

chore: agent-claim #5791 nonce=165ad2d5fdf60518ab9141cc108868c1

chore: agent-claim #5791 nonce=b5676cf23a25fa0f14d69f255f26aeb4

The native recovery checkpoint has advanced. Protected preparation run 41638754979 completed successfully at 13:06 UTC on reviewed main bcb59909aa5922828a3a93d67437414f39ab1b83. Its guarded host-memory recovery reports that original storage identities, native allocation, advertised capacity and database identity were proved. Current-main validation also passed, and repeated independent platform-health reads now return healthy. This supersedes the earlier failed-health observation; it does not erase the failed deployment and healing attempts.

Follow-up preparation 41641900502 is still in progress under the existing recovery owner. Keep the issue open for the remaining worker/storage acceptance and clean V1 baseline. Successful main restoration and its leftover-resource check still need their own accepted evidence; aggregate health and this successful host operation do not establish those separate outcomes. This run performed independent verification only and did not retry or mutate production.

Follow-up verification: run 41641900502 completed with failure at 13:20:32 UTC; the guarded step reports that host-memory maintenance is held after an external-command failure. A subsequent live platform-health read is unhealthy again. The earlier successful operation retains its bounded proof, but no portfolio-wide recovery is claimed. The recovery owner's coordination lease is live and has been renewed; this run leaves its journal, maintenance state and production workflows untouched.

chore: agent-claim #5791 nonce=43c149bd1c942aef033e482c7a775a31

chore: agent-claim #5791 nonce=e72506a32207ce3e5cc493d0f18ffec3

Read-only follow-up: preparation run 41644616964 failed at 13:37:34 UTC on bcb59909aa5922828a3a93d67437414f39ab1b83. Its guarded step stopped because the reviewed source was no longer current main, and explicitly retained its journal and owned maintenance state. This is a different stop condition from the preceding external-command failure; it supplies no further allocation or storage acceptance.

Two independent platform-health reads in this run passed. Separately, Restore Stranded Prod 41638218705 reports workflow success but ran only its queue preflight: the restore job was skipped. That run therefore does not prove main restoration or its leftover-resource check. The remaining worker/storage acceptance and clean V1 baseline stay open.

The recovery coordination lease was freshly renewed. This verification leaves recovery with its existing owner and changes no production workflow, journal or maintenance state.

The protected host-memory recovery runs 41638754979 and 41646262163 completed successfully. Live readback confirms recovered journals, preserved storage and database identities, and released maintenance fences and cordons. The follow-up platform health check exits 0.

The remaining host operation is held by an independently owned recovery workload. Resume only after its owner verifies cleanup and a fresh inventory meets the reviewed capacity limit. No V2 activation or application-volume cutover has occurred; #5827 remains a draft behind its native preparation prerequisites.

Navigation Menu

Repository navigation

Resolve the native huge-page allocation hold before V2 rollout #5791

Description

Steps to Reproduce

Expected Behavior

Actual Behavior

Acceptance Criteria

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

Resolve the native huge-page allocation hold before V2 rollout #5791

Description

Description

Steps to Reproduce

Expected Behavior

Actual Behavior

Acceptance Criteria

Activity

coderabbitai commented on Nov 2, 2026

devantler commented on Nov 3, 2026

devantler commented on Nov 3, 2026

devantler commented on Nov 3, 2026

devantler commented on Nov 3, 2026

devantler commented on Nov 3, 2026

devantler commented on Nov 3, 2026

devantler commented on Nov 3, 2026

devantler commented on Nov 3, 2026

devantler commented on Nov 3, 2026

Metadata

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

Issue actions