[iamWHTWD]

Fresh Linux workers can omit an explicitly enabled runc handler when OverlayFS or EROFS is not registered. Prepare and verify both filesystems before sandboxd starts, using the selected node image. The init container mounts the existing host /lib read-only at /host/lib; already registered filesystems need neither a module tree nor modprobe. Missing filesystems require matching host modules and are loaded with modprobe -d /host, then checked again. The default non-runc DaemonSet remains unchanged.

The change also adds a volume-prerequisite regression and a maintained Kubernetes acceptance probe. Its negative mode accepts only a constructor-stage structured scheduling deadline with zero nodes/capacity and no_workers; transport, command, rollback and deletion failures cannot pass. Probe regressions cover pagination, cleanup and JSON-safe credential redaction.

Deployment guidance and AGENTS now document kernel/image/permission requirements, whole-Pod initialization failure, usable ADX capacity, operator detection and recovery. Related guides remove obsolete fork/CI campaign narratives and repeated historical artifact pins while retaining current manifests, build receipts and runtime/restore constraints.

Validation

Tested candidate: 1cd96ce93ac55a37ba196f63cbe38e59b647e8e2, incorporating develop b43d5481258b3536406e566d76fa32ed42a7d56b.

ECS check	Result
Acceptance-probe regressions	13 passed
Core chart tests	32 passed
SDK unit tests, including real TLS regressions	441 passed
Benchmark-driver tests	35 passed
make deploy-script-check	Passed
Kind/kubelet control, capacity exclusion, targeted rejection and recovery	Passed
Fresh-kernel loading and permission rejection with the selected image's userland	Passed

Local core/umbrella × standard/Kruise render checks and Helm lint also passed. Default complete DaemonSets match the base after YAML normalization; enabled init image/pull-policy inheritance and shell syntax were checked.

• Kind, Kubernetes 1.31 on Linux 6.8.0-138-generic: a real kubelet rejects the old mandatory missing-module-directory prerequisite. With filesystem-registration/module-source fault injection, the worker stays Kubernetes Ready while its init fails and ADX exposes no usable worker. Both negative campaigns returned DEADLINE_EXCEEDED/504, zero nodes/capacity and no_workers=1. Restoring the supported profile allowed targeted runc commands and private writes on the same worker; final running/paused inventory was empty.
• Fresh-kernel QEMU: verified 22 shell/grep/uname/kmod/loader/library/configuration files extracted from the exact node image. OverlayFS and EROFS were initially absent, missing modules failed explicitly, the image's kmod loaded both from a read-only module tree, and a repeat succeeded without modules or modprobe. After unloading EROFS inside the guest and disabling guest module loading, the candidate received real EPERM and EROFS stayed unregistered. The VM used 768 MiB, one vCPU, no network and no disk; host modules were not changed.
• Dependency resolution: the declared adx-environment==0.1.0rc4 wheel was resolved through the local configured index and provided as an explicit offline find-links source because the ECS mirror had not synchronized rc4. The version pin and ECS index were unchanged. Wheel SHA-256: 4931f9c456a9e0cd76b2e5703cf1f5084b135d3500b1f568e390f92af0997fc1.

Test image ID: sha256:e177d0e1d0e034f83e6f51891c856f5b3c8012e30ff9eeab874f16aea265d779; its OCI revision equals the candidate. ADX release commit: e611b9e4a83dcd6da5346bd17c2f6af28e244952. Init script SHA-256: d82ac3ec9312a46c3bc9a625c0fa716f2e0ed7dd7beba527af96fbb6d0429765.

Qualification boundary

Kind fault injection and the separate fresh-kernel image-userland VM are combined evidence, not one complete fresh Kubernetes worker module-loading run. Recovery covers graceful DaemonSet replacement and operator restoration; hard host loss, heartbeat-only recovery and cloud autoscaler/automatic replacement are unqualified. This chart provides an explicit operator recovery contract rather than installing automatic replacement.

Owned clusters, VMs, test image tags, credentials and remote temporary files were removed. Pre-existing container identities and start times were unchanged.

GitHub Actions validation for the pushed head: CI run 42159564888. ECS self-tests above are separate from this Actions run.

[WenYuLuo]

If the host lacks EROFS/Overlay support, the required kernel modules, or permission to load them, this init container may fail indefinitely. A newly provisioned Kubernetes node could therefore join the cluster successfully while remaining unavailable to AKernel workloads.

Could we assess and validate how the autoscaling and node recovery path detects and handles this state? In particular, can it avoid counting the node as usable AKernel capacity and trigger node replacement, an alert, or another recovery action? Please document any autoscaler or node-pool configuration this depends on, along with the validation results.

[rcyw]

Reviewed 81b722e16db684079c5ed770e2976b68cd841416 against b42a6912daa061c7b2c8438c368236e76d5c2a53.

Preparing and verifying host filesystems before starting an explicitly enabled runc runtime is a reasonable direction. The default non-runc DaemonSet remains unchanged in the independent render comparisons. I recommend addressing the following before merging.

1. R1 / P2 — preserve compatibility when the kernel already supports both filesystems but the host has no /lib/modules directory. The new unconditional hostPath.type: Directory requires that directory to exist before the init command can run. An already-capable kernel needs no module files, yet the whole AKernel node Pod is now blocked before the script can skip modprobe. Please adjust the mount/precondition strategy to preserve that supported path and add a regression covering the volume prerequisite. The current six shell cases begin after volume setup; asserting the literal Directory type does not test this behavior. See the inline comment on daemonset.yaml:326. Kubernetes documents the Directory requirement and storage preparation before init execution.

2. Document and qualify initialization failure and recovery. This also follows up on WenYuLuo's existing question. Please update the deployment guidance and root AGENTS, whose maintenance rules require updates when operational assumptions change:

   • State the supported host kernel/module prerequisites, selected-image requirements, module-loading permissions, observable failure state, troubleshooting steps, and recovery procedure. Explain that failed preparation blocks the entire node Pod, including other runtimes configured in that Pod.
   • Distinguish Kubernetes NodeReady from usable AKernel capacity. Source inspection suggests an init-blocked fresh worker never starts adxlet; that is not evidence that the deployed system incorrectly counts it as capacity, nor a substitute for checking the actual capacity/schedulability signal.
   • Document any node-pool/autoscaler dependencies and how an indefinitely failing worker is detected and handled. An explicit operator recovery/alerting procedure is acceptable if that is the intended contract; this does not require introducing an entire automatic replacement system in this PR.
   • Provide isolated-worker evidence for successful preparation and for an unavailable-module/load-permission failure: resulting Pod state, AKernel capacity/schedulability, detection, and the documented recovery action. Include a genuinely not-yet-registered filesystem case, or clearly identify any equivalent test coverage and remaining qualification boundary.

Independent validation on this exact candidate:

Check	Result
Chart tests	PASS: 31 tests, including the six rendered-shell cases
make deploy-script-check	PASS
Umbrella Helm lint with runc enabled	PASS
core/umbrella × standard/Kruise render matrix	PASS: default complete DaemonSet equals the base after YAML parsing; enabled init preserves the selected node image/pull policy; rendered script passes sh -n and ShellCheck
Missing-module-directory compatibility control	Base PASS / exit 0; head FAIL / exit 1 for the new mandatory hostPath prerequisite

The negative check renders the real chart and evaluates the Directory prerequisite against a temporary host-directory fixture. On the already-capable local kernel, the head's actual unmodified init command exits 0 without invoking the substituted modprobe. This is a manifest/precondition reproduction, not a live Kubelet or Pod reproduction; the real host module directory was not removed or changed.

CI run 41995321108 has ten successful checks, including deployment contracts and Standalone E2E; the image-publish job is skipped. Its merge commit e67b36cddae08e58bcd4cb77c67ab7e658b88a76 and this head have the same source tree. These checks do not establish fresh Kubernetes worker module loading or recovery behavior.

My independent validation did not perform real module loading/unloading, native mounts/runc execution, or cluster/autoscaler recovery. The author's reported native validation on a host where EROFS was already loaded is separate evidence.

Please fix R1, add the volume/precondition regression, supply the failure contract and recovery evidence, and rerun CI and the affected checks on the resulting head before merging.

[iamWHTWD]

@rcyw pushed 1cd96ce93ac55a37ba196f63cbe38e59b647e8e2, incorporating develop b43d548. The review fixes and updated validation are ready for another look.

R1: the init now reads host modules through read-only /lib → /host/lib. It checks filesystem registration first and requires the matching kernel module tree only when loading is necessary. Added the volume-prerequisite regression and verified the old missing-directory failure with a real kubelet control; the repaired already-registered/no-modules path succeeds.

Failure and recovery contract: deployment guidance and AGENTS now cover host/image/module-loading prerequisites, whole-Pod blocking, usable ADX capacity, troubleshooting and operator recovery. The intended contract is operator detection and repair/replacement; no cloud autoscaler or automatic replacement guarantee is assumed.

Independent ECS qualification on this exact candidate:

• 13 probe regressions, 32 chart tests, 441 SDK unit tests and 35 benchmark-driver tests passed; deployment syntax and local Helm/render checks passed.
• Kind fault injection produced a failing init on a Kubernetes Ready worker with an empty ADX resource view. Both targeted create negative campaigns returned structured DEADLINE_EXCEEDED/504 with zero capacity and no_workers=1. The maintained probe catches only construction-stage scheduling rejection; communication, command, rollback and cleanup errors fail acceptance.
• Two recoveries completed targeted runc execution and private-write isolation checks on that same worker. Final running/paused inventory was empty.
• A separate fresh-kernel VM used 22 hash-verified userland files from the exact test image, including its shell, grep, uname, kmod, loader and libraries. It passed real initial OverlayFS/EROFS loading and repeat execution without module files/modprobe. Guest module-loading prohibition then produced real EPERM and left EROFS unregistered.

Kind uses synthetic filesystem-registration/module-source failure inputs; the separate VM establishes real loading and permission failure with the selected image tools. This is combined evidence rather than a complete fresh-worker kubelet loading run. Recovery was graceful Pod replacement and operator restoration; hard host loss, heartbeat-only recovery and cloud automatic replacement remain outside qualification.

The updated PR description records the exact image/script identities and rc4 dependency-resolution provenance. The maintained probe and assertions are included, with their regression tests connected to CI. Historical CI/fork campaign narratives were also removed from deployment guides while preserving current constraints and authoritative manifests.

All owned ECS test resources and credentials were cleaned up; existing container identities/start times were preserved. New-head CI has been triggered and was in progress when this response was posted. Please re-review the changes and qualification boundaries.

[rcyw]

Re-reviewed 1cd96ce93ac55a37ba196f63cbe38e59b647e8e2 against develop b43d5481258b3536406e566d76fa32ed42a7d56b, including the author’s follow-up and the changes addressing the earlier review. The previous merge prerequisites are addressed, and I found no remaining blocking issue on this head. Preparing and verifying host filesystems for an explicitly enabled runc handler is a reasonable long-term design.

• R1 is resolved: the init mounts the host’s existing /lib read-only at /host/lib. It checks filesystem registration first, and only requires a matching module tree and modprobe -d /host when loading is necessary, followed by registration verification. The documented host/image/tool/permission prerequisites and module-tree limitations make the supported profile explicit.

• The failure/recovery contract is complete for this scope: deployment guidance and AGENTS now explain whole-node-Pod blocking, Kubernetes NodeReady versus usable ADX capacity, detection signals, and operator repair/replacement with drain and revalidation. Automatic node replacement is not promised.

• Independent compatibility verification passed: with the same module-free fixture containing an ordinary or usrmerged /lib, the original 81b722e candidate rejects the missing /lib/modules prerequisite and this head passes. The unmodified already-registered shell path succeeds without invoking modprobe or uname. Across core/umbrella × standard/Kruise, the default complete DaemonSets match current develop after YAML parsing; enabled-image/pull-policy inheritance and the security context also pass.

• Independent checks passed: 13 acceptance-probe regressions, 32 chart tests, make deploy-script-check, and the four-layout render/Helm lint/sh -n/ShellCheck checks. The rendered init script SHA-256 is d82ac3ec9312a46c3bc9a625c0fa716f2e0ed7dd7beba527af96fbb6d0429765, matching the reported qualification input. The negative probe correctly rejects transport, command, rollback and deletion failures, checks complete paginated running/paused inventory, and accepts only the structured zero-capacity scheduling rejection.

CI 42159564888 is successful for this exact head: ten jobs passed, covering Python 3.10–3.14 SDK tests, lint/type checks, distributions, deployment syntax/contracts and Standalone E2E. Publish to Docker Hub is expected to be skipped for the PR. Checkout logs identify merge 6f1ac8f57b6a3698957337e39d7304d5e4cec2ef; its tree matches the reviewed head.

The author’s Kind fault-injection/capacity/recovery results and separate fresh-kernel VM tests with the selected image’s userland are combined evidence. I did not rerun those cluster/VM campaigns locally. A complete first-module-loading run on one fresh Kubernetes worker remains a useful non-blocking extension; hard host loss, heartbeat-only recovery and cloud automatic replacement remain unqualified. These boundaries are now explicit and do not block the documented operator-recovery contract.

Recommend Approve for this head. Local verification used isolated worktrees and did not change host modules or deploy services. No merge performed.

[rcyw]

Approved 1cd96ce93ac55a37ba196f63cbe38e59b647e8e2 following the follow-up review and validation results.

The /lib/modules compatibility issue and initialization failure/recovery requirements are addressed. Independent checks and exact-head CI pass; the documented fresh-worker qualification extension is non-blocking. No remaining blocking findings on this head. No merge performed.