fix(linux): remove bootstrap-kubeconfig after TLS bootstrapping

[Cameron Meissner (cameronmeissner)]

What this PR does / why we need it:

remove bootstrap-kubeconfig after TLS bootstrapping

Which issue(s) this PR fixes:

Fixes #

[github-actions]

Windows Unit Test Results

  3 files   18 suites   1m 4s ⏱️
560 tests 560 ✅ 0 💤 0 ❌
563 runs  563 ✅ 0 💤 0 ❌

Results for commit 5ed5ec9.

♻️ This comment has been updated with latest results.

[Copilot]

🟡 Changes recommended

Script delivery, logging initialization, startup-timeout coupling, and incompatible validators introduce startup and E2E regressions.

4 open findings

• Missing script delivery mapping breaks kubelet startup · New
• Unset EVENTS_LOGGING_DIR aborts cleanup hook · New
• Synchronous credential wait can time out kubelet startup · New
• Credential cleanup breaks post-provisioning validation · New

What changed in this PR

This PR reduces credential retention during Linux node provisioning by removing bootstrap inputs after use.

Changes:

• Adds bootstrap-kubeconfig cleanup through a kubelet post-start hook.
• Restricts bootstrap-kubeconfig permissions and deletes node-controller inputs.
• Adds ShellSpec coverage and an E2E cleanup assertion.

File	Description
spec/​parts/​linux/​cloud-init/​artifacts/​remove-bootstrap-kubeconfig_spec.sh	Tests cleanup behavior and deletion failures.
parts/​linux/​cloud-init/​artifacts/​remove-bootstrap-kubeconfig.sh	Waits for kubeconfig, then removes bootstrap credentials.
parts/​linux/​cloud-init/​artifacts/​kubelet.service	Invokes cleanup after kubelet starts.
parts/​linux/​cloud-init/​artifacts/​cse_config_kubelet.sh	Restricts bootstrap credentials to mode 0600.
parts/​linux/​cloud-init/​artifacts/​aks-node-controller-launcher.sh	Deletes provisioning inputs after execution.
e2e/​scenario/​validators.go	Checks that bootstrap credentials were removed.

🧠 Review effort: Balanced

---

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

🟡 Changes recommended

Cleanup can delete bootstrap credentials before a client certificate is available, preventing recovery after a kubelet restart.

1 open finding

• Verify client credentials before deleting bootstrap kubeconfig · New

4 resolved since last review

• Synchronous credential wait can time out kubelet startup
• Unset EVENTS_LOGGING_DIR aborts cleanup hook
• Missing script delivery mapping breaks kubelet startup
• Credential cleanup breaks post-provisioning validation

🧠 Review effort: Balanced

[Copilot]

🟡 Changes recommended

Deleting controller inputs after failed provisioning prevents subsequent attempts from provisioning the node.

2 open findings

• Cleanup on provisioning failure prevents subsequent recovery · New
• Verify client credentials before deleting bootstrap kubeconfig

🧠 Review effort: Balanced

[Copilot]

🟡 Changes recommended

Cleanup can trigger kubelet startup timeouts, and ANC input deletion conflicts with existing E2E validators.

2 open findings

• Unbounded ExecStartPost wait can restart kubelet during startup · New
• Cleanup deletes files required by existing E2E validators · New

2 resolved since last review

• Cleanup on provisioning failure prevents subsequent recovery
• Verify client credentials before deleting bootstrap kubeconfig

🧠 Review effort: Balanced

[aks-node-assistant]

AgentBaker Linux gate detective

Run: 186034915
Failed job/stage/task: e2e / Run AgentBaker E2E / AzureLinuxV2 node API deadline and broad scenario failures

Detective summary: Run AgentBaker E2E failed after 32 scenarios passed and 75 were skipped; AzureLinuxV2 did not appear in the Kubernetes API and node listing hit a client rate-limiter context deadline, after which the run reported 88 failed scenarios.

Likely cause / signature: e2e-shared-client-rate-limiter-context-deadline.

Confidence: Medium-high. Timeline isolates the failure to e2e/Run AgentBaker E2E, the task log shows client rate limiter context deadline during node readiness, build metadata ties this to PR 9797, and an existing wiki row tracks this generic control-plane/client-deadline family.

Recommended owner/action: Treat as shared E2E control-plane/client throttling or environment latency unless it repeats on this PR; no product code action is indicated from this single run.

Strongest alternative: PR 9797 removed bootstrap-kubeconfig after TLS bootstrapping, but the observed failure is a generic Kubernetes API/list timeout after many unrelated scenarios, not bootstrap token validation or kubelet credential cleanup.

Evidence: build run, failed timeline records for stage e2e/job Run AgentBaker E2E/task Run AgentBaker E2E, task log lines showing client rate limiter Wait returned an error: context deadline exceeded, source branch refs/pull/9797/merge, and wiki tracker.

Wiki: e2e-shared-client-rate-limiter-context-deadline

[Copilot]

🟡 Changes recommended

The new ShellSpec output expectations conflict with the helper’s diagnostics and fail on otherwise correct behavior.

1 open finding

• Cleanup script tests expect blank output despite diagnostics · New

2 resolved since last review

• Unbounded ExecStartPost wait can restart kubelet during startup
• Cleanup deletes files required by existing E2E validators

🧠 Review effort: Balanced

[Copilot]

🔵 Needs a closer look

E2E compilation failures, unsafe shared-subnet deletion, and lost Bastion write limits remain unresolved.

6 open issues

• Expected cleanup message does not match script output
• Cleanup setting missing from ANC typed configuration
• Synchronous certificate wait can time out kubelet startup
• Missing closing quote corrupts generated CSE command
• Cleanup helper missing from older supported VHDs
• OSGuard image omits bootstrap cleanup helper

1 resolved since last review

• Validator requires opt-in cleanup unconditionally

2 files not reviewed

• aks-node-controller/pkg/gen/aksnodeconfig/v1/bootstrapping_config.pb.go: Generated file
• aks-node-controller/pkg/gen/aksnodeconfig/v1/gpu_config.pb.go: Generated file

_{Review effort: Balanced}

[Copilot]

🟡 Changes recommended

Older-image compatibility and scriptless configuration and validation inconsistencies remain unresolved.

8 open issues

• Expected cleanup message does not match script output
• Cleanup setting missing from ANC typed configuration
• Synchronous certificate wait can time out kubelet startup
• Missing closing quote corrupts generated CSE command
• RemoveBootstrapKubeconfig flag is lost during config conversion · New
• Legacy VHD lacks cleanup helper, leaving credentials on disk · New
• Cleanup helper missing from older supported VHDs
• OSGuard image omits bootstrap cleanup helper

2 files not reviewed

• aks-node-controller/pkg/gen/aksnodeconfig/v1/bootstrapping_config.pb.go: Generated file
• aks-node-controller/pkg/gen/aksnodeconfig/v1/gpu_config.pb.go: Generated file

_{Review effort: Balanced}

[Copilot]

🔵 Needs a closer look

The missing closing quote in the Linux CSE template blocks provisioning even when cleanup is disabled.

2 open issues

• Legacy VHD lacks cleanup helper, leaving credentials on disk
• OSGuard image omits bootstrap cleanup helper

6 resolved since last review

• Expected cleanup message does not match script output
• Cleanup setting missing from ANC typed configuration
• Synchronous certificate wait can time out kubelet startup
• Missing closing quote corrupts generated CSE command
• RemoveBootstrapKubeconfig flag is lost during config conversion
• Cleanup helper missing from older supported VHDs

1 file not reviewed

• aks-node-controller/pkg/gen/aksnodeconfig/v1/bootstrapping_config.pb.go: Generated file

_{Review effort: Balanced}

[Copilot]

🟡 Changes recommended

Unconditional E2E enablement breaks the frozen Azure Linux V2 scenario because its image lacks the cleanup helper.

1 open issue

• Frozen AzureLinuxV2 image lacks cleanup helper · New

2 resolved since last review

• Legacy VHD lacks cleanup helper, leaving credentials on disk
• OSGuard image omits bootstrap cleanup helper

1 file not reviewed

• aks-node-controller/pkg/gen/aksnodeconfig/v1/bootstrapping_config.pb.go: Generated file

_{Review effort: Balanced}

[github-actions]

Package	Line Rate	Health
parts_windows Package 1	73%	➖
staging_cse_windows Package 1	65%	➖
debug	0%	❌
provisioningscripts	2%	❌
test	0%	❌
windows	27%	❌
Summary	38% (1645 / 6827)	❌

[Copilot]

🔵 Needs a closer look

Kubelet startup timeout and older-image compatibility concerns remain unresolved.

No open issues 🎉

1 resolved since last review

• Frozen AzureLinuxV2 image lacks cleanup helper

1 file not reviewed

• aks-node-controller/pkg/gen/aksnodeconfig/v1/bootstrapping_config.pb.go: Generated file

_{Review effort: Balanced}

[github-actions]

Package	Line Rate	Health
github.com/Azure/agentbaker/aks-node-controller	72%	➖
github.com/Azure/agentbaker/aks-node-controller/common	100%	✔
github.com/Azure/agentbaker/aks-node-controller/helpers	68%	➖
github.com/Azure/agentbaker/aks-node-controller/parser	88%	✔
github.com/Azure/agentbaker/aks-node-controller/pkg/gpu	100%	✔
github.com/Azure/agentbaker/aks-node-controller/pkg/nodeconfigutils	88%	✔
github.com/Azure/agentbaker/aks-node-controller/utils	0%	❌
github.com/Azure/agentbaker/apiserver	24%	❌
github.com/Azure/agentbaker/cmd	0%	❌
github.com/Azure/agentbaker/cmd/starter	0%	❌
github.com/Azure/agentbaker/fuzz/api	0%	❌
github.com/Azure/agentbaker/hotfix/render-nodecustomdata	0%	❌
github.com/Azure/agentbaker/pkg/agent	77%	➖
github.com/Azure/agentbaker/pkg/agent/datamodel	78%	➖
github.com/Azure/agentbaker/pkg/agent/toggles	0%	❌
github.com/Azure/agentbaker/pkg/vhdbuilder/datamodel	87%	✔
Summary	74% (5660 / 7595)	➖

[github-actions]

Package	Line Rate	Health
shellspec spec	28%	❌
Summary	28% (4548 / 15952)	❌