Skip to content

feat: update the NPD binary through LPC as a separate npdBinary component - #9833

Draft
chmill-zz wants to merge 1 commit into
mainfrom
npdBinary
Draft

chmill-zz wants to merge 1 commit into
mainfrom
npdBinary

Conversation

@chmill-zz

Copy link
Copy Markdown
Contributor

What this PR does / why we need it:

Adds the node side of NPD binary updates through generic live patching. The AKS resource provider now publishes an npdBinary component (npd-binary-package-versions), separate from npdConfig; until this PR, nodes ignored it. Follows #9725, which baked both NPD packages and added npdConfig updates.

Separate component

  • npdBinary updates only node-problem-detector-kubernetes; npdConfig updates only node-problem-detector-aks-config. Both generic dispatchers (Ubuntu and Azure Linux 3) route them separately, so each gets its own checkpoint, status and events (AKS.LivePatching.npdBinary.*).
  • npd-update.sh runs one shared transaction per component (forward-only selection, download, validation, durable marker, held install, restart, health check, rollback to the cached prior package, interrupted-transaction recovery). Each component uses its own package, marker (pending vs pending-binary) and cached rollback package. The existing config tests pass unchanged.
  • Payload shape matches npdConfig, e.g. {"ubuntuPackageVersions":{"24.04":"0.8.25-ubuntu24.04u24"},"azureLinuxPackageVersions":{"3.0":"0.8.25-24.azl3"}}. Every release has its own binary build, so the 26.04→24.04 config mapping does not apply to the binary.

Binary-specific checks

  • Before install: the staged package must contain the NPD executables, and the new node-problem-detector --version must run on the node.
  • After restart: /healthz alone can pass while every apiserver call fails (as seen with the ML-KEM/OpenSSL issue fixed in the VM extension). The handler also requires NPD to refresh its node condition heartbeat (KernelDeadlock or ReadonlyFilesystem) after the restart, otherwise it rolls back. Rollback only requires local health, so an apiserver outage can't block recovery.

VHD

  • The VHD now also caches the binary package (about 20 MB) in /opt/node-problem-detector/downloads for offline rollback; the content test checks both cached packages and holds.

Validation

  • ShellSpec: 234 examples across the NPD handler (config and binary), VHD install helper, both dispatchers, security handlers and delivery tests, including component isolation (a config update never touches the binary and vice versa), the reporting check and rollback.
  • New npd-binary-integration.sh with real published packages and the real package manager, passed on Ubuntu 24.04 (u23→u24) and Azure Linux 3 (23.azl3→24.azl3): no-op, real held upgrade from PMC (through the tdnf lock bypass on Azure Linux), interrupted recovery, and rollback when NPD stops reporting. Only the restart and the reporting check are stubbed.
  • Existing config integration test still passes; verify_shell.sh and go test ./pkg/agent/... pass.

Not yet validated: the heartbeat check on a real node (that NPD refreshes lastHeartbeatTime promptly after a restart). I'll validate on a PR-built VHD before asking for merge.

Which issue(s) this PR fixes:
Fixes #

@github-actions

Copy link
Copy Markdown
Contributor

Windows Unit Test Results

  3 files   18 suites   1m 1s ⏱️
560 tests 560 ✅ 0 💤 0 ❌
563 runs  563 ✅ 0 💤 0 ❌

Results for commit 06378aa.

@github-actions

Copy link
Copy Markdown
Contributor

Code Coverage

Package Line Rate Health
parts_windows Package 1 73% ➖
staging_cse_windows Package 1 65% ➖
debug 0% ❌
provisioningscripts 2% ❌
test 0% ❌
windows 27% ❌
Summary 38% (1645 / 6827) ❌

@github-actions

Copy link
Copy Markdown
Contributor

Code Coverage

Package Line Rate Health
github.com/Azure/agentbaker/aks-node-controller 72% ➖
github.com/Azure/agentbaker/aks-node-controller/common 100% ✔
github.com/Azure/agentbaker/aks-node-controller/helpers 68% ➖
github.com/Azure/agentbaker/aks-node-controller/parser 88% ✔
github.com/Azure/agentbaker/aks-node-controller/pkg/gpu 100% ✔
github.com/Azure/agentbaker/aks-node-controller/pkg/nodeconfigutils 88% ✔
github.com/Azure/agentbaker/aks-node-controller/utils 0% ❌
github.com/Azure/agentbaker/apiserver 24% ❌
github.com/Azure/agentbaker/cmd 0% ❌
github.com/Azure/agentbaker/cmd/starter 0% ❌
github.com/Azure/agentbaker/fuzz/api 0% ❌
github.com/Azure/agentbaker/hotfix/render-nodecustomdata 0% ❌
github.com/Azure/agentbaker/pkg/agent 77% ➖
github.com/Azure/agentbaker/pkg/agent/datamodel 78% ➖
github.com/Azure/agentbaker/pkg/agent/toggles 0% ❌
github.com/Azure/agentbaker/pkg/vhdbuilder/datamodel 87% ✔
Summary 74% (5657 / 7592) ➖

@github-actions

Copy link
Copy Markdown
Contributor

Code Coverage

Package Line Rate Health
shellspec spec 28% ❌
Summary 28% (4550 / 15970) ❌

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant