Skip to content

feat: bake Inspektor Gadget into Azure Linux 3 Kata VHDs - #9836

Merged
chmill-zz merged 1 commit into
mainfrom
kataIG
Oct 21, 2026
Merged

chmill-zz merged 1 commit into
mainfrom
kataIG

Conversation

@chmill-zz

Copy link
Copy Markdown
Contributor

What this PR does / why we need it:

Bakes Inspektor Gadget (IG) into Azure Linux 3 Kata (Pod Sandboxing) VHDs, so Kata uses the same VHD-owned IG as regular Azure Linux 3 instead of the copy the AKS VM extension installs.

Kata was left out when IG moved into the VHD (#7371) only to keep that PR's scope small. No technical reason was recorded. The Kata host kernel (kernel-mshv) has what IG needs: BTF, eBPF, kprobes, uprobes, tracepoints and syscall tracing, with lockdown off. It doesn't have BPF LSM, which IG doesn't use; Azure Linux 3 CVM already ships IG without it.

Changes

  • components.json: adds an azurelinuxkata entry for inspektor-gadget, mirroring azurelinux v3.0. Kata builds look up packages under azurelinuxkata, so before this no version resolved.
  • install-dependencies.sh / packer_source.sh: removes the Kata skips for the install and for the gadget import/remove scripts and unit. Both are removed together, since the build disables and imports through those files.
  • Content test: Kata now has to have the baked IG assets instead of having to lack them.
  • No CSE or E2E logic change. The AzureLinuxV3Gen2Kata scenario already runs ValidateInspektorGadget once skip_vhd_ig exists, including the trace_exec functional check.

What changes for Kata nodes

Kata hosts already run IG today. The extension installs IG 0.53.2 because Kata VHDs have no skip_vhd_ig marker. This PR changes who installs it and the version (0.53.2 → 0.57.0, matching regular Azure Linux 3). It does not add IG to Kata nodes for the first time.

Known limitations (same with the extension's IG today)

These come from where IG runs (on the host, outside the Kata guest VM). They apply equally to the extension-installed IG on Kata nodes, so baking keeps parity and is not a regression:

Gadget behavior on a Kata node Baked IG (this PR) Extension IG (today)
Host processes and runc pods: traced and enriched normally Yes Yes
ig list-containers lists Kata pods (via containerd) Yes Yes
top_process attributes each Kata pod's containerd-shim / cloud-hypervisor to the right pod Yes Yes
Processes inside a Kata guest VM (e.g. trace_exec of commands in a Kata pod) Not visible Not visible
top_cpu_throttle for Kata pods (throttling is visible only on the pod cgroup, which holds the VMM, shim and virtiofsd; there is no per-container cgroup on the host) Not reported Not reported

Upstream IG lists Kata Containers as unsupported for in-container visibility (inspektor-gadget/inspektor-gadget#5743). NPD's CPU-throttle IG check falls back to node-level stats on Kata (the host kernel also lacks PSI, microsoft/azurelinux#19417). This behaves the same with either IG.

Validation

  • Local: CUE validation, shellcheck, spec/vhdbuilder ShellSpec (451 examples, including install_ig_spec.sh), e2e go vet and components/config tests.
  • Disposable Kata node pool from the Kata VHD built for feat: bake node-exporter into Azure Linux 3 Kata VHDs #9810 (kernel 6.18.34.mshv3-1.azl3), with the RP-installed AKSLinuxExtension 1.519:
    • Before: the extension had installed IG 0.53.2 (no marker). Kernel prerequisites for IG were confirmed on the node.
    • Replaced it as this PR's VHD build does: PMC ig-0.57.0-3.azl3 and ig-gadgets-0.57.0-1.azl3, AgentBaker's import/remove scripts and disabled unit, gadget import (43/43) and skip_vhd_ig. The E2E trace_exec functional check passed.
    • Ran list-containers, trace_exec, top_process and top_cpu_throttle with runc, kata and kata-v2 pods on IG 0.53.2 (extension) and 0.57.0 (VHD). Results were identical; see the table above.
    • Extension coexistence: with the marker present, removing and re-adding the extension (its uninstall and install paths) only touched NPD. Baked IG and node-exporter were left untouched.
    • After a reboot: IG and its 43 gadgets were intact, trace_exec passed, and the Kata pods returned to Running.
    • Overhead: IG is a CLI with no resident service. A running gadget used about 100 MB RSS and about 11% of one CPU.
  • PR-gate pipeline on this branch: pending.

Which issue(s) this PR fixes:

Related: #9810 (node-exporter on Kata)

Kata was deferred when IG moved into the VHD, not excluded for a technical
reason. The Kata host kernel has BTF and the eBPF/tracing support IG needs.
Resolve ig through azurelinuxkata, copy the gadget import assets, and
require baked IG in content tests.

Kata hosts already run the VM extension's IG; baking it keeps the same host
view (host processes and runc pods, not workloads inside Kata guest VMs) and
moves ownership and version in line with regular Azure Linux 3.
@chmill-zz
chmill-zz requested review from a team as code owners October 20, 2026 20:31
Copilot AI balanced review requested due to automatic review settings October 20, 2026 20:31
@github-actions github-actions Bot added the components This pull request updates cached components on Linux or Windows VHDs label Oct 20, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Windows Unit Test Results

  3 files   18 suites   1m 15s ⏱️
563 tests 563 ✅ 0 💤 0 ❌
585 runs  585 ✅ 0 💤 0 ❌

Results for commit ed4e82c.

@github-actions

Copy link
Copy Markdown
Contributor

Code Coverage

Package Line Rate Health
parts_windows Package 1 73% ➖
staging_cse_windows Package 1 65% ➖
debug 0% ❌
provisioningscripts 2% ❌
test 0% ❌
windows 27% ❌
Summary 38% (1652 / 6834) ❌

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Manual validation is encouraging, but the actual Kata VHD build and PR-gate E2E results remain pending.

No open issues 🎉

What changed in this PR

Moves Inspektor Gadget ownership from the VM extension to the Azure Linux 3 Kata VHD, matching regular Azure Linux 3.

Changes:

  • Adds IG 0.57.0 for Azure Linux 3 Kata.
  • Enables baking IG packages, scripts, and service assets into Kata images.
  • Requires baked assets in content tests; updates E2E documentation without changing validation logic.
File Description
vhdbuilder/​packer/​test/​linux-vhd-content-test.sh Requires IG assets on Kata images.
vhdbuilder/​packer/​packer_source.sh Enables copying IG scripts and unit for Kata.
vhdbuilder/​packer/​install-dependencies.sh Removes the Kata installation exclusion.
parts/​common/​components.json Adds the Kata IG package version.
e2e/​scenario/​validators.go Clarifies Kata support in a comment.

Review effort: Balanced

@github-actions

Copy link
Copy Markdown
Contributor

Code Coverage

Package Line Rate Health
shellspec spec 28% ❌
Summary 28% (4510 / 15933) ❌

@github-actions

Copy link
Copy Markdown
Contributor

Code Coverage

Package Line Rate Health
github.com/Azure/agentbaker/aks-node-controller 72% ➖
github.com/Azure/agentbaker/aks-node-controller/common 100% ✔
github.com/Azure/agentbaker/aks-node-controller/helpers 68% ➖
github.com/Azure/agentbaker/aks-node-controller/parser 88% ✔
github.com/Azure/agentbaker/aks-node-controller/pkg/gpu 100% ✔
github.com/Azure/agentbaker/aks-node-controller/pkg/nodeconfigutils 88% ✔
github.com/Azure/agentbaker/aks-node-controller/utils 0% ❌
github.com/Azure/agentbaker/apiserver 24% ❌
github.com/Azure/agentbaker/cmd 0% ❌
github.com/Azure/agentbaker/cmd/starter 0% ❌
github.com/Azure/agentbaker/fuzz/api 0% ❌
github.com/Azure/agentbaker/hotfix/render-nodecustomdata 0% ❌
github.com/Azure/agentbaker/pkg/agent 77% ➖
github.com/Azure/agentbaker/pkg/agent/datamodel 78% ➖
github.com/Azure/agentbaker/pkg/agent/toggles 0% ❌
github.com/Azure/agentbaker/pkg/vhdbuilder/datamodel 87% ✔
Summary 74% (5660 / 7595) ➖

@chmill-zz
chmill-zz merged commit 11f28a8 into main Oct 21, 2026
52 checks passed
@chmill-zz
chmill-zz deleted the kataIG branch October 21, 2026 00:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

components This pull request updates cached components on Linux or Windows VHDs

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants