[Garrett Settles (gsettles01)]

Summary

On Azure VMs, WireServer SharedConfig can report an IPv4 address that is not assigned to the VM (seen: 10.10.10.10, on both Arm64 and amd64). Afterburn writes it to COREOS_AZURE_IPV4_DYNAMIC, so anything templated with {PRIVATE_IPV4} gets an address the VM does not have. That is what broke acl.etcd-member.discovery and cl.etcd-member.v2-backup-restore on Azure: etcd bound to or advertised 10.10.10.10. Those tests have since been retired on ACL along with the Docker-based etcd wrapper (#91, microsoft/azure-container-linux-mantle#48), but every other consumer of the metadata still gets the wrong address. cl.metadata.azure, made strict by microsoft/azure-container-linux-mantle#45, is now the regression test.

After Afterburn writes metadata, a new helper checks the route-selected source IPv4 against the interface's assigned addresses and IMDS (matching MAC and IP), and only then corrects COREOS_AZURE_IPV4_DYNAMIC. The first WireServer result is kept in /run/metadata/flatcar.wireserver. If any check fails, the metadata is left unchanged and coreos-metadata.service fails.

To keep CI working, merge in this order:

1. acl-pipelines !29414, which adds the testAzureMetadataIpv4 gate parameter. Until it merges, this PR's validation fails on that unknown parameter.
2. This PR.
3. [WIP] test(metadata): check Azure IPv4 against the assigned NIC azure-container-linux-mantle#45, which makes cl.metadata.azure check the address.

Change Log

• Add azure-metadata-ipv4 and a coreos-metadata.service drop-in that runs it as ExecStartPost; manglefs_rpm.sh installs both in RPM mode only.
• Add build_library/rpm/tests/test_azure_metadata_ipv4.sh (22 offline cases; needs only Bash and jq) and run it in PR validation via .pipelines/github-pr-validation.yml.
• Enforce cl.metadata.azure in acl/tests/kola_enforcing.yaml, with two exceptions:
  • QEMU: kola registers the test for Azure only, so QEMU never selects it. Without this exception the QEMU amd64 gate fails with "1 enforced test(s) not selected".
  • Azure acl-t (temporary): acl-t removes the oem-azure sysext that carries the helper. The test still runs on acl-t, but its failure is forgiven until acl-t installs the correction.

Type of Change

• Image build change (base image, sysexts, OEM images)
• Package/SPEC update
• CI/automation change
• SDK/toolchain update
• Configuration change
• Documentation update
• Bug fix

Does this affect the image build?

• Yes
• No

Adds the helper and drop-in to the Azure OEM sysext in RPM images. Other platforms and Portage builds are unchanged. acl-t images don't get the helper yet, because acl-t removes the OEM sysext.

Associated Issues

• ADO work item - AB#18833: acl.etcd-member.discovery Kola enforced test fails in Azure testing for aarch64
• Prerequisite: acl-pipelines !29414 (gsettles/18833-azure-metadata-ip-fix).
• Follow-up: [WIP] test(metadata): check Azure IPv4 against the assigned NIC azure-container-linux-mantle#45.
• Follow-up: carry the correction into acl-t, then remove the acl-t exception.
• Related: Retire Docker-based etcd and sdnotify-proxy from ACL RPM images #91 and Exclude legacy etcd-member tests on ACL azure-container-linux-mantle#48 retired the Docker-based etcd-member tests on ACL after this PR was opened.

Test Methodology

• Original failure: [ARM64]-Prod-BuildACL-Nightly 1204420.
• amd64 has the same root cause. In the aclmain nightly 1217971 (Azure amd64 kola), the three acl.etcd-member.discovery nodes got DHCP addresses 10.100.6.16, .21 and .22, and etcd failed on all three with listen tcp 10.10.10.10:2380: bind: cannot assign requested address. cl.etcd-member.v2-backup-restore advertised 10.10.10.10 while its NIC had 10.100.6.21.
• Offline: bash build_library/rpm/tests/test_azure_metadata_ipv4.sh passes 22/22 in the SDK container with no network, on a plain Ubuntu host, and in Azure Linux 3 using the pipeline step. Replacing the helper with exit 0 makes it fail.
• Azure Arm64, images built from this branch: two fresh 3-node clusters passed acl.etcd-member.discovery. A third failed on an unrelated DNS error while pulling the etcd image (systemd retried it successfully); its metadata was correct on all 3 nodes.
• Latest Azure run: on all 3 nodes the corrected IPv4 matched the NIC and IMDS, the WireServer value was preserved, and coreos-metadata.service succeeded with no restarts.
• The correction held through metadata regeneration and reboots on 3/3 nodes in each of the last two runs. An unfixed image reproduced the wrong address on 3/3 nodes.
• CI 1209202, on an earlier base: everything passed, including the new Test azure-metadata-ipv4 step on both architectures, except QEMU amd64 kola. It failed with "1 enforced test(s) not selected: cl.metadata.azure", which the QEMU exception fixes.
• Exception logic: ran acl-pipelines' scripts/evaluate_test_results.sh (from main) against the old and new kola_enforcing.yaml for QEMU, stock Azure and Azure acl-t on both architectures. The old file reproduces the 1209202 failure. The new file passes on QEMU, forgives a cl.metadata.azure failure only on Azure acl-t, and still blocks it on stock Azure. acl-pipelines' tests/test_exception_matching.sh passes 33/33 with the same tooling.
• Rebased onto aclmain at 9fe43d2. git range-diff shows four of the five patches unchanged; the one that sets testAzureMetadataIpv4: true only picked up new context from Enable IPE audit in GitHub ACL PR validation #113. On the rebased tree the offline test passes 22/22.
• Pipeline compile: a dry-run preview of github-acl-pr-validation (definition 5448) for this branch with acl-pipelines at !29414 compiles, and includes the offline test on both architectures. With acl-pipelines main it fails with "Unexpected parameter 'testAzureMetadataIpv4'", as expected until !29414 merges.
• End-to-end: acldevel run 1218369 built this branch at 12d9e9c (before Retire Docker-based etcd and sdnotify-proxy from ACL RPM images #91) with Mantle from [WIP] test(metadata): check Azure IPv4 against the assigned NIC azure-container-linux-mantle#45 at 4689fd6, for amd64 and aarch64 including acl-t, and ran the enforced (default) kola set. Every stage succeeded.
  • Stock Azure images, both architectures: acl.etcd-member.discovery, cl.etcd-member.v2-backup-restore and the strict cl.metadata.azure passed on the first attempt. In the unfixed aclmain nightly 1217971, both etcd tests failed every attempt on both architectures.
  • Node journals (5 Azure nodes per architecture): the helper logged azure-metadata-ipv4: using 10.100.6.x on eth0 (verified against IMDS) on every node, and etcd bound to and advertised the NIC address. No bind errors, no coreos-metadata.service failures, and no SELinux denials from the helper or coreos-metadata.
  • Negative control: acl-t images in the same run don't carry the helper. cl.metadata.azure failed every attempt there on both architectures with COREOS_AZURE_IPV4_DYNAMIC = "10.10.10.10", want Azure NIC private IP ..., and the acl-t exception forgave it as intended.
  • QEMU amd64 and aarch64 kola evaluations passed with cl.metadata.azure exempt, so the "not selected" failure from 1209202 is gone.
  • No new failures: every test that never passed in this run also never passed in nightly 1217971.
• Pending: acldevel 1230310 re-runs the end-to-end check above on the current head (0df0799) with Mantle from [WIP] test(metadata): check Azure IPv4 against the assigned NIC azure-container-linux-mantle#45 at b4144b3. Required CI checks on this PR can pass only after the acl-pipelines prerequisite merges; until then github-acl-pr-validation fails with "Unexpected parameter 'testAzureMetadataIpv4'" (for example 1230307).

Merge Checklist

All applicable boxes should be checked before merging.

• Prerequisite PR merged - acl-pipelines !29414
• Image builds successfully with this change (re-running on the current head in acldevel 1230310; 1218369 passed on the previous base, aclmain 6adc59d)
• Relevant kola tests pass (re-running in acldevel 1230310; 1218369 passed on Azure amd64 and aarch64 on the previous base)
• Required CI checks pass
• Ready to merge

[Copilot]

🔵 Needs a closer look

Human confirmation is needed that the shared-pipeline prerequisite has merged and required CI passes before approving this boot-time metadata change.

No open issues 🎉

What changed in this PR

Corrects Azure metadata IPv4 values in RPM images by validating the route-selected address against guest networking and Azure IMDS before releasing metadata consumers.

Changes:

• Adds synchronous validation and correction, preserving the original incorrect metadata.
• Adds 22 offline regression cases and enables their PR validation gate.
• Enforces cl.metadata.azure, with QEMU and temporary Azure acl-t exceptions.

File	Description
sdk_container/​src/​third_party/​coreos-overlay/​coreos-base/​oem-azure/​files/​manglefs_rpm.sh	Installs the helper and service drop-in for RPM images.
sdk_container/​src/​third_party/​coreos-overlay/​coreos-base/​oem-azure/​files/​azure-metadata-ipv4	Validates and corrects Azure IPv4 metadata.
sdk_container/​src/​third_party/​coreos-overlay/​coreos-base/​oem-azure/​files/​20-azure-ipv4.conf	Runs validation after Afterburn completes.
build_library/​rpm/​tests/​test_azure_metadata_ipv4.sh	Adds offline validation and packaging coverage.
acl/​tests/​kola_enforcing.yaml	Enforces Azure metadata testing with scoped exceptions.
.pipelines/​github-pr-validation.yml	Enables the offline regression gate.

_{Review effort: Balanced}

[Garrett Settles (gsettles01)]

Closing. The region survey on AB#18833 found the wrong IP is Azure's: WireServer reports 10.10.10.10 instead of the VM's IP in 5 of 9 regions tested, and in kola's region it has switched on and off since July. Nothing in ACL or AKS reads COREOS_AZURE_IPV4_DYNAMIC by default, and the etcd tests that did were retired in #91, so this workaround isn't needed. If it's ever needed, the better place is Afterburn, reading the IP from IMDS. No review needed, thanks.