[Adams Anaglo (adamsanaglo)]

What this PR does / why we need it:
Builds on the ACL VHD->COSI conversion work to upload the COSI directly to PMC via its AFD upload endpoint under PMC's service connection.

• Add cosi-upload, a Go tool that uploads via the Azure Blob SDK's chunked block-blob upload to avoid the single Put Blob size limit on large COSI images.
• Add upload-cosi-to-pmc.sh and packer.mk targets.
• convert-vhd-to-cosi.sh stages the COSI for upload; cosi-publishing-info.json uses the AFD download URL and adds sha1.
• Normalize the COSI package version to strict SemVer for Nebraska: convert-vhd-to-cosi.sh strips per-component leading zeros (e.g. 202608.06.0 -> 202608.6.0) and appends -fips for FIPS builds (driven by ENABLE_FIPS, never duplicated), writing the result to cosi-publishing-info.json.image_version. IMAGE_VERSION and SIG versions are unchanged. COSI blob names include the SKU to keep architecture/FIPS variants distinct. Covered by ShellSpec tests.
• Distinguish the ARM64 COSI conversion jobs in the release pipeline.
• Keep PMC publishing out of PR/non-ACL template expansion, and wait for immutable VHD copy completion before handing a read-only SAS to native-arm64 conversion.
• Add azblob and azidentity dependencies.

Which issue(s) this PR fixes:

Fixes #

[Copilot]

Pull request overview

This PR extends the Azure Container Linux (ACL) VHD → COSI flow by adding a dedicated upload path to PMC via an Azure Front Door (AFD) upload endpoint, wiring the new conversion + upload steps into the VHD builder release pipeline, and introducing a small Go-based uploader that uses Azure Blob SDK block uploads to avoid single-request size limits.

Changes:

• Add an ImageCustomizer-based VHD→COSI conversion script that stages the artifact and emits cosi-publishing-info.json (including sha1).
• Add a Go CLI (cmd/cosi-upload) plus a wrapper script to upload large COSI artifacts via chunked block-blob upload using Azure CLI (Entra ID) auth.
• Wire new “Convert ACL VHD to COSI” and “Upload COSI to PMC” steps into the builder release template and set ImageCustomizer image variables in pipeline definitions.

Reviewed changes

Copilot reviewed 8 out of 10 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
vhdbuilder/packer/imagecustomizer/scripts/convert-vhd-to-cosi.sh	Downloads the staged ACL VHD, converts it to COSI via ImageCustomizer, stages the COSI, and generates cosi-publishing-info.json with hashes.
vhdbuilder/packer/imagecustomizer/scripts/upload-cosi-to-pmc.sh	Validates expected environment + artifacts and invokes the Go uploader to push to the PMC AFD upload endpoint.
cmd/cosi-upload/main.go	Implements block/chunked upload using azblob with Azure CLI credential for pipeline auth.
packer.mk	Adds convert-vhd-to-cosi, build-cosi-upload, and upload-cosi-to-pmc targets used by pipeline steps.
.pipelines/templates/.builder-release-template.yaml	Adds pipeline tasks to convert ACL VHD to COSI and upload COSI to PMC, and publishes cosi-publishing-info.json.
.pipelines/.vsts-vhd-builder.yaml	Sets ImageCustomizer container image variables (primary + GHCR fallback) for relevant stages.
.pipelines/.vsts-vhd-builder-release.yaml	Sets ImageCustomizer container image variables (primary + GHCR fallback) for relevant stages.
go.mod	Adds Azure Blob SDK + Azure Identity dependencies required by the new uploader tool.
go.sum	Updates module checksums for newly introduced Go dependencies.

[aks-node-assistant]

AgentBaker Linux gate detective

Run: https://msazure.visualstudio.com/CloudNativeCompute/_build/results?buildId=182176653
Failed job/stage/task: build / build2204fipsgen2containerd / Test, Scan, and Cleanup.

Detective summary: Test/Scan completed the VHD scan, then the CIS baseline comparison again found Ubuntu 22.04 FIPS rules 5.1.18 and 5.1.6 regressed from pass to fail. This is the existing repair-linked CIS baseline-drift signature.

Likely cause / signature: $sig.

Confidence: High for recurring baseline drift.

Recommended owner/action: VHD validation owners should work repair #39802117 to reconcile the CIS baseline or fix image content.

Strongest alternative: PR-specific component changes caused the CIS diff; less likely because the same two CIS rules reproduced across unrelated PRs and manual branches.

Evidence: Timeline marks Test/Scan as succeededWithIssues; task log reports CIS regressions detected: 2 with 5.1.18|pass->fail and 5.1.6|pass->fail; existing wiki row and repair item track this recurring signature.

Wiki signature: $sig; repair #39802117

[Copilot]

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Copilot review overview

1 open finding

• If E2E_GO_TEST_TIMEOUT is unset/empty, this produces -timeout \"\" which go test treats as an… · New

1 resolved since last review

• ShellCheck flags C-style loop in POSIX shell validation

🧠 Review effort: Lite

[Copilot]

🔵 Needs a closer look

Marketplace metadata and COSI acceptance checks need correction, and the cross-architecture publishing path warrants final human review.

5 open findings

• Marketplace ACL metadata is missing from VHD publishing info · New
• COSI validation accepts malformed partition layouts · New
• Conflicting duplicate image hashes are silently overwritten · New
• COSI validation does not verify artifact architecture · New
• If E2E_GO_TEST_TIMEOUT is unset/empty, this produces -timeout \"\" which go test treats as an…

🧠 Review effort: Lite

[Copilot]

🟡 Changes recommended

An unsupported Azure CLI option blocks COSI handoff, and the content validator accepts incomplete compression metadata.

9 open findings

• Blob existence check uses unsupported --blob-url option · New
• Validator accepts unsupported disk LBA sizes · New
• COSI validation accepts invalid disk LBA sizes · New
• Compression validation does not require maxWindowLog · New
• COSI validation does not verify artifact architecture
• Conflicting duplicate image hashes are silently overwritten
• COSI validation accepts malformed partition layouts
• Marketplace ACL metadata is missing from VHD publishing info
• If E2E_GO_TEST_TIMEOUT is unset/empty, this produces -timeout \"\" which go test treats as an…

🧠 Review effort: Lite

[Copilot]

🟡 Changes recommended

Concurrent COSI scenarios can fill the E2E runner’s temp volume with multiple multi-GB downloads.

9 open findings

• Concurrent COSI downloads can exhaust runner disk and network capacity · New
• Blob existence check uses unsupported --blob-url option
• Compression validation does not require maxWindowLog
• COSI validation accepts invalid disk LBA sizes
• Validator accepts unsupported disk LBA sizes
• COSI validation does not verify artifact architecture
• Conflicting duplicate image hashes are silently overwritten
• COSI validation accepts malformed partition layouts
• If E2E_GO_TEST_TIMEOUT is unset/empty, this produces -timeout \"\" which go test treats as an…

1 resolved since last review

• Marketplace ACL metadata is missing from VHD publishing info

🧠 Review effort: Lite

[Copilot]

🟡 Changes recommended

Unresolved script and Packer compatibility issues block approval.

4 open findings

• Exporter uses wrong non-staging blob container · New
• ACL CVM build copies missing override files · New
• Blob existence check uses unsupported --blob-url option
• If E2E_GO_TEST_TIMEOUT is unset/empty, this produces -timeout \"\" which go test treats as an…

7 resolved since last review

• Concurrent COSI downloads can exhaust runner disk and network capacity
• Compression validation does not require maxWindowLog
• COSI validation accepts invalid disk LBA sizes
• Validator accepts unsupported disk LBA sizes
• COSI validation does not verify artifact architecture
• Conflicting duplicate image hashes are silently overwritten
• COSI validation accepts malformed partition layouts

🧠 Review effort: Lite

[bfjelds (bfjelds)]

in my previous testing, i modified the test framework's firewall to allow the POC nebraska server instead of this download-upload-download cycle ... when i try this approach, i see timeouts in the test.

[bfjelds (bfjelds)]

in case it is helpful, here are the firewall changes (i think) that i was using:

The Nebraska firewall change is entirely in two files, introduced by  cf69abae3e : 45 added lines, 8 removed. It has two parts—allowing the endpoint and ensuring existing shared firewalls receive the rule.

1. Define and install the Nebraska allow rule

In  e2e/scenario/aks_model.go :

Declare the endpoint:

const nebraskaPocFqdn = "nebraska-poc-download-ep-hjf7e5fseafnejha.b01.azurefd.net"

Create an application rule permitting HTTPS on port 443:

nebraskaPocRule := armnetwork.AzureFirewallApplicationRule{
    Name:            to.Ptr("nebraska-poc-fqdn"),
    SourceAddresses: []*string{to.Ptr("*")},
    Protocols: []*armnetwork.AzureFirewallApplicationRuleProtocol{
        {
            ProtocolType: to.Ptr(armnetwork.AzureFirewallApplicationRuleProtocolTypeHTTPS),
            Port:         to.Ptr[int32](443),
        },
    },
    TargetFqdns: []*string{to.Ptr(nebraskaPocFqdn)},
}

Append it to the existing Allow application-rule collection:

Rules: []*armnetwork.AzureFirewallApplicationRule{
    &aksAppRule,
    &blobStorageAppRule,
    &mooncakeMARRule,
    &dmcRule,
    &nebraskaPocRule,
},

This modifies the shared E2E firewall, not just the COSI scenario’s VM.

2. Detect existing firewalls missing the rule

In  e2e/scenario/shared_infra.go:491–533 ,  firewallAppRulesUpToDate()  previously checked only the test-storage hostname. The change requires both rules:

expectedStaticRuleFqdns := map[string]string{
    "blob-storage-fqdn": config.Config.BlobStorageAccount() + ".blob.core.windows.net",
    "nebraska-poc-fqdn": "nebraska-poc-download-ep-hjf7e5fseafnejha.b01.azurefd.net",
}

It scans the existing application rules, records their target FQDNs, and returns  false  if either expected rule is missing or points elsewhere:

for name, expectedFqdn := range expectedStaticRuleFqdns {
    if foundRuleFqdns[name] != expectedFqdn {
        return false
    }
}
return true

Why this second part matters: the firewall persists between runs. Adding a rule to its creation model alone wouldn’t update an already-existing firewall.

The pre-existing  ensureSharedFirewall()  reconciliation code then updates a stale firewall in place, preserving its public IP and waiting for completion. No new reconciliation machinery, YAML variable, or manual firewall change was needed.

[Cameron Meissner (cameronmeissner)]

I'd just use an if/else here rather than having a needless step in the acldevel case

[Cameron Meissner (cameronmeissner)]

why do we need the fallback? using not using mcr.microsoft.com will most likely raise supply chain analysis warnings

[bfjelds (bfjelds)]

agree. in early development it was because imagecustomizer azurelinux release was lagging, but at this point we probably don't need it.

[Cameron Meissner (cameronmeissner)]

would prefer if we align all parameter/variable naming style to CAPITAL_SNAKE_CASE

[Cameron Meissner (cameronmeissner)]

let's create a separate go module

[Cameron Meissner (cameronmeissner)]

this YAML Is sourced by pipelines running in other tenants, not just TME - would prefer we attach at this. via the ADO library rather than directly through YAML

[Cameron Meissner (cameronmeissner)]

do we plan on using this template elsewhere? if not it would be simpler to just include that template's content directly within this template

[Cameron Meissner (cameronmeissner)]

this is a large script, would prefer we factor it out to a dedicated bash file somewhere under vhdbuilder - that way we can also write shellspec tests against it as well

[Copilot]

🔵 Needs a closer look

The broad release and pipeline scope includes an unresolved conversion-job dependency issue requiring human review.

4 open issues

• ACL CVM build copies missing override files
• Exporter uses wrong non-staging blob container
• Blob existence check uses unsupported --blob-url option
• If E2E_GO_TEST_TIMEOUT is unset/empty, this produces -timeout \"\" which go test treats as an…

_{Review effort: Lite}

[github-actions]

Package	Line Rate	Health
parts_windows Package 1	73%	➖
staging_cse_windows Package 1	65%	➖
debug	0%	❌
provisioningscripts	2%	❌
test	0%	❌
windows	27%	❌
Summary	38% (1652 / 6834)	❌

[github-actions]

Package	Line Rate	Health
shellspec spec	28%	❌
Summary	28% (4618 / 16254)	❌

[github-actions]

Package	Line Rate	Health
github.com/Azure/agentbaker/aks-node-controller	72%	➖
github.com/Azure/agentbaker/aks-node-controller/common	100%	✔
github.com/Azure/agentbaker/aks-node-controller/helpers	68%	➖
github.com/Azure/agentbaker/aks-node-controller/parser	88%	✔
github.com/Azure/agentbaker/aks-node-controller/pkg/gpu	100%	✔
github.com/Azure/agentbaker/aks-node-controller/pkg/nodeconfigutils	89%	✔
github.com/Azure/agentbaker/aks-node-controller/utils	0%	❌
github.com/Azure/agentbaker/apiserver	24%	❌
github.com/Azure/agentbaker/cmd	0%	❌
github.com/Azure/agentbaker/cmd/starter	0%	❌
github.com/Azure/agentbaker/fuzz/api	0%	❌
github.com/Azure/agentbaker/hotfix/render-nodecustomdata	0%	❌
github.com/Azure/agentbaker/pkg/agent	78%	➖
github.com/Azure/agentbaker/pkg/agent/datamodel	77%	➖
github.com/Azure/agentbaker/pkg/agent/toggles	0%	❌
github.com/Azure/agentbaker/pkg/vhdbuilder/datamodel	87%	✔
Summary	74% (5833 / 7825)	➖

[Copilot]

🟡 Changes recommended

Unresolved critical Azure CLI bootstrap and additional pipeline, version propagation, and COSI execution issues block approval.

5 open issues

• Configure Azure CLI repository before ARM64 installation · New
• ACL CVM build copies missing override files
• Exporter uses wrong non-staging blob container
• Blob existence check uses unsupported --blob-url option
• If E2E_GO_TEST_TIMEOUT is unset/empty, this produces -timeout \"\" which go test treats as an…

_{Review effort: Lite}

[Peter Damianov (pdamianov-dev)]

I dont think we should create an image version to set if packer doesn't pass in $IMAGE_VERSION, we should probably just fail

[Peter Damianov (pdamianov-dev)]

does this IMAGE_VERSION_FILEPATH file need to be updated or does trident agent know to switch its context to the new partition's image and corresponding image version file path?

[Peter Damianov (pdamianov-dev)]

like is there a mechanism on trident side that knows it has to change to the new image version? the comments for this file suggest this file DOES NOT change during update

context: Line 120: This file lives on the root partition, so it describes the node image the node was provisioned from and is not replaced by an A/B update of /usr.

[bfjelds (bfjelds)]

this is a complicated question. the problem is that ACL creates an updatable partition /usr and AgentBaker cannot currently change that parition because it cannot reseal/resign /usr and the ukis. that is why the comment is left here.

in the future, the AKS ACL image build will need to be changed to make everything updatable that the current build puts on /opt and /etc. but that hasn't been invented yet.

[Peter Damianov (pdamianov-dev)]

Potential current/target version mismatch: this normalizes the published version (for example, 202608.06.0 -> 202608.6.0, adding -fips when enabled), but post-install-dependencies.sh writes the raw IMAGE_VERSION into /opt/azure/containers/image-version, which the agent override configures as its current-version source.

The Trident agent source currently reads that file value directly (version.rs:90-97) and uses literal string equality against request.target_version for its AlreadyAtTarget checks (stage, finalize). If requests carry the normalized published version, the same image can compare unequal and miss the already-at-target short-circuit.

Could we ensure the agent-readable stamp and requested target use the same canonical version, including FIPS handling, and add a regression case with a zero-padded component? Please also confirm the agent revision shipped in these images uses the linked comparison path; the exact runtime impact depends on that revision and the targetVersion supplied by RP.

[bfjelds (bfjelds)]

agree that we need to trace through aks-rp abupdate annotations and image-version file to ensure that these are all consistent!

trident-acl-agent should not understand fips vs not-fips (and the AlreadyAtTarget is really just a nice-to-have, as aks-rp should know what the node is), it really just passes the aks-rp request along to Nebraska.

[bfjelds (bfjelds)]

until the image version is updatable, it might be better to use this in parts/linux/cloud-init/artifacts/acl/trident-acl-agent-override.conf:

TRIDENT_ACL_AGENT_CURRENT_VERSION_PATH=/dev/null
TRIDENT_ACL_AGENT_CURRENT_VERSION_FALLBACK=0.0.0