ci(msan): retry transient network failures while building the MSan image

[DeusData]

The defect

test-msan dies on PR CI whenever the buildx layer cache misses and the builder's resolver hiccups. It fails in step "Build MSan image (cached layers)" (scripts/ci/msan-lane.sh build → buildx build of test-infrastructure/Dockerfile.msan), 2 of the last 15 PR runs — runs 33816056229 and 33818108315:

#7 [2/6] RUN apt-get update && apt-get install -y --no-install-recommends wget gnupg ca-certificates && wget -qO- https://apt.llvm.org/llvm-snapshot.gpg.key > ... && apt-get update && apt-get install ...
#7 26.77   Could not resolve 'apt.llvm.org'
#7 27.34 W: Failed to fetch http://apt.llvm.org/noble/dists/llvm-toolchain-noble-22/InRelease  Could not resolve 'apt.llvm.org'
ERROR: failed to build: failed to solve: process "/bin/sh -c apt-get update && ..." did not complete successfully

The wget of the signing key from that same host had succeeded seconds earlier in the same layer, so this is a transient DNS failure inside buildkit — not a wrong URL, not a dead mirror. One unlucky packet and the whole lane is red.

The fix

A small POSIX-sh helper is written into the image (/usr/local/bin/retry) in its own layer before anything touches the network: 5 attempts with growing backoff (5s/10s/20s/40s), and it still exits non-zero once they are exhausted — a real breakage keeps failing the build rather than being swallowed.

Wrapped with it:

• both apt-get update / apt-get install pairs (also given -o Acquire::Retries=3),
• the wget of the LLVM signing key,
• the git clone of llvm-project,
• the git clone of zlib.

Two shapes needed care:

• the key is now fetched with wget -O <file> instead of -qO- > <file>: under retry the redirect is opened once for all attempts, so a partial write from a failed attempt would be prepended to the output of a later successful one;
• each clone clears its destination first, or a half-finished clone makes every later attempt fail on destination path already exists.

Unchanged: the pinned base digest, the pinned llvmorg-22.1.0 / v1.3.1 tags, --no-install-recommends, and the layer structure and its comments. No behaviour change on a healthy network — every wrapped command succeeds on attempt 1, retry returns immediately and never sleeps (confirmed below: the real apt layer built with zero retry lines).

Verification (Colima, arm64)

The full MSan image was deliberately not built locally — it compiles libc++ with MSan. Instead the changed pieces were exercised directly.

1. Lint

$ docker buildx build --check -f test-infrastructure/Dockerfile.msan test-infrastructure/
Check complete, no warnings found.

2. The real apt.llvm.org layer — scratch image = the FROM line + the retry helper + the first apt layer, taken verbatim out of this Dockerfile, built with --no-cache:

$ docker build --no-cache -t cbm-msan-scratch-a A/     → exit 0
#6 20.08 Setting up llvm-22 (1:22.1.8~++20260714014902+ca7933e47d3a-...) ...
#6 20.09 Setting up clang-22 (1:22.1.8~++20260714014902+ca7933e47d3a-...) ...
#6 DONE 20.4s

Zero retry: attempt ... failed lines in that log — the healthy path costs nothing.

3. The retry path fires — wrapped command fails twice, then succeeds:

$ docker build --no-cache -t cbm-msan-scratch-b B/     → exit 0
#6 0.166 scratch-b attempt 1
#6 0.166 retry: attempt 1 failed, sleeping 5s before retrying: ...
#6 5.176 scratch-b attempt 2
#6 5.176 retry: attempt 2 failed, sleeping 10s before retrying: ...
#6 15.19 scratch-b attempt 3
#6 15.19 SCRATCH_B_OK retry_exit=0 attempts=3

4. Exhausted retries still fail the build — wrapped command always fails:

$ docker build --no-cache -t cbm-msan-scratch-c C/     → exit 1
#6 0.131 retry: attempt 1 failed, sleeping 5s before retrying: ...
#6 5.138 retry: attempt 2 failed, sleeping 10s before retrying: ...
#6 15.15 retry: attempt 3 failed, sleeping 20s before retrying: ...
#6 35.15 retry: attempt 4 failed, sleeping 40s before retrying: ...
#6 75.16 retry: giving up after 5 attempts: ...
#6 75.16 SCRATCH_C retry_exit=1
ERROR: failed to build: ... did not complete successfully: exit code: 1

That third case is the one that matters for honesty: a retry helper that swallowed the final failure would hide real breakage behind a green lane.

Scratch images removed afterwards.

Scope

Only test-infrastructure/Dockerfile.msan is touched — scripts/ci/msan-lane.sh, the workflow and everything else are untouched.

Not implemented, offered as a follow-up decision: the lane builds on a buildx created builder (msan-builder, docker-container driver), which resolves through its own container network — that is where the hiccup lives. Adding --network=host to the buildx invocation would sidestep the builder's resolver entirely, but it needs the network.host entitlement on the builder and changes the lane script, so it is out of this PR's scope. Retries help regardless of which resolver is in play.