ci: a failed apt.llvm.org download makes the Clang installer report success having installed nothing

[Yaraslaut]

Found while working #672 (PR for #679/#672/#675). Filed, not folded: #672's narrow fix is about a download that fails loudly in the wrong place; this one fails silently in the right place, which needs a different fix.

The finding

Ten steps across two workflows install Clang this way:

wget -qO- https://apt.llvm.org/llvm.sh | sudo bash -s -- ${{ env.CLANG_VERSION }}

nine in .github/workflows/ci.yml (lines 281, 467, 630, 849, 1589, 1855, 2210, 2277 and the new bank-sanitizers job, which copies the same line) and one in .github/workflows/mutation.yml:79.

wget -qO- does not write an error document to stdout. On an HTTP 5xx it exits 8 having written zero bytes. bash then reads an empty script, does nothing, and exits 0. GitHub runs run: under bash -e without pipefail, so the pipeline's status is bash's, not wget's.

The step therefore reports success having installed no compiler.

Verification status: reproduced locally, against a server that returns 503

Not observed in CI. The 503 is synthetic — a local http.server returning 503 Service Unavailable with a short HTML body — but the apt.llvm.org / Launchpad outages #672 records are exactly this condition, and three of them happened in one day.

=== wget -qO- <200> | bash -s -- 22 ===
llvm.sh ran
exit=0
=== wget -qO- <503> | bash -s -- 22   (the outage case) ===
exit=0
=== wget -qO- <503> alone, exit code and stdout bytes ===
wget exit=8
0

The second block is the defect: the same exit code as the first, and no output at all.

Measured on 95aa5686, GNU wget 1.25.0, bash 5.3, Arch Linux. Not verified: the exact wget version on ubuntu-24.04 runners, and whether it differs here (wget's "write the error document" behaviour is --content-on-error, off by default, and has been for a long time — but I did not check the runner's build).

What it costs

Less than a red leg, because the failure is deferred rather than hidden outright: every one of these jobs then configures with an explicit -DCMAKE_CXX_COMPILER=clang++-22, which fails when the compiler is not there. So the leg does go red — at the Configure step, saying the compiler is missing, several steps after the step that was supposed to install it and reported success.

That is #672's complaint in its purest form: the check list, and the log, both name the wrong thing. It is worse than the sccache case in one respect — there, the failing step at least was the installer.

What would resolve it

Any one of:

• set -o pipefail at the top of the step, which makes wget's exit 8 the pipeline's;
• download to a file and run the file, the shape ci: an apt/PPA outage fails a leg the same way a defect does, and every PR re-derives that from a log #672's fix used for sccache;
• wget -q -O /tmp/llvm.sh + a non-empty check + sudo bash /tmp/llvm.sh.

All three are local to the step and need no CI-wide decision. Ten copies, so whichever is chosen is a ten-line change plus one comment.

What would change the verdict

Close as invalid if wget on the runner image does exit non-zero in a way bash -e propagates, or if it writes the error document so that bash fails on it — either would need checking on ubuntu-24.04 rather than here. Close as wontfix if the deferred Configure failure is judged legible enough.

Related

• ci: an apt/PPA outage fails a leg the same way a defect does, and every PR re-derives that from a log #672 — the same class (an upstream outage failing a leg the way a defect does), of which this is the silent variant. Its narrow half is fixed in the PR this was found from; the class stays open there.

[Yaraslaut]

Triage: valid — I reproduced the mechanism, and it is the purest instance of this repository's headline defect yet

Run locally, against a URL that 404s:

$ wget -qO- https://apt.llvm.org/does-not-exist.sh > out.txt
  wget exit=8   bytes written=0
$ wget -qO- https://apt.llvm.org/does-not-exist.sh | bash
  pipeline exit=0

wget fails, writes nothing, bash reads empty input and succeeds, the pipeline reports success. A failed Clang toolchain install is indistinguishable from a successful one, at nine sites across ci.yml and mutation.yml.

triage: valid, bug, area: ci.

Why this outranks its siblings

AGENTS.md leads with "a control that reports success while measuring nothing", and this repository has spent the day finding instances — an invalid coverage config, a sanitizer job with no instrumentation, a suppression over an invariant body. This one is in the setup step of the job that installs the compiler, and its failure mode is not a wrong number but a silently absent toolchain. Everything downstream then runs against whatever clang the image already had, or fails later somewhere unrelated and far more confusing.

It is a genuinely different shape from #672's, which is why filing it separately was right: #672's failures are loud and unintelligible (gzip: stdin: not in gzip format), this one is silent and plausible. #683 fixed the loud one with curl --fail; that change cannot help here, because wget -qO- | bash has no non-zero exit to propagate.

For whoever takes it

• set -o pipefail alone is not enough. It propagates wget's exit 8 — but only if the step's shell has it, and the default bash -e {0} GitHub uses for a bare run: does not enable pipefail. Check what each of the nine steps actually runs under before assuming.
• The shape that cannot fail silently is download-then-verify-then-execute, exactly as ci: three robustness gaps — a check that refused the narrow case, a download that blamed tar, and a sanitizer matrix the bank example was never in (fixes #675, fixes #679, refs #672) #683 did for the curl sites: wget -O llvm.sh -- (a real file, no -q swallowing the error), check the exit, check it is non-empty, then bash llvm.sh. Reusing ci: three robustness gaps — a check that refused the narrow case, a download that blamed tar, and a sanitizer matrix the bank example was never in (fixes #675, fixes #679, refs #672) #683's pattern keeps one idiom rather than two.
• Nine sites, and a partial fix leaves the lottery running — ci: three robustness gaps — a check that refused the narrow case, a download that blamed tar, and a sanitizer matrix the bank example was never in (fixes #675, fixes #679, refs #672) #683's Qt pin made the same argument for doing all eleven at once.
• Prove it: point one step at a 404 and watch the job go red. A fix verified only against a working URL has verified nothing.

Not verified: the filer measured GNU wget 1.25.0 on Arch; ubuntu-24.04's wget could in principle differ. One apt-cache policy wget on a runner, or a deliberate 404 in a scratch branch, settles it — and the second doubles as the proof above.

🤖 Generated with Claude Code

https://claude.ai/code/session_01VptDWG2fKr2vBnLSJcgzgW