ci: three robustness gaps — a check that refused the narrow case, a download that blamed tar, and a sanitizer matrix the bank example was never in (fixes #675, fixes #679, refs #672)

[Yaraslaut]

Three CI-robustness tickets, one commit each, batched into one PR per the runner's instruction. Branched from origin/master at 95aa5686 (PR #673 had landed by the time the ci.yml boundary was reached, so no ticket was blocked on it).

Commit	Ticket	Outcome
29c07c3d	#675	Landed — narrow --binary mode, floor intact
6dd23963	#672	Landed, narrowly — curl --fail; the class stays open
eedacd68	#679	Landed — new bank-sanitizers job; first bill zero

---

#675 — the sanitizer-instrumentation check refused a single-target tree

scripts/check_sanitizer_instrumentation.sh conflated two questions behind one floor. "Did this check examine a representative set?" needs the floor — that is the CI invocation. "Is this one binary instrumented?" is a yes/no about a single file and needs no floor, but the script refused it (only examined 1 binaries -- too few for this check to mean anything), so the #673 lane answered it by hand with nm | grep __tsan_, re-deriving the per-mode symbol table and the SIGPIPE trap the script already encodes.

--binary <file> <mode> answers the second question against the same symbol table.

How the narrow mode is kept out of the CI invocation

Structurally, not by convention: it refuses outright, exit 2, when GITHUB_ACTIONS is set. No step of any workflow in this repository can reach it. A CI invocation that tried would fail loudly rather than pass having examined one file. The sweep is untouched — same floor, same message, same exit codes.

env -u GITHUB_ACTIONS defeats it, as it defeats any environment-keyed guard. That is deliberate circumvention, not the accident the guard is for, and the script says so.

Why a mode rather than the documented one-liner

The issue offers nm | grep __tsan_ in a comment as a legitimate close, and it nearly is. Against it: the one-liner is not one line. It is the per-mode symbol table (__asan_/__tsan_/__ubsan_), plus the reason it must be grep -c and not grep -q (under pipefail, nm takes SIGPIPE and an instrumented binary reports as uninstrumented — the script's own comment records catching that), plus the fact that a count of zero is the verdict. Writing that down creates a second copy that can drift from the first. The mode is ~40 lines and keeps one.

Verification

New scripts/test_check_sanitizer_instrumentation.sh, following the repository's test_check_*.sh convention, wired into drift-guard.yml's sanitizer-can-fail job. Twelve cases; no compiler beyond cc and no sanitizer runtime, because the gate's whole measurement is nm | grep -c __<mode>_, so a fixture that defines a function of that name exercises the gate rather than clang, and a hand-written CTestTestfile.cmake is all ctest --show-only needs.

The two load-bearing cases were confirmed to fail against a mutated checker, not assumed:

# guard replaced by `if false`
error: --binary ran under GITHUB_ACTIONS -- the floor can now be bypassed from a workflow step:
1 self-test check(s) failed

# floor lowered from 2 to 1
error: the sweep accepted a one-binary tree -- the floor has been lowered:
1 self-test check(s) failed

All twelve pass unmutated. Separately, the narrow mode was exercised against a real UBSan-instrumented binary during #679's measurement:

check_sanitizer_instrumentation: bank_gui_tests carries 52 __ubsan_ symbols -- ubsan-instrumented.
check_sanitizer_instrumentation: this examined one file and applied no floor; it is not a substitute for the build-tree sweep a CI leg runs.

---

#672 — an apt/PPA outage fails a leg the same way a defect does

Scoped deliberately to the narrow, cheap part, and the class stays open. Not implemented, because they are decisions rather than implementations: a retry policy for dependency installation, and a CI-wide marker distinguishing "the environment failed" from "the change failed" in the check list. The three apt/PPA outages that make up the rest of #672 are untouched.

What is fixed is the one case where the log actively misled. The installer piped curl -sSL <url> straight into tar -xz; plain curl treats an HTTP 5xx as a successful transfer of the error page, so the decompressor was the only thing that complained. set -o pipefail would not have helped — GitHub runs run: under bash -e without it, so the pipeline's status is tar's regardless.

Reproduced against a local server returning 503, both shapes under bash -e:

=========== OLD SHAPE, 503 (bash -e, no pipefail) ===========

gzip: stdin: not in gzip format
tar: Child returned status 1
tar: Error is not recoverable: exiting now
exit=2
=========== NEW SHAPE, 503 ===========
curl: (22) The requested URL returned error: 503
exit=22
=========== NEW SHAPE, 200 ===========
exit=0
total 4
-rwxr-xr-x 1 yaraslau yaraslau 28 Sep 21 18:36 sccache

The first block is #672's log verbatim. The third is the success path over a tarball with the real release's member layout, so --strip-components=1 still lands the binary.

Applied to all nine copies in ci.yml (the first carries the reasoning, the other eight point at it) and to the two other downloads with the identical defect and the identical one-flag fix: docs.yml's Doxygen tarball and mutation.yml's Mull .deb. Those two are the same finding, not separate ones — the Mull case is measurably worse, since a 503 puts 64 bytes of HTML in ${asset} and leaves dpkg-deb to object to the archive.

---

#679 — no sanitizer leg builds the bank example

Confirmed: linux-sanitizers, kanban-tsan, ladder-sanitizers and valgrind set zero MORPH_BUILD_BANK_* flags, against eight sites elsewhere in ci.yml. Bank is not a rung, so MORPH_BUILD_LADDER=ON does not reach it either.

The half the ticket did not have

bank's targets never called apply_sanitizers(). ladder_bank_server was the only one that did. Flipping the CI flag alone would have built bank_lib, bank_cli and all three test binaries blind. Measured, bank configured on with the blocks absent:

::error::check_sanitizer_instrumentation: 3 of 9 ctest binaries are not ubsan-instrumented

naming bank_tests, bank_gui_tests, bank_gui_qml_tests. With the blocks in, 9 of 9. They are also required together: removing bank_tests' alone, keeping bank_lib's, does not leave the binary unchecked — it fails to link (undefined reference to __ubsan_handle_type_mismatch_v1_abort).

Why a separate job, not a flag on linux-sanitizers

The ticket proposed the flag. Two measured reasons against it. That leg builds no Qt and its own comment reserves the matrix against GUI stacks, while bank's GUI is where the UB was. And, cold and cacheless, 12 cores, clang 22.1.8:

configure	build	ninja edges
36s	144s	144	linux-sanitizers clang-ubsan leg's current shape
51s	151s	150	same + Qt, no bank
64s	391s	287	this job: core + Qt + bank + bank GUI

Folding one into the other roughly triples the slowest leg of a three-leg matrix, whose duration is then the matrix's.

Wall-clock delta on existing jobs: zero. No existing job's flags are changed by this PR. Bank's isolated build cost — row 2 to row 3 — is +13s configure, +240s build, +137 ninja edges. ubsan not asan: UBSan diagnoses this class, and ASan over a Qt GUI needs the detect_leaks=0 and suppression story ladder-sanitizers carries. Bank under ASan is not closed by this.

The first bill: zero

Per the #646 (84) and #656 (97) precedents, measured before landing. bank_tests (145 assertions / 21 cases), bank_gui_tests (19 / 5), bank_gui_qml_tests (32 / 2) all clean under -fsanitize=undefined -fno-sanitize-recover=undefined; the full 1696-test suite of this configure is 100% passed, 212s serial, zero runtime error lines. Nothing suppressed, no allowlist entry.

A finding that cuts against the ticket's framing

The ticket says this gap is why #663 survived to be found by reading code. Half true, and the half that is false is the interesting one. Rebuilding Format.hpp as it stood before the fix, in this job's exact configuration:

examples/bank/gui/controllers/Format.hpp:76:38: runtime error: 9.2e+19 is
outside the range of representable values of type 'long'

exit 1 — but only because #663's fix also added the test that calls parseMinor with such a value. Measured too: with the pre-#663 header and the pre-#663 test set, this leg is green. No pre-existing bank test drove that path. The sanitizer gap was real and is what this closes; it was not on its own what let #663 through, and this job's reach is bounded by how much of bank the suites actually drive. That is written into the job's banner, not only here.

Also in this commit, because the job needs it

A bank ctest label on the three suites, so the leg runs bank's 28 tests (6s) rather than re-running the 1696 (212s) that linux-sanitizers' clang-ubsan leg already runs under identical instrumentation. A label filter matching nothing would be this repository's named failure mode; CMakePresets.json's base-test already sets noTestsAction: error — confirmed to exit 8 with No tests were found!!! against a tree built without bank.

---

Review reasoning (inline, per the runner's constraint — no /code-review, no /simplify)

• Does the ci: check_sanitizer_instrumentation.sh refuses a single-target build tree, so it cannot verify the narrow case it is most needed for #675 guard measure anything? Mutated twice, both mutations caught (output above). The self-test's fixtures deliberately do not involve a sanitizer runtime, which means the self-test proves the gate, not the instrumentation; scripts/check_sanitizer_can_fail.sh is the complementary gate that proves the instrumentation behaviourally, and the new file says so.
• Does ci: an apt/PPA outage fails a leg the same way a defect does, and every PR re-derives that from a log #672's change alter the success path? Verified against a tarball with the real release's member layout — --strip-components=1 still lands sccache in place, exit 0. The nine copies are byte-identical apart from their comments.
• Could No sanitizer leg builds the bank example, so the UB in #663 was unreachable by every ASan/UBSan job #679's new job pass while measuring nothing? Three floors stand between it and that: the instrumentation sweep (9 of 9, and it was seen failing at 3 of 9 on the tree without the CMake blocks), noTestsAction: error on a -L bank filter that stopped matching, and -fno-sanitize-recover=undefined already on the compile line, with halt_on_error=1 making the runtime agree.
• Scope discipline. Two findings filed rather than folded (below). docs.yml/mutation.yml's curls were folded into ci: an apt/PPA outage fails a leg the same way a defect does, and every PR re-derives that from a log #672 — same defect, same one-flag fix, nine lines away — and that is called out rather than slipped in.

Filed, not folded

• ci: a failed apt.llvm.org download makes the Clang installer report success having installed nothing #681 — wget -qO- https://apt.llvm.org/llvm.sh | sudo bash fails the other way. Measured against the same 503 server: wget -qO- exits 8 writing zero bytes, bash reads empty input and exits 0, the pipeline exits 0. A Clang-installation step that reports success having installed nothing. Ten sites. Different failure shape from ci: an apt/PPA outage fails a leg the same way a defect does, and every PR re-derives that from a log #672's, so a different fix.
• bank: the 21 bank_tests cases share one SQLite database and fail under ctest -j, with nothing in the tree saying so #682 — bank's 21 ctest cases share one SQLite file. ctest -j 12 fails all 21 with [SQLite]disk I/O error (10); serially they pass. CI runs ctest serially (no test preset sets a parallel level), so it is latent — which is exactly why it is worth recording.

What I did not verify

• Any of this on a hosted runner. Every timing is local (12 cores, clang 22.1.8, Qt 6.11.2, no compiler cache) and models neither the apt install nor the Qt install.
• Bank under ASan or TSan, and whether the Lightweight ORM is clean under anything but UBSan.
• ci: check_sanitizer_instrumentation.sh refuses a single-target build tree, so it cannot verify the narrow case it is most needed for #675's narrow mode against a Mach-O binary — the code path is shared with the sweep's, which has the same file -b | grep -qE 'ELF|Mach-O' test, but no macOS run was made.
• ci: a failed apt.llvm.org download makes the Clang installer report success having installed nothing #681's premise on the ubuntu-24.04 runner image specifically: wget's "do not write the error document" behaviour is the default and long-standing, but I measured GNU wget 1.25.0 on Arch, not the runner's build.
• Nothing here was run through CI before opening this PR.

Fixes #675
Fixes #679
Refs #672

🤖 Generated with Claude Code

https://claude.ai/code/session_01VptDWG2fKr2vBnLSJcgzgW

[Yaraslaut]

Runner verification

#675's guard does what it claims, measured not read:

under GITHUB_ACTIONS: exit=2   ::error:: --binary is a local, single-file mode and is refused under
                               GITHUB_ACTIONS -- a CI leg must run the build-tree sweep, whose floor
                               is the only thing that makes 'all instrumented' mean 'all were looked at'
uninstrumented binary: exit=1  ::error file=…:: true carries no __ubsan_ symbols …
self-test:             exit=0

Making the narrow mode structurally unreachable from CI is a better answer than documenting a convention, and the rejection of the issue's own wontfix alternative is well argued: the "one-liner" is really a per-mode symbol table plus the grep -c-not-grep -q SIGPIPE trap the script already records catching, and writing that down creates a second copy that drifts.

#681 is the best thing in this batch and I reproduced it:

$ wget -qO- https://apt.llvm.org/does-not-exist.sh > out.txt
  wget exit=8   bytes written=0
$ wget -qO- https://apt.llvm.org/does-not-exist.sh | bash
  pipeline exit=0

wget fails, writes nothing, bash reads empty input and succeeds, the step passes. A failed LLVM toolchain install reports success, at nine sites across ci.yml and mutation.yml. That is this repository's headline failure mode — a control that reports success having done nothing — living in its own CI setup, and it is a different shape from #672's (which at least fails loudly, just unintelligibly). Filing it separately rather than folding it into #672 was right.

The correction to my triage of #679

I wrote that the sanitizer gap "explains why #663 survived to be found by hand". That is half false and the lane did not ship around it: rebuilding pre-#663 Format.hpp under the new job does trip UBSan — but only because #663's fix also added the test that calls parseMinor with an out-of-range value. With the pre-#663 header and the pre-#663 test set, the leg is green.

The honest statement is narrower than mine: a sanitizer only catches UB on a path something executes, so the missing leg and the missing test were both necessary. The gap was real and is now closed; it was not on its own sufficient to have caught #663. Writing that into the job's banner is the right place for it — the next person to wonder why bank has its own sanitizer job gets the true reason.

That is the third of my triage claims corrected by a lane today, and the pattern is consistent: each was a causal story that sounded specific and went one step further than the evidence.

Two findings the ticket did not have

bank's targets never called apply_sanitizers() — flipping the CI flag alone would have built everything blind (3 of 9 ctest binaries are not ubsan-instrumented), and instrumenting bank_lib without the executables fails to link. So the CMake change is a prerequisite, not a nicety, and "turn the flag on" would have produced a green leg that measured nothing.

A first bill of zero, with the denominator shown — 1696 tests, 100% passed, no suppression and no allowlist entry. A zero is only meaningful next to what was examined, and this one has it.

The new job rather than a flag on linux-sanitizers is justified by measurement: bank costs +240s build and +137 edges, which would roughly triple the slowest leg of a three-leg matrix, and that leg carries its own written exclusion of GUI stacks. kanban-tsan is the precedent.

Not verified by me

The timings, the 1696-test run, the instrumentation counts, and #682's parallel-ctest failure — all need builds I did not run. And the branch's load-bearing claim (GITHUB_ACTIONS is set for every step on a hosted runner) is documented GitHub behaviour pinned by a self-test, not something either of us measured on a runner.

🤖 Generated with Claude Code

https://claude.ai/code/session_01VptDWG2fKr2vBnLSJcgzgW

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[Yaraslaut]

Runner: the one red leg is upstream, and the guard this PR adds is what caught it

Bank example / UBSan failed. The job's log is not retrievable yet (three legs still running), so I read the check run's annotations, which populate independently:

failure: check_sanitizer_instrumentation: ctest listed no tests in build/clang-ubsan
         -- this check would pass having examined nothing
failure: The template is not valid. jurplel/install-qt-action/v4/action.yml (Line: 160, Col: 11):
         Expected format {org}/{repo}[/path]@ref. Actual '$/action'

Read those in the right order. The Qt install action failed, so nothing Qt-dependent configured, so ctest listed no tests — and the instrumentation sweep refused to pass on an empty set. That is exactly the floor #675 is about, doing its job on its first real run: a check that had examined nothing declined to report success. The leg is red for the right reason.

The upstream cause

Not a defect here, and not a moved tag. jurplel/install-qt-action@v4's action.yml:160 currently reads:

# Reference same-repository actions with self-repository syntax - GitHub Changelog
#   https://github.blog/changelog/2026-07-30-reference-same-repository-actions-with-self-repository-syntax/
uses: $/action

$/action is GitHub's newer self-repository syntax. Some runner images resolve it; the one that took this job at 17:23:29Z did not. Linux / Qt6 WebSockets — same @v4, same version/modules/cache inputs, verified identical — started at 17:13:34Z and passed. Ten minutes and a different runner apart.

So this is runner-image luck against a floating action tag, which is the fifth distinct infrastructure failure today and a mechanism the earlier four did not have. I have added it to #672.

What I am doing

Re-running the failed job — GitHub refuses while the workflow still has legs in flight, so it is queued behind those. If it recurs, the durable fix is pinning install-qt-action to an immutable ref rather than the floating @v4, which would make Qt installs deterministic across every job. That is a change to four call sites and a version choice, so it belongs in its own PR rather than being bolted onto this one — and it should not gate this PR, whose own leg is correct.

I am also reproducing the job locally (same preset and flags, cold build dir) to confirm the instrumentation sweep reports all binaries instrumented on a tree that actually built. That answers the only question the red leg leaves open: whether the sweep would have passed had Qt installed.

🤖 Generated with Claude Code

https://claude.ai/code/session_01VptDWG2fKr2vBnLSJcgzgW

[Yaraslaut]

Fixed in 948e26b0 — pinned the Qt action, and proved the job itself was sound

The job was never wrong. I reproduced its exact configuration locally (clang-ubsan, MORPH_BUILD_QT/BANK_EXAMPLE/BANK_GUI=ON, cold build directory) and ran the step CI could not reach:

check_sanitizer_instrumentation: 9 ctest binaries all carry __ubsan_ symbols (0 allowlisted).
  exit=0

Nine of nine, against the 3 of 9 the lane measured before its CMake change. So the sweep would have passed had Qt installed — the red leg was the environment, and the guard correctly refused to report success on the empty set that environment left behind.

The fix. @v4 is a major alias upstream moves on every release, which is how uses: $/action arrived in our CI without any change here. Pinned all 11 call sites across ci.yml, wasm-demo.yml and wasm-ladder.yml to v4.3.1 — the last release whose action.yml does not use that syntax (v4.4.0 and v4.4.1 both do), and which declares every input this repository passes: arch, cache, dir, host, modules, target, version.

All eleven together, deliberately: a partial pin leaves the same lottery running on whichever job was missed.

Verified before pushing: no floating @v4 remains, all seven workflows parse, and banner-lint, option-coverage and catch2-pin pass.

What this does not fix, stated rather than implied

A pinned tag is still a tag. v4.3.1 is immutable by convention, not by construction; the durable answer is a commit SHA (48d3ad6db93f). I did not take that because it trades immutability for a reference no reader can identify, and that trade is a decision for this repository rather than one to make inside an unrelated PR. Recorded on #672.

It also does not address the five distinct infrastructure failures today, of which this is one; #672 remains the ticket for the class.

Why this landed here rather than in its own PR

I said earlier it belonged in a separate PR, and I have changed my mind for a reason worth stating: this PR's own leg cannot go green without it, the failure is the same class as #672 which this PR already addresses, and a re-run was a coin flip on runner image rather than a fix. Waiting for a better runner is not fixing it.

🤖 Generated with Claude Code

https://claude.ai/code/session_01VptDWG2fKr2vBnLSJcgzgW