ci: master's red bank-ubsan leg — ctest enumerates bank's PRE_TEST suites headless, and the checker was hiding why (fixes #690, refs #691)

[Yaraslaut]

master is red on Bank example / UBSan, and has been on every run since the job landed in #683 — twice on master, once on PR #689. This makes it green, and makes the next failure of its kind arrive named instead of opaque.

Two commits, two concerns.

---

1. scripts: print why ctest listed no tests instead of discarding it

check_sanitizer_instrumentation.sh enumerated the tree with ctest --show-only=json-v1 2>/dev/null, which made two different failures indistinguishable: ctest enumerated a tree that registers no tests and ctest failed before printing any JSON. Both surfaced as ctest listed no tests, with the sentence that tells them apart discarded one pipe away from the error message. On #690 it was the second, and the cause had been printed on every one of the three failing runs.

stderr now goes to a file and the exit status is kept; on the empty-list path the checker prints the status, the stdout byte count, and either the captured stderr or an explicit statement that there was none — which is the distinction, since a listing that found nothing is silent and one that failed is not.

Two self-test cases hold it, and the mutation check was run: with the script change reverted and the self-test left in place, both fail —

error: the failed listing was rejected, but ctest's own reason was discarded
       -- the caller is left with 'listed no tests' and no cause, which is morph#690
2 self-test check(s) failed

— and all 13 pass with it. drift-guard.yml runs that self-test in CI.

This commit alone does not make the leg green. It is the half that would have paid for itself even if the second half were wrong.

2. ci: the bank-ubsan sweep enumerates ctest headless, so it needs the offscreen platform

The sweep's first act is ctest --show-only=json-v1, and ctest is exactly where bank's discovery runs. Bank's three suites are the only Qt-linked targets in this repository registered with DISCOVERY_MODE PRE_TEST; every other Qt suite uses POST_BUILD, where the Build step's QT_QPA_PLATFORM=offscreen already covers the enumeration run. PRE_TEST moves that run to ctest time, so listing the tests executes bank_gui_qml_tests --list-tests, whose main constructs a QGuiApplication (examples/common/testkit/testkit_main.cpp) before Catch2 ever sees the flag. Headless and with no QT_QPA_PLATFORM it aborts; CatchAddTests.cmake turns a nonzero discovery into message(FATAL_ERROR ...); ctest exits 8 having printed no JSON at all — not bank's entries missing, the entire listing, every other suite with it. Nine binaries became zero.

The step now declares the same QT_QPA_PLATFORM: offscreen the Test step below it already declares. That is the principle and not a patch: this sweep's subject is the binaries that step will run, so it has to enumerate them in that step's environment.

The guard is untouched. #675's floor did its job here — it refused to report success having examined nothing, and it is the only reason this was visible at all. Nothing in this branch relaxes it; the diff to check_sanitizer_instrumentation.sh adds output on the failing path and changes no verdict.

---

Verification

Reproduced locally and fixed locally, on the real clang-ubsan bank configure at 24a470c4 (clang 22.1.8, Catch2 3.16.0, Qt 6.11.2 — the runner's versions differ, the code path does not). Configured from an empty build directory. The runner's one distinguishing property is that it is headless, so that is what was emulated. Same tree, same build, one environment variable apart:

$ env -u DISPLAY -u WAYLAND_DISPLAY \
      bash scripts/check_sanitizer_instrumentation.sh build/clang-ubsan ubsan
::error::check_sanitizer_instrumentation: ctest listed no tests in build/clang-ubsan
         -- this check would pass having examined nothing
check_sanitizer_instrumentation: `ctest --show-only=json-v1` exited 8 and wrote 0 bytes of stdout.
check_sanitizer_instrumentation: its stderr follows -- a non-empty stderr here means
the listing *failed*, not that the tree registers no tests:
    | CMake Error at /usr/lib/cmake/Catch2/CatchAddTests.cmake:307 (message):
    |   Error listing tests from executable
    |   '.../build/clang-ubsan/examples/bank/bank_gui_qml_tests':
    |
    |     Result: Subprocess aborted
    |     Output:

$ env -u DISPLAY -u WAYLAND_DISPLAY QT_QPA_PLATFORM=offscreen \
      bash scripts/check_sanitizer_instrumentation.sh build/clang-ubsan ubsan
check_sanitizer_instrumentation: 9 ctest binaries all carry __ubsan_ symbols (0 allowlisted).

The first block is the CI failure, locally, for the first time. The second is the fix.

The binary itself, directly:

$ env -u DISPLAY -u WAYLAND_DISPLAY ./bank_gui_qml_tests --list-tests
exit 134 (SIGABRT), no output
$ env -u DISPLAY -u WAYLAND_DISPLAY QT_QPA_PLATFORM=offscreen ./bank_gui_qml_tests --list-tests
2 test cases

And the step that has never run — headless, offscreen, UBSAN_OPTIONS=print_stacktrace=1:halt_on_error=1, as the workflow sets it:

100% tests passed out of 29
Label Time Summary:
bank    =   6.83 sec*proc (29 tests)
Total Test time (real) =   1.81 sec

So the leg has somewhere to go once the sweep lets it through.

Why two earlier sessions could not reproduce this and reported 9 ctest binaries all carry __ubsan_ symbols: a workstation has a display, so the enumeration succeeds there whether or not the variable is set. That is the runner-versus-workstation difference, and it is the whole of it.

What I did not verify

• CI itself. Handing back without waiting, per the lane's instructions. No run of this branch has happened yet.
• The Catch2 version CI resolves. Measured against 3.16.0 (local distro). CI gets v3.8.1 via morph_cache_dep, or the distro catch2 the bank job installs from apt. The execute_process + FATAL_ERROR path in CatchAddTests.cmake is long-standing, and the CI symptom (ctest listed no tests, after a successful Build) is consistent with the same path, but I did not read 3.8.1's copy.
• That the discovery abort is the only thing wrong with the leg. If something else fails after the sweep passes, commit 1 means the next report names it.

The one claim this branch rests on

That adding QT_QPA_PLATFORM=offscreen to a step cannot make anything worse. It is inert for every binary that needs no platform integration, it is the value the adjacent Test step already sets, and the only behaviour it changes is which Qt platform plugin bank_gui_qml_tests loads during a --list-tests run that currently aborts.

---

Review notes, done inline

• Does the change weaken the guard? No. check_sanitizer_instrumentation.sh's verdicts are unchanged — the diff adds output on the already-failing path and a mktemp/trap pair. The floor at checked < 2, the empty-list refusal, the per-mode symbol table and the --binary refusal under GITHUB_ACTIONS are all untouched, and the self-test still passes all 13 cases including the floor ones.
• Is the new diagnostic itself a control that measures nothing? It was mutation-tested against the unpatched script and fails there (output above). Case 3 was also tightened so a genuinely empty tree must be named as empty — otherwise "prints something on an empty list" would have been satisfied by the old code.
• Could the sweep step's ctest run leave a stale discovery cache the Test step then reuses? Yes, it caches <target>_tests.cmake, and that is fine: both steps now run with the identical QT_QPA_PLATFORM, so the cached listing is the one the Test step would have produced. Before this change the sweep step left no cache at all, because it failed.
• set -e correctness. The ctest invocation uses || ctest_status=$? rather than a command substitution, so a nonzero exit is captured instead of aborting the script. jq reads from the captured file rather than a pipe, so pipefail has nothing to trip over.
• Two comments corrected, both of which stated the fact that was missed: the Build step's claim that discovery runs "each Qt-linked test binary" at build time (in this configure that is morph_qt_tests, not bank's suites — precisely the gap), and examples/bank/CMakeLists.txt's PROPERTIES ENVIRONMENT "QT_QPA_PLATFORM=offscreen", which reads as though it makes the suite headless-safe. It does not: PROPERTIES apply to the tests Catch2 registers, and CatchAddTests.cmake's execute_process() forwards only DL_PATHS/DL_FRAMEWORK_PATHS into the discovery run.

Filed, not folded

• ci: three of the four sanitizer sweep steps still enumerate ctest without their Test step's environment #691 — three of the four sanitizer sweep steps still run ctest --show-only without their own Test step's environment. kanban-tsan and ladder-asan set offscreen on Build and Test and omit it on the sweep step between them; they are green only because every Qt-linked suite they build happens to use POST_BUILD. One word changed anywhere turns them red with the same uninformative message. Latent hazard, not an outage; exposure inferred from reading, not reproduced.

Fixes #690

🤖 Generated with Claude Code

https://claude.ai/code/session_01VptDWG2fKr2vBnLSJcgzgW

[Yaraslaut]

Runner verification — root cause confirmed, and the guard is untouched

Both load-bearing links check out.

Why linux-sanitizers passes the identical check while this job cannot:

linux-sanitizers configures:  MORPH_BUILD_NET=ON  MORPH_BUILD_OFFLINE_SQLITE=ON

No Qt at all. So it never builds a Qt-linked PRE_TEST suite, and the "tests/ is PRE_TEST too" caveat I put in the dispatch resolves exactly as you found — that caveat was the thing standing between the hypothesis and the answer, and you closed it rather than working around it.

And the abort's origin, in examples/common/testkit/testkit_main.cpp:

MORPH_LADDER_TESTKIT_GUI_APP … upgrades that object from QCoreApplication to
QGuiApplication … what it adds is a platform integration, without which Qt Quick
cannot instantiate a window at all.

So --list-tests constructs a QGuiApplication before Catch2 ever sees the flag. Headless, that aborts; CatchAddTests.cmake raises FATAL_ERROR; ctest exits 8 having printed no JSON — and the whole listing goes, not just bank's entries. 9 → 0, which is the number that made no sense until now.

The guard is intact — verified by running it, not by reading the diff:

self-test:                    all 13 checks pass
empty build tree:             exit 1
  `ctest --show-only=json-v1` exited 0 and wrote 186 bytes of stdout.
  it wrote nothing to stderr, so this is a build tree that genuinely registers
  no tests rather than a listing that failed.

That second sentence is the fix to the diagnostic gap. The floor still refuses; it now says which of the two ways it got nothing, which is precisely what three sessions could not tell from CI.

On the structure of the change

Landing the stderr capture as its own commit, first, was right — and your framing is the part I want to keep: it "would have paid for itself even if commit 2 were wrong." A diagnostic improvement that stands independent of the fix it enabled is worth more than the fix, because the next opaque failure in that script is now named rather than guessed at. Mutation-testing it (revert the script, two self-test cases fail) is what makes that a control rather than a comment.

Reproducing by removing DISPLAY is the move two earlier sessions missed — including me. We both concluded "cannot reproduce locally" without asking which property of the runner we were failing to emulate. The answer was one environment variable.

#691 is the better outcome of this ticket

Three of the four sanitizer sweep steps still run ctest --show-only without their own Test step's environment, and kanban-tsan / ladder-asan are green only because every Qt-linked suite they build happens to use POST_BUILD. One DISCOVERY_MODE word changed anywhere turns two green jobs red with the same uninformative message this ticket spent three sessions on.

Filing that as a latent hazard — mechanism reproduced, those jobs' exposure explicitly marked as inferred — rather than folding a speculative fix into this PR is the right call. The PROPERTIES ENVIRONMENT trap it records (reads as headless-safe; CatchAddTests.cmake forwards only DL_PATHS/DL_FRAMEWORK_PATHS into discovery) is the kind of thing that costs a day the second time.

Not verified by me: the headless reproduction, the 29/29 ctest -L bank run, and which CatchAddTests.cmake CI actually resolves. The last is worth keeping in view — you measured against local Catch2 3.16.0 and said so.

🤖 Generated with Claude Code

https://claude.ai/code/session_01VptDWG2fKr2vBnLSJcgzgW

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!