cmake: let a red leg name every failing test, not just the first (fixes #616)

[Yaraslaut]

All twelve test presets set stopOnFailure: true, so ctest abandoned the run at the first failure on every CI leg. One broken test and thirty broken tests produce the same log, so every diagnosis read off a red leg is made on partial information — as happened on #585, where CI was read as "all six failing jobs fail on one test" and the singularity was taken as a property of the defect. Six tests were failing.

fixes #616

What changed

CMakePresets.json only. The twelve test presets now inherit a hidden base-test preset that carries output/execution once, and stopOnFailure is gone from it.

The per-preset decision

Removed on all twelve — cl-debug, cl-qt-debug, cl-qt-release, clangcl-debug, windows-everything, gcc-debug, clang-debug, linux-everything, clang-asan, clang-tsan, clang-ubsan, clang-coverage.

The issue is explicit that this is not "delete it everywhere", and a leg whose later tests are meaningless after an early failure would be a legitimate place to keep it. Looking for one:

• Every ctest case here is its own process (catch_discover_tests), so an earlier failure does not poison a later case's state on any leg.
• The sanitizer legs (clang-asan, clang-tsan, clang-ubsan) are the obvious candidates, and they are not it: a sanitizer diagnostic aborts the process it fired in and the next case starts a fresh one. Sanitizer findings also tend to arrive in clusters, which is precisely the case where seeing one of them is worst.
• clang-coverage is the other candidate — and stopping early loses profile data rather than protecting it. The crashing test has already written its truncated .profraw; abandoning the remaining cases only makes the merged coverage wrong as well.
• "It saves CI time" does not survive the matrix: the leg that already reports in full is Valgrind, the slowest in it.

So none of the twelve earns it today. Rather than leaving twelve copies of a setting with no recorded reason, the decision now lives in one description on base-test, which also says what a future leg has to do to set it back (set it on its own preset, with its reason in its own description).

Shape: preset, not per-invocation

The issue offered "keep the preset honest for local one-shot use, pass the opposite in CI" as an alternative. That option does not exist. ctest has --stop-on-failure and no negation:

$ ctest --no-stop-on-failure
CMake Error: Unknown argument: --no-stop-on-failure
CMake Error: Run 'ctest --help' for all supported options.
$ ctest --help | grep -i stop
  --stop-on-failure            = Stop running the tests after one has failed.
  --stop-time <time>           = Set a time at which all tests should stop

(ctest 4.4.3.) The preset is the only place the choice can be made, so the choice is made there. A local one-shot run that wants to stop early still can: ctest --preset gcc-debug --stop-on-failure, which is the direction that is available per invocation.

Verification — a green run proves nothing here

Two Catch2 cases in tests/test_rational.cpp were deliberately broken, the leg run filtered to -R "^Rational::" (46 cases) on gcc-debug, GNU 16.2.1. The breakage was reverted before the commit; the tree is clean and tests/ is untouched by this PR.

Before (stopOnFailure: true, master's presets) — run abandoned at 2 of 46, one test named:

 1/46 Test #596: Rational::CommonArithmetic::PiApproximation22Over7 .....   Passed    0.01 sec
      Start 597: Rational::Comparison
 2/46 Test #597: Rational::Comparison ..................................***Failed    0.00 sec
...
50% tests passed, 1 tests failed out of 2
The following tests FAILED:
	597 - Rational::Comparison (Failed)
Errors while running CTest

After (this PR) — all 46 run, both named:

46/46 Test #641: Rational::toDouble::PrecisionRoundtrip ................   Passed    0.00 sec

96% tests passed, 2 tests failed out of 46

The following tests FAILED:
	597 - Rational::Comparison (Failed)
	599 - Rational::Constants (Failed)
Errors while running CTest

Both exit 8, so a red leg is still red. --output-on-failure still reaches both: the "after" run printed both Catch2 bodies (test_rational.cpp:451: FAILED and test_rational.cpp:303: FAILED).

Restoring the setting restores the old behaviour, which is what shows this change and nothing else is responsible. Adding "execution": { "stopOnFailure": true } to the inheriting gcc-debug preset only, same two broken tests:

The following tests FAILED:
	597 - Rational::Comparison (Failed)

Inheritance was checked, not assumed

Moving execution into a base preset is only safe if the base's settings actually reach the legs, so:

• noTestsAction: error still arrives. ctest --preset gcc-debug -R zzz_no_such_test → No tests were found!!!, exit 8. The same filter without the preset (ctest --test-dir build/gcc-debug -R zzz_no_such_test) exits 0. The check is live.
• outputOnFailure still arrives — see the failure bodies in the "after" run above.
• CMake merges execution key by key across inherits, not as a whole map: in the restore experiment the child set only stopOnFailure, and the no-such-test probe still exited 8, so noTestsAction came through alongside it. (Measured on CMake 4.4.3; the base-test description records this, because the opposite assumption would be a quiet way to lose noTestsAction later.)
• $comment is not an option in this file at all: CMake 4.4.3 rejects it both in a preset (Invalid extra field "$comment" in Preset) and at the root (... in root object). description is the only field that can carry the reasoning, which is why it reads like a comment.

Full gcc-debug suite on the new presets: 100% tests passed out of 1562, 11.2 s.

Stop-on-failure equivalents outside CMakePresets.json

Checked, as the issue asked:

• No --stop-on-failure anywhere in .github/workflows/, scripts/, or cmake/.
• No Catch2 --abort/-x (abort after N failures) in any test registration or workflow step.
• All four ci.yml job matrices already set fail-fast: false, so a red leg does not cancel its siblings.

One equivalent was found and is not fixed here — a job's later test steps are skipped once an earlier one fails, which hides the same kind of blast radius one level up. It is filed separately (see below) rather than folded in, because it needs ci.yml edits and #605 is queued to touch that file.

Not done, deliberately

• No ci.yml change. Every leg runs its suite through ctest --preset ..., so the preset change is sufficient and ci.yml stays free for ci: MORPH_BUILD_BANK_GUI is enabled by no job, and nothing checks that a declared option is built at all #605.
• No CHANGELOG.md entry. This is CI/tooling reporting behaviour, the same shape as Batch issues per PR by default, and make the cache-stats step measure the job it ran in #611 which took none; and CHANGELOG is the file every in-flight PR conflicts in. Say the word and I will add one.
• No spec update. Nothing in docs/spec/ states a stop-on-failure policy (grepped). testing_charter.md would be the place to record it, and it is held by ci: audit mutation_survivors.json's line citations instead of hoping (fixes #608) #614 — noted in the follow-up issue.

Not verified

• Only gcc-debug was run locally. The other eleven presets are changed by inheritance, checked by ctest --list-presets and cmake --list-presets=workflow resolving all of them, and by the gcc-debug inheritance probes above — not by running each leg. The Windows and sanitizer legs are unverified locally and rest on CI.
• The six-versus-one split on qt: move QtWebSocketBackend onto the structural registration surface #585's own head was not re-reproduced here; the mechanism was reproduced instead, with the output above.

🤖 Generated with Claude Code

https://claude.ai/code/session_01AbwhcguQFkhvVi2AH19sWk

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!