ci: every ctest preset sets stopOnFailure, so a leg with six failures reports one — and which one is a race

[Yaraslaut]

Found while fixing #593, whose own body reports six failing tests while CI on the same commit reported one. The discrepancy is not a difference in behaviour between the two environments. It is this setting, and resolving it took a full local build to do.

The finding

Every test preset in CMakePresets.json sets stopOnFailure:

$ python3 -c "
import json;d=json.load(open('CMakePresets.json'))
for p in d['testPresets']: print(p['name'], p['execution'])
"
cl-debug {'noTestsAction': 'error', 'stopOnFailure': True}
cl-qt-debug {'noTestsAction': 'error', 'stopOnFailure': True}
cl-qt-release {'noTestsAction': 'error', 'stopOnFailure': True}
clangcl-debug {'noTestsAction': 'error', 'stopOnFailure': True}
windows-everything {'noTestsAction': 'error', 'stopOnFailure': True}
gcc-debug {'noTestsAction': 'error', 'stopOnFailure': True}
clang-debug {'noTestsAction': 'error', 'stopOnFailure': True}
linux-everything {'noTestsAction': 'error', 'stopOnFailure': True}
clang-asan {'noTestsAction': 'error', 'stopOnFailure': True}
clang-tsan {'noTestsAction': 'error', 'stopOnFailure': True}
clang-ubsan {'noTestsAction': 'error', 'stopOnFailure': True}
clang-coverage {'noTestsAction': 'error', 'stopOnFailure': True}

All twelve, and .github/workflows/ci.yml drives every leg through ctest --preset ${{ matrix.preset }}. So a leg with six failures reports one, and the failure it reports is whichever one ctest reached first — which depends on the -j scheduling, so it is not even stably the same one across legs.

Measured, on one commit

PR #585 head a08db4f9, CI run 35467846733. Six legs failed. Each reported exactly one failing test:

Linux / clang-asan      1663 - SocketBackend: action result delivered via then (Failed)
Linux / clang-tsan      1663 - SocketBackend: action result delivered via then (Failed)
Linux / clang-ubsan     1669 - SocketBackend: action result delivered via then (Failed)
Linux / clang-coverage  2786 - SocketBackend: action result delivered via then (Failed)

The same branch, rebased onto master a020e69c and run locally without a preset (GCC 16.2.1, Debug, -DMORPH_BUILD_QT=ON -DMORPH_BUILD_NET=ON, ctest -j4):

99% tests passed, 6 tests failed out of 1833

The following tests FAILED:
	1660 - SocketBackend: action result delivered via then (Failed)
	1676 - SocketBackend: executeTimeout surfaces as backend::TimeoutError, not a generic runtime_error (Failed)
	1681 - SocketBackend: many concurrent in-flight executes all resolve, matched by callId (Failed)
	1689 - SocketBackend: server dropping mid-call resolves the pending completion with DisconnectedError (Failed)
	1692 - SocketBackend: two backends share one server with isolated model state (Failed)
	1833 - SocketBackend interop: connects to a QtWebSocketServer and completes an action (Failed)

One cause, six symptoms, and CI showed one of them.

Note that the Valgrind memcheck leg on that same run, which does not go through a test preset, reported all of its five:

tests/net/test_socket_backend.cpp:242: FAILED:
tests/net/test_socket_backend.cpp:299: FAILED:
tests/net/test_socket_backend.cpp:327: FAILED:
tests/net/test_socket_backend.cpp:1241: FAILED:
tests/net/test_socket_backend.cpp:1440: FAILED:
test cases:  189 |  184 passed | 5 failed
==10010== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

That is the shape of report the preset legs suppress.

Why it matters

1. Blast radius is invisible. "One test fails" and "every test that registers a handler over this transport fails" are different verdicts about whether a PR is one line from green. qt: move QtWebSocketBackend onto the structural registration surface #585's own PR body read the CI logs correctly and still described the failure as singular for three revisions.
2. Each extra symptom costs a CI cycle. Fix the first failure, push, wait 26–74 minutes, discover the second. With six symptoms that is potentially six cycles to learn what one uninterrupted run would have printed.
3. It interacts badly with -j. The reported test is a race, so two runs of the same red commit can name different tests and look like two different regressions.

What it is not

stopOnFailure is a defensible setting for a leg whose later tests are meaningless once an early one fails, and for one that is expensive to run to completion. This ticket is not "delete it everywhere" — it is that the twelve presets adopt it uniformly and nothing records why, while the one leg that does not use a preset produces a strictly more useful report at apparently no cost (that leg is Valgrind, the slowest in the matrix, which is evidence against "it is there to save time").

Verification status

Reproduced. The preset listing above is real output from the tree at a020e69c. The CI counts are read from run 35467846733's job logs, not inferred. The six-failure local run is measured, on the rebased branch, with the build configuration stated.

Not measured: how much wall-clock a full run costs on each leg versus a stop-at-first — i.e. the actual price of turning it off. Somebody proposing a change should measure that, per leg, before changing all twelve.

What would change the verdict

• Close this if a leg-by-leg measurement shows running to completion materially lengthens the matrix and the current behaviour is a deliberate trade — but then it wants a comment in CMakePresets.json saying so, because it is currently unexplained.
• Close this if ctest's --stop-on-failure can be left on while still printing the full failing set, which would make this a reporting fix rather than a policy one. I did not check whether it can.

🤖 Generated with Claude Code

https://claude.ai/code/session_01AbwhcguQFkhvVi2AH19sWk

[Yaraslaut]

Triage: valid

Premise reproduced independently, on master @ a020e69c, before reading the filer's numbers:

test presets: 12
  cl-debug                     stopOnFailure=True
  cl-qt-debug                  stopOnFailure=True
  cl-qt-release                stopOnFailure=True
  clangcl-debug                stopOnFailure=True
  windows-everything           stopOnFailure=True
  gcc-debug                    stopOnFailure=True
  clang-debug                  stopOnFailure=True
  linux-everything             stopOnFailure=True
  clang-asan                   stopOnFailure=True
  clang-tsan                   stopOnFailure=True
  clang-ubsan                  stopOnFailure=True
  clang-coverage               stopOnFailure=True
--> 12 of 12 set stopOnFailure

Twelve of twelve. Confirmed.

The "don't undo a deliberate fix" check — run, and it supports the ticket

The triage skill's standing warning is to git log -S anything odd-looking before changing it, because a per-commit cache key once looked wasteful until its history showed why it existed. Run here:

$ git log --oneline -S"stopOnFailure" -- CMakePresets.json
d255723f cmake: add windows-everything and linux-everything presets (#124)
54f54286 [morph] Fix documentation and CI
00d480d5 [morph] Init

00d480d5 is the initial commit and 54f54286 a general CI fix; neither names the setting or a problem it solves. d255723f is the only one with a substantive message, and it is about adding two presets that enable every optional feature — it inherits stopOnFailure along with everything else in the pattern and never mentions it.

So there is no commit anywhere that chose this setting for a stated reason. That is the ticket's central claim — "the twelve presets adopt it uniformly and nothing records why" — and it now has history behind it rather than only absence of a comment. This is not a deliberate fix being undone.

This one bit me directly, which is the strongest evidence in the ticket

I am the reader this defect misled. Earlier in this sweep I read CI on a08db4f9 and reported, in as many words, that all six failing jobs failed on one test — and treated the singularity as a fact about the defect, noting it as "deterministic, not a flake." It was deterministic. It was not singular: six tests were failing and ctest stopped at the first. I then handed a lane the 1-vs-6 discrepancy as something to resolve rather than assume, which is the only reason it got resolved at all.

That is cost #1 in the ticket ("blast radius is invisible") landing on a live decision, not a hypothetical. The ticket notes #585's own PR body described the failure as singular for three revisions; add this sweep's triage to that list.

Why valid rather than rescope

The ticket is careful not to over-reach — "This ticket is not 'delete it everywhere'" — and that restraint is what makes it implementable rather than a decision nobody in a lane can make. It also pre-empts the one serious counter-argument with evidence: the leg that reports in full is Valgrind, the slowest in the matrix, so "it is there to save time" does not survive contact with the data.

Acceptance is clear enough to hand to a lane:

• A red CI leg reports every failing test, not the first one -j happened to reach.
• Any preset that keeps stopOnFailure carries a comment saying why, so the next reader inherits the reason rather than re-deriving it.
• Per AGENTS.md's "would this still pass if the feature did nothing": the change is only demonstrated by a run that fails. Deliberately break two tests, run the affected leg, and show both named in the output. A green run proves nothing here.

One implementation note, offered not prescribed: ctest's stop-on-failure is settable per invocation as well as per preset, so "keep the preset honest for local one-shot use, pass the opposite in CI" is available if anyone wants the local behaviour preserved. Whoever takes it should decide; either satisfies the acceptance above.

Contention

Touches CMakePresets.json and possibly .github/workflows/ci.yml. PR #614 holds .github/workflows/drift-guard.yml and #605 is queued to touch ci.yml — check both before dispatching. Nothing holds CMakePresets.json.

What I did not verify

• I did not re-run qt: move QtWebSocketBackend onto the structural registration surface #585's failing commit to reproduce the six-vs-one split myself; I verified the setting and its history, and am relying on the filer's local run for the six-failure output. Those are different claims and only the first is mine.
• I did not check whether noTestsAction: error (set on all twelve alongside it) has a recorded rationale — it looks deliberate and useful, and is out of scope here.
• I did not check for stop-on-failure equivalents outside CMakePresets.json, e.g. a bare ctest flag in a workflow step.

Labels: triage: valid, plus the bug / area: ci already applied.

[Yaraslaut]

Implementing this in #617. One correction to the issue, since it affects which shapes are actually available.

The "keep it in the preset, pass the opposite in CI" option does not exist. ctest has --stop-on-failure and no negation:

$ ctest --no-stop-on-failure
CMake Error: Unknown argument: --no-stop-on-failure
CMake Error: Run 'ctest --help' for all supported options.
$ ctest --help | grep -i stop
  --stop-on-failure            = Stop running the tests after one has failed.
  --stop-time <time>           = Set a time at which all tests should stop

(ctest 4.4.3, Linux.) Stop-on-failure is settable per invocation only in the enabling direction, so the preset is the only place the decision can be made, and #617 makes it there. A local one-shot run that wants the old behaviour still gets it with ctest --preset gcc-debug --stop-on-failure.

Two other things measured while implementing, both recorded in #617:

• CMake rejects $comment in a presets file, in a preset (Invalid extra field "$comment" in Preset) and at the root. description is the only field that can carry a rationale, which is why the one in cmake: let a red leg name every failing test, not just the first (fixes #616) #617 reads like a comment.
• The step-level twin of this defect — a job whose later test steps are skipped once an earlier one fails — is real and is filed as A red CI job reports only its first failing test *step*, hiding the rest (the step-level twin of #616) #618 rather than folded in, since it needs ci.yml and ci: MORPH_BUILD_BANK_GUI is enabled by no job, and nothing checks that a declared option is built at all #605 is queued for that file.

No recorded rationale for stopOnFailure turned up beyond the three commits in the triage comment.