Skip to content

cmake: let a red leg name every failing test, not just the first (fixes #616) #617

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Yaraslaut merged 1 commit into from
Sep 20, 2026
Merged

cmake: let a red leg name every failing test, not just the first (fixes #616) #617

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
87 changes: 17 additions & 70 deletions CMakePresets.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -226,77 +226,24 @@
],
"testPresets": [
{
"name": "cl-debug",
"configurePreset": "cl-debug",
"output": { "outputOnFailure": true },
"execution": { "noTestsAction": "error", "stopOnFailure": true }
},
{
"name": "cl-qt-debug",
"configurePreset": "cl-qt-debug",
"output": { "outputOnFailure": true },
"execution": { "noTestsAction": "error", "stopOnFailure": true }
},
{
"name": "cl-qt-release",
"configurePreset": "cl-qt-release",
"output": { "outputOnFailure": true },
"execution": { "noTestsAction": "error", "stopOnFailure": true }
},
{
"name": "clangcl-debug",
"configurePreset": "clangcl-debug",
"output": { "outputOnFailure": true },
"execution": { "noTestsAction": "error", "stopOnFailure": true }
},
{
"name": "windows-everything",
"configurePreset": "windows-everything",
"output": { "outputOnFailure": true },
"execution": { "noTestsAction": "error", "stopOnFailure": true }
},
{
"name": "gcc-debug",
"configurePreset": "gcc-debug",
"output": { "outputOnFailure": true },
"execution": { "noTestsAction": "error", "stopOnFailure": true }
},
{
"name": "clang-debug",
"configurePreset": "clang-debug",
"output": { "outputOnFailure": true },
"execution": { "noTestsAction": "error", "stopOnFailure": true }
},
{
"name": "linux-everything",
"configurePreset": "linux-everything",
"output": { "outputOnFailure": true },
"execution": { "noTestsAction": "error", "stopOnFailure": true }
},
{
"name": "clang-asan",
"configurePreset": "clang-asan",
"output": { "outputOnFailure": true },
"execution": { "noTestsAction": "error", "stopOnFailure": true }
},
{
"name": "clang-tsan",
"configurePreset": "clang-tsan",
"output": { "outputOnFailure": true },
"execution": { "noTestsAction": "error", "stopOnFailure": true }
},
{
"name": "clang-ubsan",
"configurePreset": "clang-ubsan",
"output": { "outputOnFailure": true },
"execution": { "noTestsAction": "error", "stopOnFailure": true }
},
{
"name": "clang-coverage",
"configurePreset": "clang-coverage",
"name": "base-test",
"hidden": true,
"description": "Shared test-run policy for every leg. Two things a reader needs. (1) stopOnFailure is deliberately NOT set here. A red CI leg has to name every test that failed, not whichever one ctest happened to reach first: stopping early makes one broken test and thirty broken tests look identical in the log, and a diagnosis made from that log is made on partial information (morph#616). ctest has no --no-stop-on-failure, so 'set it in the preset and override it in CI' is not available -- the preset is the only place the decision can be made. (2) A leg whose later tests are genuinely meaningless after an early failure may set stopOnFailure on its own preset, but must say why in its own description. Inheritance merges 'execution' key by key -- measured on CMake 4.4.3: a child preset that sets only stopOnFailure still gets noTestsAction from here -- so setting it back is a one-key change, not a copy of this whole block.",
"output": { "outputOnFailure": true },
"execution": { "noTestsAction": "error", "stopOnFailure": true }
}
"execution": { "noTestsAction": "error" }
},
{ "name": "cl-debug", "configurePreset": "cl-debug", "inherits": "base-test" },
{ "name": "cl-qt-debug", "configurePreset": "cl-qt-debug", "inherits": "base-test" },
{ "name": "cl-qt-release", "configurePreset": "cl-qt-release", "inherits": "base-test" },
{ "name": "clangcl-debug", "configurePreset": "clangcl-debug", "inherits": "base-test" },
{ "name": "windows-everything","configurePreset": "windows-everything","inherits": "base-test" },
{ "name": "gcc-debug", "configurePreset": "gcc-debug", "inherits": "base-test" },
{ "name": "clang-debug", "configurePreset": "clang-debug", "inherits": "base-test" },
{ "name": "linux-everything", "configurePreset": "linux-everything", "inherits": "base-test" },
{ "name": "clang-asan", "configurePreset": "clang-asan", "inherits": "base-test" },
{ "name": "clang-tsan", "configurePreset": "clang-tsan", "inherits": "base-test" },
{ "name": "clang-ubsan", "configurePreset": "clang-ubsan", "inherits": "base-test" },
{ "name": "clang-coverage", "configurePreset": "clang-coverage", "inherits": "base-test" }
],
"workflowPresets": [
{
Expand Down
Loading