ci+mutation: the campaign could not report a failure at all -- three compounding defects, a pipefail gate, and TSan's second stack (fixes #730, #731, #732, #736)

[Yaraslaut]

Four of six tickets, one commit each. The first three are mutation.yml and
they compound: on master the campaign is unable to report a failure at
all — the step goes green, and anything that does surface is swallowed by an
unrelated open issue. Fixed in that order, because #730 has to land first for
the other two to be checkable.

Two tickets are handed back with evidence rather than implemented; both are
boundary problems, not design problems, and both are recorded on their issues.

ticket	what landed
#730	set -o pipefail on the campaign step, the other four unguarded pipes in the tree, and scripts/check_workflow_pipefail.py + self-test + a pipefail-lint job so the class stays closed
#731	the failure report moves to scripts/report_mutation_failure.sh, which comments on the scope's open issue instead of skipping — self-tested against a stub gh
#732	mull's per-mutant timeout is derived from a baseline measured on the runner, not pinned to one machine's 22 s
#736	second_deadlock_stack=1 in both ci.yml sites and CMakePresets.json's clang-tsan preset
#674	handed back (third time) — every file the finished change needs belongs to another lane
#650	handed back — the surviving half is documentation whose home is in tests/**

---

#730 — a failed campaign reported success

mutation.yml ran bash scripts/mutation.sh … | tee … under bash -e {0},
which is -e but not -o pipefail, so the step's status was tee's.
scripts/mutation.sh takes deliberate care to exit 1 when mull-runner writes
no report; that 1 was discarded and the failure surfaced one step later as
check_mutation_regression.py failing to open a missing file.

The deliberate breakage, driving the step's own shell fragment the way
GitHub does against a stub mutation.sh that fails the way run 35592789912 did:

BEFORE: campaign step exit = 0
AFTER:  campaign step exit = 1
AFTER (a campaign that prints nothing and exits 0): exit = 1

The third line is a test -s on the log: pipefail cannot see a script that
prints nothing and exits 0.

The unguarded-pipe sweep

Parsing all 169 run: blocks across the seven workflow files found 13
pipelines, 6 unguarded:

site	pipeline	verdict
mutation.yml:119	scripts/mutation.sh | tee	the defect — now guarded
ci.yml:1435, :1898	echo "$changed" | grep -qE	benign (echo cannot fail) — guarded anyway
ci.yml:2627	tr -cd '\0' < … | wc -c	benign: the -lt 100 check below already refuses a vacuous run — guarded anyway
ci.yml:2965	find A B … | head -1	deliberate: the left side is expected to fail. Carries # pipefail-ok: with the reason morph#479's own comment already gives
wasm-ladder.yml:209	find … | sort	printed nothing and exited 0 whether the tree was empty or absent — guarded

So one live defect, one deliberate exemption, four hardened. ci.yml:286
already carried a comment about this exact hazard and pipefail was set by hand
in three steps — the guard was known and applied inconsistently, which is what
the gate is for.

scripts/check_workflow_pipefail.py makes it the rule: a pipeline must be
covered by a pipefail shell, a set … pipefail earlier in the block, or a
# pipefail-ok: <reason> marker — and a marker that excuses nothing is an
error, since it sits inert ready to excuse whatever is written on that line
next. Two anti-vacuity floors, on run blocks parsed and pipelines found, because
a scanner that stops matching would otherwise report a clean tree it never read.

The gate shown failing on broken input (13 cases, all green on this branch):

ok: the unmodified tree passes
ok: caught: morph#730: the mutation campaign's pipefail removed
ok: caught: morph#479: the clang-tidy step's pipefail removed
ok: caught: a newly added unguarded pipeline
ok: caught: the run-block syntax moved out from under the parser
ok: caught: every pipeline rewritten out of the shape the gate reads
ok: caught: an exemption with no reason
ok: caught: an exemption left behind after its pipeline went away
ok: accepted: a disjunction is not a pipeline
ok: accepted: a quoted pipe character is not a pipeline
ok: accepted: a GitHub expression containing ||
ok: accepted: a pipeline under an explicit shell: bash
ok: accepted: a pipeline in a pwsh step

Note the last-but-two: shell: bash is bash --noprofile --norc -eo pipefail {0} and does carry the guard. Only the absent key does not, and that asymmetry
is most of why this recurs.

---

#731 — the report was silenced by an unrelated open issue

The old step built a fixed title per scope and skipped when any issue with that
title was open. The title carries the scope and nothing about the failure, so
the first failure silenced every later, different one — which is how #732 stayed
invisible behind #517.

The body moves to scripts/report_mutation_failure.sh, which comments on the
open issue instead, with the cause it classified and the tail of each log the
run produced. The dedup is kept: one thread per scope, and the thread is the
campaign's history.

It moves to a script because the acceptance condition is "force two different
failures in one scope and confirm both are recorded", and what has to be
confirmed is the absence of a record — no issue, no comment, no label. There
is nothing to inspect afterwards, so it has to be driven.
scripts/test_report_mutation_failure.sh drives it against a stub gh, using
the two runs #731 cites verbatim:

ok: two different failures in one scope: the first opens the issue, the second comments on it
ok: the comment carries the second failure's own log, not the first's
ok: the comment names the cause it classified
ok: the comment names the run it came from
ok: the opening issue quotes the regression rather than paraphrasing it
ok: an issue whose title merely contains the phrase is not commented on
ok: a failure in a different scope opens its own thread
ok: an unrecognised failure is reported as unclassified, with its log
ok: a log the run never wrote is named as missing, and the report still lands
ok: a failing search is fatal, and the failure still reaches the run log

Shown failing on the old behaviour — reinstating the exit 0 branch in a
scratch copy:

error: both failures must be recorded. Transcript was:
create 900
expected:
create 900
comment 900
error: the comment does not quote the warm-up timeout it was reporting:
error: the comment does not name the classified cause:
error: the comment does not name run 2
4 case(s) failed.

Two details that are not incidental: the title match is exact, and the
filter is in shell rather than in --jq, because a filter inside jq is one only
the real gh can run and the near-miss case could not be tested at all. And a
failing gh issue list is fatal rather than "no issue open", which would
file a duplicate every week.

---

#732 — the campaign could not complete, and the baseline nobody had

The hosted-runner baseline, measured — it was in the campaign's own logs:

$ gh run view 34836153375 --log | grep -i "warm up" -A 2
[info] Warm up run (threads: 1)
       [################################] 1/1. Finished in 24.21s

run	date	warm-up (1 thread)	mutant phase	build
34349442137	2026-09-09	completed	completed	—
34836153375	2026-09-14	24.21 s	121m13.7s / 784	~14 min
35592789912	2026-09-21	> 60 s (killed at the cap)	never started	~29 min

This corrects the ticket's premise. A hosted runner is not materially
slower: 24.21 s against the workstation's 22 s is about 10%, and two of the
three hosted runs completed the campaign at the old cap. What it is, is
variable — the same job's instrumented build took 14 minutes on 09-14 and 29
on 09-21, and the baseline moved with it, from 2.5× under the cap to over it.
So a second constant derived from 24.21 s would have the same shape as the first
and a cliff in a new place.

scripts/mutation.sh now runs the unmutated suite once, single-threaded,
exactly as mull's warm-up will, and sets --timeout to 2.7× what it measured,
floored at the old 60000 ms. Driving the derivation lines as they appear in the
script:

baseline   1s -> --timeout 60000 ms (60.0x)      <- the floor
baseline  22s -> --timeout 60000 ms (2.7x)       <- today's workstation, unchanged
baseline  24s -> --timeout 64800 ms (2.7x)       <- the measured hosted baseline
baseline  60s -> --timeout 162000 ms (2.7x)      <- the run that aborted would have proceeded
baseline 150s -> --timeout 405000 ms (2.7x)
baseline 150s with MULL_TIMEOUT_MS=99000 -> --timeout 99000 ms

The measurement is also the only place the unmutated suite's own output is ever
seen — mull reports a failing baseline as Original test failed (warmup run)
with stdout: ''. Against a stub binary that fails:

refusal exit = 1
scripts/mutation.sh: the unmutated suite exited 1 after 0s.
  Every mutant is scored against this run, so a failing baseline makes the
  whole campaign meaningless. Mull reports this as 'Original test failed
  (warmup run)' with the suite's own output stripped out; here it is:
test_outbox.cpp:412: FAILED: REQUIRE( queue.size() == 1 )

Not verified: no campaign was run — no Mull on this box, and the campaign is
~76 minutes at best. What is measured is the derivation, the refusal, and the
three runs' figures, which come from CI's logs rather than an estimate.

Consequence: no mutation score since 2026-09-09. This unblocks #517,
which cannot be closed without a campaign run to triage its four survivors —
that was left alone here, as instructed.

---

#736 — TSan now names where the already-held mutex was taken

Both ci.yml sites and CMakePresets.json's clang-tsan test preset. Measured
on a two-mutex inversion, clang 22.1.8:

$ TSAN_OPTIONS=""                        ./deadlock   ->  36 lines
$ TSAN_OPTIONS="second_deadlock_stack=1" ./deadlock   ->  55 lines
$ grep -c "previously acquired by the same thread here" plain.txt   ->  0
$ grep -c "previously acquired by the same thread here" sds.txt     ->  2

The 19 extra lines are two stacks naming the acquiring function and source line
(take_a_then_b() deadlock.cpp:11:33). Without them TSan prints only the hint
to set the option — advice nobody can take after the fact, since #578 and #717
are intermittent and the run that fires is the only evidence there will be.

The triage's own closing condition ("invalid if the option costs meaningful
runtime") measured: 2M lock acquisitions across 4 threads over 16 mutexes under
TSan, which is the worst case for an option that retains a stack per
acquisition —

without:                       0.275  0.275  0.242
with second_deadlock_stack=1:  0.263  0.273  0.278

No measurable difference. Proxy, not the suite — a lock-heavy microbenchmark,
not morph_tests under the CI preset.

The option is colon-separated into the existing value; a second
TSAN_OPTIONS: key silently replaces the first and drops the suppressions file:

>>> yaml.safe_load("env:\n  TSAN_OPTIONS: suppressions=…\n  TSAN_OPTIONS: second_deadlock_stack=1\n")
{'env': {'TSAN_OPTIONS': 'second_deadlock_stack=1'}}

cmake/tsan.supp is untouched — it is held by an open PR and nothing here needs
it.

---

Handed back

#674 — third hand-back, same boundary, one new measurement: the half this
lane does own (ci.yml's CATCH2_VERSION and 22 catch2 apt lines) is the
half that must not ship first. CMakeLists.txt:555-562's fallback fetches
v3.8.1 without SYSTEM, so dropping the apt installs alone makes every
Catch2 macro expansion user code — the #684 lane measured 1 finding → 45 on a
single TU. Recorded on the issue, with the corollary that any contributor
without the distro package is already in that configuration today.

#650 — the rescope verdict stands and needs one correction: the coupling
is four checkers, not three, and they point in opposite directions. Three
prune tests/lint from a tree-wide scan; check_bidi_controls.py names its
fixture root to scan it deliberately. So the unwritten convention is two rules,
not one, which strengthens the documentation case and weakens the gate case.
The remaining work is tests/lint/README.md plus a two-way correspondence
check; the README is in tests/**, held by another PR this sweep, and the
check is worthless without it. Recorded on the issue.

---

Filed

• test_check_*.sh: the shared edit() helper cannot tell an applied mutation from one that silently truncated the file #746 — the edit() helper shared by ten test_check_*.sh self-tests
  cannot tell an applied mutation from one that silently truncated the file.
  Reproduced here: five consecutive ok: accepted lines over a docs.yml that
  had been emptied by a sed syntax error. The repaired helper is in this
  branch's new self-test; the other ten are other lanes' files.
• mutation.yml: timeout-minutes 240 has a 1.6x margin over the only completed hosted run, and the runner variance is 2x #747 — mutation.yml's timeout-minutes: 240 has a 1.6× margin over the
  only completed hosted run (2h16m35s), against 2× observed runner variance.
  Surfaced by The mutation campaign cannot complete on a hosted runner: mull's warm-up run of the unmutated suite times out at 60 s #732: a slow runner now reaches the mutant phase instead of
  aborting in the warm-up. Extrapolated, not observed.
• TSAN_OPTIONS is spelled out in three places that must agree, and nothing compares them #748 — TSAN_OPTIONS is now spelled out in three places that must agree,
  with nothing comparing them. That divergence is what ci: five test suites are never sanitizer-instrumented, including the one a job was widened to cover #542 and A relative TSAN_OPTIONS suppressions path makes ctest's Catch2 discovery exit 66 with no message, and no doc says to use an absolute one #688 each were.

Nothing staled a citation in either allowlist JSON. scripts/mutation_survivors.json
mentions --timeout 60000 in three places; all three are records of specific
past runs on the 12-core workstation, where the derived value is still 60000, so
they remain accurate and were not touched.

Gates

All run after the final edit, exit codes captured directly from the command
rather than through a pipe:

check_allowlist_citations                    exit=0
check_dep_pins                               exit=0
check_dep_pins_selftest                      exit=0
check_mutation_survivors                     exit=0
check_mutation_regression_st                 exit=0
check_branch_coverage_selftest               exit=0
check_ci_clang_pin                           exit=0
check_rung_filters                           exit=0
check_nolint_directives                      exit=0
check_workflow_job_banners                   exit=0
check_workflow_option_coverage               exit=0
check_workflow_job_references                exit=0
test_dep_cache                               exit=0
check_workflow_pipefail                      exit=0
test_check_workflow_pipefail                 exit=0
test_report_mutation_failure                 exit=0

0 gate(s) failed

Plus check_spec_citations.sh, check_tidy_suppression_scope.sh and
check_catch2_pin.sh, all exit=0.

🤖 Generated with Claude Code

https://claude.ai/code/session_01VptDWG2fKr2vBnLSJcgzgW

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!