tests/ci: a race regression test that detects its race, and a Catch2 version that is checked rather than asserted (fixes #668, fixes #666, refs #667)

[Yaraslaut]

Three tickets about controls that report success while measuring nothing. Two
commits. One of the three turns out to be false, and is not "fixed" here —
see the #667 section, which is the part worth reading first if you only read
one.

• test_strand_race.cpp does not detect the race it is the regression test for -- 10/10 passes under TSan against a build with the two-step drain-and-erase restored #668 — the named regression test for the strand's per-key serialisation
  invariant does not detect that invariant's regression. Reproduced, rewritten,
  mutation-tested in both directions. fixes #668.
• clang-tidy-diff run locally is silently blind to Catch2 TEST_CASE cognitive-complexity findings the CI job reports #666 — a local clang-tidy-diff is silently green on a diff CI fails,
  and nothing records or checks which Catch2 the job measured against. A gate
  now does. fixes #666.
• tests/.clang-tidy (and nine copies) name the wrong Catch2 version for the clang-tidy job #667 — the nine .clang-tidy copies name the wrong Catch2 version. They
  do not. The runner's Catch2 is 3.4.0, and both tickets' shared premise is
  false. refs #667; the copies are reworded so the number is checked rather
  than merely asserted, and the ticket is answered with the evidence.

---

#668 — commit 1

Reproduced the miss before touching anything, as the ticket asks

Two-step drain restored in scheduleNext (flip running under strand->mtx,
release it, then erase under _mapMtx in a separate critical section), base
7a343e6f, x86-64 Linux, clang 22.1.8, clang-tsan preset,
TSAN_OPTIONS=suppressions=cmake/tsan.supp:

run 1..10: PASS   All tests passed (41 assertions in 2 test cases)
=== [race] : 10 pass / 0 fail out of 10 ===

Not a blind run: the binary carries 173 __tsan_ symbols.
(scripts/check_sanitizer_instrumentation.sh refuses a build directory holding
one binary — only examined 1 binaries ... too few for this check to mean anything — so the symbol count is read directly, since only morph_tests was
built.)

The finding is not stale. It reproduces exactly as filed.

The miss is the load shape

The erase only fires when a drain finds the pending queue empty, and eight
threads posting 3200 tasks back-to-back onto one key keep it non-empty almost
throughout. The header comment's "very short tasks maximise the drain/re-arm
interleaving that triggered the bug" is false in the direction that matters:
they maximise re-arm, and never produce the quiet moment a drain needs. That
sentence is corrected in place, with the measurement beside it.

The new case manufactures the rendezvous

A pilot task is posted alone on the key and publishes the round number as its
last act. Four chaser threads spin on that publication and post the instant it
flips — i.e. while the drain block following the pilot's body is deciding
keep-running vs. erase — each with a per-thread stagger that walks its post
across the handful of instructions that decision spans. A round ends only once
every one of its tasks has run, so the strand genuinely empties before the next
pilot. The gap is the point.

Three detectors, because the defect and its symptom are not the same event:

detector	what it sees	needs a sanitizer?
maxInFlight (kept from the old case)	the symptom: two tasks for one key at the same wall-clock moment	no
plain, non-atomic state touched by every task	the defect: two strands for one key leave those accesses unordered by any happens-before edge — TSan reports it whether or not the tasks overlap; lost updates to it are visible anyway	no (both ways)
per-producer FIFO	an orphaned strand runs one producer's later task before its earlier one, with no overlap required	no

Mutation-tested, with the parameters as committed

clang-tsan,  mutant, 10 runs        :  0 pass / 10 fail
                                       2-3 TSan data races per run
                                       outOfOrder 65-114, maxInFlight 2
clang-debug, mutant, 15 runs        :  0 pass / 15 fail
                                       outOfOrder 1-9, maxInFlight 2
clang-tsan,  mutant, taskset 4 CPUs :  0 pass / 10 fail
clang-tsan,  mutant, taskset 2 CPUs :  0 pass / 10 fail
the OLD case, same mutant binary    : 10 pass /  0 fail

The TSan report names the mechanism:

WARNING: ThreadSanitizer: data race (pid=520470)
  Write of size 4 at 0x721800008400 by thread T1:
    #0 ...::$_2::operator()(int, int) const tests/test_strand_race.cpp:210:22
    #1 ...::$_0::operator()() const::'lambda'()::operator()() const tests/test_strand_race.cpp:245:71
    #6 morph::exec::detail::StrandExecutor::scheduleNext(...)::'lambda'()::operator()() const strand.hpp:158:17
  Previous write of size 4 at 0x721800008400 by thread T2:
    ... the same two frames, the other pool thread

Two strand lambdas for one key writing the same plain int.

And it does not flake on the fixed tree

[race] passes 20/20 under clang-tsan and 20/20 under clang-debug; the
full morph_tests binary passes (1562 cases, 23007 assertions, the one
failed as expected). Cost: 1.0 s under TSan on 4 CPUs, 1.3 s on 2,
0.098 s without a sanitizer, against the suite's TIMEOUT 120.

include/morph/core/strand.hpp is untouched (PR #671 holds it). The mutant
was applied to the file and reverted from a byte-identical backup; md5sum
checked against the pristine copy after each cycle, and git diff --stat for
the branch names no header.

---

#666 + #667 — commit 2

Establishing the version first, as instructed — and it settles #667 the other way

apt-cache policy catch2 is not available here (Arch workstation), so two
independent sources:

Launchpad, ubuntu/+archive/primary for noble:

catch2 3.4.0-1build1 in noble | Release  | Published  | 2024-04-22
catch2 3.4.0-1build1 in noble | Proposed | Deleted    | 2024-04-22
catch2 3.4.0-1       in noble | Release  | Superseded | 2023-11-19
catch2 2.13.10-1     in noble | Release  | Superseded | 2023-10-23

No -updates or -security publication. 3.4.0-1build1.

The CI artifact both tickets read as 3.5.x, checked against the upstream
tarballs:

v3.4.0  catch_test_registry.hpp:121
  INTERNAL_CATCH_TESTCASE2( INTERNAL_CATCH_UNIQUE_NAME( dummyFunction ) )
v3.5.3  catch_test_registry.hpp:121
  INTERNAL_CATCH_TESTCASE2( INTERNAL_CATCH_UNIQUE_NAME( dummyFunction ), __VA_ARGS__ )

The artifact prints it without , __VA_ARGS__ — v3.4.0's text exactly, and
the one line #667 itself flags as not matching 3.5.3. catch_test_macros.hpp:142
and catch_unique_name.hpp:12/13/15 match v3.4.0 too; they are simply
unchanged between the two releases, so they never distinguished them.

The premise both tickets rest on is false. #667: "dummyFunction exists
only inside #ifdef CATCH_CONFIG_EXPERIMENTAL_STATIC_ANALYSIS_SUPPORT, which
Catch2 did not have in 3.4.0 — it arrives in 3.5.0." v3.4.0 has it, at
catch_test_registry.hpp:77/91/124, and #cmakedefines it in
catch_user_config.hpp.in:172. So the nine copies were right all along, and
#667 meets its own invalidation condition verbatim — "the naming mechanism is
not what anyone thinks".

Two smaller corrections to #667's text while we are here: the sentence is in
nine files, and tests/.clang-tidy (which the title names) is not one
of them — it is the eight examples/*/tests/.clang-tidy plus
examples/common/testkit/.clang-tidy.

What actually needed fixing

#666's divergence is real and its measurement stands: which Catch2 is on the
include path decides whether a readability-function-cognitive-complexity
finding on a TEST_CASE body reaches the job. What was missing is that nothing
recorded or checked the version, and apt-get install -y catch2 pins nothing.

scripts/check_catch2_pin.sh, modelled on scripts/check_ci_clang_pin.sh
(#457) down to the rule-B rewording trap and the anti-vacuity failure:

• A. Textual. Every CI pins catch2 <V> must name ci.yml's new
  CATCH2_VERSION; any other line naming a Catch2 version beside a CI
  reference is rejected as a phrasing the gate cannot check; zero canonical
  assertions is a failure, not a pass.
• B. Behavioural. It reads the version out of the Catch2 headers actually
  on the include path. Under --strict — how the clang-tidy job runs it, right
  after the apt install and before the configure — a moved runner package, or
  no Catch2 at all, fails the job before it measures anything. Without
  --strict, a workstation run prints the divergence and says in as many words
  that its clang-tidy-diff result is not the measurement CI makes:

WARNING: this machine's Catch2 is 3.16.0 (/usr/include); CI pins catch2 3.4.0.
         A local clang-tidy-diff run is therefore NOT the measurement the
         clang-tidy-diff job makes. ...
         A green local run is not evidence for those checks. It is still
         evidence for every check whose finding lands on a line you wrote.

scripts/test_check_catch2_pin.sh drives both halves from the wrong side as
well as the right one — 13 cases, each drift reintroduced one at a time into a
scratch git tree or a synthetic include directory and required to fail for the
stated reason, plus four false-positive mirrors. All 13 pass; the two that
matter most:

ok: caught: --strict against a runner whose Catch2 package has moved
ok: caught: --strict with no Catch2 installed at all

The nine copies are reworded, not corrected: same fact, in the canonical
phrasing the gate reads, so the number is now checked rather than asserted —
which is what #667's triage actually wanted from nine hand-written copies of a
version.

What this deliberately does not do

It does not make a local clang-tidy-diff agree with CI's. Only the job not
depending on the runner's Catch2 does that (#666's first closing condition):
have the clang-tidy configure use the Catch2 CMakeLists.txt already pins for
FetchContent instead of apt's. That needs examples/common/CMakeLists.txt,
which PR #665 holds, and it is filed separately rather than folded in.

Nothing here tells a contributor to run clang-tidy differently — the local half
reports a measurement, the CI half fails a job.

---

The one claim this branch rests on

tests/test_strand_race.cpp now fails against the two-step drain-and-erase
mutant, and its new case's CHECKs cannot pass while two Strand objects
serve one key — measured 10/10 under clang-tsan and 15/15 without a
sanitizer, at 12, 4 and 2 CPUs, against 10/10 passes for the case it sits
beside.

Everything else on the branch is a gate over prose and a package version.

Review notes, done inline

• Could the new case flake on the fixed tree? Its three CHECKs are all
  invariants the spec states unconditionally (at most one task in flight per
  key, per-producer FIFO, no lost updates to state the strand is the only
  synchronisation for). None depends on a deadline or a task count reached in
  time: the round barrier is a synchronisation point, and ~StrandExecutor is
  the final drain, exactly as the neighbouring case's comment argues. 40 clean
  runs across two build configurations found none.
• Does the spin-wait starve a small runner? Four chasers spin while four
  pool threads work. The spin yields once per 4096 iterations as a starvation
  guard; pinned to 2 CPUs the case takes 1.3 s rather than 1.0 s and still
  catches the mutant 10/10, so the guard costs the rendezvous nothing.
• Is the plain state actually data-race-free on the fixed tree? Every
  handoff between two tasks for one key passes through _mapMtx/strand->mtx
  and the pool's own queue mutex, so each task's writes happen-before the next
  task's reads. 20 TSan runs report zero races; the mutant reports 2-3 per run.
• Could the new gate's rule B fire on existing prose? Surveyed before
  writing it: exactly one line in the tree names a Catch2 version by the gate's
  regex (docs/superpowers/specs/2026-09-15-opt-in-logging-design.md:119), and
  it carries no CI reference, so it is out of scope by construction. The
  self-test pins that as an expect_accepted case.

Gates run here

check_mutation_survivors.py, check_nolint_directives.sh,
check_bidi_controls.py, check_rung_filters.sh (58 checks, including all
nine reworded .clang-tidy files), check_spec_citations.sh,
check_ci_clang_pin.sh, check_workflow_job_banners.py,
check_workflow_option_coverage.py, check_catch_test_names.sh,
check_test_type_names.sh, check_ctest_name_collisions.sh build/clang-debug,
test_check_catch2_pin.sh, and clang-format --dry-run -Werror over
tests/test_strand_race.cpp. All pass.

Not verified: the --strict half has never run on an actual ubuntu-24.04
runner — it is exercised here against synthetic include directories, and this
PR's own clang-tidy leg is its first real execution. If it fails there, the
interesting outcome is what version it prints. Also not verified: the
Kanban / ThreadSanitizer leg and every non-Linux leg of the new test case.

🤖 Generated with Claude Code

https://claude.ai/code/session_01VptDWG2fKr2vBnLSJcgzgW

[Yaraslaut]

Runner verification

#668's fix matches the mechanism, not just the symptom. My triage argued the old case missed because 8 threads × 400 posts is sustained saturation while the race needs a strand to reach empty. The new case is built on exactly that:

"A pilot task is posted alone on the key. Its last act is to publish the round number… kChasers threads spin on that publication and post the instant it flips — that is, while the drain block following the pilot's body is deciding 'keep running vs. erase'… The round ends only once every one of its tasks has run, so the strand really does empty before the next pilot. The gap is the point."

And the mutation table is the part that makes it a control rather than a test: 0 pass / 10 fail against the mutant under clang-tsan, 0/15 without a sanitizer, unchanged at 4 and 2 CPUs — against 10/10 passes for the old case on the same mutant binary. Three independent detectors (maxInFlight, plain non-atomic state, per-producer FIFO), of which the last breaks on an orphaned strand with no overlap at all, which is the case maxInFlight structurally cannot see.

include/morph/core/strand.hpp is untouched — confirmed, empty diff against base. This is a test fix, not a source fix, which is what the ticket asked for.

The new gate works: test_check_catch2_pin.sh passes, and check_catch2_pin.sh reports all 9 catch2-pin assertion(s) agree with .github/workflows/ci.yml, exit 0.

#667 was mine to get wrong, and this branch is right

I triaged #667 valid. It is invalid, and I have relabelled it and recorded the correction there. One command settles it:

v3.4.0 catch_test_registry.hpp:
   77:#if !defined(CATCH_CONFIG_EXPERIMENTAL_STATIC_ANALYSIS_SUPPORT)
  121:        INTERNAL_CATCH_TESTCASE2( INTERNAL_CATCH_UNIQUE_NAME( dummyFunction ) )
v3.5.3 same file:
  121:        INTERNAL_CATCH_TESTCASE2( INTERNAL_CATCH_UNIQUE_NAME( dummyFunction ), __VA_ARGS__ )

v3.4.0 has the macro, and the artifact line has no , __VA_ARGS__ — so it is 3.4.0, and the nine .clang-tidy copies were right all along. Rejecting the ticket rather than implementing it is the outcome AGENTS.md asks for, and the three independent lines of evidence are why it is convincing.

Rewording the nine into a phrasing the gate reads, instead of editing a number, is the better fix — it converts prose into something checkable, which is the same move #659 made for AUTOMOC pairing.

Two things before this can land

1. It needs a rebase. Base is 7a343e6f; master is 563502ab since bank/gui + ci: clear the eleven bank-GUI sources' clang-tidy debt, and gate the Q_OBJECT header split that no build catches early (fixes #656, fixes #659) #665 merged, and that added scripts/check_qobject_moc_pairing.py plus a drift-guard.yml job — a gate this branch has never been judged by. Same verdict I applied to core/cmake: stop a serial dispatch rebuilding a std::deque it never fills, and say what -Wno-shadow-uncaptured-local really suppresses (fixes #660, refs #662) #671 an hour ago.
2. --strict has never run on a real runner, as the lane says. Its first real execution is this PR's own clang-tidy leg, and if it fails there the interesting output is which version it prints — that is the direct measurement neither ticket had.

Filing the check_sanitizer_instrumentation.sh finding the lane offered rather than leaving it in a hand-back: a guard that refuses a single-target build tree makes the script unusable for exactly the narrow verification it is most needed for.

🤖 Generated with Claude Code

https://claude.ai/code/session_01VptDWG2fKr2vBnLSJcgzgW

[Yaraslaut]

Runner: rebased onto 563502ab and fixed the one finding — 00ff98cc

Rebase, on the gate limb. #665 landed scripts/check_qobject_moc_pairing.py and its drift-guard.yml job — a gate this branch had never been judged by. Same verdict I applied to #671. It passes on the rebased tree, along with the branch's own new gate:

moc-pairing OK (the new gate from #665)
ok: all 9 catch2-pin assertion(s) agree with .github/workflows/ci.yml
nolint OK   bidi OK

clang-tidy-diff had one finding, read from the run's artifact since the run was still in progress:

tests/test_strand_race.cpp:289:40: error: use a ranges version of this algorithm
    [modernize-use-ranges,-warnings-as-errors]

Fixed: std::minmax_element(cells.begin(), cells.end()) → std::ranges::minmax_element(cells). <algorithm> was already included. The result is an aggregate of two iterators rather than a pair, so the structured binding and both dereferences are unchanged — I compiled that shape standalone under clang 22.1.8 -std=c++23 -Werror rather than assuming it, since "structured binding still decomposes" is exactly the kind of thing that reads as obvious and is not.

Worth noting what did not fail: the new --strict Catch2 check. The lane flagged that its first real execution on a runner would be this PR's own clang-tidy leg, and that the interesting output would be which version it prints. It did not fail, which is the first real evidence the pin matches — though the printed version is in the leg's log, and that log is still unavailable while the run finishes. Worth reading next sweep, because it is the direct apt-cache policy measurement neither #666 nor #667 ever had.

A correction about my own monitoring

I armed a watcher on #671 that reported "settled" while every check was still pending. It exited immediately because I started it seconds after a push, before the new run's checks existed — so pending == 0 was true for the wrong reason. That is the same "control that reports success while measuring nothing" failure this repository files issues about, in my own tooling rather than the repo's. Re-armed with a total-count floor, which is the same remedy check_bidi_controls.py and check_nolint_directives.sh already use: a scan that examined nothing must not read as clean.

🤖 Generated with Claude Code

https://claude.ai/code/session_01VptDWG2fKr2vBnLSJcgzgW

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[Yaraslaut]

Runner: the red legs are environmental, re-run requested

Both runs are complete, so I read every failing job rather than inferring. Failing step, per leg:

Not one of them is a compile, a test or a lint finding. They are package installs, a vcpkg checkout, and a dependency-cache configure — the same class as #672, which now has four distinct upstreams recorded in one day.

The two signatures seen here:

• Install sccache / Install GCC 15 / Setup vcpkg — gzip: stdin: not in gzip format (a download that returned something other than a tarball, piped to tar unchecked) and pathspec 'D:\a\morph\morph\vcpkg' did not match any file(s) known to git.

• Configure — inside the vendored Lightweight ORM's own CMakeLists in the dep cache:

  CMake Error at ~/.cache/morph-dep-cache/Lightweight_bbb972a78e1962b9/CMakeLists.txt:98 (CPMAddPackage):
  -- reflection-cpp not found, downloading...
    Unknown CMake command "CPMAddPackage".
  

  That is a cached third-party tree missing CPM, not anything either branch changed.

Why I am confident this is not the branches: master's own 14:27Z run was success, and PR #671 settled 54/54 green at 18:12 — passing the very Configure step that failed here at ~15:21. The failure window is bounded and shared, which is what an environment looks like and not what a defect looks like.

Re-run requested. If a leg comes back red with a compile or test error, that is a real finding and I will treat it as one.

🤖 Generated with Claude Code

https://claude.ai/code/session_01VptDWG2fKr2vBnLSJcgzgW

[Yaraslaut]

Runner: rebased onto 5fc5e788 — and this one is not routine

include/morph/core/strand.hpp changed on master (#671's PendingQueue), and tests/test_strand_race.cpp includes it directly. Second staleness limb: the combination was never tested.

Why it matters more than the usual rebase. This PR's entire contribution is a race test, and its evidence is a mutation result — 0 pass / 10 fail against the two-step-drain mutant, versus 10/10 passes for the old case on the same binary. That was measured against the old strand, whose queue was a std::deque rebuilt per dispatch. #671 replaced it with PendingQueue, which holds the head task inline.

So the open question is not whether the test still compiles or passes — CI answers that — but whether it still detects the race it was written to detect. A green run cannot distinguish "the test works" from "the test no longer provokes the window". That is precisely the failure #668 exists to fix, and merging on a green tick alone would reintroduce it one layer up.

Rebased cleanly, three commits replayed, pushed as f6191588. Cheap gates on the rebased tree:

ok: all 9 catch2-pin assertion(s) agree with .github/workflows/ci.yml
moc-pairing OK
ok: 15 structured citation(s) in scripts/mutation_survivors.json resolve to the line they name.
bidi-control lint OK: 1249 text file(s) scanned, 12 codepoint(s) searched for, 0 raw occurrence(s).

Before this merges, the mutation needs re-running against PendingQueue — restore the two-step drain in the new strand and confirm the new case still fails. I am not treating a green CI run as a substitute for that.

🤖 Generated with Claude Code

https://claude.ai/code/session_01VptDWG2fKr2vBnLSJcgzgW