tests/.clang-tidy (and nine copies) name the wrong Catch2 version for the clang-tidy job

[Yaraslaut]

tests/.clang-tidy's header states, as a fact a reader is meant to rely on:

The clang-tidy-diff job installs ubuntu-24.04's catch2, 3.4.0, which
predates that comment -- which is why the finding reaches CI here and not on
a workstation with a current Catch2.

(The same sentence is repeated in examples/bank/tests/.clang-tidy and, per
scripts/check_rung_filters.sh, in the eight other examples/*/tests/
copies.)

The version is wrong. The runner's Catch2 is in the 3.5.x series.

Verification status

Measured from CI's own artifact, not reproduced by installing the package.
The clang-tidy-report artifact of run 35581623269 (PR #665, head
aa84b126) prints the runner's headers verbatim:

/usr/include/catch2/internal/catch_test_registry.hpp:121:35: note: expanded from macro 'INTERNAL_CATCH_TESTCASE'
  121 |         INTERNAL_CATCH_TESTCASE2( INTERNAL_CATCH_UNIQUE_NAME( dummyFunction ) )

dummyFunction exists only inside
#ifdef CATCH_CONFIG_EXPERIMENTAL_STATIC_ANALYSIS_SUPPORT, which Catch2 did
not have in 3.4.0 — it arrives in 3.5.0. In 3.4.0 the same macro names the
function CATCH2_INTERNAL_TEST_. A 3.4.0 runner could not print that line.

Three further line numbers in the same artifact match Catch2 3.5.3 exactly
(checked against the v3.5.3 tarball): catch_test_macros.hpp:142 is
#define TEST_CASE( ... ) INTERNAL_CATCH_TESTCASE( __VA_ARGS__ ), and
catch_unique_name.hpp:12/13/15 are ..._LINE2 / ..._LINE /
INTERNAL_CATCH_UNIQUE_NAME.

Not verified: the exact package version. One line does not match 3.5.3 —
the artifact renders catch_test_registry.hpp:121 without the
, __VA_ARGS__ that upstream 3.5.3 carries there, so it may be a
distribution patch or a neighbouring 3.5.x point release. "3.5.x" is as far
as the evidence goes; apt-cache policy catch2 on an ubuntu-24.04 runner
would settle it in one line.

Why it is worth fixing rather than shrugging at

The sentence is not decoration: it is the stated reason a suppression exists,
and the reason a workstation is expected to disagree with CI. #666 is about
that disagreement being real and silent; this ticket is about the record of it
naming the wrong version, which is the first thing anyone chasing #666 would
try to reproduce against.

What would change the verdict

• Close it when the copies say what the runner actually has — ideally printed
  by the job itself rather than asserted in a comment, so it cannot go stale
  the next time the runner image moves.
• Close it as moot if clang-tidy-diff run locally is silently blind to Catch2 TEST_CASE cognitive-complexity findings the CI job reports #666 is resolved by pinning the job's Catch2, since the
  sentence then describes a pin rather than whatever apt happens to ship.
• Re-open if apt-cache policy catch2 on the runner image reports 3.4.0
  after all, which would mean the artifact is being misread.

[Yaraslaut]

Triage: valid — nine copies of a number, and the number is the load-bearing part of the sentence

Confirmed on 7a343e6f: nine examples/*/tests/.clang-tidy files carry

"The clang-tidy-diff job installs ubuntu-24.04's catch2, 3.4.0, which predates that comment — which is why the finding reaches CI here and not on a workstation with a current Catch2."

triage: valid, documentation, area: ci.

The reasoning in that sentence is right and the fact in it is probably wrong, which is the awkward combination. The filer's evidence: 3.4.0 has no CATCH_CONFIG_EXPERIMENTAL_STATIC_ANALYSIS_SUPPORT, so it cannot generate the dummyFunctionNN symbol names that the CI artifact is full of — and those names are all over the report that failed #665. That places the installed package in 3.5.x. The filer explicitly did not verify the exact package version, and neither did I.

Why it is worth fixing rather than leaving: the sentence exists to tell a reader why their local run disagrees with CI. A reader who checks the claim, finds their distro's 3.4.0 has different behaviour than described, and concludes the comment is stale, learns the wrong thing. And there are nine copies, so the correction is nine edits — which is itself the argument for not writing a version number into prose nine times.

For whoever takes it

• Establish the actual version first. apt-cache policy catch2 on an ubuntu-24.04 runner, or read it out of a CI log. Do not substitute one unverified number for another.
• Then consider not writing it down at all. The nine copies exist because the suppression is duplicated per rung; the version inside them is incidental to what each file is for. A single citation — or having the job print its Catch2 version, so the log is the source of truth — removes a class of staleness rather than one instance of it. scripts/check_rung_filters.sh already keeps those nine files in step, so whatever shape you choose should stay checkable by it.
• This is the same fix site as clang-tidy-diff run locally is silently blind to Catch2 TEST_CASE cognitive-complexity findings the CI job reports #666. If that ticket pins or detects the version, this one's number should come from there rather than being independently corrected. Worth doing them together, or doing clang-tidy-diff run locally is silently blind to Catch2 TEST_CASE cognitive-complexity findings the CI job reports #666 first.

What would change the verdict

Close as invalid if ubuntu-24.04's catch2 really is 3.4.0 and the dummyFunctionNN names come from somewhere else — in which case the finding is that the naming mechanism is not what anyone thinks, which is a better ticket than this one.

🤖 Generated with Claude Code

https://claude.ai/code/session_01VptDWG2fKr2vBnLSJcgzgW

[Yaraslaut]

The version is 3.4.0. The nine copies were right, and this ticket's premise is false.

Taken as part of PR #673's batch, which was told to establish the version
before replacing one unverified number with another. It establishes 3.4.0.

The premise

dummyFunction exists only inside
#ifdef CATCH_CONFIG_EXPERIMENTAL_STATIC_ANALYSIS_SUPPORT, which Catch2 did
not have in 3.4.0 — it arrives in 3.5.0. In 3.4.0 the same macro names the
function CATCH2_INTERNAL_TEST_. A 3.4.0 runner could not print that line.

v3.4.0 has it. From the v3.4.0 tarball,
src/catch2/internal/catch_test_registry.hpp:

 77:#if !defined(CATCH_CONFIG_EXPERIMENTAL_STATIC_ANALYSIS_SUPPORT)
 91:#else  // ^^ !CATCH_CONFIG_EXPERIMENTAL_STATIC_ANALYSIS_SUPPORT | vv CATCH_CONFIG_EXPERIMENTAL_STATIC_ANALYSIS_SUPPORT
121:        INTERNAL_CATCH_TESTCASE2( INTERNAL_CATCH_UNIQUE_NAME( dummyFunction ) )
124:#endif // CATCH_CONFIG_EXPERIMENTAL_STATIC_ANALYSIS_SUPPORT

and src/catch2/catch_user_config.hpp.in:172 is
#cmakedefine CATCH_CONFIG_EXPERIMENTAL_STATIC_ANALYSIS_SUPPORT. So a 3.4.0
runner whose package was built with that option on prints exactly that line —
which is what the artifact shows.

The line that does distinguish the two releases points at 3.4.0, not 3.5.3

This ticket already noticed it and read it the wrong way round:

One line does not match 3.5.3 — the artifact renders
catch_test_registry.hpp:121 without the , __VA_ARGS__ that upstream 3.5.3
carries there, so it may be a distribution patch or a neighbouring 3.5.x
point release.

It is not a distribution patch. It is v3.4.0's text:

v3.4.0  catch_test_registry.hpp:121
  INTERNAL_CATCH_TESTCASE2( INTERNAL_CATCH_UNIQUE_NAME( dummyFunction ) )
v3.5.3  catch_test_registry.hpp:121
  INTERNAL_CATCH_TESTCASE2( INTERNAL_CATCH_UNIQUE_NAME( dummyFunction ), __VA_ARGS__ )

The three "further line numbers matching 3.5.3 exactly" —
catch_test_macros.hpp:142 and catch_unique_name.hpp:12/13/15 — match v3.4.0
exactly as well; they are unchanged between the two releases, so they never
distinguished them. Checked against the v3.4.0 tarball:
catch_test_macros.hpp:142 is #define TEST_CASE( ... ) INTERNAL_CATCH_TESTCASE( __VA_ARGS__ ),
and catch_unique_name.hpp lines 12/13/15 are ..._LINE2 / ..._LINE /
INTERNAL_CATCH_UNIQUE_NAME.

And the archive agrees

apt-cache policy catch2 was not available (Arch workstation), so Launchpad's
ubuntu/+archive/primary for noble:

catch2 3.4.0-1build1 in noble | Release  | Published  | 2024-04-22
catch2 3.4.0-1build1 in noble | Proposed | Deleted    | 2024-04-22
catch2 3.4.0-1       in noble | Release  | Superseded | 2023-11-19
catch2 2.13.10-1     in noble | Release  | Superseded | 2023-10-23

No -updates or -security publication. 3.4.0-1build1, which is what the
nine files say.

Two smaller corrections to the text

• The sentence is in nine files, and tests/.clang-tidy — which the title
  names — is not one of them. It is the eight examples/*/tests/.clang-tidy
  plus examples/common/testkit/.clang-tidy. tests/.clang-tidy discusses
  Catch2 3.16.0 locally, which is a different claim and a correct one.
• The mechanism paragraph in clang-tidy-diff run locally is silently blind to Catch2 TEST_CASE cognitive-complexity findings the CI job reports #666 that this ticket leans on — "which of those a
  given Catch2 release produces … is what changed between 3.5.x and 3.16.0" —
  should read 3.4.0 and 3.16.0. The divergence clang-tidy-diff run locally is silently blind to Catch2 TEST_CASE cognitive-complexity findings the CI job reports #666 measured is real; only the
  version at CI's end of it was wrong.

Verification status

• Measured: the upstream header texts (v3.4.0 and v3.5.3, fetched from
  raw.githubusercontent.com and read), the local Catch2 (3.16.0, with
  CATCH_CONFIG_EXPERIMENTAL_STATIC_ANALYSIS_SUPPORT undefined in
  catch_user_config.hpp:172), and the Launchpad publication history above.
• Not measured: apt-cache policy catch2 on an actual ubuntu-24.04 runner.
  PR tests/ci: a race regression test that detects its race, and a Catch2 version that is checked rather than asserted (fixes #668, fixes #666, refs #667) #673 adds the step that will print and assert it —
  scripts/check_catch2_pin.sh . --strict in the clang-tidy job — so the next
  run of that job is the direct measurement. If it disagrees with 3.4.0, this
  comment is wrong and the job goes red saying so, which is the point of
  building it that way.
• Not re-measured: clang-tidy-diff run locally is silently blind to Catch2 TEST_CASE cognitive-complexity findings the CI job reports #666's two clang-tidy exit codes. Nothing here
  contradicts them.

Disposition

This ticket's own closing condition is met:

Close it as invalid if ubuntu-24.04's catch2 really is 3.4.0 and the
dummyFunctionNN names come from somewhere else — in which case the finding
is that the naming mechanism is not what anyone thinks, which is a better
ticket than this one.

Both halves hold. The naming mechanism is indeed not what anyone thought:
dummyFunctionNN is not a 3.5.0 feature, it is the
CATCH_CONFIG_EXPERIMENTAL_STATIC_ANALYSIS_SUPPORT branch, present in 3.4.0
(earlier releases not checked) and switched on by whoever built the package
rather than by the version. That
now has somewhere to live: PR #673 records the version in ci.yml's
CATCH2_VERSION and reads it back off the runner, so it is checked rather
than argued from macro archaeology.

Closing as invalid. The nine copies are reworded, not corrected, in #673 —
same fact, in the canonical phrasing the new gate reads — which gives this
ticket's real concern (nine hand-written copies of a number that nothing
checks) a control rather than a ninth edit.

🤖 Generated with Claude Code

https://claude.ai/code/session_01VptDWG2fKr2vBnLSJcgzgW

[Yaraslaut]

Closed as invalid on the evidence above: ubuntu-24.04's catch2 is 3.4.0-1build1, which is what the nine copies already said. Labels left untouched -- the stale triage: valid is an assessment's output and AGENTS.md reserves changing it to the triage-issue skill, not to whoever worked the ticket.

[Yaraslaut]

Runner: correcting my own triage — this is invalid, and I labelled it valid on a premise I did not check

I triaged this triage: valid and wrote that the reasoning was right and the fact wrong. Both halves of that were backwards. Relabelled triage: invalid; the issue is closed.

The shared premise — mine and the filer's — was "3.4.0 has no CATCH_CONFIG_EXPERIMENTAL_STATIC_ANALYSIS_SUPPORT, so it cannot produce the dummyFunctionNN names in the CI artifact." I accepted it because it was specific and plausible. It is false, and one curl settles it:

$ curl -s https://raw.githubusercontent.com/catchorg/Catch2/v3.4.0/src/catch2/internal/catch_test_registry.hpp
 77:#if !defined(CATCH_CONFIG_EXPERIMENTAL_STATIC_ANALYSIS_SUPPORT)
 91:#else  // ^^ !CATCH_CONFIG_EXPERIMENTAL_STATIC_ANALYSIS_SUPPORT | vv …
121:        INTERNAL_CATCH_TESTCASE2( INTERNAL_CATCH_UNIQUE_NAME( dummyFunction ) )
124:#endif // CATCH_CONFIG_EXPERIMENTAL_STATIC_ANALYSIS_SUPPORT

v3.4.0 has the macro. dummyFunctionNN is a packager's build option, not a version feature.

And the artifact line that was read as evidence for 3.5.x is v3.4.0's text exactly:

v3.4.0:  INTERNAL_CATCH_TESTCASE2( INTERNAL_CATCH_UNIQUE_NAME( dummyFunction ) )
v3.5.3:  INTERNAL_CATCH_TESTCASE2( INTERNAL_CATCH_UNIQUE_NAME( dummyFunction ), __VA_ARGS__ )

The artifact has no , __VA_ARGS__. So it is 3.4.0, the nine .clang-tidy copies were right all along, and the thing I called "a number that is probably wrong" was the only checked number in the discussion.

What I should have done

The check above took one command and no build. I had already written "do not substitute one unverified number for another" into my own triage of this issue — and then labelled it valid without running the verification I was demanding of whoever picked it up. That is the failure mode this repository files issues about, performed by the person doing the filing.

The general lesson, which is worth more than the correction: a premise stated as a specific mechanism ("3.4.0 lacks feature X, therefore Y") reads as verified and usually is not. The filer labelled the package version explicitly unverified and I read that caveat as covering only the version string, when it covered the inference too.

What survives

Nothing of this ticket. #666 stands on its own — the local/CI divergence is real and independently measured — and its text has been corrected from 3.5.x to 3.4.0. The nine files are being reworded by PR #673 into a phrasing a gate can check, which is the right outcome: the number stops being prose and becomes something scripts/check_catch2_pin.sh verifies.

🤖 Generated with Claude Code

https://claude.ai/code/session_01VptDWG2fKr2vBnLSJcgzgW