Skip to content

tests: block the strand drain-race case's coarse rendezvous instead of spinning - #834

Merged
Yaraslaut merged 1 commit into
masterfrom
fix/826-strand-race-blocking-wait
Sep 26, 2026
Merged

Yaraslaut merged 1 commit into
masterfrom
fix/826-strand-race-blocking-wait

Conversation

@Yaraslaut

Copy link
Copy Markdown
Member

Summary

tests/test_strand_race.cpp's "keeps one strand per key when a post races the drain" case synchronised its threads by spinning: four chaser threads spun on an atomic gate (yielding once every 4096 spins), and the main thread spun on an atomic completed counter. Neither wait yields the CPU meaningfully to the actual work, so under Valgrind -- which runs one thread at a time and lets a yielding thread take the lock straight back -- the spin burns real instrumented CPU time for the whole round instead of letting the (slow, instrumented) work proceed. Measured on real CI runs (cited in #826): ~45 of a 54-minute job, and a separate run cancelled after >3h.

This PR replaces both coarse per-round waits with a std::condition_variable blocking wait. The fine-grained stagger loop that samples the actual few-instruction drain-and-erase race window is untouched -- it needs the spin's precision, and a blocking wait's wake latency would overshoot a window that small.

As part of /simplify, completed/gate were also changed from std::atomic<int> to plain int (every access is now under the same mutex the condition variables use, so the atomics were redundant), and body()'s notify was narrowed to fire only when a task's completion crosses the round's threshold, rather than on every one of a round's kPerRound tasks.

Verification

  • Correctness / no regression: full local suite green (ctest, 1605/1605 tests) on a plain Release build, and the fixed case passes cleanly under ThreadSanitizer (36/36 assertions, 3 runs).
  • Detection power preserved (the risk this fix could quietly break): the pilot task's post-task drain decision runs on the same worker thread within nanoseconds of the gate publish, while a blocking wait's OS wake latency is 1-50+ us -- so it's not obvious a blocking rendezvous still lands chaser posts inside that window. Verified empirically rather than assumed: reintroduced the historical two-step drain bug this test regresses (post() releasing _mapMtx before re-arming under strand->mtx only, and scheduleNext's drain flipping running/erasing in two separate critical sections instead of one) in a local, uncommitted mutant of include/morph/core/strand.hpp, then ran the drain-race case under TSan 10 times against both the original spin-based test and the new blocking-wait version. Both caught the reintroduced race 10/10 (a TSan data-race report plus a plain-state assertion failure each time). include/morph/core/strand.hpp itself is unchanged in this PR -- the mutant was local-only, for verification, and reverted before committing.
  • CPU-burn mechanism, measured directly: a standalone microbenchmark reproducing just the two rendezvous shapes (4 threads waiting out a fixed 3-second artificial stall) shows the old spin-based wait burning 9.81 CPU-seconds over that stall, versus 0.000025 CPU-seconds for the new blocking wait -- directly reproducing the mechanism behind the cited Valgrind numbers, independent of Valgrind's own overhead (not available on this development machine, macOS/arm64).

Adjacent finding (out of scope here)

tests/test_strand_race.cpp's third case ("recycles a drained strand under the key that asked for it") has a structurally identical per-round spin-wait, not covered by #826's issue text or triage. Filed as #833 rather than folded into this PR.

Test plan

  • ctest full suite green (1605/1605) on a plain Release build
  • Fixed case passes under ThreadSanitizer (3 runs, 36 assertions each)
  • Detection power verified: 10/10 catch rate against a reintroduced historical bug, matching the pre-fix spin-based test's own 10/10 catch rate
  • CPU-burn mechanism confirmed via a standalone microbenchmark
  • CI Valgrind job (this branch has no --fair-sched=yes workaround from Move morph onto core-cpp v0.5.0: coroutine model handlers on core-cpp strands, CPM dependencies #806, since that PR is unmerged -- this fix should let the case complete without CI's own workaround)

Fixes #826

🤖 Generated with Claude Code

https://claude.ai/code/session_018fEUahMFF32wQLiWjbsfkc

…f spinning

The drain-race case's per-round rendezvous (a chaser waiting for the pilot
to open `gate`, and the main thread waiting for the round's `completed`
count) busy-spun. Nothing blocks a spinning thread from being scheduled, so
under Valgrind -- which runs one thread at a time and lets a yielding thread
take the lock straight back -- the spin burns real instrumented CPU time for
the whole round instead of yielding it to the actual work, stalling the case
for tens of minutes.

Both coarse waits now block on a condition variable instead, notified when
the round's task count crosses its threshold. The fine-grained stagger loop
that samples the actual few-instruction drain-and-erase window is untouched:
it is not a rendezvous wait, and needs the spin's precision to land inside a
window a blocking wait's wake latency would overshoot.

Fixes #826

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018fEUahMFF32wQLiWjbsfkc
Signed-off-by: Yaraslau Tamashevich <yaraslau.tamashevich@gmail.com>
@codecov

codecov Bot commented Sep 26, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@Yaraslaut
Yaraslaut merged commit 150c881 into master Sep 26, 2026
37 checks passed
@Yaraslaut
Yaraslaut deleted the fix/826-strand-race-blocking-wait branch September 26, 2026 15:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

tests: the strand drain-race case's rendezvous depends on spinning, and starves under Valgrind

1 participant