test: assert parallelism, not wall-clock, in concurrent workers - #17
Merged
Merged
Conversation
test_passing_suite spawned 5 workers over 5 two-second spec files and asserted the run finished in under 5 seconds. That conflates "the work overlapped" with "the runner was fast": booting five rubies on a loaded CI box eats the margin even when parallelism worked. It failed two of five matrix legs on its first CI run, at 5.02s and 5.19s. Compare wall clock against the summed job time the queue already records instead. A serialized run takes at least as long as the sum, so the bound scales with runner speed rather than assuming it. Verified both ways: parallel passes with ~2x headroom (5s vs 10s), and forcing a single worker still fails it (10.65s vs 10.03s). Raising the constant to 10 was the alternative, but 10s is the serial time, so it would have passed a fully serialized run. refs DE-1818 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
test_passing_suitespawns 5 workers across 5 spec files that each sleep 2s, then asserted the run finished in under 5 seconds — which conflates "the work overlapped" with "the runner was fast". It failed two of five matrix legs on its first CI run, at 5.02s and 5.19s.Compares wall clock against the summed job time the queue already records instead. A serialized run takes at least as long as the sum, so the bound scales with runner speed rather than assuming it.
Verified both directions: parallel passes with ~2x headroom (5s vs 10s), and forcing a single worker still fails it (10.65s vs 10.03s). Raising the constant to 10 was the alternative, but 10s is the serial time, so it would have passed a fully serialized run.
refs DE-1818
🤖 Generated with Claude Code