Repository navigation
CI: run perpr as four legs balanced by recorded test time - #2161
Merged
Merged
Conversation
scripts/ci-tests-shard.sh dealt the serial scripts/ci-tests.d registrations out round-robin by sorted position, so the slowest leg set the job's wall clock, and one inserted registration moved every later test to the next leg. scripts/ci-durations.tsv now holds whole seconds per serial registration: the seconds between consecutive "== <registration>" lines of the three perpr job logs of public CI run 41148962559 (2026-10-31), rounded, at least 1. The helper takes the serial registrations longest first (ties by name, C locale) and gives each to the least-loaded leg (ties to the lowest index). A registration with no valid row counts as the table's median, so adding a test needs no table edit. Background (*.bg.sh) registrations stay round-robin, so only serial seconds are balanced. The slices are still disjoint and complete for any N, a malformed SHARD still exits 2 and an empty slice still exits 1. On the committed table (every test counted at least 1 s, so these are predictions from the table, not CI measurements) the old split gives legs of 283, 489 and 682 s of serial tests, and 544, 355 and 556 s once this change's own new registration moves every later test to the next leg; the new split gives 485, 485 and 485 s, so the slowest predicted leg drops from 682 to 485 s. New test scripts/test-perpr-shard-balance.sh, registered as scripts/ci-tests.d/perpr-shard-balance.sh: the table format, disjoint and complete slices for N = 1 to 5, the slowest of 3 legs within 1.1x of the mean (it fails on the old split), a fixture that pins the assignment, the median default and the background round-robin, and the awk command CONTRIBUTING.md gives for rebuilding the table, run on synthetic job logs. test-perpr-shard.sh drops its "slices differ by at most 2 registrations" case, which only the round-robin guaranteed. CONTRIBUTING.md describes the new split, says when to refresh the table (the slowest leg's Per-PR tests step more than 20% over the fastest on a PR run) and gives the rebuild command. Tests: 10/10 affected sections passed (run-ci-tests.sh --affected 2e7e099, including perpr-shard-balance, perpr-shard, ci-shards and conflict-free-prs); size ratchet, fragments gate and privacy clean. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The perpr job ran three legs, balanced by recorded test time. It now runs four: the matrix is shard 1 to 4, each leg passes SHARD=i/4, and the comment above the job says so. The selftests job keeps three legs. On the committed scripts/ci-durations.tsv (predictions from the table, not CI measurements) the old helper with three legs on the tree before this change gave legs of 283, 489 and 682 s of serial tests; the new split over four legs gives 364, 364, 364 and 363 s. The wrapper that runs a background test (*.bg.sh) now writes the end time to <test>.end before it publishes <test>.rc. The "Per-PR tests" step prints one "bg elapsed: <test> <seconds> s" line after the test's log: the end time minus the background start, or "?" when no end time was written. The line does not start with "==", so the awk command in CONTRIBUTING.md that rebuilds the table still ignores it. scripts/test-perpr-shard.sh checks that every fixture leg logs exactly one. scripts/test-perpr-shard-balance.sh no longer hard-codes three legs: it reads N from the SHARD=i/N that ci.yml's perpr legs pass, with the same 1.1x-of-mean bar. CONTRIBUTING.md and a comment in scripts/ci-tests-shard.sh no longer name the number of legs, and the rebuild command reads leg*.log, so the next change of N needs no doc edit. The changelog fragment now describes four legs. Tests: test-perpr-shard-balance 6/6, test-perpr-shard 21/21, test-ci-shards 1/1; 26 of 26 sections passed (run-ci-tests.sh --affected 2e7e099, privacy and trigger lint included); size ratchet, fragments gate and privacy clean. On the old helper the four-leg balance case fails (legs 380/247/544/284 s). Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
remigiusz-antczak
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Wave 13 runs the CI
perprjob as four legs balanced by recorded test time; on the committed table the slowest leg's serial tests drop from 682 to 364 seconds.What changes
scripts/ci-tests-shard.shgives each serial test, longest first, to the least-loaded leg, by seconds inscripts/ci-durations.tsv(CI run 41148962559). Unlisted tests count as the median.bg elapsed: <test> <seconds> s.scripts/test-perpr-shard-balance.sh;CONTRIBUTING.mdsays when and how to rebuild the table.Verification
Keep bar: revert unless, on each of the next 3 PRs, the slowest
perprleg is ≤ 9:10 and totalperprrunner minutes rise ≤ 10%.🤖 Generated with Claude Code