[dmihalcik-virtru]

Found while reviewing #669. This is a pre-existing bug on main, so this PR is based on main and is independent of the stage 2/3 stacks.

The bug

Every invocation's timeout is clipped to the cell's deadline. At CI defaults that is nearly every invocation: 1500 s across seven cells per SDK gives each cell about 214 s, against a 600 s timeout_s.

• A hang looks like the budget running out. A build that never finishes times out at the deadline. one_round reclassified any MeasurementError that ended at or after the deadline as BudgetExhausted. That included a timeout, a crash that happened to land there, and even a full 600 s timeout that overran it.
• A skip doesn't fail the run. test_benchmarks.py records BudgetExhausted as a skip, and skips fail the gate only when every cell skips (nothing_measured).
• Net effect: a candidate that hung on every call read budget ran out during warm-up after 0 of 5 rounds. The other cells passed, and the run was green.
• An intermittent hang after min_rounds ended the loop as stopped_because: "budget", which the step summary does not show.

What it does

• Timeouts get their own type. measure() raises MeasurementTimeout(MeasurementError) for a timeout, and only that can become BudgetExhausted. A non-zero exit or a failure to start is a MeasurementError whenever it happens.

• When a timeout counts as budget. Both of these must hold:

  • the deadline set it, so the invocation got less than timeout_s;
  • the cell has no evidence the invocation should have finished. Either nothing has completed yet, or it got less than _HANG_FACTOR (10) times the longest invocation completed in this cell (any arm, warm-up included).

  Any other timeout fails the cell, in any round, naming the arm, the time it had and that longest invocation.

• The reference runs first in the cell's first warm-up round, so a candidate's first run always has evidence to be judged against. Warm-up samples are dropped, and the shuffle still runs in that round, so a seed draws the same measured interleaving as before.

• If the reference times out before anything has completed, it counts as budget. No candidate has run yet. A reference that hangs everywhere still fails the run as NOTHING MEASURED.

• Skip messages keep their cause. The warm-up and "only N rounds" BudgetExhausted messages append the reason one_round gave and chain it as __cause__.

• README: a new design section, "A timeout fails the cell unless the budget caused it", a "failed cell" note in section 1, and invariant 11.

Behaviour changes

• A full-length (600 s) timeout now always fails the cell, even if the deadline also passed. So does a crash past the deadline.
• Genuine budget exhaustion still skips. The stopping rule and the statistics are unchanged.

Open questions for review

• A cell left with little time can still go red. Example: the cell starts with 20 s left, the reference's first run takes 1 s, and the candidate's cold first run legitimately needs 25 s. That fails rather than skips. Adding a minimum (e.g. given time ≥ 10× longest and ≥ N s) would close it. An absolute floor was left out deliberately, because it would compare against a fixed number rather than this run's evidence.
• The "first warm-up round" check is implicit. It is detected as "nothing has completed yet" (not longest_s), not as an explicit flag, so that one_round's signature stays the same and the change does not collide with chore(xtest): measure benchmark cells over a tuple of arms #669.
• --bench-warmup 0 has no warm-up round to reorder. Whichever arm is drawn first is judged with no evidence. This is documented.

Validation

• uv run pytest test_bench_*.py: 238 passed. ruff, ruff format and pyright are clean.
• TestTimeouts has 10 new tests. All 10 fail against main's runner.py. Each uses CI-like numbers (214 s window, 600 s timeout) and covers:
  • an always-hanging candidate, under both shuffle draws;
  • reference-first in round 0, with the RNG sequence preserved;
  • cases that still skip: seconds left, slow rounds, and a reference timeout with no evidence;
  • the cause chain;
  • an intermittent hang after min_rounds;
  • a full-length timeout;
  • a crash at the deadline.
• test_bench_measure.py asserts that a timeout is a MeasurementTimeout, and that a non-zero exit and a failure to start are not.

Interaction with #669

• runner.py conflicts in four places in one_round's invocation block. Keep this PR's structure, use chore(xtest): measure benchmark cells over a tuple of arms #669's who (f"{arm.name} ({arm.label})") in the messages, and keep the inner BudgetExhausted messages without the cell_id prefix.
• test_bench_runner.py conflicts on one import line.
• After the rebase, TestTimeouts' run_cell calls take a list of arms, and the expected message text changes to who.

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[github-actions]

X-Test Results

✅ java@v0.19.1-main
✅ js@v0.23.0-main
✅ js@main-v0.29.0
✅ js@v0.23.0-v0.29.0
✅ js@main-main
✅ go@main-main
✅ go@main-v0.29.0
✅ java@main-v0.29.0
✅ go@v0.39.0-main
✅ java@main-main
✅ java@v0.19.1-v0.29.0
✅ go@v0.39.0-v0.29.0