[nasser1941]

Found on 0.61.0's gate run. uv's cache was emptied while the run was going, so its last 18 SWE-bench attempts couldn't start. lcode bench --compare still gave a normal verdict and exit 0, and its noise came out too small.

What was wrong

• Nothing said that attempts were missing. --compare scores only the attempts that ran, which is right, but it didn't mention that one side had 54 attempts and the other 72.
• A task with one attempt counted as certain. The noise comes from how much a task's attempts disagree, and one attempt can't disagree with itself. So a task with one attempt added nothing, however often it passes or fails on the other side.

What changes

• --compare warns when the two runs didn't have the same number of attempts over the tasks in both. The exit code still depends only on the verdict.
• A task with one attempt takes its pass rate from both runs. Tasks with two or more attempts count as before, so a normal comparison gives the same numbers.

Two single attempts that disagree used to be "a drop beyond the noise (±0)". Now they're within the noise.

Real test

0.60.0's run against the night's 0.61.0 run, the same two folders:

	Before	With this
Verdict	pass@1 47% → 50%, within the noise	The same
Noise	±10	±13
Missing attempts	Not mentioned	Not the same number of attempts ran: 72 before and 54 after, over the tasks in both. With fewer attempts, a result says less.

New tests cover both cases. The full suite, ruff, basedpyright and mkdocs build --strict pass. This doesn't change anything the model sees.

🤖 Generated with Claude Code