Summary
benchmarks/public/runner/run_subprocess.py passes --limit into
load_questions(), which returns the first N rows in file order. The shuffle
then only permutes that prefix, so --limit N runs the same N questions no
matter what --seed says.
Details
In run_eval (run_subprocess.py:267):
questions = load_questions(args.benchmark, limit=args.limit, offset=args.offset, ...)
if not args.no_shuffle:
random.Random(seed).shuffle(questions)
if args.limit:
questions = questions[:args.limit]registry.load_questions breaks out at len(questions) >= limit
(registry.py:230), so it hands back a head-of-file slice. That makes the
questions[:args.limit] line below the shuffle a no-op. The fact that the line
is there at all reads like the intended order was shuffle first, limit second.
I reproduced it against a 200-row dataset with --limit 10:
seed=42 -> ['q000', 'q001', ..., 'q009']
seed=1234 -> ['q000', 'q001', ..., 'q009']
identical question set across seeds: True
Impact
--limit N scores a fixed prefix of the dataset rather than a sample. On any
dataset grouped by category or difficulty, that biases the number.
--runs N assigns seed + run_index - 1 per run, which only makes sense if the
seed changes the sample. With --limit in play, every run scores the identical
question set, so the spread across runs captures agent nondeterminism and
nothing else. Any error bars computed that way come out too small.
docs/eval-frontier-search.md describes --no-shuffle as keeping "canonical
query order", which implies the default gives you a randomised subset. It
doesn't.
Suggested fix
Drop limit= from the load_questions() call in run_eval and let the slice
after the shuffle do the work. Leave offset where it is. Under --no-shuffle
nothing changes, since it's the same prefix either way.
A test that runs two different seeds at the same --limit and asserts the
selected sets differ would catch a regression. tests/ has no sampling coverage
today.
Two smaller things in the same file
--seed help text says "default: 42", but the argparse default is None, and
config.json gets written from vars(args) before base_seed resolves. A run
that used 42 records "seed": null.
With --runs N, JUDGE_SESSION sits behind if "JUDGE_SESSION" not in os.environ, so whichever concurrent run gets there first decides the session id
for all of them.
I can send a PR for the sampling fix if you want it.
Summary
benchmarks/public/runner/run_subprocess.pypasses--limitintoload_questions(), which returns the first N rows in file order. The shufflethen only permutes that prefix, so
--limit Nruns the same N questions nomatter what
--seedsays.Details
In
run_eval(run_subprocess.py:267):registry.load_questionsbreaks out atlen(questions) >= limit(registry.py:230), so it hands back a head-of-file slice. That makes the
questions[:args.limit]line below the shuffle a no-op. The fact that the lineis there at all reads like the intended order was shuffle first, limit second.
I reproduced it against a 200-row dataset with
--limit 10:Impact
--limit Nscores a fixed prefix of the dataset rather than a sample. On anydataset grouped by category or difficulty, that biases the number.
--runs Nassignsseed + run_index - 1per run, which only makes sense if theseed changes the sample. With
--limitin play, every run scores the identicalquestion set, so the spread across runs captures agent nondeterminism and
nothing else. Any error bars computed that way come out too small.
docs/eval-frontier-search.mddescribes--no-shuffleas keeping "canonicalquery order", which implies the default gives you a randomised subset. It
doesn't.
Suggested fix
Drop
limit=from theload_questions()call inrun_evaland let the sliceafter the shuffle do the work. Leave
offsetwhere it is. Under--no-shufflenothing changes, since it's the same prefix either way.
A test that runs two different seeds at the same
--limitand asserts theselected sets differ would catch a regression.
tests/has no sampling coveragetoday.
Two smaller things in the same file
--seedhelp text says "default: 42", but the argparse default isNone, andconfig.jsongets written fromvars(args)beforebase_seedresolves. A runthat used 42 records
"seed": null.With
--runs N,JUDGE_SESSIONsits behindif "JUDGE_SESSION" not in os.environ, so whichever concurrent run gets there first decides the session idfor all of them.
I can send a PR for the sampling fix if you want it.