diff --git a/benchmarks/frontierchallenge/README.md b/benchmarks/frontierchallenge/README.md index a4a654c..73f59b1 100644 --- a/benchmarks/frontierchallenge/README.md +++ b/benchmarks/frontierchallenge/README.md @@ -29,7 +29,7 @@ simulation, electrochemistry, quantitative imaging, and molecular biology. Tasks97 (74 hard, 23 medium) Taxonomy6 domains, 21 subdomains - Runtime81 open-image tasks, 16 user-supplied ORCA tasks + Runtime81 open-image tasks, 16 tasks executing user-supplied ORCA Gradingdeterministic checks; 77 tasks also judge the report HarnessHarbor 0.20.0 Outputnamed files under /app/output @@ -59,10 +59,10 @@ its SHA-256, and load it into Docker: HF_TOKEN=hf_... ./scripts/setup.sh --track open ``` -The full track adds 16 normally released ORCA tasks. FrontierChallenge does -not distribute ORCA or an image containing it. After obtaining ORCA 6.0.1 from -its official provider, build and smoke-test the private local runtime, then -validate the full track: +The full track adds 16 normally released tasks that execute ORCA. +FrontierChallenge does not distribute ORCA or an image containing it. After +obtaining ORCA 6.0.1 from its official provider, build and smoke-test the +private local runtime, then validate the full track: ```bash ./scripts/build_orca_runtime.sh \ @@ -74,6 +74,10 @@ HF_TOKEN=hf_... ./scripts/setup.sh --track full Do not push, export, publish, or share the resulting ORCA image. See the [ORCA setup tutorial](docs/providers/orca.md). +Track membership describes what a task executes, not where its input files +came from. For example, `task_098_orca_claisen_thermochemistry` reads supplied +ORCA output but does not run ORCA, so it belongs to the open track. + ### 2. Run a real task Fill `.env`, then run Harbor with the Claude Code agent: diff --git a/benchmarks/frontierchallenge/TASKS.md b/benchmarks/frontierchallenge/TASKS.md index 6fa43fd..527bcb6 100644 --- a/benchmarks/frontierchallenge/TASKS.md +++ b/benchmarks/frontierchallenge/TASKS.md @@ -14,7 +14,10 @@ HF_TOKEN=hf_... ./scripts/setup.sh --track open ``` The table below previews benchmark coverage without exposing evaluator data. -Keywords come from each task and describe technique rather than answers. +Keywords come from each task and describe technique rather than answers. The +`Image` column records what the task executes, not software named in supplied +files: `task_098_orca_claisen_thermochemistry`, for example, reads precomputed +ORCA output and therefore uses the open image. | Task | Difficulty | Image | Judge | Agent budget | Techniques | |---|---|---|---|---|---| diff --git a/benchmarks/frontierchallenge/docs/huggingface-release.md b/benchmarks/frontierchallenge/docs/huggingface-release.md index 5b83e3a..9304e09 100644 --- a/benchmarks/frontierchallenge/docs/huggingface-release.md +++ b/benchmarks/frontierchallenge/docs/huggingface-release.md @@ -19,9 +19,12 @@ workspace. HF_TOKEN=hf_... ./scripts/setup.sh --track open ``` -Setup downloads the current `main` branches, runs the verification tool bundled -with each dataset, and requires both `source_registry.json` files to equal this -checkout's `registry.json`. A mixed or incomplete dataset is refused. +Setup downloads the exact solve and reference commits declared in +`release/datasets.json`, runs the verification tool bundled with each dataset, +and requires both `source_registry.json` files to equal this checkout's +`registry.json`. A mixed or incomplete dataset is refused. Release maintainers +may test newer snapshots with `--revision` and `--reference-revision`; published +runtime changes should update both pins together. Use local directories instead of HF repository IDs for an offline handoff: @@ -44,3 +47,9 @@ the encrypted verifier hash. GitHub contains neither payload. The solve dataset must contain no `tests/`, verifier archive, rubric, fixture, or reference output; the reference dataset must contain no instruction, input, or runtime environment. + +The top-level Hugging Face `README.md` is intentionally outside +`checksums.sha256`: it is a mutable dataset card whose citation and links may be +edited without changing the benchmark payload. Task files, task-level READMEs, +registries, manifests, image artifacts, and verifier archives remain covered by +the checksum manifests and registry commitments. diff --git a/benchmarks/frontierchallenge/docs/providers/docker.md b/benchmarks/frontierchallenge/docs/providers/docker.md index 34370d8..c791448 100644 --- a/benchmarks/frontierchallenge/docs/providers/docker.md +++ b/benchmarks/frontierchallenge/docs/providers/docker.md @@ -92,7 +92,7 @@ path. ## Two things that bite **ORCA is user-supplied and writes beside its input.** Before selecting one of -the 16 ORCA tasks, create the licensed local runtime described in +the 16 tasks that execute ORCA, create the licensed local runtime described in [orca.md](orca.md). Copy ORCA inputs into a writable directory (`/app/data`, `/tmp`) before running; invoking ORCA directly on a read-only bind-mounted file fails. diff --git a/benchmarks/frontierchallenge/docs/quickstart.md b/benchmarks/frontierchallenge/docs/quickstart.md index 8a11fe9..85f3b63 100644 --- a/benchmarks/frontierchallenge/docs/quickstart.md +++ b/benchmarks/frontierchallenge/docs/quickstart.md @@ -61,19 +61,25 @@ This is the shortest path for most evaluators: HF_TOKEN=hf_... ./scripts/setup.sh --track open ``` -Setup downloads the solve and reference datasets from their current `main` branches, -verifies both packages, binds them to this checkout's `registry.json`, then +Setup downloads the solve and reference revisions pinned by this Git checkout in +`release/datasets.json`, verifies both packages, binds them to this checkout's +`registry.json`, then downloads `images/frontierchallenge-cpu-open-2026.08.docker.tar.zst` from the solve dataset. It checks the declared size, SHA-256 and image ID before loading the `linux/amd64` image into Docker. No container registry is used. Evaluator- local paths are written to `.frontierchallenge/config.env`. +For release development only, `--revision main` overrides both pins; +`--reference-revision` can override the reference revision independently. Normal +evaluation should keep the checkout's pins so later dataset changes cannot alter +an otherwise identical run. + ### Full track: build the private ORCA runtime -All 16 ORCA task statements and inputs are released normally. Only ORCA and a -configured ORCA image are absent. Obtain ORCA 6.0.1 from its official provider, -install it outside this checkout, and keep the complete directory together. -Then run: +All statements and inputs for the 16 tasks that execute ORCA are released +normally. Only ORCA and a configured ORCA image are absent. Obtain ORCA 6.0.1 +from its official provider, install it outside this checkout, and keep the +complete directory together. Then run: ```bash ./scripts/build_orca_runtime.sh \ @@ -118,6 +124,11 @@ tasks into evaluator staging, decrypts the matching verifier there, starts the agent, and invokes Harbor's verifier after the agent exits. By default Claude Code's `WebSearch` and `WebFetch` tools are disabled. +Selection comes from each task's declared `task.json.environment`, validated +against the registry. Include/exclude filters are applied before image preflight, +staging, verifier decryption, resume, and Harbor invocation; stale directories +from an older run cannot add tasks to the effective run. + A healthy run reaches messages like: ```text diff --git a/benchmarks/frontierchallenge/docs/running.md b/benchmarks/frontierchallenge/docs/running.md index faff611..1de1e0a 100644 --- a/benchmarks/frontierchallenge/docs/running.md +++ b/benchmarks/frontierchallenge/docs/running.md @@ -12,7 +12,8 @@ The open track contains the 81 tasks that use the redistributable image: ./scripts/run_eval.sh --agent claude-code --model ``` -The full track adds 16 ORCA tasks. Prepare the licensed local runtime first: +The full track adds 16 tasks that execute ORCA. Prepare the licensed local +runtime first: ```bash ./scripts/build_orca_runtime.sh --orca-root /path/to/orca-6.0.1 @@ -22,6 +23,10 @@ The full track adds 16 ORCA tasks. Prepare the licensed local runtime first: Setup writes verified local paths under `.frontierchallenge/`. The runner validates the GitHub/solve/reference registries again before staging anything. +Track membership follows each task's declared execution environment, not +software names in its instruction or supplied files. Thus +`task_098_orca_claisen_thermochemistry`, which only reads precomputed ORCA +output, remains an open-track task. ## Runtime diff --git a/benchmarks/frontierchallenge/docs/troubleshooting.md b/benchmarks/frontierchallenge/docs/troubleshooting.md index 9bece3f..1782740 100644 --- a/benchmarks/frontierchallenge/docs/troubleshooting.md +++ b/benchmarks/frontierchallenge/docs/troubleshooting.md @@ -40,6 +40,11 @@ or distribute it. After obtaining and installing ORCA officially, run: The build helper runs a real ORCA calculation, and the runner checks the local image again before any selected ORCA task starts. +An open task may still mention ORCA because it reads supplied output files. +That does not require the licensed runtime: preflight follows the task's +registry-backed execution environment, not instruction text. For example, +`task_098_orca_claisen_thermochemistry` belongs to the open track. + ## Every task fails during agent setup Check that `.env` contains the key required by the selected agent and that the diff --git a/benchmarks/frontierchallenge/release/datasets.json b/benchmarks/frontierchallenge/release/datasets.json new file mode 100644 index 0000000..f7bdea4 --- /dev/null +++ b/benchmarks/frontierchallenge/release/datasets.json @@ -0,0 +1,11 @@ +{ + "schema_version": 1, + "solve": { + "repo": "apodex/FrontierChallenge", + "revision": "8a73bb02a5709aa007e1a146d0cdc5d36ef5ea4d" + }, + "reference": { + "repo": "apodex/FrontierChallenge-reference", + "revision": "59fbf007ad5c6816caf7d379eef32d9fb384edea" + } +} diff --git a/benchmarks/frontierchallenge/scripts/run_eval.sh b/benchmarks/frontierchallenge/scripts/run_eval.sh index b522b63..433fa47 100755 --- a/benchmarks/frontierchallenge/scripts/run_eval.sh +++ b/benchmarks/frontierchallenge/scripts/run_eval.sh @@ -233,33 +233,55 @@ fi mkdir -p "$STAGE_DIR" "$JOBS_DIR" -# Mirror Harbor's include/exclude matching so only selected tasks are staged. -task_selected() { - local task_id="$1" - if [[ ${#INCLUDE_PATTERNS[@]} -gt 0 ]]; then - local matched=0 - for pattern in "${INCLUDE_PATTERNS[@]}"; do - [[ "$task_id" == $pattern ]] && { matched=1; break; } - done - [[ "$matched" -eq 1 ]] || return 1 - fi +# Resolve selection once from the verified solve package. Persistent staging may +# contain tasks from older runs, so it must never define preflight or grading. +SELECTION_ARGS=( + --tasks-root "$SOLVE_TASKS" + --registry "$SOLVE_DIR/source_registry.json" + --track "$TRACK" +) +if [[ ${#INCLUDE_PATTERNS[@]} -gt 0 ]]; then + for pattern in "${INCLUDE_PATTERNS[@]}"; do + SELECTION_ARGS+=(--include "$pattern") + done +fi +if [[ ${#EXCLUDE_PATTERNS[@]} -gt 0 ]]; then for pattern in "${EXCLUDE_PATTERNS[@]}"; do - [[ "$task_id" == $pattern ]] && return 1 + SELECTION_ARGS+=(--exclude "$pattern") done - return 0 -} +fi + +if ! SELECTION_OUTPUT="$( + python3 "$ROOT/scripts/task_selection.py" "${SELECTION_ARGS[@]}" +)"; then + echo "FATAL: task selection is invalid." >&2 + exit 1 +fi +EFFECTIVE_TASK_IDS=() +EFFECTIVE_TASK_ENVS=() +EFFECTIVE_TASK_SOURCES=() +while IFS=$'\t' read -r task_id task_environment task_source; do + [[ -n "$task_id" ]] || continue + EFFECTIVE_TASK_IDS+=("$task_id") + EFFECTIVE_TASK_ENVS+=("$task_environment") + EFFECTIVE_TASK_SOURCES+=("$task_source") +done <<< "$SELECTION_OUTPUT" +unset SELECTION_OUTPUT +if [[ ${#EFFECTIVE_TASK_IDS[@]} -eq 0 ]]; then + echo "FATAL: task selection is empty." >&2 + exit 1 +fi echo "== Staging $TRACK-track tasks from $SOLVE_TASKS into $STAGE_DIR ==" staged=0 skipped=0 -for task_dir in "$SOLVE_TASKS"/*/; do - task_id="$(basename "$task_dir")" - [[ -f "$task_dir/task.toml" ]] || continue - if [[ "$TRACK" == "open" ]] && ! grep -q '"environment": "open"' "$task_dir/task.json"; then - continue - fi - task_selected "$task_id" || continue +EFFECTIVE_TASK_DIRS=() +for index in "${!EFFECTIVE_TASK_IDS[@]}"; do + task_id="${EFFECTIVE_TASK_IDS[$index]}" + task_environment="${EFFECTIVE_TASK_ENVS[$index]}" + task_dir="${EFFECTIVE_TASK_SOURCES[$index]}" dest="$STAGE_DIR/$task_id" + EFFECTIVE_TASK_DIRS+=("$dest") source_identity="$SOLVE_DIR|$(grep -m1 '"source_task_sha256"' "$task_dir/task.json" | tr -d ' ,\"')|$OPEN_IMAGE" if [[ "$FORCE_RESTAGE" -eq 0 && -f "$dest/task.toml" \ && -f "$dest/instruction.md" && ! -e "$dest/statement.fcref" \ @@ -271,9 +293,12 @@ for task_dir in "$SOLVE_TASKS"/*/; do # Copy the verified solve task and pin its open-image Dockerfile to setup's # selected reference. The HF source is immutable; only the staged copy changes. rm -rf "$dest" - cp -a "$task_dir" "$dest" + # Hugging Face snapshots may expose files as cache-relative symlinks. Copy + # their contents so the evaluator stage cannot contain broken links after it + # leaves the snapshot directory hierarchy. + cp -aL "$task_dir" "$dest" printf '%s\n' "$source_identity" > "$dest/.frontier-source" - if grep -q '"environment": "open"' "$dest/task.json"; then + if [[ "$task_environment" == "open" ]]; then OPEN_IMAGE="$OPEN_IMAGE" python3 - "$dest/environment/Dockerfile" <<'PIN_OPEN_IMAGE' import os import pathlib @@ -316,15 +341,15 @@ if [[ "$AGENT" == "claude-code" && ${#AGENT_KWARGS[@]} -eq 0 ]]; then # Prevent direct web lookup unless the evaluator deliberately overrides this. AGENT_KWARG_ARGS+=(--agent-kwarg "disallowed_tools=WebSearch WebFetch") fi -AGENT_KWARG_ARGS+=("${AGENT_KWARGS[@]}") +if [[ ${#AGENT_KWARGS[@]} -gt 0 ]]; then + AGENT_KWARG_ARGS+=("${AGENT_KWARGS[@]}") +fi INCLUDE_ARGS=() -for pattern in "${INCLUDE_PATTERNS[@]}"; do - INCLUDE_ARGS+=("--include-task-name" "$pattern") -done -EXCLUDE_ARGS=() -for pattern in "${EXCLUDE_PATTERNS[@]}"; do - EXCLUDE_ARGS+=("--exclude-task-name" "$pattern") +# Give Harbor the exact resolved IDs. This makes stale directories in a reused +# stage invisible even when the user supplied no include/exclude flags. +for task_id in "${EFFECTIVE_TASK_IDS[@]}"; do + INCLUDE_ARGS+=("--include-task-name" "$task_id") done # A command-line verifier env takes precedence over the task declaration. @@ -388,18 +413,9 @@ fi # Fail before evaluation if a selected task needs ORCA but the evaluator-local # licensed runtime is unavailable. orca_tasks=() -for task_dir in "$STAGE_DIR"/*/; do - [[ -d "$task_dir" ]] || continue - task_name="$(basename "$task_dir")" - if [[ ${#INCLUDE_PATTERNS[@]} -gt 0 ]]; then - matched=0 - for pattern in "${INCLUDE_PATTERNS[@]}"; do - [[ "$task_name" == *"$pattern"* ]] && { matched=1; break; } - done - [[ "$matched" -eq 1 ]] || continue - fi - if grep -qil 'orca' "$task_dir/task.toml" "$task_dir/instruction.md" "$task_dir/environment/Dockerfile" 2>/dev/null; then - orca_tasks+=("$task_name") +for index in "${!EFFECTIVE_TASK_IDS[@]}"; do + if [[ "${EFFECTIVE_TASK_ENVS[$index]}" == "licensed-orca" ]]; then + orca_tasks+=("${EFFECTIVE_TASK_IDS[$index]}") fi done if [[ ${#orca_tasks[@]} -gt 0 ]]; then @@ -434,15 +450,14 @@ fi # completed trial directories but archive stale job-level metadata. RESUME_JOB=0 if [[ -f "$JOBS_DIR/$JOB_NAME/config.json" ]]; then - if REQUESTED="${INCLUDE_PATTERNS[*]-}" python3 - "$JOBS_DIR/$JOB_NAME/lock.json" <<'PY' + if REQUESTED="${EFFECTIVE_TASK_IDS[*]}" python3 - "$JOBS_DIR/$JOB_NAME/lock.json" <<'PY' import json, os, sys requested = set(os.environ.get("REQUESTED", "").split()) try: recorded = {t["task"]["name"] for t in json.load(open(sys.argv[1]))["trials"]} except Exception: sys.exit(1) # unreadable lock -> treat as new work -# No --include means "the whole staged set", which resume also covers. -sys.exit(0 if not requested or requested == recorded else 1) +sys.exit(0 if requested == recorded else 1) PY then RESUME_JOB=1 @@ -469,10 +484,8 @@ if [[ -f "$REFERENCE_DIR/tools/verify_reference_dataset.py" ]]; then fi echo "== Injecting encrypted verifier archives from $REFERENCE_DIR ==" injected=0 -for task_dir in "$STAGE_DIR"/*/; do - [[ -f "$task_dir/task.toml" ]] || continue +for task_dir in "${EFFECTIVE_TASK_DIRS[@]}"; do task_id="$(basename "$task_dir")" - task_selected "$task_id" || continue source_verifier="$REFERENCE_TASKS/$task_id/verifier.fcref" if [[ ! -f "$source_verifier" ]]; then echo "FATAL: encrypted verifier missing for $task_id: $source_verifier" >&2 @@ -497,8 +510,7 @@ echo "Injected $injected encrypted verifier archive(s)." if [[ -f "$ROOT/scripts/reference_archive.py" ]]; then echo "== Unsealing encrypted verifiers with the published archive password ==" unsealed=0 - for task_dir in "$STAGE_DIR"/*/; do - [[ -f "$task_dir/task.toml" ]] || continue + for task_dir in "${EFFECTIVE_TASK_DIRS[@]}"; do if [[ ! -f "$task_dir/instruction.md" || ! -f "$task_dir/verifier.fcref" ]]; then echo "FATAL: $(basename "$task_dir") lacks plaintext instruction or verifier archive." >&2 exit 1 @@ -534,7 +546,7 @@ else --artifact /app/output \ --jobs-dir "$JOBS_DIR" --job-name "$JOB_NAME" \ --env-file "$ENV_FILE" \ - "${AGENT_KWARG_ARGS[@]}" "${INCLUDE_ARGS[@]}" "${EXCLUDE_ARGS[@]}" "${VERIFIER_ENV_ARGS[@]}" \ + "${AGENT_KWARG_ARGS[@]}" "${INCLUDE_ARGS[@]}" "${VERIFIER_ENV_ARGS[@]}" \ --yes fi diff --git a/benchmarks/frontierchallenge/scripts/setup_release.py b/benchmarks/frontierchallenge/scripts/setup_release.py index f8cc997..48ea539 100755 --- a/benchmarks/frontierchallenge/scripts/setup_release.py +++ b/benchmarks/frontierchallenge/scripts/setup_release.py @@ -13,11 +13,22 @@ import sys from pathlib import Path - ROOT = Path(__file__).resolve().parents[1] -DEFAULT_SOLVE_REPO = "apodex/FrontierChallenge" -DEFAULT_REFERENCE_REPO = "apodex/FrontierChallenge-reference" -DEFAULT_REVISION = "main" + + +def select_revisions( + release: dict, + revision_override: str | None, + reference_revision_override: str | None, +) -> tuple[str, str]: + """Return pinned defaults, preserving the legacy --revision override.""" + solve_revision = revision_override or release["solve"]["revision"] + reference_revision = ( + reference_revision_override + or revision_override + or release["reference"]["revision"] + ) + return solve_revision, reference_revision def digest(path: Path) -> str: @@ -284,14 +295,16 @@ def write_config( reference: Path, track: str, open_image: str, - revision: str, + solve_revision: str, + reference_revision: str, ) -> None: values = { "FRONTIER_SOLVE_DIR": str(solve), "FRONTIER_REFERENCE_DIR": str(reference), "FRONTIER_TRACK": track, "FRONTIER_OPEN_IMAGE": open_image, - "FRONTIER_DATASET_REVISION": revision, + "FRONTIER_SOLVE_REVISION": solve_revision, + "FRONTIER_REFERENCE_REVISION": reference_revision, } path.parent.mkdir(parents=True, exist_ok=True) path.write_text( @@ -303,11 +316,16 @@ def write_config( def main() -> int: image_manifest = json.loads((ROOT / "release" / "images.json").read_text()) + dataset_release = json.loads((ROOT / "release" / "datasets.json").read_text()) default_image = image_manifest["images"]["open"]["ref"] parser = argparse.ArgumentParser(description=__doc__) - parser.add_argument("--solve-source", default=DEFAULT_SOLVE_REPO) - parser.add_argument("--reference-source", default=DEFAULT_REFERENCE_REPO) - parser.add_argument("--revision", default=DEFAULT_REVISION) + parser.add_argument("--solve-source", default=dataset_release["solve"]["repo"]) + parser.add_argument("--reference-source", default=dataset_release["reference"]["repo"]) + parser.add_argument( + "--revision", + default=None, + help="override both pinned dataset revisions (for example, main)", + ) parser.add_argument("--reference-revision", default=None) parser.add_argument("--track", choices=("open", "full"), default="open") parser.add_argument("--cache-dir", type=Path, default=Path.home() / ".cache/frontierchallenge") @@ -316,18 +334,21 @@ def main() -> int: ) parser.add_argument("--skip-image", action="store_true", help="verify datasets only") args = parser.parse_args() + solve_revision, reference_revision = select_revisions( + dataset_release, args.revision, args.reference_revision + ) token = os.environ.get("HF_TOKEN") solve = resolve_dataset( args.solve_source, - args.revision, + solve_revision, args.cache_dir, token=token, ignore_patterns=["images/*.tar.zst"], ) reference = resolve_dataset( args.reference_source, - args.reference_revision or args.revision, + reference_revision, args.cache_dir, token=token, ) @@ -340,7 +361,7 @@ def main() -> int: load_hf_image_archive( solve=solve, solve_source=args.solve_source, - revision=args.revision, + revision=solve_revision, cache_dir=args.cache_dir, archive_config=image_manifest["images"]["open"]["hf_archive"], token=token, @@ -357,7 +378,8 @@ def main() -> int: reference=reference, track=args.track, open_image=default_image, - revision=args.revision, + solve_revision=solve_revision, + reference_revision=reference_revision, ) print(f"ready: {len(selected)} tasks; configuration written to {args.config.resolve()}") return 0 diff --git a/benchmarks/frontierchallenge/scripts/task_selection.py b/benchmarks/frontierchallenge/scripts/task_selection.py new file mode 100755 index 0000000..aafa65c --- /dev/null +++ b/benchmarks/frontierchallenge/scripts/task_selection.py @@ -0,0 +1,108 @@ +#!/usr/bin/env python3 +"""Resolve the exact FrontierChallenge task set for an evaluation run.""" + +from __future__ import annotations + +import argparse +import fnmatch +import json +from dataclasses import dataclass +from pathlib import Path + +VALID_ENVIRONMENTS = {"open", "licensed-orca"} + + +@dataclass(frozen=True) +class SelectedTask: + task_id: str + environment: str + path: Path + + +def select_tasks( + tasks_root: Path, + *, + track: str, + include: tuple[str, ...] = (), + exclude: tuple[str, ...] = (), + registry_path: Path | None = None, +) -> list[SelectedTask]: + """Select from verified solve tasks, never from persistent staging.""" + if track not in {"open", "full"}: + raise ValueError(f"unsupported track: {track}") + + registry_environments: dict[str, str] | None = None + if registry_path is not None: + registry = json.loads(registry_path.read_text(encoding="utf-8")) + registry_environments = {} + for record in registry.get("tasks", []): + task_id = record.get("id") + environment = record.get("image") + if not isinstance(task_id, str) or environment not in VALID_ENVIRONMENTS: + raise ValueError(f"invalid registry task record: {record!r}") + if task_id in registry_environments: + raise ValueError(f"duplicate registry task ID: {task_id}") + registry_environments[task_id] = environment + + selected: list[SelectedTask] = [] + for task_path in sorted(tasks_root.iterdir()): + if not task_path.is_dir() or not (task_path / "task.toml").is_file(): + continue + metadata_path = task_path / "task.json" + if not metadata_path.is_file(): + raise ValueError(f"task metadata missing: {metadata_path}") + metadata = json.loads(metadata_path.read_text(encoding="utf-8")) + task_id = metadata.get("task_id") + environment = metadata.get("environment") + if task_id != task_path.name: + raise ValueError(f"task ID/path mismatch: {task_path.name} != {task_id!r}") + if environment not in VALID_ENVIRONMENTS: + raise ValueError(f"invalid environment for {task_id}: {environment!r}") + if registry_environments is not None: + registered = registry_environments.get(task_id) + if registered != environment: + raise ValueError( + f"task/registry environment mismatch for {task_id}: " + f"{environment!r} != {registered!r}" + ) + if track == "open" and environment != "open": + continue + if include and not any(fnmatch.fnmatchcase(task_id, pattern) for pattern in include): + continue + if any(fnmatch.fnmatchcase(task_id, pattern) for pattern in exclude): + continue + selected.append(SelectedTask(task_id, environment, task_path.resolve())) + return selected + + +def main() -> int: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--tasks-root", type=Path, required=True) + parser.add_argument("--registry", type=Path, required=True) + parser.add_argument("--track", choices=("open", "full"), required=True) + parser.add_argument("--include", action="append", default=[]) + parser.add_argument("--exclude", action="append", default=[]) + args = parser.parse_args() + + try: + selected = select_tasks( + args.tasks_root, + track=args.track, + include=tuple(args.include), + exclude=tuple(args.exclude), + registry_path=args.registry, + ) + except (OSError, ValueError, json.JSONDecodeError) as exc: + parser.error(str(exc)) + if not selected: + parser.error("task selection is empty") + for task in selected: + path = str(task.path) + if any(character in path for character in ("\t", "\n", "\r")): + parser.error(f"task path contains a control character: {task.path}") + print(f"{task.task_id}\t{task.environment}\t{path}") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/benchmarks/frontierchallenge/tests/test_run_eval_contract.py b/benchmarks/frontierchallenge/tests/test_run_eval_contract.py new file mode 100644 index 0000000..96fc7db --- /dev/null +++ b/benchmarks/frontierchallenge/tests/test_run_eval_contract.py @@ -0,0 +1,29 @@ +from pathlib import Path + +SCRIPT = Path(__file__).parents[1] / "scripts" / "run_eval.sh" + + +def test_orca_preflight_uses_declared_environment_not_instruction_text(): + text = SCRIPT.read_text(encoding="utf-8") + + assert 'EFFECTIVE_TASK_ENVS[$index]' in text + assert "grep -qil 'orca'" not in text + + +def test_harbor_receives_exact_effective_task_ids(): + text = SCRIPT.read_text(encoding="utf-8") + + assert 'for task_id in "${EFFECTIVE_TASK_IDS[@]}"' in text + assert 'INCLUDE_ARGS+=("--include-task-name" "$task_id")' in text + + +def test_staging_dereferences_hugging_face_cache_symlinks(): + text = SCRIPT.read_text(encoding="utf-8") + + assert 'cp -aL "$task_dir" "$dest"' in text + + +def test_verifier_work_is_limited_to_effective_task_directories(): + text = SCRIPT.read_text(encoding="utf-8") + + assert text.count('for task_dir in "${EFFECTIVE_TASK_DIRS[@]}"') == 2 diff --git a/benchmarks/frontierchallenge/tests/test_setup_release.py b/benchmarks/frontierchallenge/tests/test_setup_release.py index 38bd35f..b1b2391 100644 --- a/benchmarks/frontierchallenge/tests/test_setup_release.py +++ b/benchmarks/frontierchallenge/tests/test_setup_release.py @@ -1,12 +1,11 @@ from __future__ import annotations import json -from pathlib import Path import sys +from pathlib import Path from types import SimpleNamespace import pytest - import setup_release @@ -91,16 +90,44 @@ def test_write_config_quotes_evaluator_paths(tmp_path): reference=tmp_path / "reference package", track="open", open_image="example/image@sha256:123", - revision="main", + solve_revision="solve-sha", + reference_revision="reference-sha", ) text = config.read_text() assert "FRONTIER_SOLVE_DIR=" in text assert "'" in text assert "example/image@sha256:123" in text + assert "FRONTIER_SOLVE_REVISION=solve-sha" in text + assert "FRONTIER_REFERENCE_REVISION=reference-sha" in text assert "FRONTIER_IMAGE_SOURCE" not in text +def test_select_revisions_uses_independent_pins_by_default(): + release = { + "solve": {"revision": "solve-sha"}, + "reference": {"revision": "reference-sha"}, + } + + assert setup_release.select_revisions(release, None, None) == ( + "solve-sha", + "reference-sha", + ) + + +def test_revision_override_preserves_legacy_both_dataset_behavior(): + release = { + "solve": {"revision": "solve-sha"}, + "reference": {"revision": "reference-sha"}, + } + + assert setup_release.select_revisions(release, "main", None) == ("main", "main") + assert setup_release.select_revisions(release, "solve-next", "ref-next") == ( + "solve-next", + "ref-next", + ) + + def test_validate_orca_runtime_accepts_wrapper_contract(monkeypatch): calls = [] diff --git a/benchmarks/frontierchallenge/tests/test_task_selection.py b/benchmarks/frontierchallenge/tests/test_task_selection.py new file mode 100644 index 0000000..6f089b0 --- /dev/null +++ b/benchmarks/frontierchallenge/tests/test_task_selection.py @@ -0,0 +1,103 @@ +from __future__ import annotations + +import json +from pathlib import Path + +import pytest +import task_selection + + +def make_task(root: Path, task_id: str, environment: str, instruction: str = "") -> Path: + task = root / task_id + (task / "environment").mkdir(parents=True) + (task / "task.toml").write_text('schema_version = "1.1"\n') + (task / "task.json").write_text( + json.dumps({"task_id": task_id, "environment": environment}) + ) + (task / "instruction.md").write_text(instruction) + (task / "environment" / "Dockerfile").write_text("FROM example/open\n") + return task + + +def test_open_task_can_mention_orca_without_requiring_licensed_runtime(tmp_path): + make_task( + tmp_path, + "task_098_orca_claisen_thermochemistry", + "open", + "Read the supplied ORCA output files; do not execute ORCA.", + ) + + selected = task_selection.select_tasks(tmp_path, track="open") + + assert [(task.task_id, task.environment) for task in selected] == [ + ("task_098_orca_claisen_thermochemistry", "open") + ] + + +def test_open_track_excludes_declared_licensed_tasks(tmp_path): + make_task(tmp_path, "task_011_open", "open") + make_task(tmp_path, "task_199_orca", "licensed-orca") + + assert [task.task_id for task in task_selection.select_tasks(tmp_path, track="open")] == [ + "task_011_open" + ] + assert [task.task_id for task in task_selection.select_tasks(tmp_path, track="full")] == [ + "task_011_open", + "task_199_orca", + ] + + +def test_include_and_exclude_use_glob_semantics(tmp_path): + for task_id in ("task_011_alpha", "task_012_beta", "task_199_orca"): + make_task(tmp_path, task_id, "open") + + selected = task_selection.select_tasks( + tmp_path, + track="open", + include=("task_0*",), + exclude=("*_beta",), + ) + + assert [task.task_id for task in selected] == ["task_011_alpha"] + + +def test_selection_reads_solve_source_not_stale_stage(tmp_path): + solve = tmp_path / "solve" + stale_stage = tmp_path / "stage" + make_task(solve, "task_011_open", "open") + make_task(stale_stage, "task_199_orca", "licensed-orca") + + selected = task_selection.select_tasks(solve, track="open") + + assert [task.task_id for task in selected] == ["task_011_open"] + + +def test_invalid_environment_is_rejected(tmp_path): + make_task(tmp_path, "task_011_bad", "orca-by-text-search") + + with pytest.raises(ValueError, match="invalid environment"): + task_selection.select_tasks(tmp_path, track="full") + + +def test_task_environment_must_match_registry_commitment(tmp_path): + make_task(tmp_path, "task_098_orca_claisen_thermochemistry", "open") + registry = tmp_path / "source_registry.json" + registry.write_text( + json.dumps( + { + "tasks": [ + { + "id": "task_098_orca_claisen_thermochemistry", + "image": "licensed-orca", + } + ] + } + ) + ) + + with pytest.raises(ValueError, match="task/registry environment mismatch"): + task_selection.select_tasks( + tmp_path, + track="full", + registry_path=registry, + )