fix(frontierchallenge): make open-track selection reproducible

benchmarks/frontierchallenge/README.md

@@ -29,7 +29,7 @@ simulation, electrochemistry, quantitative imaging, and molecular biology.
   <tbody>
     <tr><td>Tasks</td><td>97 (74 hard, 23 medium)</td></tr>
     <tr><td>Taxonomy</td><td>6 domains, 21 subdomains</td></tr>
-    <tr><td>Runtime</td><td>81 open-image tasks, 16 user-supplied ORCA tasks</td></tr>
+    <tr><td>Runtime</td><td>81 open-image tasks, 16 tasks executing user-supplied ORCA</td></tr>
     <tr><td>Grading</td><td>deterministic checks; 77 tasks also judge the report</td></tr>
     <tr><td>Harness</td><td>Harbor 0.20.0</td></tr>
     <tr><td>Output</td><td>named files under <code>/app/output</code></td></tr>
@@ -59,10 +59,10 @@ its SHA-256, and load it into Docker:
 HF_TOKEN=hf_... ./scripts/setup.sh --track open
 ```
 
-The full track adds 16 normally released ORCA tasks. FrontierChallenge does
-not distribute ORCA or an image containing it. After obtaining ORCA 6.0.1 from
-its official provider, build and smoke-test the private local runtime, then
-validate the full track:
+The full track adds 16 normally released tasks that execute ORCA.
+FrontierChallenge does not distribute ORCA or an image containing it. After
+obtaining ORCA 6.0.1 from its official provider, build and smoke-test the
+private local runtime, then validate the full track:
 
 ```bash
 ./scripts/build_orca_runtime.sh \
@@ -74,6 +74,10 @@ HF_TOKEN=hf_... ./scripts/setup.sh --track full
 Do not push, export, publish, or share the resulting ORCA image. See the
 [ORCA setup tutorial](docs/providers/orca.md).
 
+Track membership describes what a task executes, not where its input files
+came from. For example, `task_098_orca_claisen_thermochemistry` reads supplied
+ORCA output but does not run ORCA, so it belongs to the open track.
+
 ### 2. Run a real task
 
 Fill `.env`, then run Harbor with the Claude Code agent:

benchmarks/frontierchallenge/TASKS.md

@@ -14,7 +14,10 @@ HF_TOKEN=hf_... ./scripts/setup.sh --track open
 ```
 
 The table below previews benchmark coverage without exposing evaluator data.
-Keywords come from each task and describe technique rather than answers.
+Keywords come from each task and describe technique rather than answers. The
+`Image` column records what the task executes, not software named in supplied
+files: `task_098_orca_claisen_thermochemistry`, for example, reads precomputed
+ORCA output and therefore uses the open image.
 
 | Task | Difficulty | Image | Judge | Agent budget | Techniques |
 |---|---|---|---|---|---|

benchmarks/frontierchallenge/docs/huggingface-release.md

@@ -19,9 +19,12 @@ workspace.
 HF_TOKEN=hf_... ./scripts/setup.sh --track open
 ```
 
-Setup downloads the current `main` branches, runs the verification tool bundled
-with each dataset, and requires both `source_registry.json` files to equal this
-checkout's `registry.json`. A mixed or incomplete dataset is refused.
+Setup downloads the exact solve and reference commits declared in
+`release/datasets.json`, runs the verification tool bundled with each dataset,
+and requires both `source_registry.json` files to equal this checkout's
+`registry.json`. A mixed or incomplete dataset is refused. Release maintainers
+may test newer snapshots with `--revision` and `--reference-revision`; published
+runtime changes should update both pins together.
 
 Use local directories instead of HF repository IDs for an offline handoff:
 
@@ -44,3 +47,9 @@ the encrypted verifier hash. GitHub contains neither payload. The solve dataset
 must contain no `tests/`, verifier archive, rubric, fixture, or reference
 output; the reference dataset must contain no instruction, input, or runtime
 environment.
+
+The top-level Hugging Face `README.md` is intentionally outside
+`checksums.sha256`: it is a mutable dataset card whose citation and links may be
+edited without changing the benchmark payload. Task files, task-level READMEs,
+registries, manifests, image artifacts, and verifier archives remain covered by
+the checksum manifests and registry commitments.

benchmarks/frontierchallenge/docs/providers/docker.md

@@ -92,7 +92,7 @@ path.
 ## Two things that bite
 
 **ORCA is user-supplied and writes beside its input.** Before selecting one of
-the 16 ORCA tasks, create the licensed local runtime described in
+the 16 tasks that execute ORCA, create the licensed local runtime described in
 [orca.md](orca.md). Copy ORCA inputs into a writable directory (`/app/data`,
 `/tmp`) before running; invoking ORCA directly on a read-only bind-mounted file
 fails.

benchmarks/frontierchallenge/docs/quickstart.md

@@ -61,19 +61,25 @@ This is the shortest path for most evaluators:
 HF_TOKEN=hf_... ./scripts/setup.sh --track open
 ```
 
-Setup downloads the solve and reference datasets from their current `main` branches,
-verifies both packages, binds them to this checkout's `registry.json`, then
+Setup downloads the solve and reference revisions pinned by this Git checkout in
+`release/datasets.json`, verifies both packages, binds them to this checkout's
+`registry.json`, then
 downloads `images/frontierchallenge-cpu-open-2026.08.docker.tar.zst` from the
 solve dataset. It checks the declared size, SHA-256 and image ID before loading
 the `linux/amd64` image into Docker. No container registry is used. Evaluator-
 local paths are written to `.frontierchallenge/config.env`.
 
+For release development only, `--revision main` overrides both pins;
+`--reference-revision` can override the reference revision independently. Normal
+evaluation should keep the checkout's pins so later dataset changes cannot alter
+an otherwise identical run.
+
 ### Full track: build the private ORCA runtime
 
-All 16 ORCA task statements and inputs are released normally. Only ORCA and a
-configured ORCA image are absent. Obtain ORCA 6.0.1 from its official provider,
-install it outside this checkout, and keep the complete directory together.
-Then run:
+All statements and inputs for the 16 tasks that execute ORCA are released
+normally. Only ORCA and a configured ORCA image are absent. Obtain ORCA 6.0.1
+from its official provider, install it outside this checkout, and keep the
+complete directory together. Then run:
 
 ```bash
 ./scripts/build_orca_runtime.sh \
@@ -118,6 +124,11 @@ tasks into evaluator staging, decrypts the matching verifier there, starts the
 agent, and invokes Harbor's verifier after the agent exits. By default Claude
 Code's `WebSearch` and `WebFetch` tools are disabled.
 
+Selection comes from each task's declared `task.json.environment`, validated
+against the registry. Include/exclude filters are applied before image preflight,
+staging, verifier decryption, resume, and Harbor invocation; stale directories
+from an older run cannot add tasks to the effective run.
+
 A healthy run reaches messages like:
 
 ```text

benchmarks/frontierchallenge/docs/running.md

@@ -12,7 +12,8 @@ The open track contains the 81 tasks that use the redistributable image:
 ./scripts/run_eval.sh --agent claude-code --model <model>
 ```
 
-The full track adds 16 ORCA tasks. Prepare the licensed local runtime first:
+The full track adds 16 tasks that execute ORCA. Prepare the licensed local
+runtime first:
 
 ```bash
 ./scripts/build_orca_runtime.sh --orca-root /path/to/orca-6.0.1
@@ -22,6 +23,10 @@ The full track adds 16 ORCA tasks. Prepare the licensed local runtime first:
 
 Setup writes verified local paths under `.frontierchallenge/`. The runner
 validates the GitHub/solve/reference registries again before staging anything.
+Track membership follows each task's declared execution environment, not
+software names in its instruction or supplied files. Thus
+`task_098_orca_claisen_thermochemistry`, which only reads precomputed ORCA
+output, remains an open-track task.
 
 ## Runtime
 

benchmarks/frontierchallenge/docs/troubleshooting.md

@@ -40,6 +40,11 @@ or distribute it. After obtaining and installing ORCA officially, run:
 The build helper runs a real ORCA calculation, and the runner checks the local
 image again before any selected ORCA task starts.
 
+An open task may still mention ORCA because it reads supplied output files.
+That does not require the licensed runtime: preflight follows the task's
+registry-backed execution environment, not instruction text. For example,
+`task_098_orca_claisen_thermochemistry` belongs to the open track.
+
 ## Every task fails during agent setup
 
 Check that `.env` contains the key required by the selected agent and that the

benchmarks/frontierchallenge/release/datasets.json

@@ -0,0 +1,11 @@
+{
+  "schema_version": 1,
+  "solve": {
+    "repo": "apodex/FrontierChallenge",
+    "revision": "8a73bb02a5709aa007e1a146d0cdc5d36ef5ea4d"
+  },
+  "reference": {
+    "repo": "apodex/FrontierChallenge-reference",
+    "revision": "59fbf007ad5c6816caf7d379eef32d9fb384edea"
+  }
+}

benchmarks/frontierchallenge/scripts/run_eval.sh

@@ -233,33 +233,55 @@ fi
 
 mkdir -p "$STAGE_DIR" "$JOBS_DIR"
 
-# Mirror Harbor's include/exclude matching so only selected tasks are staged.
-task_selected() {
-  local task_id="$1"
-  if [[ ${#INCLUDE_PATTERNS[@]} -gt 0 ]]; then
-    local matched=0
-    for pattern in "${INCLUDE_PATTERNS[@]}"; do
-      [[ "$task_id" == $pattern ]] && { matched=1; break; }
-    done
-    [[ "$matched" -eq 1 ]] || return 1
-  fi
+# Resolve selection once from the verified solve package. Persistent staging may
+# contain tasks from older runs, so it must never define preflight or grading.
+SELECTION_ARGS=(
+  --tasks-root "$SOLVE_TASKS"
+  --registry "$SOLVE_DIR/source_registry.json"
+  --track "$TRACK"
+)
+if [[ ${#INCLUDE_PATTERNS[@]} -gt 0 ]]; then
+  for pattern in "${INCLUDE_PATTERNS[@]}"; do
+    SELECTION_ARGS+=(--include "$pattern")
+  done
+fi
+if [[ ${#EXCLUDE_PATTERNS[@]} -gt 0 ]]; then
   for pattern in "${EXCLUDE_PATTERNS[@]}"; do
-    [[ "$task_id" == $pattern ]] && return 1
+    SELECTION_ARGS+=(--exclude "$pattern")
   done
-  return 0
-}
+fi
+
+if ! SELECTION_OUTPUT="$(
+  python3 "$ROOT/scripts/task_selection.py" "${SELECTION_ARGS[@]}"
+)"; then
+  echo "FATAL: task selection is invalid." >&2
+  exit 1
+fi
+EFFECTIVE_TASK_IDS=()
+EFFECTIVE_TASK_ENVS=()
+EFFECTIVE_TASK_SOURCES=()
+while IFS=$'\t' read -r task_id task_environment task_source; do
+  [[ -n "$task_id" ]] || continue
+  EFFECTIVE_TASK_IDS+=("$task_id")
+  EFFECTIVE_TASK_ENVS+=("$task_environment")
+  EFFECTIVE_TASK_SOURCES+=("$task_source")
+done <<< "$SELECTION_OUTPUT"
+unset SELECTION_OUTPUT
+if [[ ${#EFFECTIVE_TASK_IDS[@]} -eq 0 ]]; then
+  echo "FATAL: task selection is empty." >&2
+  exit 1
+fi
 
 echo "== Staging $TRACK-track tasks from $SOLVE_TASKS into $STAGE_DIR =="
 staged=0
 skipped=0
-for task_dir in "$SOLVE_TASKS"/*/; do
-  task_id="$(basename "$task_dir")"
-  [[ -f "$task_dir/task.toml" ]] || continue
-  if [[ "$TRACK" == "open" ]] && ! grep -q '"environment": "open"' "$task_dir/task.json"; then
-    continue
-  fi
-  task_selected "$task_id" || continue
+EFFECTIVE_TASK_DIRS=()
+for index in "${!EFFECTIVE_TASK_IDS[@]}"; do
+  task_id="${EFFECTIVE_TASK_IDS[$index]}"
+  task_environment="${EFFECTIVE_TASK_ENVS[$index]}"
+  task_dir="${EFFECTIVE_TASK_SOURCES[$index]}"
   dest="$STAGE_DIR/$task_id"
+  EFFECTIVE_TASK_DIRS+=("$dest")
   source_identity="$SOLVE_DIR|$(grep -m1 '"source_task_sha256"' "$task_dir/task.json" | tr -d ' ,\"')|$OPEN_IMAGE"
   if [[ "$FORCE_RESTAGE" -eq 0 && -f "$dest/task.toml" \
         && -f "$dest/instruction.md" && ! -e "$dest/statement.fcref" \
@@ -271,9 +293,12 @@ for task_dir in "$SOLVE_TASKS"/*/; do
 # Copy the verified solve task and pin its open-image Dockerfile to setup's
 # selected reference. The HF source is immutable; only the staged copy changes.
   rm -rf "$dest"
-  cp -a "$task_dir" "$dest"
+  # Hugging Face snapshots may expose files as cache-relative symlinks. Copy
+  # their contents so the evaluator stage cannot contain broken links after it
+  # leaves the snapshot directory hierarchy.
+  cp -aL "$task_dir" "$dest"
   printf '%s\n' "$source_identity" > "$dest/.frontier-source"
-  if grep -q '"environment": "open"' "$dest/task.json"; then
+  if [[ "$task_environment" == "open" ]]; then
     OPEN_IMAGE="$OPEN_IMAGE" python3 - "$dest/environment/Dockerfile" <<'PIN_OPEN_IMAGE'
 import os
 import pathlib
@@ -316,15 +341,15 @@ if [[ "$AGENT" == "claude-code" && ${#AGENT_KWARGS[@]} -eq 0 ]]; then
   # Prevent direct web lookup unless the evaluator deliberately overrides this.
   AGENT_KWARG_ARGS+=(--agent-kwarg "disallowed_tools=WebSearch WebFetch")
 fi
-AGENT_KWARG_ARGS+=("${AGENT_KWARGS[@]}")
+if [[ ${#AGENT_KWARGS[@]} -gt 0 ]]; then
+  AGENT_KWARG_ARGS+=("${AGENT_KWARGS[@]}")
+fi
 
 INCLUDE_ARGS=()
-for pattern in "${INCLUDE_PATTERNS[@]}"; do
-  INCLUDE_ARGS+=("--include-task-name" "$pattern")
-done
-EXCLUDE_ARGS=()
-for pattern in "${EXCLUDE_PATTERNS[@]}"; do
-  EXCLUDE_ARGS+=("--exclude-task-name" "$pattern")
+# Give Harbor the exact resolved IDs. This makes stale directories in a reused
+# stage invisible even when the user supplied no include/exclude flags.
+for task_id in "${EFFECTIVE_TASK_IDS[@]}"; do
+  INCLUDE_ARGS+=("--include-task-name" "$task_id")
 done
 
 # A command-line verifier env takes precedence over the task declaration.
@@ -388,18 +413,9 @@ fi
 # Fail before evaluation if a selected task needs ORCA but the evaluator-local
 # licensed runtime is unavailable.
 orca_tasks=()
-for task_dir in "$STAGE_DIR"/*/; do
-  [[ -d "$task_dir" ]] || continue
-  task_name="$(basename "$task_dir")"
-  if [[ ${#INCLUDE_PATTERNS[@]} -gt 0 ]]; then
-    matched=0
-    for pattern in "${INCLUDE_PATTERNS[@]}"; do
-      [[ "$task_name" == *"$pattern"* ]] && { matched=1; break; }
-    done
-    [[ "$matched" -eq 1 ]] || continue
-  fi
-  if grep -qil 'orca' "$task_dir/task.toml" "$task_dir/instruction.md" "$task_dir/environment/Dockerfile" 2>/dev/null; then
-    orca_tasks+=("$task_name")
+for index in "${!EFFECTIVE_TASK_IDS[@]}"; do
+  if [[ "${EFFECTIVE_TASK_ENVS[$index]}" == "licensed-orca" ]]; then
+    orca_tasks+=("${EFFECTIVE_TASK_IDS[$index]}")
   fi
 done
 if [[ ${#orca_tasks[@]} -gt 0 ]]; then
@@ -434,15 +450,14 @@ fi
 # completed trial directories but archive stale job-level metadata.
 RESUME_JOB=0
 if [[ -f "$JOBS_DIR/$JOB_NAME/config.json" ]]; then
-  if REQUESTED="${INCLUDE_PATTERNS[*]-}" python3 - "$JOBS_DIR/$JOB_NAME/lock.json" <<'PY'
+  if REQUESTED="${EFFECTIVE_TASK_IDS[*]}" python3 - "$JOBS_DIR/$JOB_NAME/lock.json" <<'PY'
 import json, os, sys
 requested = set(os.environ.get("REQUESTED", "").split())
 try:
     recorded = {t["task"]["name"] for t in json.load(open(sys.argv[1]))["trials"]}
 except Exception:
     sys.exit(1)          # unreadable lock -> treat as new work
-# No --include means "the whole staged set", which resume also covers.
-sys.exit(0 if not requested or requested == recorded else 1)
+sys.exit(0 if requested == recorded else 1)
 PY
   then
     RESUME_JOB=1
@@ -469,10 +484,8 @@ if [[ -f "$REFERENCE_DIR/tools/verify_reference_dataset.py" ]]; then
 fi
 echo "== Injecting encrypted verifier archives from $REFERENCE_DIR =="
 injected=0
-for task_dir in "$STAGE_DIR"/*/; do
-  [[ -f "$task_dir/task.toml" ]] || continue
+for task_dir in "${EFFECTIVE_TASK_DIRS[@]}"; do
   task_id="$(basename "$task_dir")"
-  task_selected "$task_id" || continue
   source_verifier="$REFERENCE_TASKS/$task_id/verifier.fcref"
   if [[ ! -f "$source_verifier" ]]; then
     echo "FATAL: encrypted verifier missing for $task_id: $source_verifier" >&2
@@ -497,8 +510,7 @@ echo "Injected $injected encrypted verifier archive(s)."
 if [[ -f "$ROOT/scripts/reference_archive.py" ]]; then
   echo "== Unsealing encrypted verifiers with the published archive password =="
   unsealed=0
-  for task_dir in "$STAGE_DIR"/*/; do
-    [[ -f "$task_dir/task.toml" ]] || continue
+  for task_dir in "${EFFECTIVE_TASK_DIRS[@]}"; do
     if [[ ! -f "$task_dir/instruction.md" || ! -f "$task_dir/verifier.fcref" ]]; then
       echo "FATAL: $(basename "$task_dir") lacks plaintext instruction or verifier archive." >&2
       exit 1
@@ -534,7 +546,7 @@ else
     --artifact /app/output \
     --jobs-dir "$JOBS_DIR" --job-name "$JOB_NAME" \
     --env-file "$ENV_FILE" \
-    "${AGENT_KWARG_ARGS[@]}" "${INCLUDE_ARGS[@]}" "${EXCLUDE_ARGS[@]}" "${VERIFIER_ENV_ARGS[@]}" \
+    "${AGENT_KWARG_ARGS[@]}" "${INCLUDE_ARGS[@]}" "${VERIFIER_ENV_ARGS[@]}" \
     --yes
 fi