Skip to content

fix(frontierchallenge): make open-track selection reproducible - #30

Open
fzp0424 wants to merge 1 commit into
ApodexAI:mainfrom
fzp0424:fix/frontierchallenge-open-track
Open

fix(frontierchallenge): make open-track selection reproducible#30
fzp0424 wants to merge 1 commit into
ApodexAI:mainfrom
fzp0424:fix/frontierchallenge-open-track

Conversation

@fzp0424

@fzp0424 fzp0424 commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

Summary

  • resolve one registry-validated effective task set and reuse it for staging, ORCA preflight, verifier handling, resume, and Harbor
  • keep stale or excluded task directories out of evaluation and classify task_098 as open from metadata rather than instruction text
  • dereference Hugging Face cache symlinks while staging
  • pin solve and reference dataset revisions independently
  • clarify the execution-based ORCA track boundary across the runtime documentation

Hugging Face release

  • solve: 8a73bb02a5709aa007e1a146d0cdc5d36ef5ea4d
  • reference: 59fbf007ad5c6816caf7d379eef32d9fb384edea
  • removed the obsolete solve-side statement unsealer; instructions remain plaintext

Verification

  • 45 FrontierChallenge tests pass
  • fresh-cache verification passes for 97 solve tasks and 97 encrypted verifiers
  • task_098 runs through the runner without ORCA preflight, even with a stale licensed task in staging
  • Harbor receives only the exact selected task and staged HF files contain no broken symlinks
  • public leak, restricted-software, shell syntax, Python compile, and diff gates pass

Addresses #27 and #28.

@zhanghanduo

Copy link
Copy Markdown
Collaborator

Small upgrade-path edge case: the stage cache identity is unchanged from the previous runner, so an existing default stage created with cp -a can still pass the reuse check and skip the new cp -aL. If that cached copy contains Hugging Face cache-relative symlinks, they may remain broken after upgrading unless the evaluator uses --force-restage or a fresh stage directory.

It may be worth adding a staging-format/version value to .frontier-source, checking cached tasks for symlinks before reuse, or otherwise forcing a one-time restage.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants