Non-agentic evals: parameterize evalchemy sbatch paths by $USER

[mrinal-bespoke]

Problem

The four evalchemy sbatch scripts hardcode /leonardo_work/AIFAC_5C0_290/bfeuer00/...
for every path — conda root, the evalchemy-marin clone, nltk data, HF cache, JIT
caches, and the output root. A second operator cannot run a non-agentic eval at all.

Change

Split paths by whether they are read-only shared runtime or per-user writable:

• Read-only runtime (conda env, evalchemy-marin clone, nltk data) — use the
  submitter's own copy if present, else fall back to the shared bfeuer00 one. A new
  operator does not have to duplicate a ~20 GB conda env to run an eval.
• Writable (HF_HOME/HF_HUB_CACHE, CACHE_ROOT, OUT_ROOT) — always per-user.
• #SBATCH --output uses SLURM's %u.

Each is ${VAR:-<default>}, so an explicit env var still wins.

This is a no-op for the original operator: with USER=bfeuer00 every path resolves
to the same byte-identical string as before.

Notable fixes

Conda fallback must validate the env, not the root. Testing only for a miniforge3
directory is not enough — an operator can have a partial install (e.g. otagent but not
evalchemy-marin) and would then fail with EnvironmentNameNotFound: evalchemy-marin
instead of falling back to the shared root. The check is now for
<root>/envs/evalchemy-marin.

--output directories must pre-exist. SLURM opens the log file before the job script
runs and does not create intermediate directories, so a new operator gets an instant
"Unable to open file" that reads like a launch bug rather than a missing mkdir. The script
cannot self-heal this — its own mkdir -p runs far too late — so the one-time setup is
documented inline next to each path.

Reviewer note

With HF_HUB_OFFLINE=1 and HF_HOME now per-user, a new operator's cache starts empty,
so models will not resolve until they pre-cache or symlink them in. That is inherent to
making the cache writable (one shared HF cache across operators is not safe), and the
symlink workaround is documented in the diff — but it is a genuine behaviour change for a
new user.

Testing

• bash -n clean on all four scripts.
• Path resolution simulated for three cases: original operator (all-own, byte-identical),
  new operator (shared fallback for runtime, per-user for writable), and partial install
  (correctly prefers shared over an incomplete own root).
• Comments-only in the final commit.

🤖 Generated with Claude Code

[Nativeatom]

It looks good now, with a few adjustment options that potentially increase the robustness.

Force-set → env-overridable: a real behavior change masked by the "no-op" claim (medium)

In the base, HF_HOME, HF_HUB_CACHE, and CACHE_ROOT were force-set (export HF_HOME=…/data/hub, etc.), overriding any ambient value. OUT_ROOT was also force-set in delphi_rl_think_eval.sbatch and qwen3_eval.sbatch (OUT in qwen3). In the head these become ${VAR:-default} (evalchemy_eval.sbatch:80-81,96, delphi_rl_think_eval.sbatch:97-98,103,116, qwen3_eval.sbatch:64-65,72,90).

Consequence: if the submitting shell exports any of these (e.g. a HF_HOME pointing at the read-only $HOME, which is a common Hugging-Face habit, or a CACHE_ROOT from another workflow), SLURM propagates it and the script now honours it instead of the work-FS path. For the original operator this can silently redirect the HF/JIT caches away from the pre-populated work-FS copy and fail the eval (or, worse, write into read-only $HOME). This contradicts the letter of the "no-op" claim, which is only guaranteed when none of these variables are set in the submit environment.

This is consistent with the PR's stated design ("explicit env var still wins"), so it is not an accidental bug — but it is a semantic change that the "no-op" wording understates. A one-line mitigation is to keep these force-set (or force-set to _OTA_WORK default unless an explicit opt-in var is given), or to document that a new/other operator must ensure these are unset.

Fallback validation is inconsistent: _pick uses -e, not a marker (low)

The PR's own "Notable fixes" argues that testing only for a directory is insufficient because a partial install can exist. That reasoning was applied to conda (_pick_conda checks /envs/evalchemy-marin), but not to the other two read-only fallbacks:

evalchemy-marin clone — _pick "${_OTA_WORK}/code/evalchemy-marin" … (evalchemy_eval.sbatch:71, delphi_rl_think_eval.sbatch:93, qwen3_eval.sbatch:57) only tests -e on the directory. A half-created/empty own clone would be chosen over the complete shared one, and the job would then fail at python -m eval.eval instead of falling back.
nltk_data — _pick "${_OTA_WORK}/data/nltk_data" … (evalchemy_eval.sbatch:92) has the same weakness: an empty own dir would shadow the populated shared one.
Same bug class the author already fixed for conda, just at lower likelihood. Recommend testing a marker (e.g. pyproject.toml/eval/eval.py for the clone, tokenizers/punkt for nltk) or reusing _pick_conda-style subdir validation.

OUT_ROOT semantics differ between qwen3 and the others (low)

evalchemy_eval.sbatch:110 and delphi_rl_think_eval.sbatch:116 define OUT = OUT_ROOT/RUN_NAME with OUT_ROOT defaulting to …/experiments/delphi-eval. qwen3_eval.sbatch:90 defines OUT = OUT_ROOT/RUN_NAME but with OUT_ROOT defaulting to …/experiments/delphi-eval/qwen3-baseline. A user who sets OUT_ROOT=/custom gets /custom/ from qwen3, dropping the qwen3-baseline grouping that the default (and the --output log path) provides. Not a bug for default usage, but the meaning of OUT_ROOT is not uniform across the four scripts.

Repo-probe ordering can prefer a stale canonical checkout (informational)

_ota_first_repo checks ${_OTA_WORK}/code/OpenThoughts-Agent (candidate #2) before SLURM_SUBMIT_DIR (candidate #3). If an operator has an outdated checkout at the canonical path but is submitting a newer script from a different clone, candidate #2 wins and the job runs against the stale tree. This preserves the original operator's historical behavior (the old code also hardcoded that checkout), so it is not a regression, but it is a subtlety worth knowing for any operator who keeps multiple checkouts.