Skip to content

docs: refresh v1 authoring and evaluation skills - #2555

Draft
willccbb wants to merge 1 commit into
mainfrom
wb-717-skills
Draft

docs: refresh v1 authoring and evaluation skills#2555
willccbb wants to merge 1 commit into
mainfrom
wb-717-skills

Conversation

@willccbb

@willccbb willccbb commented Sep 7, 2026

Copy link
Copy Markdown
Member

Refresh the environment-authoring and evaluation skills against the current v1 APIs. Correct the environment class and trace metadata names, explain Codex session continuation, and configure the solver runtime in the agentic-judge example to match how the judge inherits its runtime policy.

Also document the default Prime runtime, evaluation upload and server behavior, both retry scopes, and the current resolved-config and attempt-log paths used when resuming a run.

Validation: deterministic tests (pytest tests/ -m 'not e2e' -q), Ruff, all pre-commit checks, and the pre-push type check passed. Model-free checks verified the documented API names, defaults, authoring snippets, and output paths. No live model or sandbox runs were performed.


Note

Low Risk
Documentation-only changes to skill markdown; no runtime or application code modified.

Overview
Updates the create-environments and evaluate-environments skill docs so they match current v1 naming and CLI behavior.

Authoring (create-environments): Multi-agent guidance now refers to vf.Env / vf.Env[YourConfig] and trace.agent.name instead of Environment and trace.agent_name. Assistant turn continuation is described as transcript resume vs a stateful harness overriding session() (e.g. Codex via ACPHarness), replacing the older resume() wording.

Evaluation (evaluate-environments): Documents default Prime agent runtimes, --no-push upload behavior, and default environment-server vs --no-serve. The agentic-judge example configures --env.solver.runtime.* (judge follows the solver runtime policy; judge runtime flags are ignored). Retries are split into per-agent (--env.<agent>.retries) vs whole-episode (--env.retries). Run output and resume paths use configs/resolved/eval.json and per-launch logs/attempt_<n>/eval.log with logs/latest.

Reviewed by Cursor Bugbot for commit 7d80377. Bugbot is set up for automated code reviews on this repo. Configure here.

Note

Refresh v1 authoring and evaluation skills docs

  • Updates skills/create-environments/SKILL.md: changes resume contract from overriding resume() to overriding session() for persistent live processes; switches extension type from Environment to vf.Env; updates trace agent identity from trace.agent_name to trace.agent.name
  • Updates skills/evaluate-environments/SKILL.md: adds Prime sandbox defaults and runtime selection guidance; splits retries into per-agent (--env.<agent>.retries) and whole-episode (--env.retries) scopes; revises output layout to use configs/resolved/eval.json, per-attempt log directories, and a logs/latest symlink; changes agentic-judge example to use solver runtime policy and ignore judge runtime settings
  • Risk: these are docs-only changes, but reviewers should confirm the updated session(), vf.Env, trace.agent.name, retry flags, and output paths match current tooling behavior

Macroscope summarized 7d80377.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant