docs: refresh v1 authoring and evaluation skills - #2555
Draft
willccbb wants to merge 1 commit into
Draft
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Refresh the environment-authoring and evaluation skills against the current v1 APIs. Correct the environment class and trace metadata names, explain Codex session continuation, and configure the solver runtime in the agentic-judge example to match how the judge inherits its runtime policy.
Also document the default Prime runtime, evaluation upload and server behavior, both retry scopes, and the current resolved-config and attempt-log paths used when resuming a run.
Validation: deterministic tests (
pytest tests/ -m 'not e2e' -q), Ruff, all pre-commit checks, and the pre-push type check passed. Model-free checks verified the documented API names, defaults, authoring snippets, and output paths. No live model or sandbox runs were performed.Note
Low Risk
Documentation-only changes to skill markdown; no runtime or application code modified.
Overview
Updates the create-environments and evaluate-environments skill docs so they match current v1 naming and CLI behavior.
Authoring (
create-environments): Multi-agent guidance now refers tovf.Env/vf.Env[YourConfig]andtrace.agent.nameinstead ofEnvironmentandtrace.agent_name. Assistant turn continuation is described as transcript resume vs a stateful harness overridingsession()(e.g. Codex viaACPHarness), replacing the olderresume()wording.Evaluation (
evaluate-environments): Documents default Prime agent runtimes,--no-pushupload behavior, and default environment-server vs--no-serve. The agentic-judge example configures--env.solver.runtime.*(judge follows the solver runtime policy; judge runtime flags are ignored). Retries are split into per-agent (--env.<agent>.retries) vs whole-episode (--env.retries). Run output and resume paths useconfigs/resolved/eval.jsonand per-launchlogs/attempt_<n>/eval.logwithlogs/latest.Reviewed by Cursor Bugbot for commit 7d80377. Bugbot is set up for automated code reviews on this repo. Configure here.
Note
Refresh v1 authoring and evaluation skills docs
skills/create-environments/SKILL.md: changes resume contract from overridingresume()to overridingsession()for persistent live processes; switches extension type fromEnvironmenttovf.Env; updates trace agent identity fromtrace.agent_nametotrace.agent.nameskills/evaluate-environments/SKILL.md: adds Prime sandbox defaults and runtime selection guidance; splits retries into per-agent (--env.<agent>.retries) and whole-episode (--env.retries) scopes; revises output layout to useconfigs/resolved/eval.json, per-attempt log directories, and alogs/latestsymlink; changesagentic-judgeexample to use solver runtime policy and ignore judge runtime settingssession(),vf.Env,trace.agent.name, retry flags, and output paths match current tooling behaviorMacroscope summarized 7d80377.