fix(v1): eval writes summary.json beside traces.jsonl

[willccbb]

What the run dir lacked

A finished eval run dir held traces.jsonl, configs/resolved/eval.json and logs/, and nothing that said how the run went. The dashboard computed a mean reward and an error share for display only, so anyone reading a run afterwards (a script, a pipeline, a person with jq) re-derived pass rate, mean reward and failures from traces.jsonl with their own reading of Episode.ok, Trace.reward and agent.trainable. Those readings drift. The constraint here is one reading of the record, owned by verifiers.

What changes

• cli/output.py gains summarize(episodes) -> Summary and write_summary. run_eval writes summary.json beside traces.jsonl when the run completes and logs one summary: episodes=… failed=… reward=… line (on the console under --no-rich, in logs/latest/eval.log always). A --resume drops the previous summary before re-running the owed rollouts, since resume.load rewrites traces.jsonl underneath it, and writes a fresh one on completion. The file's presence therefore means the run completed.
• The dashboard footer's reward … · err … reads summarize instead of its own formula; the per-signal breakdown rows keep format_mean.
• docs/v1/evaluation.md and the evaluate-environments skill name the file.
• No new config, flag or dependency.

What summary.json holds

{
  "episodes": 6,
  "failed": 2,
  "errors": {"ProviderError": 2},
  "reward": 0.5,
  "tasks": {
    "echo:hello world": {"name": null, "rollouts": 2, "failed": 0, "reward": 1.0},
    "echo:ping":        {"name": null, "rollouts": 2, "failed": 2, "reward": null},
    "echo:verifiers":   {"name": null, "rollouts": 2, "failed": 0, "reward": 0.0}
  }
}

• episodes: finished episodes, one per rollout (on a resume, the kept ones plus the re-run ones).
• failed: episodes that ended not ok (Episode.ok is false).
• errors: failed episodes by the type of the error that failed them: a failed trace's last error, else the episode's own last error (a hook's). These are the live errors episode_should_retry reads; an ok trace's errors are history its per-agent retry recovered from, and are never the cause. A failed episode that recorded no error is counted nowhere here, so the counts sum to at most failed.
• reward: mean over scored rollouts. A rollout's reward is the mean Trace.reward over its policy traces (agent.trainable, or every trace when no seat is trainable) that were scored; a trace whose rewards are all None (or empty) was never scored, and a rollout with no scored trace is unscored and outside every mean. null when nothing was scored.
• tasks: the same per task, keyed by task.key (task.hash when unset; rows written before either was recorded hash their data, as resume does), with TaskData.name when the task has one.

utils/platform.run_metrics (the --push metadata) keeps its v0-shaped avg_reward / avg_error, where an errored trace counts as 0. That is a platform contract and is untouched here.

How it was checked

• tests/v1/test_e2e.py::test_summary runs echo-v1 on the null harness (subprocess runtime, in-process) against a local stub endpoint that answers every request with "hello world" and fails the fail phrase upstream, so it needs no key and no network. It asserts summary.json verbatim: 3 episodes, 1 failed, {"ProviderError": 1}, reward 0.5, per task 1.0 / 0.0 / null. test_summary_rules pins the rules that run can't reach over hand-built episodes: a frozen seat's reward is not counted, every trace counts when no seat is trainable, an unscored trace leaves its rollout out of the mean, and a failure is attributed to the failed trace's error rather than to history a recovered sibling kept.
• The default served path through the CLI (uv run eval echo-v1 --env.agent.harness.id null --env.agent.runtime.type subprocess --no-rich -n 3 -r 2, same stub): the run dir gains the summary.json above, the log ends with summary: episodes=6 failed=2 reward=0.500, and a --resume of that dir re-runs the two failed rollouts and rewrites the summary.
• uv run ruff check --fix ., uv run ruff format ., uv run pytest tests/v1 -n auto -m "not e2e" (85 passed), uv run pre-commit run --all-files and ty check verifiers are green.

🤖 Generated with Claude Code

---

Note

Low Risk
Additive run output and aligned dashboard display; resume behavior only removes a stale summary file before recomputation.

Overview
Finished eval runs now emit summary.json next to traces.jsonl, giving pipelines and humans a single canonical rollup instead of re-parsing episodes with ad hoc reward/error rules.

summarize(episodes) in cli/output.py defines that rollup: episode and failure counts, failures grouped by error type, mean reward over scored rollouts (policy/trainable traces only, with documented fallbacks for multi-agent and unscored cases), plus the same stats per task key. run_eval writes the file on completion, logs a one-line summary, and deletes the old summary.json on --resume before rewriting it when the resumed run finishes. The live dashboard reward · err footer now uses summarize so on-screen numbers match the file.

Docs/skills mention summary.json. Tests add a local stub_model HTTP stub (no API key/network), an e2e check of the echo run’s summary shape, and unit tests for aggregation edge cases.

^{Reviewed by Cursor Bugbot for commit 2331dcd. Bugbot is set up for automated code reviews on this repo. Configure here.}

Note

Write summary.json beside traces.jsonl on eval completion

• Adds Summary and TaskSummary Pydantic models plus summarize and write_summary helpers in output.py to aggregate episode counts, failure counts by error type, scored reward means, and per-task results
• Integrates summary generation into runner.py: run_eval writes summary.json after all episodes finish and deletes any stale summary before a resumed run's owed rollouts
• Updates the live dashboard in eval.py so headline reward and error values follow the Summary aggregation rules (reward mean excludes unscored rollouts; error ratio uses failed/total episodes)\n- Adds a local HTTP stub-model fixture and end-to-end/aggregation tests in test_e2e.py and conftest.py, plus docs in evaluation.md and SKILL.md
• Behavioral Change: resumed evaluations now remove and rewrite summary.json; reward means prefer trainable-agent traces and exclude unscored rollouts, which may change displayed dashboard values versus prior direct calculation

^{Macroscope summarized 2331dcd.}

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

Want reviews to match your repository better? Bugbot Learning can learn team-specific rules from PR activity. A team admin can enable Learning in the Cursor dashboard.

^{Reviewed by Cursor Bugbot for commit 502cc78. Configure here.}

[macroscopeapp]

Approvability

Verdict: Not approved

Macroscope's review found this PR not approvable — This PR adds a new persistent summary.json contract for completed evaluations and changes reward/error aggregation shown by the rich evaluation dashboard. The scope is focused and tested, but the behavior is enabled for existing evaluation runs rather than isolated behind an opt-in path.

^{You can add or adjust custom eligibility rules. Learn more.}

[xeophon]

summary.json is the run-completion artifact, but Path.write_text() creates or truncates the final path before the write completes. An interruption or full-disk write can leave malformed JSON that a watcher observes as completed, and --resume will not repair it when no rollouts are owed. Please serialize first, write a sibling temporary file, and atomically replace the final path.

[xeophon]

resume.load() above atomically rewrites traces.jsonl to the kept rows before this removes the old summary. If the process stops between those operations, a stale summary.json remains beside a changed trace set and still looks like a completion marker. Please invalidate the summary before calling resume.load(), while preserving or republishing a valid summary for the no-owed path.