feat: add rubric-based scenario judging

[Copilot]

This PR updates rubric-based judging to work with the current scenario, trial, and portable result bundle structure. Scenarios can define weighted 1-5 criteria with examples and thresholds, and each trial records either a scored rubric result or an explicit unavailable result when the judge cannot complete.

The judge only receives the original task, final response, and read-only workspace tools. The scenario config and rubric are excluded from the candidate workspace, so the evaluation criteria are not leaked into the task environment.

Golden eval scorer alignment

I also compared this with the rubric and scorer setup in github/prototyping:

• Deterministic checkable criteria map to scenario.test.ts and browser.test.ts in agent-eval.
• Subjective felt criteria map to the rubric judge added here.
• Prototyping keeps target criteria, weighted rubric policy, judge configuration, and scorer composition as separate layers. This PR follows the same boundary by keeping deterministic tests and qualitative judging as independent signals on the trial result.
• Prototyping treats missing evaluator evidence as inconclusive rather than failed. Rubric results now use status: 'unavailable' for that case instead of producing a false failing score.
• A composite scorer registry similar to evals/goldens/scorers.json could be added later if agent-eval needs pluggable VRT, static analysis, or DOM scorers. It is not needed to introduce rubric judging, and we should avoid combining these signals into one reward until the scoring policy is calibrated.

Before / After

Before	After

Changelog

New

• Add optional scenario rubrics with a configured judge model, weighted criteria, concrete score descriptions, good and bad examples, and minimum score thresholds.
• Add structured rubric results to trial artifacts and experiment and benchmark output bundles.
• Add rubric configuration and result views to the website.

Changed

• Update scenario loading, trial execution, and portable output parsing for the current repository structure.
• Preserve judge failures as explicit unavailable results so missing evidence is not treated as a failing evaluation.

Removed

• None.

[joshblack]

@copilot add support for showing good and bad examples for each criteria and make sure this is provided into the prompt for the judge agent. Also make sure no part of the judge prompt is in the environment when running the scenario

[joshblack]

@copilot include support for this in the UI

[joshblack]

@copilot allow for explicit examples to be provided for each criteria (e.g. code samples) of examples of the criteria or good/bad to help it assess

[Copilot]

Pull request overview

Adds rubric-based, repeatable LLM judging for scenarios in @primer/agent-eval, including schema validation, structured judgeResult output, and website UI surfacing rubric configs and per-run rubric results.

Changes:

• Introduces scenario rubric configuration (model + optional reasoning effort + weighted criteria with optional minimum thresholds) and validates it during scenario loading/output parsing.
• Adds a new judge runner that evaluates the workspace + final agent response, producing a structured, weighted judgeResult.
• Exposes rubric configuration/results in the website scenario and experiment run pages, and documents the new scoring dimension.

Show a summary per file

File	Description
website/src/scenarios.ts	Extends website scenario model to include optional rubric config from resolved scenarios.
website/src/app/scenarios/[id]/components/Page.tsx	Renders rubric configuration details on the scenario detail page.
website/src/app/experiments/[id]/runs/[date]/page.tsx	Includes judgeResult in run details derived from AgentEvalOutput.
website/src/app/experiments/[id]/runs/[date]/components/Page.tsx	Displays rubric evaluation results (score + per-criterion judgments) in run UI.
README.md	Documents rubric score as an additional scoring dimension.
packages/agent-eval/src/treatment.ts	Extends TreatmentResult to optionally carry a typed JudgeResult.
packages/agent-eval/src/scenarios.ts	Adds runtime validation for rubric in scenario configs (model/effort/criteria validation).
packages/agent-eval/src/scenarios.test.ts	Adds tests for accepting valid rubrics and rejecting incomplete score descriptions.
packages/agent-eval/src/scenario-config.ts	Re-exports rubric-related types for scenario authoring.
packages/agent-eval/src/run.ts	Runs rubric judging after the agent run (with workspace lock-down) and includes judgeResult in results; factors copy excludes into a constant.
packages/agent-eval/src/run.test.ts	Adds a small check ensuring scenario config is excluded from copied workspace via SCENARIO_COPY_EXCLUDES.
packages/agent-eval/src/output.ts	Extends output schema to include optional rubric config in resolved scenarios and optional judgeResult in results.
packages/agent-eval/src/output.test.ts	Adds fixture coverage for parsing/serializing judgeResult in output.
packages/agent-eval/src/judge.ts	Implements judge prompt/args, response parsing/validation, weighted scoring, and threshold enforcement.
packages/agent-eval/src/judge.test.ts	Adds unit tests for prompt composition, CLI args, parsing, weighting, thresholds, and fenced JSON handling.
packages/agent-eval/src/index.ts	Exports new judge result types from the package entrypoint.
packages/agent-eval/src/experiment-config.ts	Adds Rubric, RubricCriterion, and RubricScore types and wires rubric into ScenarioConfig.
packages/agent-eval/README.md	Documents how to configure rubrics and what judge results contain.
.changeset/tasty-steaks-give.md	Declares a minor release for adding rubric-based judging.

Review details

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

• Files reviewed: 19/19 changed files
• Comments generated: 3
• Review effort level: Lite

[Copilot]

🔵 Needs a closer look

It introduces a new judge execution path (Copilot CLI invocation) and threads new optional rubric data through core output schemas and the website, warranting final human review for runtime/cost/compatibility implications.

Review details

• Files reviewed: 20/20 changed files
• Comments generated: 0 new
• Review effort level: Lite