Add YAML scenario profiles and recursive evaluation aggregation

benchmarks/scenario_suite/README.md

@@ -1,205 +1,138 @@
-# Benchmark Runner
+# Scenario Suite Runner
 
-This folder contains the scenario list and usage notes for the benchmark.
+The runner executes selected scenarios sequentially, saves each trajectory, and
+runs evaluation unless `--no-evaluate` is set.
 
-In the benchmark, users can add the scenario IDs they want to execute.
+## Select scenarios
 
-The benchmark runner executes each scenario sequentially, saves trajectories, and then invokes the existing evaluation pipeline to generate per-scenario and aggregate reports.
-
-## Scenario ID file
-
-The benchmark registry is a plain text file:
-
-```text
-benchmarks/scenario_suite/scenarios.txt
-```
-
-Each line contains one scenario id:
+`--scenario-ids` accepts a named selector:
 
 ```text
-11
-12
-14
-15
+<category>[+<category>...]_<all|lite>
 ```
 
-Blank lines and lines starting with `#` are ignored, so you can also use comments:
+Categories are `car`, `fcc`, `fmsr`, `health`, `tsfm`, and `wosr`. The `all`
+and `lite` shorthands select every category from that profile.
 
-```text
-# User 1
-11
-12
-14
-15
-
-# User 2
-21
-22
-23
-```
+Profiles are loaded from `all.yaml` and `lite.yaml` in this directory. The Lite
+profile contains:
 
-## Expected scenario folder layout
+| Category | Scenario IDs |
+| -------- | ------------ |
+| CAR | 151, 152, 153, 156, 167, 178, 180, 182, 183, 193 |
+| FCC | 301, 303, 305, 308, 314, 316, 320, 323, 325, 327 |
+| FMSR | 902, 904, 905, 906, 915, 916, 920, 923, 928, 932 |
+| Health | 401–410 |
+| TSFM | 1001–1005 |
+| WOSR | 5, 9, 13, 20, 24, 31, 43, 50, 61, 66 |
 
-The runner expects a scenario root directory containing folders like:
+Examples:
 
-```text
-scenarios_data/
-  scenario_11/
-    question.txt
-    manifest.json
-    groundtruth.txt
-  scenario_12/
-    question.txt
-    manifest.json
-    groundtruth.txt
+```bash
+--scenario-ids fcc_lite
+--scenario-ids fcc+fmsr_all
+--scenario-ids lite
+--scenario-ids all
 ```
 
-For each scenario:
-
-- `question.txt` is passed to the agent
-- `manifest.json` is used by couchdb to load the data
-- `groundtruth.txt` is used by the evaluator
-
-The scenario folder name must match the id from `scenarios.txt`:
-
-- `11` → `scenario_11`
-- `12` → `scenario_12`
-
-## Run direct LLM
-
-Run the direct LLM baseline sequentially over the listed scenarios:
+A profile YAML file can also be passed directly:
 
 ```bash
-uv run python -m benchmark.scenario_suite_runner   --scenario-ids benchmarks/scenario_suite/scenarios.txt   --scenario-root /.../scenarios_data   --agent_name direct_llm --model-id tokenrouter/MiniMax-M3
+--scenario-ids benchmarks/scenario_suite/lite.yaml
 ```
 
-This writes trajectories to:
+Plain-text files are supported too. Put one scenario ID on each line; blank
+lines and `#` comments are ignored.
 
-```text
-traces/trajectories/scenario_suite/direct_llm/
-```
+## Scenario data layout
 
-and reports to:
+The scenario root must contain one directory per selected ID:
 
 ```text
-reports/scenario_suite/direct_llm/
+scenarios_data/
+  scenario_151/
+    question.txt
+    manifest.json
+    groundtruth.txt
 ```
 
-## Run Stirrup agent
+`question.txt` is passed to the agent, `manifest.json` loads the scenario into
+CouchDB, and `groundtruth.txt` is required by evaluation.
 
-Run the Stirrup agent sequentially over the listed scenarios:
+## Run scenarios
 
-```bash
-uv run python -m benchmark.scenario_suite_runner   --scenario-ids benchmarks/scenario_suite/scenarios.txt   --scenario-root /.../scenarios_data   --agent_name stirrup_agent
-```
-
-Run the Stirrup agent sequentially over the listed scenarios using the MiniMax model
+Direct LLM baseline:
 
 ```bash
 uv run python -m benchmark.scenario_suite_runner \
-  --scenario-ids benchmarks/scenario_suite/scenarios.txt \
-  --scenario-root /.../scenarios_data \
-  --agent_name stirrup_agent \
+  --scenario-ids lite \
+  --scenario-root /path/to/scenarios_data \
+  --agent_name direct_llm \
   --model-id tokenrouter/MiniMax-M3
 ```
 
-This writes trajectories to:
-
-```text
-traces/trajectories/scenario_suite/stirrup_agent/
-```
-
-and reports to:
-
-```text
-reports/scenario_suite/stirrup_agent/
-```
-
-## Run all agents
-
-Run all supported agents one after the other:
-
-```bash
-uv run python -m benchmark.scenario_suite_runner   --scenario-ids benchmarks/scenario_suite/scenarios.txt   --scenario-root /.../scenarios_data   --agent_name all
-```
+Available agent names are `direct_llm`, `stirrup_agent`, `opencode_agent`,
+`gemini_cli_agent`, `openclaw_cli_agent`, and `all`.
 
 ## Useful options
 
-### Dry run
+| Option | Behavior |
+| ------ | -------- |
+| `--dry-run` | Print commands without executing them. |
+| `--skip-existing` | Skip a scenario when its expected trajectory already exists; default is false. |
+| `--continue-on-error` | Continue after a scenario fails. |
+| `--no-evaluate` | Save trajectories without running evaluation. |
+| `--preserve-workspaces` | Keep existing per-run workspaces. |
 
-Print the commands without executing them:
+With `--skip-existing`, the runner checks:
 
-```bash
-uv run python -m benchmark.scenario_suite_runner   --scenario-ids benchmarks/scenario_suite/scenarios.txt   --scenario-root /.../scenarios_data   --agent_name direct_llm   --dry-run
-```
-
-### Skip existing trajectories
-
-Skip scenarios whose trajectory files already exist:
-
-```bash
-uv run python -m benchmark.scenario_suite_runner   --scenario-ids benchmarks/scenario_suite/scenarios.txt   --scenario-root /.../scenarios_data   --agent_name direct_llm   --skip-existing
+```text
+<trajectory-root>/<agent>/<model-slug>/<agent>_<scenario-id>.json
 ```
 
-### Continue after errors
-
-Keep running later scenarios even if one fails:
+For example:
 
 ```bash
-uv run python -m benchmark.scenario_suite_runner   --scenario-ids benchmarks/scenario_suite/scenarios.txt   --scenario-root /.../scenarios_data   --agent_name direct_llm   --continue-on-error
+uv run python -m benchmark.scenario_suite_runner \
+  --scenario-ids fcc+fmsr_all \
+  --scenario-root /path/to/scenarios_data \
+  --agent_name direct_llm \
+  --model-id tokenrouter/MiniMax-M3 \
+  --skip-existing
 ```
 
-## Environment variables
+## Environment
 
-The direct LLM baseline uses TokenRouter by default. Set these before running:
+For `tokenrouter/*` models, set:
 
 ```bash
 export TOKENROUTER_API_KEY=your_tokenrouter_key
 export TOKENROUTER_BASE_URL=https://api.tokenrouter.com/v1
 ```
 
-If you use a different model or backend, set the corresponding environment variables required by that backend.
+For `litellm_proxy/*` models, set `LITELLM_API_KEY` and `LITELLM_BASE_URL`.
 
 ## Output layout
 
-Typical outputs look like this:
+Outputs are nested by agent and model slug:
 
 ```text
 traces/trajectories/scenario_suite/
   direct_llm/
-    direct_llm_11.json
-    direct_llm_12.json
-    direct_llm_14.json
-    direct_llm_15.json
-  stirrup_agent/
-    stirrup_agent_11.json
-    stirrup_agent_12.json
-```
+    tokenrouter-MiniMax-M3/
+      direct_llm_151.json
 
-```text
 reports/scenario_suite/
   direct_llm/
-    direct_llm_11.json
-    direct_llm_12.json
-    _aggregate.json
-  stirrup_agent/
-    stirrup_agent_11.json
-    stirrup_agent_12.json
-    _aggregate.json
+    tokenrouter-MiniMax-M3/
+      _aggregate.json
 ```
 
-Each per-scenario report contains the final answer, score, and operational metrics. The aggregate report summarizes the full batch.
+Each aggregate report contains matched scenario results, operational metrics,
+and score summaries for that agent/model pair.
 
 ## Tests
 
-Run the benchmark runner tests with:
-
-```bash
-uv run pytest src/benchmark/tests/test_scenario_suite_runner.py -v
-```
-
-Run all benchmark tests with:
-
 ```bash
-uv run pytest src/benchmark/tests -v
+uv run pytest src/benchmark/tests/test_scenario_suite_runner.py -q
 ```