diff --git a/benchmarks/scenario_suite/README.md b/benchmarks/scenario_suite/README.md index 2fd083358..05d031597 100644 --- a/benchmarks/scenario_suite/README.md +++ b/benchmarks/scenario_suite/README.md @@ -1,205 +1,138 @@ -# Benchmark Runner +# Scenario Suite Runner -This folder contains the scenario list and usage notes for the benchmark. +The runner executes selected scenarios sequentially, saves each trajectory, and +runs evaluation unless `--no-evaluate` is set. -In the benchmark, users can add the scenario IDs they want to execute. +## Select scenarios -The benchmark runner executes each scenario sequentially, saves trajectories, and then invokes the existing evaluation pipeline to generate per-scenario and aggregate reports. - -## Scenario ID file - -The benchmark registry is a plain text file: - -```text -benchmarks/scenario_suite/scenarios.txt -``` - -Each line contains one scenario id: +`--scenario-ids` accepts a named selector: ```text -11 -12 -14 -15 +[+...]_ ``` -Blank lines and lines starting with `#` are ignored, so you can also use comments: +Categories are `car`, `fcc`, `fmsr`, `health`, `tsfm`, and `wosr`. The `all` +and `lite` shorthands select every category from that profile. -```text -# User 1 -11 -12 -14 -15 - -# User 2 -21 -22 -23 -``` +Profiles are loaded from `all.yaml` and `lite.yaml` in this directory. The Lite +profile contains: -## Expected scenario folder layout +| Category | Scenario IDs | +| -------- | ------------ | +| CAR | 151, 152, 153, 156, 167, 178, 180, 182, 183, 193 | +| FCC | 301, 303, 305, 308, 314, 316, 320, 323, 325, 327 | +| FMSR | 902, 904, 905, 906, 915, 916, 920, 923, 928, 932 | +| Health | 401–410 | +| TSFM | 1001–1005 | +| WOSR | 5, 9, 13, 20, 24, 31, 43, 50, 61, 66 | -The runner expects a scenario root directory containing folders like: +Examples: -```text -scenarios_data/ - scenario_11/ - question.txt - manifest.json - groundtruth.txt - scenario_12/ - question.txt - manifest.json - groundtruth.txt +```bash +--scenario-ids fcc_lite +--scenario-ids fcc+fmsr_all +--scenario-ids lite +--scenario-ids all ``` -For each scenario: - -- `question.txt` is passed to the agent -- `manifest.json` is used by couchdb to load the data -- `groundtruth.txt` is used by the evaluator - -The scenario folder name must match the id from `scenarios.txt`: - -- `11` → `scenario_11` -- `12` → `scenario_12` - -## Run direct LLM - -Run the direct LLM baseline sequentially over the listed scenarios: +A profile YAML file can also be passed directly: ```bash -uv run python -m benchmark.scenario_suite_runner --scenario-ids benchmarks/scenario_suite/scenarios.txt --scenario-root /.../scenarios_data --agent_name direct_llm --model-id tokenrouter/MiniMax-M3 +--scenario-ids benchmarks/scenario_suite/lite.yaml ``` -This writes trajectories to: +Plain-text files are supported too. Put one scenario ID on each line; blank +lines and `#` comments are ignored. -```text -traces/trajectories/scenario_suite/direct_llm/ -``` +## Scenario data layout -and reports to: +The scenario root must contain one directory per selected ID: ```text -reports/scenario_suite/direct_llm/ +scenarios_data/ + scenario_151/ + question.txt + manifest.json + groundtruth.txt ``` -## Run Stirrup agent +`question.txt` is passed to the agent, `manifest.json` loads the scenario into +CouchDB, and `groundtruth.txt` is required by evaluation. -Run the Stirrup agent sequentially over the listed scenarios: +## Run scenarios -```bash -uv run python -m benchmark.scenario_suite_runner --scenario-ids benchmarks/scenario_suite/scenarios.txt --scenario-root /.../scenarios_data --agent_name stirrup_agent -``` - -Run the Stirrup agent sequentially over the listed scenarios using the MiniMax model +Direct LLM baseline: ```bash uv run python -m benchmark.scenario_suite_runner \ - --scenario-ids benchmarks/scenario_suite/scenarios.txt \ - --scenario-root /.../scenarios_data \ - --agent_name stirrup_agent \ + --scenario-ids lite \ + --scenario-root /path/to/scenarios_data \ + --agent_name direct_llm \ --model-id tokenrouter/MiniMax-M3 ``` -This writes trajectories to: - -```text -traces/trajectories/scenario_suite/stirrup_agent/ -``` - -and reports to: - -```text -reports/scenario_suite/stirrup_agent/ -``` - -## Run all agents - -Run all supported agents one after the other: - -```bash -uv run python -m benchmark.scenario_suite_runner --scenario-ids benchmarks/scenario_suite/scenarios.txt --scenario-root /.../scenarios_data --agent_name all -``` +Available agent names are `direct_llm`, `stirrup_agent`, `opencode_agent`, +`gemini_cli_agent`, `openclaw_cli_agent`, and `all`. ## Useful options -### Dry run +| Option | Behavior | +| ------ | -------- | +| `--dry-run` | Print commands without executing them. | +| `--skip-existing` | Skip a scenario when its expected trajectory already exists; default is false. | +| `--continue-on-error` | Continue after a scenario fails. | +| `--no-evaluate` | Save trajectories without running evaluation. | +| `--preserve-workspaces` | Keep existing per-run workspaces. | -Print the commands without executing them: +With `--skip-existing`, the runner checks: -```bash -uv run python -m benchmark.scenario_suite_runner --scenario-ids benchmarks/scenario_suite/scenarios.txt --scenario-root /.../scenarios_data --agent_name direct_llm --dry-run -``` - -### Skip existing trajectories - -Skip scenarios whose trajectory files already exist: - -```bash -uv run python -m benchmark.scenario_suite_runner --scenario-ids benchmarks/scenario_suite/scenarios.txt --scenario-root /.../scenarios_data --agent_name direct_llm --skip-existing +```text +///_.json ``` -### Continue after errors - -Keep running later scenarios even if one fails: +For example: ```bash -uv run python -m benchmark.scenario_suite_runner --scenario-ids benchmarks/scenario_suite/scenarios.txt --scenario-root /.../scenarios_data --agent_name direct_llm --continue-on-error +uv run python -m benchmark.scenario_suite_runner \ + --scenario-ids fcc+fmsr_all \ + --scenario-root /path/to/scenarios_data \ + --agent_name direct_llm \ + --model-id tokenrouter/MiniMax-M3 \ + --skip-existing ``` -## Environment variables +## Environment -The direct LLM baseline uses TokenRouter by default. Set these before running: +For `tokenrouter/*` models, set: ```bash export TOKENROUTER_API_KEY=your_tokenrouter_key export TOKENROUTER_BASE_URL=https://api.tokenrouter.com/v1 ``` -If you use a different model or backend, set the corresponding environment variables required by that backend. +For `litellm_proxy/*` models, set `LITELLM_API_KEY` and `LITELLM_BASE_URL`. ## Output layout -Typical outputs look like this: +Outputs are nested by agent and model slug: ```text traces/trajectories/scenario_suite/ direct_llm/ - direct_llm_11.json - direct_llm_12.json - direct_llm_14.json - direct_llm_15.json - stirrup_agent/ - stirrup_agent_11.json - stirrup_agent_12.json -``` + tokenrouter-MiniMax-M3/ + direct_llm_151.json -```text reports/scenario_suite/ direct_llm/ - direct_llm_11.json - direct_llm_12.json - _aggregate.json - stirrup_agent/ - stirrup_agent_11.json - stirrup_agent_12.json - _aggregate.json + tokenrouter-MiniMax-M3/ + _aggregate.json ``` -Each per-scenario report contains the final answer, score, and operational metrics. The aggregate report summarizes the full batch. +Each aggregate report contains matched scenario results, operational metrics, +and score summaries for that agent/model pair. ## Tests -Run the benchmark runner tests with: - -```bash -uv run pytest src/benchmark/tests/test_scenario_suite_runner.py -v -``` - -Run all benchmark tests with: - ```bash -uv run pytest src/benchmark/tests -v +uv run pytest src/benchmark/tests/test_scenario_suite_runner.py -q ``` diff --git a/benchmarks/scenario_suite/all.txt b/benchmarks/scenario_suite/all.txt deleted file mode 100644 index abc4e0fc9..000000000 --- a/benchmarks/scenario_suite/all.txt +++ /dev/null @@ -1,60 +0,0 @@ -1 -5 -13 -20 -24 -31 -40 -52 -61 -66 -151 -152 -153 -156 -167 -178 -180 -182 -183 -193 -301 -303 -306 -310 -312 -316 -320 -323 -325 -327 -401 -402 -403 -404 -405 -406 -407 -408 -409 -410 -902 -904 -905 -906 -915 -916 -920 -923 -928 -932 -1001 -1002 -1003 -1004 -1005 -1026 -1027 -1028 -1029 -1030 diff --git a/benchmarks/scenario_suite/all.yaml b/benchmarks/scenario_suite/all.yaml new file mode 100644 index 000000000..4051fd6cf --- /dev/null +++ b/benchmarks/scenario_suite/all.yaml @@ -0,0 +1,221 @@ +car: + - 151 + - 152 + - 153 + - 154 + - 155 + - 156 + - 157 + - 158 + - 159 + - 160 + - 161 + - 162 + - 163 + - 164 + - 165 + - 166 + - 167 + - 168 + - 169 + - 170 + - 171 + - 172 + - 173 + - 174 + - 175 + - 176 + - 177 + - 178 + - 179 + - 180 + - 181 + - 182 + - 183 + - 184 + - 185 + - 186 + - 187 + - 188 + - 189 + - 190 + - 191 + - 192 + - 193 + - 194 + - 195 + - 196 + - 197 + - 198 + - 199 + - 200 +fcc: + - 301 + - 302 + - 303 + - 304 + - 305 + - 306 + - 307 + - 308 + - 309 + - 310 + - 311 + - 312 + - 313 + - 314 + - 315 + - 316 + - 317 + - 318 + - 319 + - 320 + - 321 + - 322 + - 323 + - 324 + - 325 + - 326 + - 327 +fmsr: + - 901 + - 902 + - 903 + - 904 + - 905 + - 906 + - 907 + - 908 + - 909 + - 910 + - 911 + - 912 + - 913 + - 914 + - 915 + - 916 + - 917 + - 918 + - 919 + - 920 + - 921 + - 922 + - 923 + - 924 + - 925 + - 926 + - 927 + - 928 + - 929 + - 930 + - 931 + - 932 +health: + - 401 + - 402 + - 403 + - 404 + - 405 + - 406 + - 407 + - 408 + - 409 + - 410 +tsfm: + - 1001 + - 1002 + - 1003 + - 1004 + - 1005 + - 1006 + - 1007 + - 1008 + - 1009 + - 1010 + - 1011 + - 1012 + - 1013 + - 1014 + - 1015 + - 1016 + - 1017 + - 1018 + - 1019 + - 1020 + - 1021 + - 1022 + - 1023 + - 1024 + - 1025 + - 1026 + - 1027 + - 1028 + - 1029 + - 1030 +wosr: + - 1 + - 2 + - 3 + - 4 + - 5 + - 6 + - 7 + - 8 + - 9 + - 10 + - 11 + - 12 + - 13 + - 14 + - 15 + - 16 + - 17 + - 18 + - 19 + - 20 + - 21 + - 22 + - 23 + - 24 + - 25 + - 26 + - 27 + - 28 + - 29 + - 30 + - 31 + - 32 + - 33 + - 34 + - 35 + - 36 + - 37 + - 38 + - 39 + - 40 + - 41 + - 42 + - 43 + - 44 + - 45 + - 46 + - 47 + - 48 + - 49 + - 50 + - 51 + - 52 + - 53 + - 54 + - 55 + - 56 + - 57 + - 58 + - 59 + - 60 + - 61 + - 62 + - 63 + - 64 + - 65 + - 66 diff --git a/benchmarks/scenario_suite/clarification_abstein_response.txt b/benchmarks/scenario_suite/clarification_abstein_response.txt deleted file mode 100644 index 0160ca6b9..000000000 --- a/benchmarks/scenario_suite/clarification_abstein_response.txt +++ /dev/null @@ -1,10 +0,0 @@ -151 -152 -153 -156 -167 -178 -180 -182 -183 -193 diff --git a/benchmarks/scenario_suite/fcc.txt b/benchmarks/scenario_suite/fcc.txt deleted file mode 100644 index 0d6fc4060..000000000 --- a/benchmarks/scenario_suite/fcc.txt +++ /dev/null @@ -1,10 +0,0 @@ -301 -303 -306 -310 -312 -316 -320 -323 -325 -327 diff --git a/benchmarks/scenario_suite/fmsr.txt b/benchmarks/scenario_suite/fmsr.txt deleted file mode 100644 index 662d46677..000000000 --- a/benchmarks/scenario_suite/fmsr.txt +++ /dev/null @@ -1,10 +0,0 @@ -902 -904 -905 -906 -915 -916 -920 -923 -928 -932 diff --git a/benchmarks/scenario_suite/health.txt b/benchmarks/scenario_suite/health.txt deleted file mode 100644 index 1f0237054..000000000 --- a/benchmarks/scenario_suite/health.txt +++ /dev/null @@ -1,10 +0,0 @@ -401 -402 -403 -404 -405 -406 -407 -408 -409 -410 diff --git a/benchmarks/scenario_suite/lite.yaml b/benchmarks/scenario_suite/lite.yaml new file mode 100644 index 000000000..b977e9897 --- /dev/null +++ b/benchmarks/scenario_suite/lite.yaml @@ -0,0 +1,61 @@ +car: + - 151 + - 152 + - 153 + - 156 + - 167 + - 178 + - 180 + - 182 + - 183 + - 193 +fcc: + - 301 + - 303 + - 305 + - 308 + - 314 + - 316 + - 320 + - 323 + - 325 + - 327 +fmsr: + - 902 + - 904 + - 905 + - 906 + - 915 + - 916 + - 920 + - 923 + - 928 + - 932 +health: + - 401 + - 402 + - 403 + - 404 + - 405 + - 406 + - 407 + - 408 + - 409 + - 410 +tsfm: + - 1001 + - 1002 + - 1003 + - 1004 + - 1005 +wosr: + - 5 + - 9 + - 13 + - 20 + - 24 + - 31 + - 43 + - 50 + - 61 + - 66 diff --git a/benchmarks/scenario_suite/semantic_reasoning.txt b/benchmarks/scenario_suite/semantic_reasoning.txt deleted file mode 100644 index b4446ce7f..000000000 --- a/benchmarks/scenario_suite/semantic_reasoning.txt +++ /dev/null @@ -1,10 +0,0 @@ -1 -5 -13 -20 -24 -31 -40 -52 -61 -66 diff --git a/benchmarks/scenario_suite/tsfm.txt b/benchmarks/scenario_suite/tsfm.txt deleted file mode 100644 index fc517efed..000000000 --- a/benchmarks/scenario_suite/tsfm.txt +++ /dev/null @@ -1,10 +0,0 @@ -1001 -1002 -1003 -1004 -1005 -1026 -1027 -1028 -1029 -1030 diff --git a/src/benchmark/scenario_suite_runner.py b/src/benchmark/scenario_suite_runner.py index 70a0a08f2..4f6c3d514 100644 --- a/src/benchmark/scenario_suite_runner.py +++ b/src/benchmark/scenario_suite_runner.py @@ -1,13 +1,14 @@ """Sequential runner for the benchmark scenarios. -This runner reads a simple scenario-id file, runs each scenario with the -selected agent method, saves trajectories through AGENT_TRAJECTORY_DIR, and -optionally invokes the existing evaluator to generate reports. +This runner resolves a named scenario selection (or reads a custom scenario-id +file), runs each scenario with the selected agent method, saves trajectories +through AGENT_TRAJECTORY_DIR, and optionally invokes the existing evaluator to +generate reports. Example: uv run python -m benchmark.scenario_suite_runner \ - --scenario-ids benchmarks/scenario_suite/scenarios.txt \ + --scenario-ids fcc+fmsr_all \ --scenario-root /path/to/scenarios_data \ --agent_name direct_llm \ --model-id tokenrouter/MiniMax-M3 @@ -34,11 +35,81 @@ from dataclasses import dataclass from pathlib import Path +import yaml + REPO_ROOT = Path(__file__).resolve().parents[2] _DEFAULT_MODEL_ID = "tokenrouter/MiniMax-M3" _DEFAULT_GEMINI_MODEL_ID = "tokenrouter_gemini/google/gemma-4-26b-a4b-it" +SCENARIO_CATEGORY_ORDER = ("car", "fcc", "fmsr", "health", "tsfm", "wosr") +SCENARIO_PROFILE_PATHS = { + "all": REPO_ROOT / "benchmarks/scenario_suite/all.yaml", + "lite": REPO_ROOT / "benchmarks/scenario_suite/lite.yaml", +} + + +def load_scenario_profile(path: Path) -> dict[str, tuple[str, ...]]: + """Load and validate a category-to-scenario-ids YAML profile.""" + if not path.exists(): + raise FileNotFoundError(f"Scenario profile not found: {path}") + + raw_profile = yaml.safe_load(path.read_text(encoding="utf-8")) + if not isinstance(raw_profile, dict): + raise ValueError(f"Scenario profile must be a YAML mapping: {path}") + if any(not isinstance(category, str) for category in raw_profile): + raise ValueError(f"Scenario profile category names must be strings: {path}") + + expected_categories = set(SCENARIO_CATEGORY_ORDER) + actual_categories = set(raw_profile) + if actual_categories != expected_categories: + missing = sorted(expected_categories - actual_categories) + unknown = sorted(actual_categories - expected_categories) + details = [] + if missing: + details.append(f"missing categories: {', '.join(missing)}") + if unknown: + details.append(f"unknown categories: {', '.join(unknown)}") + raise ValueError(f"Invalid scenario profile {path}: {'; '.join(details)}") + + profile: dict[str, tuple[str, ...]] = {} + for category in SCENARIO_CATEGORY_ORDER: + raw_ids = raw_profile[category] + if not isinstance(raw_ids, list) or not raw_ids: + raise ValueError( + f"Scenario profile category {category!r} must be a non-empty list: " + f"{path}" + ) + + scenario_ids: list[str] = [] + for raw_id in raw_ids: + if isinstance(raw_id, bool) or not isinstance(raw_id, (str, int)): + raise ValueError( + f"Invalid scenario id {raw_id!r} in category {category!r}: {path}" + ) + scenario_id = str(raw_id).strip() + if not scenario_id: + raise ValueError( + f"Empty scenario id in category {category!r}: {path}" + ) + scenario_ids.append(scenario_id) + + if len(set(scenario_ids)) != len(scenario_ids): + raise ValueError( + f"Duplicate scenario ids in category {category!r}: {path}" + ) + profile[category] = tuple(scenario_ids) + + return profile + + +SCENARIO_IDS_ALL = load_scenario_profile(SCENARIO_PROFILE_PATHS["all"]) +SCENARIO_IDS_LITE = load_scenario_profile(SCENARIO_PROFILE_PATHS["lite"]) +SCENARIO_ID_PROFILES = { + "all": SCENARIO_IDS_ALL, + "lite": SCENARIO_IDS_LITE, +} + @dataclass(frozen=True) class MethodConfig: @@ -104,6 +175,72 @@ def load_scenario_ids(path: Path) -> list[str]: return scenario_ids +def _scenario_selector_error(selector: str) -> ValueError: + categories = ", ".join(SCENARIO_CATEGORY_ORDER) + return ValueError( + f"Invalid scenario selector {selector!r}. Use " + f"[+...]_; categories: {categories}. " + "The shorthands 'all' and 'lite' select every category." + ) + + +def scenario_ids_for_selector(selector: str) -> list[str]: + """Resolve a named category/profile selector into scenario ids. + + Examples: + + fcc_lite + fcc+fmsr_all + all + lite + """ + normalized = selector.strip().lower() + if normalized in SCENARIO_ID_PROFILES: + profile_name = normalized + categories = list(SCENARIO_CATEGORY_ORDER) + else: + category_expression, separator, profile_name = normalized.rpartition("_") + if not separator or profile_name not in SCENARIO_ID_PROFILES: + raise _scenario_selector_error(selector) + categories = category_expression.split("+") + if not categories or any( + not category or category not in SCENARIO_CATEGORY_ORDER + for category in categories + ): + raise _scenario_selector_error(selector) + + profile = SCENARIO_ID_PROFILES[profile_name] + scenario_ids: list[str] = [] + seen: set[str] = set() + for category in categories: + for scenario_id in profile[category]: + if scenario_id not in seen: + scenario_ids.append(scenario_id) + seen.add(scenario_id) + return scenario_ids + + +def resolve_scenario_ids(value: str | Path) -> list[str]: + """Resolve a selector expression, YAML profile, or plain scenario-id file.""" + raw_value = str(value).strip() + if not raw_value: + raise _scenario_selector_error(raw_value) + + path = Path(raw_value).expanduser() + if path.exists(): + if path.suffix.lower() in {".yaml", ".yml"}: + profile = load_scenario_profile(path) + return [ + scenario_id + for category in SCENARIO_CATEGORY_ORDER + for scenario_id in profile[category] + ] + return load_scenario_ids(path) + if path.suffix or "/" in raw_value or "\\" in raw_value: + raise FileNotFoundError(f"Scenario id file not found: {path}") + return scenario_ids_for_selector(raw_value) + + def scenario_dir_for_id(scenario_root: Path, scenario_id: str) -> Path: """Return the expected scenario folder path for a scenario id.""" return scenario_root / f"scenario_{scenario_id}" @@ -404,9 +541,12 @@ def _build_parser() -> argparse.ArgumentParser: ) parser.add_argument( "--scenario-ids", - type=Path, - default=Path("benchmarks/scenario_suite/scenarios.txt"), - help="Plain text file containing one scenario id per line.", + default="benchmarks/scenario_suite/scenarios.txt", + metavar="SELECTOR_OR_FILE", + help=( + "Scenario selector such as fcc_lite, fcc+fmsr_all, all, or lite; " + "a YAML profile or plain text file is also accepted." + ), ) parser.add_argument( "--scenario-root", @@ -669,13 +809,16 @@ def main() -> None: "files, bash, or edits" ) - scenario_ids = load_scenario_ids(args.scenario_ids) + try: + scenario_ids = resolve_scenario_ids(args.scenario_ids) + except (FileNotFoundError, ValueError) as exc: + parser.error(str(exc)) methods = selected_methods( method_name=args.agent_name, methods=build_methods(args), ) - print(f"Loaded {len(scenario_ids)} scenario ids from {args.scenario_ids}") + print(f"Resolved {len(scenario_ids)} scenario ids from {args.scenario_ids}") print(f"Selected methods: {', '.join(method.agent_name for method in methods)}") for method in methods: diff --git a/src/benchmark/tests/test_scenario_suite_runner.py b/src/benchmark/tests/test_scenario_suite_runner.py index fb786df2b..35cddc5e0 100644 --- a/src/benchmark/tests/test_scenario_suite_runner.py +++ b/src/benchmark/tests/test_scenario_suite_runner.py @@ -4,6 +4,7 @@ from pathlib import Path import pytest +import yaml from benchmark import scenario_suite_runner as mr @@ -34,6 +35,166 @@ def test_load_scenario_ids_raises_for_missing_file(tmp_path: Path) -> None: mr.load_scenario_ids(p) +def test_scenario_mappings_cover_expected_categories() -> None: + expected = {"car", "fcc", "fmsr", "health", "tsfm", "wosr"} + + assert set(mr.SCENARIO_IDS_ALL) == expected + assert set(mr.SCENARIO_IDS_LITE) == expected + assert all( + len(mr.SCENARIO_IDS_ALL[category]) == 10 + for category in expected - {"car", "fcc", "fmsr", "tsfm", "wosr"} + ) + assert mr.SCENARIO_IDS_ALL["car"] == tuple( + str(scenario_id) for scenario_id in range(151, 201) + ) + assert mr.SCENARIO_IDS_ALL["fcc"] == tuple( + str(scenario_id) for scenario_id in range(301, 328) + ) + assert mr.SCENARIO_IDS_ALL["fmsr"] == tuple( + str(scenario_id) for scenario_id in range(901, 933) + ) + assert mr.SCENARIO_IDS_ALL["tsfm"] == tuple( + str(scenario_id) for scenario_id in range(1001, 1031) + ) + assert mr.SCENARIO_IDS_ALL["wosr"] == tuple( + str(scenario_id) for scenario_id in range(1, 67) + ) + assert all( + len(mr.SCENARIO_IDS_LITE[category]) == 10 + for category in {"car", "fcc", "fmsr", "health", "wosr"} + ) + assert mr.SCENARIO_IDS_LITE["car"] == ( + "151", + "152", + "153", + "156", + "167", + "178", + "180", + "182", + "183", + "193", + ) + assert mr.SCENARIO_IDS_LITE["health"] == tuple( + str(scenario_id) for scenario_id in range(401, 411) + ) + assert mr.SCENARIO_IDS_LITE["tsfm"] == tuple( + str(scenario_id) for scenario_id in range(1001, 1006) + ) + assert mr.SCENARIO_IDS_LITE["wosr"] == ( + "5", + "9", + "13", + "20", + "24", + "31", + "43", + "50", + "61", + "66", + ) + + +def test_scenario_profiles_are_loaded_from_yaml() -> None: + assert mr.SCENARIO_IDS_ALL == mr.load_scenario_profile( + mr.SCENARIO_PROFILE_PATHS["all"] + ) + assert mr.SCENARIO_IDS_LITE == mr.load_scenario_profile( + mr.SCENARIO_PROFILE_PATHS["lite"] + ) + + +def test_scenario_profile_yaml_uses_integer_ids() -> None: + for path in mr.SCENARIO_PROFILE_PATHS.values(): + raw_profile = yaml.safe_load(path.read_text(encoding="utf-8")) + assert all( + isinstance(scenario_id, int) + for scenario_ids in raw_profile.values() + for scenario_id in scenario_ids + ) + + +def test_scenario_ids_for_selector_resolves_combined_all_categories() -> None: + assert mr.scenario_ids_for_selector("fcc+fmsr_all") == [ + *mr.SCENARIO_IDS_ALL["fcc"], + *mr.SCENARIO_IDS_ALL["fmsr"], + ] + + +def test_scenario_ids_for_selector_resolves_lite_category() -> None: + assert mr.scenario_ids_for_selector("fcc_lite") == list( + mr.SCENARIO_IDS_LITE["fcc"] + ) + + +def test_scenario_ids_for_selector_resolves_profile_shorthands() -> None: + assert mr.scenario_ids_for_selector("lite") == [ + scenario_id + for category in mr.SCENARIO_CATEGORY_ORDER + for scenario_id in mr.SCENARIO_IDS_LITE[category] + ] + assert len(mr.scenario_ids_for_selector("all")) == 215 + + +@pytest.mark.parametrize( + "selector", + ["fcc", "fcc_fast", "unknown_lite", "fcc++fmsr_all", "_lite"], +) +def test_scenario_ids_for_selector_rejects_invalid_selector(selector: str) -> None: + with pytest.raises(ValueError, match="Invalid scenario selector"): + mr.scenario_ids_for_selector(selector) + + +def test_resolve_scenario_ids_keeps_file_compatibility(tmp_path: Path) -> None: + path = tmp_path / "custom.txt" + path.write_text("301\n# skip\n902\n", encoding="utf-8") + + assert mr.resolve_scenario_ids(path) == ["301", "902"] + + +def test_resolve_scenario_ids_accepts_yaml_profile(tmp_path: Path) -> None: + path = tmp_path / "profile.yaml" + path.write_text( + """ +car: [151] +fcc: [301] +fmsr: [902] +health: [401] +tsfm: [1001] +wosr: [1] +""".strip(), + encoding="utf-8", + ) + + assert mr.resolve_scenario_ids(path) == ["151", "301", "902", "401", "1001", "1"] + + +def test_load_scenario_profile_rejects_missing_category(tmp_path: Path) -> None: + path = tmp_path / "invalid.yaml" + path.write_text("fcc: [301]\n", encoding="utf-8") + + with pytest.raises(ValueError, match="missing categories"): + mr.load_scenario_profile(path) + + +def test_resolve_scenario_ids_raises_for_missing_file_path(tmp_path: Path) -> None: + with pytest.raises(FileNotFoundError, match="Scenario id file not found"): + mr.resolve_scenario_ids(tmp_path / "missing.txt") + + +def test_parser_accepts_named_scenario_selector() -> None: + args = mr._build_parser().parse_args( + [ + "--scenario-ids", + "fcc+fmsr_all", + "--scenario-root", + "/tmp/scenarios_data", + ] + ) + + assert args.scenario_ids == "fcc+fmsr_all" + + def test_scenario_dir_for_id() -> None: root = Path("/tmp/scenarios_data") assert mr.scenario_dir_for_id(root, "11") == root / "scenario_11" diff --git a/src/evaluation/cli.py b/src/evaluation/cli.py index 66ee508d3..475546d76 100644 --- a/src/evaluation/cli.py +++ b/src/evaluation/cli.py @@ -24,22 +24,33 @@ def _build_parser() -> argparse.ArgumentParser: "--trajectories", type=Path, required=True, - help="Directory of {run_id}.json trajectory files (or a single file).", + help=( + "Directory recursively containing trajectory JSON files " + "(or a single JSON file)." + ), ) p.add_argument( "--scenarios", type=Path, nargs="+", required=True, - help="One or more scenario JSON / JSONL files.", + help="One or more scenario JSON / JSONL files or scenario directories.", + ) + p.add_argument( + "--scenario-ids", + default=None, + metavar="SELECTOR", + help=( + "Optional benchmark YAML selector such as fcc_lite or " + "fcc+fmsr_all. Only matching scenario IDs are evaluated." + ), ) p.add_argument( "--reports-dir", type=Path, default=Path("reports"), help=( - "Directory to write per-run JSON reports (one file per run, " - "named '.json'), plus '_aggregate.json' for the rollup. " + "Directory to write the combined '_aggregate.json' report. " "Default: reports/." ), ) @@ -85,13 +96,28 @@ def _validate_scorer_default(name: str) -> None: raise SystemExit(str(exc)) +def _resolve_scenario_ids(selector: str | None) -> set[str] | None: + """Resolve an optional benchmark scenario selector from all/lite YAML.""" + if selector is None: + return None + + from benchmark.scenario_suite_runner import scenario_ids_for_selector + + return set(scenario_ids_for_selector(selector)) + + def main(argv: list[str] | None = None) -> int: - args = _build_parser().parse_args(argv) + parser = _build_parser() + args = parser.parse_args(argv) logging.basicConfig( level=logging.INFO if args.verbose else logging.WARNING, format="%(asctime)s %(levelname)s %(name)s: %(message)s", ) + try: + scenario_ids = _resolve_scenario_ids(args.scenario_ids) + except (FileNotFoundError, ValueError) as exc: + parser.error(str(exc)) _maybe_install_judge(args.judge_model) _validate_scorer_default(args.scorer_default) @@ -101,12 +127,12 @@ def main(argv: list[str] | None = None) -> int: ).evaluate( trajectories_path=args.trajectories, scenarios_paths=list(args.scenarios), + scenario_ids=scenario_ids, ) out_dir = write_reports_dir(report, args.reports_dir) print(render_summary(report)) - print(f"\nReports written: {out_dir}/.json ({len(report.results)} files)") - print(f"Aggregate: {out_dir}/_aggregate.json") + print(f"\nAggregate report written: {out_dir}/_aggregate.json") return 0 diff --git a/src/evaluation/evaluator.py b/src/evaluation/evaluator.py index 9beb1cc8f..924f784f8 100644 --- a/src/evaluation/evaluator.py +++ b/src/evaluation/evaluator.py @@ -10,6 +10,7 @@ import json import logging import re +from collections.abc import Collection from pathlib import Path from . import scorers as scorer_registry @@ -43,8 +44,12 @@ def evaluate( *, trajectories_path: Path, scenarios_paths: list[Path], + scenario_ids: Collection[str] | None = None, ) -> EvalReport: scenarios = load_scenarios(scenarios_paths) + if scenario_ids is not None: + selected_ids = {str(scenario_id) for scenario_id in scenario_ids} + scenarios = [scenario for scenario in scenarios if scenario.id in selected_ids] trajectories = load_trajectories(trajectories_path) results: list[ScenarioResult] = [] @@ -121,4 +126,4 @@ def _normalize_model_id(model_id: str | None) -> str: normalized = model_id.strip() if normalized.startswith("litellm_proxy/"): normalized = normalized[len("litellm_proxy/") :] - return normalized \ No newline at end of file + return normalized diff --git a/src/evaluation/loader.py b/src/evaluation/loader.py index 1d5c0c9db..04278e1a6 100644 --- a/src/evaluation/loader.py +++ b/src/evaluation/loader.py @@ -13,7 +13,7 @@ def load_trajectories(path: Path) -> list[PersistedTrajectory]: - """Load every ``*.json`` trajectory under ``path``. + """Recursively load every ``*.json`` trajectory under ``path``. ``path`` may be a directory or a single JSON file. If a trajectory has ``scenario_id`` set to null, the filename stem is used as a fallback. @@ -25,7 +25,7 @@ def load_trajectories(path: Path) -> list[PersistedTrajectory]: return [_load_one_trajectory(p)] if p.suffix == ".json" else [] out: list[PersistedTrajectory] = [] - for child in sorted(p.glob("*.json")): + for child in sorted(p.rglob("*.json")): try: out.append(_load_one_trajectory(child)) except Exception: @@ -154,4 +154,4 @@ def join_records( continue scenario = by_id.get(traj.scenario_id) if scenario is not None: - yield scenario, traj \ No newline at end of file + yield scenario, traj diff --git a/src/evaluation/report.py b/src/evaluation/report.py index e24341e6b..6738cfffe 100644 --- a/src/evaluation/report.py +++ b/src/evaluation/report.py @@ -27,10 +27,10 @@ def _avg(values: list[float]) -> float | None: def _aggregate_score_summary(results: list[ScenarioResult]) -> dict[str, Any]: - """Aggregate static_json-style score.details across all results. + """Aggregate static_json-style score.details across one result group. Per-scenario key-level details stay in each result. Here we summarize the - numeric metrics and count totals across the full batch. + numeric metrics and count totals across the runner/model group. """ metric_names = [ "partial_match_accuracy", @@ -133,6 +133,20 @@ def _aggregate_score_summary(results: list[ScenarioResult]) -> dict[str, Any]: } +def _score_summaries_by_runner_model( + results: list[ScenarioResult], +) -> dict[str, dict[str, Any]]: + """Group score summaries under ``_`` keys.""" + grouped: dict[str, list[ScenarioResult]] = defaultdict(list) + for result in results: + grouped[f"{result.runner}_{result.model}"].append(result) + + return { + key: _aggregate_score_summary(grouped[key]) + for key in sorted(grouped) + } + + def build_report(results: list[ScenarioResult]) -> EvalReport: total = len(results) passed = sum(1 for r in results if r.score.passed) @@ -163,7 +177,7 @@ def build_report(results: list[ScenarioResult]) -> EvalReport: }, by_scenario_type=breakdown, ops=aggregate_ops(results), - score_summary=_aggregate_score_summary(results), + score_summary=_score_summaries_by_runner_model(results), results=results, ) @@ -176,24 +190,10 @@ def write_report(report: EvalReport, output: Path) -> Path: def write_reports_dir(report: EvalReport, reports_dir: Path) -> Path: - """Write one JSON file per result (``.json``) plus an aggregate. - - Results without a ``run_id`` fall back to ``.json`` so - nothing is dropped. Returns the directory path. - """ + """Write only the aggregate report and return its directory path.""" reports_dir = Path(reports_dir) reports_dir.mkdir(parents=True, exist_ok=True) - used: dict[str, int] = {} - for r in report.results: - stem = r.run_id or f"scenario-{r.scenario_id}" - suffix = used.get(stem, 0) - used[stem] = suffix + 1 - name = stem if suffix == 0 else f"{stem}-{suffix}" - (reports_dir / f"{name}.json").write_text( - r.model_dump_json(indent=2), encoding="utf-8" - ) - (reports_dir / _AGGREGATE_FILENAME).write_text( report.model_dump_json(indent=2), encoding="utf-8" ) @@ -210,62 +210,11 @@ def render_summary(report: EvalReport) -> str: ) if report.score_summary: - s = report.score_summary lines.append("") - lines.append("Static JSON summary:") - if s.get("score_avg") is not None: - lines.append(f" score_avg: {s['score_avg']:.4f}") - if s.get("score_min") is not None: - lines.append(f" score_min: {s['score_min']:.4f}") - if s.get("score_max") is not None: - lines.append(f" score_max: {s['score_max']:.4f}") - if s.get("partial_match_accuracy_avg") is not None: - lines.append( - f" partial_match_avg: {s['partial_match_accuracy_avg']:.4f}" - ) - if s.get("partial_exact_match_accuracy_avg") is not None: - lines.append( - f" partial_exact_match_avg: {s['partial_exact_match_accuracy_avg']:.4f}" - ) - if s.get("strict_exact_match_accuracy_avg") is not None: - lines.append( - f" strict_exact_match_avg: {s['strict_exact_match_accuracy_avg']:.4f}" - ) - if s.get("partial_similarity_score_avg") is not None: - lines.append( - f" partial_similarity_avg: {s['partial_similarity_score_avg']:.4f}" - ) - if s.get("partial_numeric_match_accuracy_avg") is not None: - lines.append( - f" partial_numeric_match_avg: {s['partial_numeric_match_accuracy_avg']:.4f}" - ) - if s.get("range_match_accuracy_avg") is not None: - lines.append( - f" range_match_avg: {s['range_match_accuracy_avg']:.4f}" - ) - if s.get("delta_1_match_accuracy_avg") is not None: - lines.append( - f" delta_1_match_avg: {s['delta_1_match_accuracy_avg']:.4f}" - ) - if s.get("precision_avg") is not None: - lines.append(f" precision_avg: {s['precision_avg']:.4f}") - if s.get("recall_avg") is not None: - lines.append(f" recall_avg: {s['recall_avg']:.4f}") - if s.get("f1_avg") is not None: - lines.append(f" f1_avg: {s['f1_avg']:.4f}") - if s.get("total_gold_keys_avg") is not None: - lines.append(f" total_gold_keys_avg: {s['total_gold_keys_avg']:.4f}") - if s.get("total_model_keys_avg") is not None: - lines.append(f" total_model_keys_avg: {s['total_model_keys_avg']:.4f}") - if s.get("matched_keys_avg") is not None: - lines.append(f" matched_keys_avg: {s['matched_keys_avg']:.4f}") - if s.get("exact_value_matches_avg") is not None: - lines.append( - f" exact_value_matches_avg: {s['exact_value_matches_avg']:.4f}" - ) - lines.append(f" missing_keys_total: {s.get('missing_keys_total', 0)}") - lines.append(f" extra_keys_total: {s.get('extra_keys_total', 0)}") - lines.append(f" detail_entries_total: {s.get('detail_entries_total', 0)}") + lines.append("Score summary by runner/model:") + for key, summary in sorted(report.score_summary.items()): + lines.append(f" {key}:") + _append_score_summary(lines, summary, indent=" ") if report.by_scenario_type: lines.append("") @@ -290,6 +239,41 @@ def render_summary(report: EvalReport) -> str: return "\n".join(lines) +def _append_score_summary( + lines: list[str], summary: dict[str, Any], *, indent: str +) -> None: + """Append one runner/model score summary to rendered CLI output.""" + metric_labels = { + "score_avg": "score_avg", + "score_min": "score_min", + "score_max": "score_max", + "partial_match_accuracy_avg": "partial_match_avg", + "partial_exact_match_accuracy_avg": "partial_exact_match_avg", + "strict_exact_match_accuracy_avg": "strict_exact_match_avg", + "partial_similarity_score_avg": "partial_similarity_avg", + "partial_numeric_match_accuracy_avg": "partial_numeric_match_avg", + "range_match_accuracy_avg": "range_match_avg", + "delta_1_match_accuracy_avg": "delta_1_match_avg", + "precision_avg": "precision_avg", + "recall_avg": "recall_avg", + "f1_avg": "f1_avg", + "total_gold_keys_avg": "total_gold_keys_avg", + "total_model_keys_avg": "total_model_keys_avg", + "matched_keys_avg": "matched_keys_avg", + "exact_value_matches_avg": "exact_value_matches_avg", + } + for metric, label in metric_labels.items(): + value = summary.get(metric) + if value is not None: + lines.append(f"{indent}{label}: {value:.4f}") + + lines.append(f"{indent}missing_keys_total: {summary.get('missing_keys_total', 0)}") + lines.append(f"{indent}extra_keys_total: {summary.get('extra_keys_total', 0)}") + lines.append( + f"{indent}detail_entries_total: {summary.get('detail_entries_total', 0)}" + ) + + def report_to_json(report: EvalReport) -> str: """Convenience JSON dump that round-trips through pydantic.""" return json.dumps(json.loads(report.model_dump_json()), indent=2) diff --git a/src/evaluation/runner.py b/src/evaluation/runner.py index c86889b37..0e36abea2 100644 --- a/src/evaluation/runner.py +++ b/src/evaluation/runner.py @@ -2,6 +2,7 @@ from __future__ import annotations +from collections.abc import Collection from pathlib import Path from .evaluator import Evaluator @@ -14,6 +15,7 @@ def evaluate( scenarios_paths: list[Path], default_scoring_method: str = "llm_judge", judge_model: str | None = None, + scenario_ids: Collection[str] | None = None, ) -> EvalReport: """Load, score, and aggregate. @@ -26,4 +28,5 @@ def evaluate( ).evaluate( trajectories_path=trajectories_path, scenarios_paths=scenarios_paths, + scenario_ids=scenario_ids, ) diff --git a/src/evaluation/tests/test_cli.py b/src/evaluation/tests/test_cli.py new file mode 100644 index 000000000..dc050bbd8 --- /dev/null +++ b/src/evaluation/tests/test_cli.py @@ -0,0 +1,50 @@ +"""Tests for the evaluation CLI argument surface.""" + +from __future__ import annotations + +from evaluation.cli import _build_parser, _resolve_scenario_ids + + +def test_cli_accepts_optional_scenario_selector() -> None: + args = _build_parser().parse_args( + [ + "--trajectories", + "trajectories", + "--scenarios", + "scenarios", + "--scenario-ids", + "fcc+fmsr_all", + ] + ) + + assert args.scenario_ids == "fcc+fmsr_all" + + +def test_resolve_scenario_ids_loads_all_yaml_categories() -> None: + selected = _resolve_scenario_ids("fcc+fmsr_all") + + assert selected is not None + assert "301" in selected + assert "327" in selected + assert "901" in selected + assert "932" in selected + assert "401" not in selected + + +def test_resolve_scenario_ids_loads_lite_yaml_category() -> None: + assert _resolve_scenario_ids("fcc_lite") == { + "301", + "303", + "305", + "308", + "314", + "316", + "320", + "323", + "325", + "327", + } + + +def test_resolve_scenario_ids_is_optional() -> None: + assert _resolve_scenario_ids(None) is None diff --git a/src/evaluation/tests/test_evaluator.py b/src/evaluation/tests/test_evaluator.py index b1109213e..66ebe8229 100644 --- a/src/evaluation/tests/test_evaluator.py +++ b/src/evaluation/tests/test_evaluator.py @@ -35,6 +35,42 @@ def test_evaluator_routes_to_default_scorer(tmp_path: Path, make_persisted_recor assert report.results[0].score.scorer == "stub-evaluator" +def test_evaluator_filters_to_selected_scenario_ids( + tmp_path: Path, make_persisted_record +): + for scenario_id in (301, 302): + record = make_persisted_record( + run_id=f"run-{scenario_id}", + scenario_id=scenario_id, + ) + (tmp_path / f"run-{scenario_id}.json").write_text( + json.dumps(record), encoding="utf-8" + ) + + scenarios_path = tmp_path / "scenarios.json" + scenarios_path.write_text( + json.dumps( + [ + {"id": 301, "text": "Q301", "type": "fcc"}, + {"id": 302, "text": "Q302", "type": "fcc"}, + ] + ), + encoding="utf-8", + ) + + registry.register("stub-evaluator", _stub_scorer) + report = Evaluator(default_scorer="stub-evaluator").evaluate( + trajectories_path=tmp_path, + scenarios_paths=[scenarios_path], + scenario_ids={"302"}, + ) + + assert report.totals["scenarios"] == 1 + assert [result.scenario_id for result in report.results] == ["302"] + summary = report.score_summary["plan-execute_watsonx/ibm/granite"] + assert summary["scored_results"] == 1 + + def test_evaluator_strips_think_blocks_before_scoring( tmp_path: Path, make_persisted_record ): @@ -203,4 +239,4 @@ def test_evaluator_allows_non_llm_judge_even_with_matching_model( scenarios_paths=[scenarios_path], ) - assert report.totals["passed"] == 1 \ No newline at end of file + assert report.totals["passed"] == 1 diff --git a/src/evaluation/tests/test_loader.py b/src/evaluation/tests/test_loader.py index 27d5c9a92..e47b0741b 100644 --- a/src/evaluation/tests/test_loader.py +++ b/src/evaluation/tests/test_loader.py @@ -20,6 +20,20 @@ def test_load_trajectories_from_dir(trajectory_dir: Path): assert records[0].scenario_id == "1" +def test_load_trajectories_recurses_into_nested_directories( + tmp_path: Path, make_persisted_record +): + nested = tmp_path / "direct-llm-agent" / "tokenrouter-MiniMax-M3" + nested.mkdir(parents=True) + record = make_persisted_record(run_id="nested-run", scenario_id=901) + (nested / "nested-run.json").write_text(json.dumps(record), encoding="utf-8") + + records = load_trajectories(tmp_path) + + assert [record.run_id for record in records] == ["nested-run"] + assert records[0].scenario_id == "901" + + def test_load_trajectories_skips_unparseable(tmp_path: Path, make_persisted_record): (tmp_path / "good.json").write_text(json.dumps(make_persisted_record()), encoding="utf-8") (tmp_path / "bad.json").write_text("{not json", encoding="utf-8") @@ -108,4 +122,4 @@ def test_load_scenarios_from_groundtruth_folders(tmp_path): assert len(scenarios) == 1 assert scenarios[0].id == "11" assert scenarios[0].expected_answer == "{'energy': 14, 'material': 48}" - assert scenarios[0].scoring_method == "static_json" \ No newline at end of file + assert scenarios[0].scoring_method == "static_json" diff --git a/src/evaluation/tests/test_report.py b/src/evaluation/tests/test_report.py index 0c821f5c6..2ad58a27f 100644 --- a/src/evaluation/tests/test_report.py +++ b/src/evaluation/tests/test_report.py @@ -69,31 +69,20 @@ def test_write_report_round_trips(tmp_path: Path): assert data["by_scenario_type"]["iot"]["pass_rate"] == 1.0 -def test_write_reports_dir_per_run_files(tmp_path: Path): +def test_write_reports_dir_writes_only_aggregate(tmp_path: Path): results = [ _result("iot", True, run_id="run-a"), _result("tsfm", False, run_id="run-b"), ] out_dir = write_reports_dir(build_report(results), tmp_path / "reports") - assert (out_dir / "run-a.json").exists() - assert (out_dir / "run-b.json").exists() assert (out_dir / "_aggregate.json").exists() - - per_run = json.loads((out_dir / "run-a.json").read_text()) - assert per_run["run_id"] == "run-a" - assert per_run["score"]["passed"] is True + assert sorted(path.name for path in out_dir.glob("*.json")) == ["_aggregate.json"] agg = json.loads((out_dir / "_aggregate.json").read_text()) assert agg["totals"]["scenarios"] == 2 - - -def test_write_reports_dir_falls_back_to_scenario_id(tmp_path: Path): - # ScenarioResult.run_id is empty when the trajectory pre-dates the - # run_id field; the writer must still produce a file. - results = [_result("iot", True)] - out_dir = write_reports_dir(build_report(results), tmp_path / "reports") - assert (out_dir / "scenario-x.json").exists() + assert set(agg["score_summary"]) == {"plan-execute_watsonx/ibm/granite"} + assert len(agg["results"]) == 2 def test_render_summary_includes_headlines(): @@ -103,6 +92,7 @@ def test_render_summary_includes_headlines(): ] text = render_summary(build_report(results)) assert "Pass rate" in text + assert "plan-execute_watsonx/ibm/granite" in text assert "iot" in text assert "tokens_in_total" in text @@ -154,6 +144,41 @@ def test_build_report_includes_score_summary(): report = build_report(results) assert report.score_summary is not None - assert report.score_summary["partial_exact_match_accuracy_avg"] == 0.0 - assert report.score_summary["strict_exact_match_accuracy_avg"] == 0.0 - assert report.score_summary["missing_keys_total"] == 0 \ No newline at end of file + summary = report.score_summary["direct-llm-agent_tokenrouter/MiniMax-M3"] + assert summary["partial_exact_match_accuracy_avg"] == 0.0 + assert summary["strict_exact_match_accuracy_avg"] == 0.0 + assert summary["missing_keys_total"] == 0 + + +def test_build_report_groups_score_summary_by_runner_and_model(): + results = [ + _result("iot", True), + ScenarioResult( + scenario_id="y", + scenario_type="iot", + run_id="run-direct", + runner="direct-llm-agent", + model="tokenrouter/MiniMax-M3", + question="q", + answer="a", + score=ScorerResult( + scorer="static_json", + passed=False, + score=0.25, + ), + ops=OpsMetrics(), + ), + ] + + report = build_report(results) + + assert set(report.score_summary or {}) == { + "direct-llm-agent_tokenrouter/MiniMax-M3", + "plan-execute_watsonx/ibm/granite", + } + assert ( + report.score_summary["direct-llm-agent_tokenrouter/MiniMax-M3"][ + "score_avg" + ] + == 0.25 + ) diff --git a/src/evaluation/tests/test_runner.py b/src/evaluation/tests/test_runner.py index f8a936db0..ec119b837 100644 --- a/src/evaluation/tests/test_runner.py +++ b/src/evaluation/tests/test_runner.py @@ -46,6 +46,34 @@ def test_evaluate_end_to_end(tmp_path: Path, make_persisted_record): assert report.ops.tokens_in_total > 0 +def test_evaluate_function_filters_scenario_ids(tmp_path: Path, make_persisted_record): + rec_a = make_persisted_record(run_id="run-a", scenario_id=1, answer="A") + rec_b = make_persisted_record(run_id="run-b", scenario_id=2, answer="B") + (tmp_path / "run-a.json").write_text(json.dumps(rec_a), encoding="utf-8") + (tmp_path / "run-b.json").write_text(json.dumps(rec_b), encoding="utf-8") + + scenarios_path = tmp_path / "scenarios.json" + scenarios_path.write_text( + json.dumps( + [ + {"id": 1, "text": "Q1", "type": "iot"}, + {"id": 2, "text": "Q2", "type": "tsfm"}, + ] + ), + encoding="utf-8", + ) + + registry.register("stub", _always_pass_scorer) + report = evaluate( + trajectories_path=tmp_path, + scenarios_paths=[scenarios_path], + default_scoring_method="stub", + scenario_ids={"2"}, + ) + + assert [result.scenario_id for result in report.results] == ["2"] + + def _always_fail_scorer(scenario: Scenario, answer: str, trajectory_text: str) -> ScorerResult: return ScorerResult(scorer="stub-fail", passed=False, score=0.0)