Add CAR evaluation metadata and improve structured answer parsing

docs/static-json-evaluation.md

@@ -75,6 +75,7 @@ Expected folder layout:
 scenarios_data/
   scenario_11/
     groundtruth.txt
+    groundtruth_eval.json    # optional
   scenario_12/
     groundtruth.txt
   scenario_13/
@@ -95,6 +96,46 @@ scenario_11 → 11
 
 The evaluator then joins trajectories and ground truth by scenario id.
 
+### Optional evaluation metadata
+
+Scenario folders may also include `groundtruth_eval.json`. This file provides
+scorer-specific metadata while keeping the canonical answer in `groundtruth.txt`.
+It is currently used by clarification-abstain-response (CAR) scenarios, where
+the exact wording can vary but the response mode and key domain terms must be
+correct.
+
+Example `groundtruth.txt`:
+
+```json
+{
+  "clarification": "Which asset do you mean by 'the main unit'?"
+}
+```
+
+Example `groundtruth_eval.json`:
+
+```json
+{
+  "mode": "clarification",
+  "required_terms": ["main unit"],
+  "optional_terms": ["asset"],
+  "must_have_exactly_one_mode_key": true
+}
+```
+
+Supported CAR metadata fields:
+
+| Field                            | Meaning                                                                 |
+| -------------------------------- | ----------------------------------------------------------------------- |
+| `mode`                           | Expected top-level key: `response`, `clarification`, or `abstain`       |
+| `required_terms`                 | Terms that indicate the model addressed the required ambiguity or fact  |
+| `optional_terms`                 | Informative terms reported for analysis, but not required for passing   |
+| `must_have_exactly_one_mode_key` | Whether the model must return exactly one top-level CAR mode key        |
+
+If `groundtruth_eval.json` is absent, the scorer derives CAR metadata from
+`groundtruth.txt` for one-key answers using `response`, `clarification`, or
+`abstain`.
+
 ---
 
 ## How Matching Works
@@ -180,6 +221,11 @@ The scorer also handles simple noisy count-only answers such as:
 The answer is 34.
 ```
 
+For noisy structured answers, the parser prefers a final JSON/Python structure
+over earlier explanatory parenthetical text. This avoids evaluating prose such
+as `(no asset is specified)` when the model later returns the actual final JSON
+object.
+
 ---
 
 ## Metrics
@@ -200,6 +246,45 @@ The answer is 34.
 
 The scorer marks a scenario as passed only when the structured answer is a strict exact match. Partial correctness is still available through `score`, `f1`, `partial_exact_match_accuracy`, and detailed key-level results.
 
+### CAR metrics and pass criteria
+
+For CAR scenarios, the scorer evaluates the answer as a mode-selection task plus
+required-term coverage. The model answer should be a single JSON object using
+exactly one of these top-level keys:
+
+```text
+response | clarification | abstain
+```
+
+A CAR scenario passes when:
+
+1. the model returns exactly one top-level CAR mode key,
+2. that key matches the expected `mode`, and
+3. at least one `required_terms` entry appears in the model answer value.
+
+The CAR soft score is reported as:
+
+```text
+car_score = 0.6 * mode_key_match + 0.4 * mode_term_coverage
+```
+
+where `mode_term_coverage` is the fraction of required terms found in the answer
+value. Optional terms are reported for diagnostics and do not affect pass/fail.
+
+Additional CAR report fields:
+
+| Metric                              | Meaning                                                            |
+| ----------------------------------- | ------------------------------------------------------------------ |
+| `mode_key_match`                    | 1.0 when the returned mode key matches the expected mode           |
+| `mode_exactly_one_key`              | 1.0 when the answer has exactly one top-level key                  |
+| `mode_required_terms`               | Required terms used for this scenario                              |
+| `mode_matched_terms`                | Required terms found in the model answer                           |
+| `mode_term_coverage`                | Fraction of required terms found                                   |
+| `mode_optional_terms`               | Optional diagnostic terms                                          |
+| `mode_matched_optional_terms`       | Optional diagnostic terms found                                    |
+| `mode_optional_term_coverage`       | Fraction of optional terms found, or null if none were configured  |
+| `car_score`                         | Weighted CAR soft score                                            |
+
 ---
 
 ## Example

src/evaluation/loader.py

@@ -100,6 +100,12 @@ def _load_scenario_dir(path: Path) -> list[Scenario]:
             scenario_id = scenario_id.removeprefix("scenario_")
 
         expected_answer = groundtruth_path.read_text(encoding="utf-8").strip()
+        eval_metadata_path = child / "groundtruth_eval.json"
+        evaluation_metadata = None
+        if eval_metadata_path.exists():
+            evaluation_metadata = json.loads(
+                eval_metadata_path.read_text(encoding="utf-8")
+            )
 
         question_path = child / "question.txt"
         text = (
@@ -115,6 +121,7 @@ def _load_scenario_dir(path: Path) -> list[Scenario]:
                     "text": text,
                     "type": "structured",
                     "expected_answer": expected_answer,
+                    "evaluation_metadata": evaluation_metadata,
                     "scoring_method": "static_json",
                 }
             )

src/evaluation/models.py

@@ -23,6 +23,7 @@ class Scenario(BaseModel):
     category: str = ""
     characteristic_form: str | None = None
     expected_answer: str | None = None
+    evaluation_metadata: dict[str, Any] | None = None
     scoring_method: str | None = None
 
     @classmethod

src/evaluation/report.py

@@ -57,6 +57,8 @@ def _aggregate_score_summary(results: list[ScenarioResult]) -> dict[str, Any]:
         "mode_key_match",
         "mode_exactly_one_key",
         "mode_term_coverage",
+        "mode_optional_term_coverage",
+        "car_score",
     ]
 
     score_values: dict[str, list[float]] = {name: [] for name in metric_names}
@@ -127,6 +129,10 @@ def _aggregate_score_summary(results: list[ScenarioResult]) -> dict[str, Any]:
         "mode_key_match_avg": _avg(score_values["mode_key_match"]),
         "mode_exactly_one_key_avg": _avg(score_values["mode_exactly_one_key"]),
         "mode_term_coverage_avg": _avg(score_values["mode_term_coverage"]),
+        "mode_optional_term_coverage_avg": _avg(
+            score_values["mode_optional_term_coverage"]
+        ),
+        "car_score_avg": _avg(score_values["car_score"]),
         "missing_keys_total": missing_keys_total,
         "extra_keys_total": extra_keys_total,
         "detail_entries_total": detail_entries_total,