Experiment: use a structured finish answer for Stirrup

INSTRUCTIONS.md

@@ -137,7 +137,7 @@ See [MCP Servers](#mcp-servers) for available tools and [docs/mcp-servers.md](do
 
 | Variable             | Default            | Description                                                                |
 | -------------------- | ------------------ | -------------------------------------------------------------------------- |
-| `STIRRUP_CODE_IMAGE` | `python:3.12-slim` | Docker image for the code sandbox (build `assetops-code` for numpy/pandas/scipy) |
+| `STIRRUP_CODE_IMAGE` | `assetops-code`    | Docker image for the Stirrup code sandbox; build the bundled image for NumPy, pandas, and SciPy |
 | `DOCKER_HOST`        | *(SDK default)*    | Daemon socket if non-default (e.g. Rancher: `unix:///<home>/.rd/docker.sock`) |
 
 ---

benchmarks/scenario_suite/README.md

@@ -19,12 +19,16 @@ profile contains:
 
 | Category | Scenario IDs |
 | -------- | ------------ |
-| CAR | 151, 152, 153, 156, 167, 178, 180, 182, 183, 193 |
+| CAR | 151, 152, 153, 156 |
 | FCC | 301, 303, 305, 308, 314, 316, 320, 323, 325, 327 |
-| FMSR | 902, 904, 905, 906, 915, 916, 920, 923, 928, 932 |
-| Health | 401–410 |
-| TSFM | 1001–1005 |
-| WOSR | 5, 9, 13, 20, 24, 31, 43, 50, 61, 66 |
+| FMSR | 902, 904, 905, 906 |
+| Health | 401–404 |
+| TSFM | `[]` (none) |
+| WOSR | 5, 9, 13, 20 |
+
+Individual categories may use an empty list, such as `tsfm: []`; the profile as
+a whole must contain at least one scenario ID. Profile shorthands such as `lite`
+simply skip empty categories.
 
 Examples:
 

benchmarks/scenario_suite/lite.yaml

@@ -3,12 +3,6 @@ car:
   - 152
   - 153
   - 156
-  - 167
-  - 178
-  - 180
-  - 182
-  - 183
-  - 193
 fcc:
   - 301
   - 303
@@ -25,37 +19,14 @@ fmsr:
   - 904
   - 905
   - 906
-  - 915
-  - 916
-  - 920
-  - 923
-  - 928
-  - 932
 health:
   - 401
   - 402
   - 403
   - 404
-  - 405
-  - 406
-  - 407
-  - 408
-  - 409
-  - 410
-tsfm:
-  - 1001
-  - 1002
-  - 1003
-  - 1004
-  - 1005
+tsfm: []
 wosr:
   - 5
   - 9
   - 13
   - 20
-  - 24
-  - 31
-  - 43
-  - 50
-  - 61
-  - 66

docs/evaluation.md

@@ -89,6 +89,8 @@ uv run evaluate \
   --judge-model litellm_proxy/azure/gpt-5.4
 ```
 
+Evaluation scores the top-level `answer` field in each trajectory.
+
 Output:
 
 ```

docs/static-json-evaluation.md

@@ -75,6 +75,7 @@ Expected folder layout:
 scenarios_data/
   scenario_11/
     groundtruth.txt
+    groundtruth_eval.json    # optional
   scenario_12/
     groundtruth.txt
   scenario_13/
@@ -95,6 +96,46 @@ scenario_11 → 11
 
 The evaluator then joins trajectories and ground truth by scenario id.
 
+### Optional evaluation metadata
+
+Scenario folders may also include `groundtruth_eval.json`. This file provides
+scorer-specific metadata while keeping the canonical answer in `groundtruth.txt`.
+It is currently used by clarification-abstain-response (CAR) scenarios, where
+the exact wording can vary but the response mode and key domain terms must be
+correct.
+
+Example `groundtruth.txt`:
+
+```json
+{
+  "clarification": "Which asset do you mean by 'the main unit'?"
+}
+```
+
+Example `groundtruth_eval.json`:
+
+```json
+{
+  "mode": "clarification",
+  "required_terms": ["main unit"],
+  "optional_terms": ["asset"],
+  "must_have_exactly_one_mode_key": true
+}
+```
+
+Supported CAR metadata fields:
+
+| Field                            | Meaning                                                                 |
+| -------------------------------- | ----------------------------------------------------------------------- |
+| `mode`                           | Expected top-level key: `response`, `clarification`, or `abstain`       |
+| `required_terms`                 | Terms that indicate the model addressed the required ambiguity or fact  |
+| `optional_terms`                 | Informative terms reported for analysis, but not required for passing   |
+| `must_have_exactly_one_mode_key` | Whether the model must return exactly one top-level CAR mode key        |
+
+If `groundtruth_eval.json` is absent, the scorer derives CAR metadata from
+`groundtruth.txt` for one-key answers using `response`, `clarification`, or
+`abstain`.
+
 ---
 
 ## How Matching Works
@@ -180,6 +221,11 @@ The scorer also handles simple noisy count-only answers such as:
 The answer is 34.
 ```
 
+For noisy structured answers, the parser prefers a final JSON/Python structure
+over earlier explanatory parenthetical text. This avoids evaluating prose such
+as `(no asset is specified)` when the model later returns the actual final JSON
+object.
+
 ---
 
 ## Metrics
@@ -200,6 +246,45 @@ The answer is 34.
 
 The scorer marks a scenario as passed only when the structured answer is a strict exact match. Partial correctness is still available through `score`, `f1`, `partial_exact_match_accuracy`, and detailed key-level results.
 
+### CAR metrics and pass criteria
+
+For CAR scenarios, the scorer evaluates the answer as a mode-selection task plus
+required-term coverage. The model answer should be a single JSON object using
+exactly one of these top-level keys:
+
+```text
+response | clarification | abstain
+```
+
+A CAR scenario passes when:
+
+1. the model returns exactly one top-level CAR mode key,
+2. that key matches the expected `mode`, and
+3. at least one `required_terms` entry appears in the model answer value.
+
+The CAR soft score is reported as:
+
+```text
+car_score = 0.6 * mode_key_match + 0.4 * mode_term_coverage
+```
+
+where `mode_term_coverage` is the fraction of required terms found in the answer
+value. Optional terms are reported for diagnostics and do not affect pass/fail.
+
+Additional CAR report fields:
+
+| Metric                              | Meaning                                                            |
+| ----------------------------------- | ------------------------------------------------------------------ |
+| `mode_key_match`                    | 1.0 when the returned mode key matches the expected mode           |
+| `mode_exactly_one_key`              | 1.0 when the answer has exactly one top-level key                  |
+| `mode_required_terms`               | Required terms used for this scenario                              |
+| `mode_matched_terms`                | Required terms found in the model answer                           |
+| `mode_term_coverage`                | Fraction of required terms found                                   |
+| `mode_optional_terms`               | Optional diagnostic terms                                          |
+| `mode_matched_optional_terms`       | Optional diagnostic terms found                                    |
+| `mode_optional_term_coverage`       | Fraction of optional terms found, or null if none were configured  |
+| `car_score`                         | Weighted CAR soft score                                            |
+
 ---
 
 ## Example

docs/stirrup-agent.md

@@ -113,18 +113,19 @@ The runner's default model is
 | `--model-id` prefix     | Client                          | Notes                                                       |
 | ----------------------- | ------------------------------- | ----------------------------------------------------------- |
 | `<provider>/<model>`    | Stirrup `LiteLLMClient`         | Native LiteLLM. `watsonx/...`, `anthropic/...`, etc. work directly. |
-| `litellm_proxy/` or `tokenrouter/` | `OpenResponsesClient`, then `ChatCompletionsClient` | Prefer Responses; fall back when unsupported or unavailable after retries. |
+| `litellm_proxy/` or `tokenrouter/` | `ChatCompletionsClient` | Uses only the OpenAI-compatible Chat Completions API. |
 
 Required env vars match the rest of the repo: the standard watsonx vars for the
 native route, `LITELLM_BASE_URL` / `LITELLM_API_KEY` for the LiteLLM proxy, or
 `TOKENROUTER_BASE_URL` / `TOKENROUTER_API_KEY` for TokenRouter.
 
-For context management, the runner assumes every configured model has a
-1,000,000-token context window. A client adapter reports that value to
-Stirrup's summarization logic while leaving each underlying client's 64,000
-maximum-output-token default unchanged. The runner requests summarization at
-85% context usage, or approximately 850,000 tokens. This is a benchmark
-assumption rather than model metadata supplied by TokenRouter.
+For context management, the runner gives Stirrup a 100,000-token working-context
+budget while leaving each underlying client's 64,000 maximum-output-token default
+unchanged. The runner requests summarization at 75% of that budget, or
+approximately 75,000 tokens. This budget controls context compaction and is not
+model metadata supplied by TokenRouter. When summarization occurs, the complete
+generated summary is printed during the run rather than the truncated preview
+used by Stirrup's default logger.
 
 ---
 
@@ -149,12 +150,14 @@ Backends (`--code-backend`):
 > inputs you control; prefer `docker` for unattended or untrusted runs.
 
 When code execution is enabled, the runner appends backend-specific guidance to
-the shared agent prompt. It directs the model to retrieve domain data through
-the MCP tools, use `code_exec` for computation and validation, keep work inside
-the persistent execution workspace, and include verified conclusions in the
-Stirrup `finish` reason. Docker runs also identify `/workspace` and the default
-image's package availability; local runs warn that commands execute with the
-current user's host permissions. `--no-code` keeps the shared prompt unchanged.
+the shared agent prompt. It directs the model to answer directly when domain
+knowledge, MCP results, or basic reasoning are sufficient; reserve `code_exec`
+for necessary computation, data processing, workspace inspection, or validation;
+and never use it for planning, comments, placeholders, or empty scripts. When
+code is needed, the prompt prefers one bounded inspect/analyze/verify script and
+small outputs. Docker runs identify `/workspace` and the installed NumPy, pandas,
+and SciPy packages; local runs warn that commands execute with the current user's
+host permissions. `--no-code` keeps the shared prompt unchanged.
 
 ---
 
@@ -176,19 +179,21 @@ Rancher Desktop must use the **dockerd (moby)** engine, not containerd.
 
 ### 2. Sandbox image with the scientific stack
 
-`python:3.12-slim` has no numpy/pandas/scipy. Build the bundled image, which adds them:
+The default image name is `assetops-code`. Build it once from the bundled
+Dockerfile, which adds NumPy, pandas, and SciPy to `python:3.12-slim`:
 
 ```bash
 docker build -f src/agent/stirrup_agent/Dockerfile.code -t assetops-code .
-export STIRRUP_CODE_IMAGE=assetops-code
 ```
 
+Set `STIRRUP_CODE_IMAGE` only when using a different tag.
+
 `Dockerfile.code`:
 
 ```dockerfile
 FROM python:3.12-slim
 RUN pip install --no-cache-dir \
-    "numpy>=1.24" "pandas>=2.0" "scipy>=1.10" "matplotlib>=3.7"
+    "numpy>=1.24" "pandas>=2.0" "scipy>=1.10"
 ```
 
 ### 3. Run
@@ -209,6 +214,45 @@ call and the right answer (479001600).
 
 ---
 
+## Reading tool-produced files
+
+On the code track, MCP text results larger than 100 KiB (102,400 UTF-8 bytes) are
+automatically written under `mcp_results/` in the active code workspace. The
+tool response returned to the model is a compact JSON handle containing the
+relative path, tool arguments, byte count, and SHA-256 digest. `code_exec` can
+read that path directly without copying the original response through another
+model turn.
+The file contains the complete, unmodified MCP response—including null fields—
+rather than a projected subset. Code should inspect only the schema, counts, a
+small sample, or the specific rows or fields needed, then process the artifact
+in place. For artifacts larger than 200 KiB, extract and process the relevant
+subset in bounded batches instead of printing the full payload.
+
+Within one agent run, identical read calls reuse an intact existing artifact.
+Successful work-order mutations, catalog mutations, and TSFM run-producing tools
+clear the read cache so later reads observe their changes. This is a run-local
+snapshot cache; it does not detect updates made externally during the run.
+Artifacts are content-addressed, so a refreshed response does not overwrite an
+earlier snapshot. Smaller responses remain inline, and persistence failures
+safely fall back to the original inline MCP result.
+
+For example:
+
+```bash
+python3 - <<'PY'
+import json
+
+with open("mcp_results/wo__list_workorders_<query-id>_<content-id>.json") as f:
+    result = json.load(f)
+
+print(len(result["work_orders"]))
+PY
+```
+
+The workspace is temporary unless `--preserve-workspace` is enabled.
+
+---
+
 ## Preserving code workspaces
 
 By default, Stirrup creates a temporary execution directory for `code_exec` and
@@ -249,7 +293,7 @@ In addition to the [common flags](../INSTRUCTIONS.md#common-flags) (`--model-id`
 | `--workspace-dir PATH` | Host base directory for Docker/local code-execution workspaces.                     |
 | `--preserve-workspace` | Copy final code-execution files into `--workspace-dir` before cleanup.              |
 
-Environment variable: `STIRRUP_CODE_IMAGE` (Docker image; default `python:3.12-slim`).
+Environment variable: `STIRRUP_CODE_IMAGE` (Docker image; default `assetops-code`).
 
 ---
 
@@ -315,7 +359,7 @@ uv run stirrup-agent --no-code --run-id stirrup-smoke --scenario-id 101 \
 | ------- | ----------- |
 | `docker.errors.DockerException: Error while fetching server API version ... FileNotFoundError` | SDK can't find the daemon socket. `export DOCKER_HOST=unix://<path from 'docker context inspect | grep Host'>`. |
 | Docker connects but `code_exec` hits `ModuleNotFoundError` | Sandbox image lacks the library; build/point `STIRRUP_CODE_IMAGE` at `assetops-code` (or an image with the stack). |
-| Agent's code can't open a tool-produced file path in Docker | Expected — the host path isn't in the sandbox. Use `--code-backend local` for file-reading scenarios. |
+| Agent code cannot open a server-returned dataset or result file pointer in Docker | Those host-side pointers are separate from automatically bridged `mcp_results/` snapshots. Use an MCP tool that consumes the pointer, or use `--code-backend local` when direct host-file access is required. |
 | `uv sync` fails to resolve | A pin (e.g. `litellm==...`) clashing with Stirrup's range; relax to a compatible range. |
 | `stirrup` import errors | `stirrup[mcp,litellm,docker]` not installed; re-run `uv sync`. |
 | A server missing from the tool list | Its subprocess failed to start (missing CouchDB creds for `iot`/`wo`/`vibration`, or model load for `tsfm`). |
@@ -324,9 +368,10 @@ uv run stirrup-agent --no-code --run-id stirrup-smoke --scenario-id 101 \
 
 ## What was added
 
-- `src/agent/stirrup_agent/` — `runner.py`, `cli.py`, `trajectory.py`, `__init__.py`,
-  `Dockerfile.code`, and `tests/test_runner.py`.
+- `src/agent/stirrup_agent/` — runner, trajectory mapping, workspace bridging,
+  CLI, Docker image, and focused tests.
+- `src/agent/_prompts.py` — concise enforcement of user-requested response formats.
 - `pyproject.toml` — `stirrup[mcp,litellm,docker]` dependency and the `stirrup-agent`
   entry point.
 
-No other parts of the repo are modified; the MCP servers are unchanged.
+The MCP servers themselves are unchanged.

src/agent/_prompts.py

@@ -12,5 +12,6 @@
 forecasting models, and work order management.
 
 Answer the user's question concisely and accurately using the available tools.
-When you retrieve data, include the key numbers or names in your answer.
+Follow the user's requested output format exactly. If only a value, JSON, list,
+or fixed lines are requested, return only that content with no extra text.
 """

src/agent/stirrup_agent/Dockerfile.code

@@ -1,3 +1,3 @@
 FROM python:3.12-slim
 RUN pip install --no-cache-dir \
-    "numpy>=1.24" "pandas>=2.0" "scipy>=1.10" "matplotlib>=3.7"
+    "numpy>=1.24" "pandas>=2.0" "scipy>=1.10"

src/agent/stirrup_agent/cli.py

@@ -31,8 +31,7 @@ def _build_parser() -> argparse.ArgumentParser:
                           directly here, e.g.
                           watsonx/meta-llama/llama-4-maverick-17b-128e-instruct-fp8
 
-Router models use the Responses API first and fall back to Chat Completions
-when Responses is unsupported or remains unavailable after retries.
+Router models use the Chat Completions API.
 
 tracks:
   --code-enabled (default)  Add a sandboxed code-execution tool (code track).
@@ -44,7 +43,7 @@ def _build_parser() -> argparse.ArgumentParser:
   TOKENROUTER_API_KEY   TokenRouter API key  (for tokenrouter/* models)
   TOKENROUTER_BASE_URL  TokenRouter base URL (e.g. https://api.tokenrouter.com/v1)
   WATSONX_APIKEY/...    Standard LiteLLM watsonx vars (for watsonx/* models)
-  STIRRUP_CODE_IMAGE    Docker image for the code track (default python:3.12-slim)
+  STIRRUP_CODE_IMAGE    Docker image for the code track (default assetops-code)
 
 examples:
   stirrup-agent "What assets are at site MAIN?"