Improve Stirrup large-result handling and trajectory evaluation

[ShuxinLin]

Summary

• bridge Stirrup MCP text responses larger than 100 KiB into the active code workspace, returning compact artifact handles instead of repeatedly placing large payloads in model context
• preserve complete MCP responses with content-addressed filenames and SHA-256 integrity checks; reuse identical reads within a run and invalidate cached reads after known state-changing operations
• use a 100,000-token Stirrup working-context budget with summarization at 75,000 tokens while retaining the provider client's 64,000-token output cap
• tighten response-format and code-execution guidance so the agent answers directly when possible, avoids planning or placeholder scripts, and processes large artifacts in bounded subsets
• replace Stirrup's default finish(reason, paths) contract with an AssetOps-specific finish(answer, reason, paths) tool so the final response is captured explicitly at termination
• improve structured-answer parsing and add clarification-abstain-response (CAR) evaluation metadata, required/optional term metrics, and aggregate CAR scores
• allow scenario profiles to contain empty categories while still requiring at least one scenario overall; the Lite profile skips TSFM

Stirrup large-result behavior

• The workspace bridge is enabled only on the code track; --no-code keeps the normal MCP provider behavior.
• Responses at or below 100 KiB remain inline.
• Larger responses are saved without field projection, including null fields, and represented by a compact artifact handle.
• Same-argument reads reuse an intact artifact during the current run. Known work-order, catalog, and TSFM mutations clear the relevant cache; external changes made outside the run are not detected.
• Artifact inspection is limited to schemas, counts, small samples, or the specific rows and fields needed. Artifacts exceeding 200 KiB are processed in bounded batches and are not printed in full.
• OpenAI-compatible Stirrup models use Chat Completions. The default assetops-code image includes NumPy, pandas, and SciPy.

Structured finish contract

Stirrup 0.1.12 supports a custom finish tool. The runner now requires the agent to provide the response separately from internal completion metadata:

finish(
  answer=<complete response to send directly to the user>,
  reason=<optional internal termination note>,
  paths=<optional output files>
)

• answer is required and rejects empty or whitespace-only values.
• The answer must match the output format requested in the original question and omit status updates, internal reasoning, and extra commentary unless requested.
• final_answer() prefers the validated finish_params.answer, while retaining assistant-text and finish.reason fallbacks for legacy or incomplete runs.
• The finish executor returns the final response as its tool result and validates reported files when a code execution environment is active.
• Trajectories persist one authoritative answer field. There is no post-run repair LLM call or answer_repair field.
• Offline evaluation always scores the trajectory's answer field.

Structured JSON and CAR scoring

• scenario directories may provide optional groundtruth_eval.json metadata alongside groundtruth.txt
• CAR scenarios can specify the expected response, clarification, or abstain mode; required terms; optional terms; and whether exactly one mode key is required
• structured-answer extraction evaluates complete content first, then balanced object/array candidates, then tuple candidates, reducing failures caused by explanatory prose or parenthetical text
• reports include required-term coverage, optional-term coverage, and the weighted CAR score

Documentation

• document artifact persistence, cache semantics, context compaction, full-summary logging, bounded artifact processing, Chat Completions routing, and the Docker scientific stack
• document static-JSON parsing and optional CAR evaluation metadata
• align the environment reference, CLI help, Lite scenario profile, and empty-category behavior

Testing

• uv run pytest src/agent/stirrup_agent/tests src/observability/tests src/evaluation/tests src/benchmark/tests -q — 188 passed
• uv run python -m compileall -q src/agent/stirrup_agent src/observability src/evaluation
• git diff --check

A full repository test run previously reached 391 passing tests before an unrelated FMSR integration test failed because its database was not connected.