refactor(evals): generalize the bench harness seam (registry-derived harnesses, shared resolvers, external-runner skeleton)

[miguelg719]

What

Generalizes the packages/evals harness seam so the Nth external harness is a registry entry plus a trajectory adapter, instead of ~900 lines of copy-paste. Behaviour-preserving for claude_code/codex.

• Registry-derived harness type — Harness is no longer a closed literal union; parseBenchHarness/isExecutableBenchHarness, help text and planner error copy derive from harnessRegistry. ClaudeCodeHarnessConfig/CodexHarnessConfig collapse into one ExternalHarnessConfig.
• defineExternalHarness({ harness, supportedToolSurfaces, defaultModels, prepareToolAdapter, runAgent }) + registerBenchHarness — owns verifier-carrier construction, adapter prepare/cleanup and the generic start() error. claudeCodeHarness/codexHarness are expressed with it; default model lists move onto the definition (EVAL_<HARNESS>_MODELS override preserved).
• Shared resolveToolSurface / resolveStartupProfile (framework/harnesses/toolSurfaceResolution.ts) replace the byte-identical per-harness resolvers.
• runExternalHarnessTask skeleton (framework/harnesses/externalRunner.ts) — prompt assembly, EVAL_RESULT parsing (marker + structured-output contracts), bounded evidence capture, step-observation drain, grading, TaskResult assembly. Both runners use it.
• Normalized harnessStatus/harnessStopReason and harness_input_tokens/harness_output_tokens/harness_total_tokens/harness_cost_usd metrics, emitted alongside the existing prefixed names; leaderboard alias table updated.
• stagehand_facade gets captureEvidence — the runner now owns the facade stdio server and hands the agent a loopback relay, so facade runs produce a final url/ariaTree/screenshot like stagehand_code.
• evals core --tool stagehand_facade is rejected up front instead of failing per case.

Testing

• turbo build typecheck + full unit gate: evals 455/455 (was 418), all integrations suites green.
• Live: exercised end-to-end by the mastra/pi/eve/deepagents harness smokes stacked on this PR (facade relay + evidence capture working on Browserbase).
• Not yet re-smoked live with claude_code/codex on the facade surface (unit-tested against a fake JSON-RPC facade).

Stacked on #2750. Implemented with Codex (gpt-5.6) under supervision; reviewed by Claude + codex exec review.

---

Summary by cubic

Refactors the eval harness layer to a registry-driven design, adds a shared external-harness runner, and makes the Stagehand facade a runner-owned bridge. Behavior for claude_code and codex is preserved; metrics are normalized (no Codex double-counting), and evals core rejects the agent-mount-only stagehand_facade up front.

• Review/rollout
  • Focus: framework/benchHarness.ts (registry + defineExternalHarness/registerBenchHarness), framework/harnesses/externalRunner.ts, core/tools/stagehandFacadeBridge.ts, framework/harnesses/toolSurfaceResolution.ts, planner/runner/TUI.
  • Registry-derived harnesses: parsing/help/default models (EVAL_<HARNESS>_MODELS) now come from the registry; CLI/help copy and guidance use listBenchHarnesses* and formatBenchHarnessFlags().
  • Executability: BenchHarness.start is optional; planning-only harnesses cannot execute and are rejected with guidance in bench runner and TUI.
  • Tool surfaces: shared resolveToolSurface/resolveStartupProfile replace per-harness resolvers; core dry-runs use listCoreRunnableTools(). Required: do not use --tool stagehand_facade with core runs; select a harness (e.g., --harness claude_code or --harness codex) and a supported surface.
  • External runner: owns prompt assembly (marker or structured output), result parsing, bounded evidence capture and step-observation drain, grading, and TaskResult assembly; claude_code/codex runners are thin wrappers.
  • Facade bridge: runner spawns the MCP stdio server and exposes a loopback relay; evidence (final url/ariaTree/screenshot) is captured from the agent’s browser. Hardening: handshake replay/backpressure/fail-fast, carrier cleanup, and gated sdk_error with sanitized errors. Agents receive STAGEHAND_EVALS_FACADE_BRIDGE_PORT.
  • Metrics/results: prefer harness_input_tokens, harness_cached_input_tokens, harness_output_tokens, harness_reasoning_output_tokens, harness_total_tokens, and harness_cost_usd (legacy per-harness metrics still emit). TaskResult adds harnessStatus/harnessStopReason. Unified parsing accepts marker and no-marker shapes; SDK iteration errors take precedence over result text.

^{Written for commit 1a47385. Summary will update on new commits.}

[changeset-bot]

⚠️ No Changeset found

Latest commit: 1a47385

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

[cubic-dev-ai]

All reported issues were addressed across 33 files

Architecture diagram

sequenceDiagram
    participant User as CLI / TUI
    participant Planner as benchPlanner
    participant Registry as Harness Registry
    participant Runner as External Runner Skeleton
    participant Bridge as Facade Bridge (Relay)
    participant SDK as Agent SDK (Claude/Codex)
    participant Verifier as verifierAdapter

    Note over User,Verifier: PR Focus: Generalized External Harness Flow

    User->>Planner: evals run --harness claude_code
    
    Planner->>Registry: NEW: getBenchHarness(harness)
    Registry-->>Planner: Return harness definition (models, surfaces)
    
    Planner->>Planner: NEW: resolveToolSurface() & resolveStartupProfile()
    Note right of Planner: Uses shared toolSurfaceResolution.ts

    alt Harness initialization
        Planner->>Runner: execute(row)
        Runner->>Runner: NEW: buildVerifierCarrierV3()
        
        opt Using stagehand_facade
            Runner->>Bridge: NEW: startStagehandFacadeBridge()
            Bridge->>Bridge: Spawn MCP stdio server process
            Bridge-->>Runner: Return loopback port & mcpServerSpec
        end
    end

    Note over Runner,SDK: Agent Execution Phase

    Runner->>Runner: NEW: buildExternalHarnessPrompt()
    Note right of Runner: Handles marker vs structured_output contracts

    Runner->>SDK: runSession(prompt, mcpServers)
    
    loop Agent Steps
        SDK->>Bridge: Tool Call (via loopback relay)
        Bridge->>Bridge: CHANGED: Forward to facade process
        Bridge-->>SDK: Tool Result
    end

    SDK-->>Runner: Return raw results & usage metrics

    Note over Runner,Verifier: Result Processing & Grading

    Runner->>Runner: NEW: parseEvalResult(raw)
    Note right of Runner: Resilience parsing for markers/JSON

    opt Capture Evidence
        Runner->>Bridge: NEW: captureEvidence()
        Bridge->>Bridge: Call facade tools (snapshot/screenshot)
        Bridge-->>Runner: Return ariaTree/binary screenshot
    end

    Runner->>Verifier: gradeExternalTrajectory()
    Verifier-->>Runner: Grade (Success/Failure)

    Runner->>Runner: NEW: buildNormalizedHarnessMetrics()
    Note right of Runner: Maps SDK usage to harness_input_tokens, etc.

    Runner-->>User: TaskResult (with normalized metrics & status)

    alt Unhappy Path: Agent Crash or Timeout
        SDK-->>Runner: SDK Error / Iteration Error
        Runner->>Runner: CHANGED: Extract error from stopReason
        Runner-->>User: TaskResult (success: false, error: msg)
    else Unhappy Path: Invalid Harness for Tool
        User->>Planner: evals core --tool stagehand_facade
        Planner->>Planner: NEW: rejectAgentMountOnlyCoreTool()
        Planner-->>User: Error: Guidance to use specific harnesses
    end

Loading

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>