feat(evals): add mastra bench harness (integrations/mastra-sdk)

[miguelg719]

What

Adds --harness mastra to evals, backed by a new private @browserbasehq/stagehand-integrations-mastra-sdk package.

• packages/integrations/mastra-sdk — runMastraSession: in-process @mastra/core Agent + @mastra/mcp MCPClient (per-session UUID id), streamed tool-call/result/usage events, abort forwarding, status/stopReason normalization, redaction.
• packages/evals/framework/mastraToolAdapter.ts — via:"mcp" mounts (stagehand_facade, playwright_mcp, chrome_devtools_mcp) with idempotent cleanup; step observations recorded.
• packages/evals/framework/harnesses/mastraAdapter.ts — events → NormalizedToolCall[].
• Registered via defineExternalHarness on the shared external-runner skeleton from the base PR.

Testing

• Full unit gate green: evals 488, mastra-sdk 12, all other suites unchanged.
• Connected smoke (evals run b:webvoyager --harness mastra --tool stagehand_facade -l 1 -t 1 -e browserbase): harness drove 7 Stagehand tool calls through the facade and produced a final answer.

Stacked on the wave-core PR. Implemented with Codex (gpt-5.6) under supervision; Claude + codex exec review findings (MCPClient id collision, env leakage, redaction) fixed in-branch.

---

Summary by cubic

Adds a Mastra-backed bench harness to evals so --harness mastra runs tasks through @mastra/core + @mastra/mcp. This extends the external-runner flow to Mastra with normalized trajectories, harness_* metrics, and sanitized logs.

• New private package @browserbasehq/stagehand-integrations-mastra-sdk wraps Agent.stream and MCPClient with per-session client IDs, abort forwarding, bounded disconnect, token-usage normalization, and sanitized transcripts.
• New evals adapters:
  • mastraToolAdapter: mounts Stagehand via stdio MCP (stagehand_facade, playwright_mcp, chrome_devtools_mcp) and exposes code surfaces (stagehand_code, playwright_code, cdp_code) as an in-process tool (stagehand_browser_run) backed by the existing code bridge; records step evidence with idempotent cleanup.
  • mastraAdapter: converts Mastra fullStream events into Trajectory with deep-sanitized args/results and image extraction for observations.
  • mastraRunner: plugs into the shared external-runner to build prompts, stream/parse results, and map status/usage; success JSON emitted before an SDK error does not count as a pass.
• CLI: Mastra is a selectable harness; default tool surface is stagehand_facade; default model is openai/gpt-5.4-mini; planner respects EVAL_MASTRA_MODELS.

Review and rollout notes

• Install @mastra/core and @mastra/mcp in the workspace; the loader throws a clear error if missing.
• MCP servers must be stdio definitions with string command (optional args, env); non-stdio configs are rejected.
• Timeouts and limits:
  • Steps: EVAL_MASTRA_MAX_STEPS (fallback AGENT_EVAL_MAX_STEPS, default 50).
  • MCP connect/list: EVAL_MASTRA_MCP_TIMEOUT_MS; disconnect is bounded.
  • Evidence capture: EVAL_CAPTURE_EVIDENCE_TIMEOUT_MS (default 15s).
  • Runtime teardown: EVAL_AGENT_MOUNT_CLEANUP_TIMEOUT_MS (default 30s).
• Security/robustness: event logs, transcripts, and tool args/results/errors are sanitized.
• CI/build: caches packages/integrations/mastra-sdk/dist, adds Turbo/Vitest targets; no changes to other harnesses’ behavior.

^{Written for commit c0a2370. Summary will update on new commits.}

[changeset-bot]

⚠️ No Changeset found

Latest commit: c0a2370

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

[cubic-dev-ai]

All reported issues were addressed across 24 files

Architecture diagram

sequenceDiagram
    participant CLI as Evals CLI
    participant Runner as mastraRunner
    participant Adapter as mastraToolAdapter
    participant SDK as mastra-sdk (Integration)
    participant Mastra as @mastra/core (Agent)
    participant Tool as Stagehand (MCP/Bridge)

    Note over CLI,Tool: NEW: Mastra Bench Harness Execution Flow

    CLI->>Runner: runAgent(harness: "mastra")
    
    Runner->>Adapter: NEW: prepareMastraToolAdapter()
    Adapter->>Adapter: Resolve Tool Surface (e.g., stagehand_facade)
    
    alt NEW: Mount via MCP (stdio)
        Adapter->>Adapter: buildMastraMcpServers()
    else NEW: Mount via handles (in-process code)
        Adapter->>Tool: Start Code Bridge
        Adapter->>SDK: createTool(stagehand_browser_run)
    end
    
    Adapter-->>Runner: Prepared Adapter (MCP configs / Tools)
    
    Runner->>SDK: NEW: runMastraSession(prompt, model, tools)
    
    SDK->>SDK: NEW: Generate unique session agentId/MCP ID
    opt If MCP enabled
        SDK->>Tool: createMcpClient & List Tools
    end
    
    SDK->>Mastra: Agent.stream(prompt)
    
    loop Agent Step Execution
        Mastra->>SDK: tool-call event
        
        alt Call via MCP
            SDK->>Tool: stdio request to MCP Server
        else Call via Browser Run Tool
            SDK->>Tool: execute JS via Code Bridge
        end
        
        Tool-->>SDK: tool-result (JSON/Images)
        
        Note over SDK,Tool: NEW: ObservationRecorder captures step evidence
        SDK->>Adapter: NEW: onToolResult() notification
        Adapter->>Tool: captureEvidence()
        
        SDK->>Mastra: Return result to Agent
        Mastra->>SDK: text-delta / reasoning-delta
    end
    
    Mastra-->>SDK: finish event (Token Usage)
    SDK-->>Runner: MastraSessionResult (Events, Usage, Text)
    
    Runner->>Runner: NEW: mastraAdapter.fromHarnessResult()
    Note right of Runner: CHANGED: Map Mastra events to<br/>NormalizedTrajectory (Steps + Metrics)
    
    Runner->>Adapter: cleanup()
    Adapter->>Tool: Teardown MCP/Bridge
    Adapter->>Adapter: NEW: Recursive RM of temp workspace
    
    Runner-->>CLI: TaskResult (Trajectory, Success/Fail)

    Note over SDK,Mastra: NEW: Sensitive data (keys, tokens)<br/>redacted from event stream & logs

Loading

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>