feat(evals): add Deep Agents bench harness (integrations/deepagents-sdk + Python runner)

[miguelg719]

What

Adds --harness deepagents to evals. Deep Agents (LangChain/LangGraph) is a Python library with no CLI or Node SDK, so this adds a Python runner plus a TS session layer that spawns it.

• packages/integrations/deepagents/runner/run_eval.py — create_deep_agent + MultiServerMCPClient over the stagehand facade stdio server; emits JSONL events (tool_call, tool_result, final with EVAL_RESULT, usage); harness tool profiles disable summarization/subagents/filesystem/todo tools; pytest coverage.
• packages/integrations/deepagents-sdk — runDeepagentsSession: spawns the runner via uv, parses JSONL with bounded draining and malformed-line tolerance, process-group termination with SIGTERM→SIGKILL escalation, signal-terminated runs reported as sdk_error, recursion-limit → max_turns, redaction.
• packages/evals/framework/deepagentsToolAdapter.ts / harnesses/deepagentsAdapter.ts — via:"mcp" mounts with bounded cleanup; events → NormalizedToolCall[].
• Registered via defineExternalHarness. CI: new Deep Agents runner checks job (uv lock check + pytest) gated on runner path changes.

Testing

• Full unit gate green (evals + deepagents-sdk 12 + all other suites); uv run --locked pytest 6/6.
• Connected smoke (evals run b:webvoyager --harness deepagents --tool stagehand_facade -l 1 -t 1 -e browserbase): 5 Stagehand tool calls via the facade, final answer produced.

Stacked on the eve PR. Implemented with Codex (gpt-5.6) under supervision; review blocker (signal exit reported as completed) fixed in-branch.

---

Summary by cubic

Adds a Deep Agents harness and Python runner so evals can run LangChain/LangGraph agents over MCP. Previously Deep Agents weren’t supported; now --harness deepagents mounts MCP servers and streams JSONL events to the verifier.

• Registers --harness deepagents with default model openai/gpt-5.4-mini and supported tool surfaces: stagehand_facade (default), playwright_mcp, chrome_devtools_mcp; resolves --tool/--startup via shared helpers.
• Uses the shared external-harness runner: deepagentsRunner runs with runExternalHarnessTask, normalized harnessStatus/harnessStopReason, and usage mapping.
• Python runner (packages/integrations/deepagents/runner/run_eval.py) keeps persistent MCP sessions and emits JSONL events: assistant, tool_call, tool_result, final (with eval result), usage, error; applies sanitize-then-clip and a full redaction policy (redacted error/iterationError); disables filesystem/todo/summarization via harness tool profiles (guidance moved into the system prompt).
• New SDK @browserbasehq/stagehand-integrations-deepagents-sdk spawns the runner via uv, parses JSONL with bounded drain and malformed-line tolerance, forwards AbortSignal to the process group (SIGTERM→SIGKILL), reports signal exits as sdk_error, and normalizes stop reason and token usage.
• Evals wiring: deepagentsToolAdapter mounts MCP servers and startup profiles; harnesses/deepagentsAdapter converts events to Trajectory; tests cover registry, adapter, runner, and session behavior.
• CI adds “Deep Agents runner checks” (uv lock + pytest) gated on packages/integrations/deepagents/runner/**; vitest/turbo include the new SDK.

Review/rollout

• Review the JSONL event contract, status/usage mapping, redaction, and signal/abort handling between the SDK and the Python runner.
• Local runs require Python 3.11+ and uv. Example: evals run b:webvoyager --harness deepagents --tool stagehand_facade -l 1 -t 1 -e browserbase.
• Only MCP tool mounts are supported.

^{Written for commit bd1e627. Summary will update on new commits.}

[changeset-bot]

⚠️ No Changeset found

Latest commit: bd1e627

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

[socket-security]

Review the following changes in direct dependencies. Learn more about Socket for GitHub.

Diff	Package	Supply Chain Security	Vulnerability	Quality	Maintenance	License
	pypi/​deepagents@​0.7.8
	pypi/​langchain-openai@​1.6.0

View full report

[cubic-dev-ai]

All reported issues were addressed across 25 files

Architecture diagram

sequenceDiagram
    participant EV as Evals Framework
    participant TA as DeepagentsToolAdapter
    participant SDK as Deepagents SDK (TS)
    participant PY as Python Runner (uv/run_eval.py)
    participant MCP as MCP Tool Servers

    Note over EV,MCP: NEW: Deep Agents Harness Integration Flow

    EV->>TA: prepareDeepagentsToolAdapter()
    TA->>TA: Resolve ToolSurface & Startup Profile
    TA->>MCP: NEW: Mount MCP Servers (stdio)
    TA-->>EV: Return config (cwd, env, mcpServers)

    EV->>SDK: runDeepagentsSession(prompt, config)
    SDK->>PY: NEW: Spawn process via 'uv'
    SDK->>PY: NEW: Write JSON config to stdin (prompt, model, mcp_servers)

    loop Agent Loop (LangGraph)
        PY->>PY: Reason/Generate Tool Call
        PY->>SDK: NEW: Emit "assistant" & "tool_call" events (JSONL)
        PY->>MCP: Invoke Tool (via MultiServerMCPClient)
        MCP-->>PY: Return Observation
        PY->>SDK: NEW: Emit "tool_result" event (JSONL)
        
        opt AbortSignal triggered
            SDK->>PY: CHANGED: Send SIGTERM to process group
            Note over SDK,PY: Escalates to SIGKILL after 5s
        end
    end

    PY->>SDK: NEW: Emit "final" & "usage" events (JSONL)
    PY-->>SDK: Process exit
    SDK->>SDK: NEW: Parse JSONL & normalize token usage
    
    alt Signal Exit or Truncated Output
        SDK-->>EV: Return status: "sdk_error"
    else Success
        SDK-->>EV: Return DeepagentsSessionResult
    end

    EV->>EV: NEW: deepagentsAdapter.fromHarnessResult()
    Note over EV: Normalize JSONL events into Trajectory for verification

Loading

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>