feat(evals): add fx bench harness (integrations/fx-sdk)

[miguelg719]

What

Adds --harness fx to evals, backed by a new private @browserbasehq/stagehand-integrations-fx-sdk package. fx (Vercel's terminal agent, v0.0.3) is CLI-only with no SDK.

• packages/integrations/fx-sdk — runFxSession: spawns fx ask --json --auto with the prompt on stdin and a per-run $HOME (generated ~/.fx/mcp.json + settings.json), tails ~/.fx/sessions/<id>/events.jsonl live for per-tool-call steps (fx only emits one JSON at exit), reads usage-v2.json for tokens; step-limit detection via fx's notice text / configured FX_MAX_AGENT_STEPS; tracked child process groups with SIGTERM→SIGKILL on abort and on process exit/SIGINT/SIGTERM; redaction.
• packages/evals/framework/fxToolAdapter.ts — via:"mcp" mounts; permissions are a wildcard deny plus mcp_<server>_* allows (real fx builtin tool names — glob_files, grep_files, open_file, file_info, semantic_search, vision, … all denied), MCP child env passes the runner's HOME/pnpm/XDG caches with EVAL_FX_MCP_STARTUP_TIMEOUT_MS (default 120 s); sanitized cleanup logs.
• packages/evals/framework/harnesses/fxAdapter.ts — events → NormalizedToolCall[].
• Registered via defineExternalHarness. Binary path via EVAL_FX_PATH.

Testing

• Full unit gate green (evals + fx-sdk 15 + all other suites).
• Connected smoke not yet run: fx needs a Vercel AI Gateway key (AI_GATEWAY_API_KEY/FX_AI_GATEWAY_API_KEY) in the runner env. Command when available: EVAL_FX_PATH=<fx> evals run b:webvoyager --harness fx --tool stagehand_facade -l 1 -t 1 -e browserbase.

Stacked on the deepagents PR. Implemented with Codex (gpt-5.6) under supervision; two review rounds (Claude + codex exec review) fixed in-branch.

---

Summary by cubic

Adds an fx eval harness that runs Vercel’s CLI-only fx via a new @browserbasehq/stagehand-integrations-fx-sdk, enabling apples-to-apples benchmarks against Stagehand tasks. Previously fx was unsupported; now --harness fx plans, dry-runs, and executes with MCP-only tools and normalized trajectories, usage, and status.

• New @browserbasehq/stagehand-integrations-fx-sdk: spawns fx ask --json --auto with an isolated HOME; tails ~/.fx/sessions/<id>/events.jsonl live to emit tool-step observations and observed call keys; reads usage-v2.json and normalizes models/usage; resolves status with stricter precedence (empty stdout, non-zero ask exit_code, or failed/cancelled turns → sdk_error); redacts summaries/transcripts; signals and kills the fx process group on abort/exit.
• Harness integration: fxRunner, fxToolAdapter, and harnesses/fxAdapter registered via defineExternalHarness("fx"); supported tool surfaces stagehand_facade (default), playwright_mcp, chrome_devtools_mcp; runs through the shared external runner for prompts, normalized harness_* metrics, and harness status; trajectory adapter converts tool_step events to normalized calls and pairs evidence by call key; model overrides via EVAL_FX_MODELS (default openai/gpt-5.4-mini).
• Permissions/runtime: deny all non-MCP fx built-ins (web_search/web_fetch/file tools included); pre-allow all discovered mcp_<server>_<tool> so --auto never adjudicates; MCP children inherit HOME and pnpm/XDG/proxy caches; startup timeout via EVAL_FX_MCP_STARTUP_TIMEOUT_MS; step-limit detection via fx notices or FX_MAX_AGENT_STEPS; sanitized cleanup logs with enforced temp-root removal.
• Repo plumbing and tests: CI artifacts, Turbo targets, and Vitest include the new package; bench/planner registry recognizes fx; unit tests added for the session layer, runner, tool adapter, and trajectory adapter.
• Requirements to run: install fx ≥ 0.0.3 and set EVAL_FX_PATH; provide AI_GATEWAY_API_KEY or FX_AI_GATEWAY_API_KEY; Node ≥ 22.18.

^{Written for commit 09b9b07. Summary will update on new commits.}

[changeset-bot]

⚠️ No Changeset found

Latest commit: 09b9b07

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

[cubic-dev-ai]

5 issues found across 20 files

Confidence score: 2/5

• In packages/evals/framework/fxRunner.ts, failure states (error/nonzero exit/step-limit) can still be marked successful when structured output is present, which risks downstream consumers treating failed runs as valid results — gate resultText/_success on completed sessions only.
• In packages/integrations/fx-sdk/src/session.ts, clipping stderr before sanitizeErrorMessage() can let credentials slip through near the truncation boundary, creating a concrete secret-leak path in user-visible errors — sanitize first, then clip (or clip in a redaction-safe way).
• In packages/integrations/fx-sdk/src/session.ts, maxAgentSteps can accept fractional input and floor to FX_MAX_AGENT_STEPS=0, which may cause immediate fx failure or skip intended step-limit behavior — enforce integer validation and reject non-integer values before env assignment.
• In packages/integrations/fx-sdk/src/session.ts and packages/integrations/fx-sdk/tests/session.test.ts, generic new Error() for user-facing integration failures and timer-racy polling tests both weaken reliability/diagnostics under failure conditions — switch to the typed integration error and make the polling test synchronization deterministic.

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="packages/integrations/fx-sdk/tests/session.test.ts">

<violation number="1" location="packages/integrations/fx-sdk/tests/session.test.ts:295">
P2: These live-observation tests race the polling loop against a 5 ms timer, so CI scheduling can make them flaky. Replace the short wall-clock delay with deterministic synchronization, or use a substantially longer process delay plus an explicit poll barrier before allowing the process to exit.</violation>
</file>

<file name="packages/evals/framework/fxRunner.ts">

<violation number="1" location="packages/evals/framework/fxRunner.ts:104">
P1: When fx returns an output alongside an error, nonzero exit, or step-limit status, this line lets the structured output set `_success` to true. Only expose `finalMessage` as `resultText` for a completed session so failed runs cannot count as successful.</violation>
</file>

<file name="packages/integrations/fx-sdk/src/session.ts">

<violation number="1" location="packages/integrations/fx-sdk/src/session.ts:325">
P2: Fractional `maxAgentSteps` values produce an invalid zero-step budget. `positiveInteger` accepts `0.5`, then `Math.floor` writes `FX_MAX_AGENT_STEPS=0`, which can make fx fail or bypass intended step-limit handling. Clamp after flooring to at least 1 before writing the env value.</violation>

<violation number="2" location="packages/integrations/fx-sdk/src/session.ts:479">
P2: Custom agent: **Exception and error message sanitization**

Use a dedicated typed integration error here instead of `new Error()`. Rule 2 forbids generic `Error` for user-visible failures, even when the message is sanitized.</violation>

<violation number="3" location="packages/integrations/fx-sdk/src/session.ts:616">
P1: Custom agent: **Exception and error message sanitization**

When stderr contains a credential near the 500-character boundary, `clip()` removes part of it before `sanitizeErrorMessage()` runs, so the redactor can miss the token and return its sensitive prefix in `stopReason`. Sanitize the complete stderr before clipping, and apply the same ordering to assistant and stderr log summaries.</violation>
</file>

Architecture diagram

sequenceDiagram
    participant Runner as Evals Framework
    participant Adapter as FxToolAdapter
    participant SDK as Fx-SDK (runFxSession)
    participant CLI as fx CLI (Process)
    participant FS as File System (~/.fx)
    participant MCP as MCP Server (Stagehand)

    Note over Runner,MCP: NEW: fx Benchmarking Flow

    Runner->>Adapter: prepareToolAdapter()
    Adapter->>FS: NEW: Create per-run HOME & workspace
    Adapter->>FS: NEW: Write mcp.json (tool config)
    Adapter->>FS: NEW: Write settings.json (Wildcard deny built-ins)
    Adapter-->>Runner: Prepared adapter (paths, env)

    Runner->>SDK: runFxAgent()
    SDK->>CLI: NEW: spawn "fx ask --json --auto"
    Note right of CLI: Uses EVAL_FX_PATH bin<br/>Passes AI_GATEWAY_API_KEY

    activate CLI
    loop Live Observation
        CLI->>FS: NEW: Append events to sessions/<id>/events.jsonl
        SDK->>FS: NEW: Tail events.jsonl
        SDK-->>Runner: recordObservation (NormalizedToolCall)
    end

    CLI->>MCP: Call tool (e.g., mcp_stagehand_run)
    MCP-->>CLI: Tool Result
    
    deactivate CLI
    CLI-->>SDK: Exit with final JSON output

    SDK->>FS: NEW: Read usage-v2.json
    FS-->>SDK: Token & cost data

    alt Abort / Process Exit
        SDK->>CLI: NEW: SIGTERM to process group
        opt Grace period expires
            SDK->>CLI: NEW: SIGKILL
        end
    end

    SDK-->>Runner: FxSessionResult (Events, Usage, Status)
    Runner->>Runner: CHANGED: Map fx trajectory to NormalizedToolCall[]
    Runner-->>Runner: Log metrics (fx_total_tokens, etc.)

Loading

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[cubic-dev-ai]

P1: When fx returns an output alongside an error, nonzero exit, or step-limit status, this line lets the structured output set _success to true. Only expose finalMessage as resultText for a completed session so failed runs cannot count as successful.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At packages/evals/framework/fxRunner.ts, line 104:

<comment>When fx returns an output alongside an error, nonzero exit, or step-limit status, this line lets the structured output set `_success` to true. Only expose `finalMessage` as `resultText` for a completed session so failed runs cannot count as successful.</comment>

<file context>
@@ -0,0 +1,170 @@
+      const usage = normalizeFxUsage(sessionResult.tokenUsage);
+      return {
+        raw: sessionResult,
+        resultText: sessionResult.finalMessage,
+        transcriptText: buildFxTranscript(sessionResult.events),
+        iterationError: sessionResult.iterationError,
</file context>

Suggested change

	resultText: sessionResult.finalMessage,
	resultText: sessionResult.status === "completed" ? sessionResult.finalMessage : "",

[cubic-dev-ai]

P1: Custom agent: Exception and error message sanitization

When stderr contains a credential near the 500-character boundary, clip() removes part of it before sanitizeErrorMessage() runs, so the redactor can miss the token and return its sensitive prefix in stopReason. Sanitize the complete stderr before clipping, and apply the same ordering to assistant and stderr log summaries.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At packages/integrations/fx-sdk/src/session.ts, line 616:

<comment>When stderr contains a credential near the 500-character boundary, `clip()` removes part of it before `sanitizeErrorMessage()` runs, so the redactor can miss the token and return its sensitive prefix in `stopReason`. Sanitize the complete stderr before clipping, and apply the same ordering to assistant and stderr log summaries.</comment>

<file context>
@@ -0,0 +1,787 @@
+    const stderr = input.stderr?.trim();
+    return {
+      status: "sdk_error",
+      stopReason: `fx produced no JSON output${stderr ? `: ${clip(stderr, 500)}` : ""}`,
+    };
+  }
</file context>

[cubic-dev-ai]

P2: These live-observation tests race the polling loop against a 5 ms timer, so CI scheduling can make them flaky. Replace the short wall-clock delay with deterministic synchronization, or use a substantially longer process delay plus an explicit poll barrier before allowing the process to exit.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At packages/integrations/fx-sdk/tests/session.test.ts, line 295:

<comment>These live-observation tests race the polling loop against a 5 ms timer, so CI scheduling can make them flaky. Replace the short wall-clock delay with deterministic synchronization, or use a substantially longer process delay plus an explicit poll barrier before allowing the process to exit.</comment>

<file context>
@@ -0,0 +1,474 @@
+      pollIntervalMs: 1,
+      onToolStep,
+      runProcess: async () => {
+        await new Promise((resolve) => setTimeout(resolve, 5));
+        processExited = true;
+        return { stdout: JSON.stringify({ output: "done" }), stderr: "", exitCode: 0 };
</file context>

[cubic-dev-ai]

P2: Fractional maxAgentSteps values produce an invalid zero-step budget. positiveInteger accepts 0.5, then Math.floor writes FX_MAX_AGENT_STEPS=0, which can make fx fail or bypass intended step-limit handling. Clamp after flooring to at least 1 before writing the env value.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At packages/integrations/fx-sdk/src/session.ts, line 325:

<comment>Fractional `maxAgentSteps` values produce an invalid zero-step budget. `positiveInteger` accepts `0.5`, then `Math.floor` writes `FX_MAX_AGENT_STEPS=0`, which can make fx fail or bypass intended step-limit handling. Clamp after flooring to at least 1 before writing the env value.</comment>

<file context>
@@ -0,0 +1,787 @@
+    HOME: input.home,
+    ...(model && { FX_MODEL: model }),
+    ...(positiveInteger(input.maxAgentSteps) && {
+      FX_MAX_AGENT_STEPS: String(Math.floor(input.maxAgentSteps!)),
+    }),
+    FX_PERMISSION_MODE: permissionMode,
</file context>

Suggested change

	FX_MAX_AGENT_STEPS: String(Math.floor(input.maxAgentSteps!)),
	FX_MAX_AGENT_STEPS: String(Math.max(1, Math.floor(input.maxAgentSteps!))),

[cubic-dev-ai]

P2: Custom agent: Exception and error message sanitization

Use a dedicated typed integration error here instead of new Error(). Rule 2 forbids generic Error for user-visible failures, even when the message is sanitized.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At packages/integrations/fx-sdk/src/session.ts, line 479:

<comment>Use a dedicated typed integration error here instead of `new Error()`. Rule 2 forbids generic `Error` for user-visible failures, even when the message is sanitized.</comment>

<file context>
@@ -0,0 +1,787 @@
+    : undefined;
+  let iterationError: unknown;
+  if (resolution.status !== "completed") {
+    iterationError = new Error(stopReason ?? "fx stopped before a normal result");
+    input.logger.warn({
+      category: "fx",
</file context>