evals: configurable trace sinks (env + evals config tracing)

[miguelg719]

why

Follow-up to @akeimach's comment on #2758 (#2758 (comment)): the OTEL transport hardcoded the Braintrust project to stagehand[-core][-dev] and offered no way to pick a LangSmith project short of editing code. #2758 shipped the precedence fix for BRAINTRUST_OTEL_PARENT; this PR adds proper sink configurability via env vars and the TUI.

what changed

Shared resolver — resolveBraintrustProjectName(tier) in framework/braintrust.ts centralizes the tier × CI project matrix and honors BRAINTRUST_PROJECT_NAME. runner.ts (native Eval(...), otel parent, summary) and otel.ts (fallback parent) both use it, removing otel.ts's duplicated default. Parent precedence stays BRAINTRUST_OTEL_PARENT > runner-computed > project_name:<resolved>.

evals config tracing — new tracing section in evals.config.json with set <key> <value> / reset [key] / path. Each key is a persisted default for exactly one env var, and the env var always wins:

key	env var	applies to
transport (native | otel)	EVAL_TRACE_TRANSPORT	—
braintrustProject	BRAINTRUST_PROJECT_NAME	both transports
langsmithProject	LANGSMITH_PROJECT	otel only (read natively by the LangSmith exporter)

resolveRunOptions turns the config section into env overrides only when the env var is unset. API keys are never stored in config.

evals doctor — new Tracing block (human + --json) showing the effective transport / braintrust project / langsmith project with its source (env / config / none) and whether LangSmith export is on. Verdict bumps to warn when EVAL_TRACE_TRANSPORT=otel but no sink is configured.

Docs — .env.example documents BRAINTRUST_PROJECT_NAME, BRAINTRUST_OTEL_PARENT, LANGSMITH_PROJECT; evals config tracing help added.

No changeset: @browserbasehq/stagehand-evals is in the changeset ignore list.

test plan

• pnpm --filter @browserbasehq/stagehand-evals typecheck, oxlint, oxfmt --check
• vitest run packages/evals/tests — 56 files / 438 tests pass
  • tests/framework/runner.test.ts: resolveBraintrustProjectName matrix, override, blank override
  • tests/tui/parse.test.ts: tracing config → env overrides; env wins over config
  • tests/tui/tracing.test.ts (new): set/reset/invalid transport/unknown key, other sections untouched, resolveTracingValue precedence
  • tests/tui/doctor.test.ts: tracing block in --json, env-over-config, no-sink warning
• Manual: evals config tracing set braintrustProject <proj> then EVAL_TRACE_TRANSPORT=otel evals run ... lands the experiment in <proj>; BRAINTRUST_PROJECT_NAME=other in the shell overrides it

---

Summary by cubic

Configures trace sinks for eval runs via env and evals config tracing, unifies Braintrust project resolution, and upgrades evals doctor to show effective tracing and warn on misconfigurations. Previously OTEL hardcoded Braintrust projects; now users can set projects without code changes.

• Centralizes Braintrust project selection in resolveBraintrustProjectName(tier). It honors BRAINTRUST_PROJECT_NAME and defaults to stagehand[-core][-dev] by tier/CI; both the runner and OTEL use it. Parent precedence stays: BRAINTRUST_OTEL_PARENT > runner-computed > project_name:<resolved>.

• Adds evals config tracing with set <key> <value>, reset [key], and path. Keys map 1:1 to env vars and env always wins:

  • transport → EVAL_TRACE_TRANSPORT (native | otel)
  • braintrustProject → BRAINTRUST_PROJECT_NAME
  • langsmithProject → LANGSMITH_PROJECT (OTEL only)
    resolveRunOptions applies these as env overrides only when the env var is unset. API keys are never stored in evals.config.json.

• evals doctor adds a Tracing block (human and --json) with value sources. It mirrors resolveTraceTransport() by treating an unrecognized EVAL_TRACE_TRANSPORT as a native fallback and warning, reports LangSmith export as on only under OTEL, and warns when EVAL_TRACE_TRANSPORT=otel but no sink is configured.

• .env.example documents BRAINTRUST_PROJECT_NAME, BRAINTRUST_OTEL_PARENT, and LANGSMITH_PROJECT. The build step preserves user-owned core and tracing sections when regenerating the built evals.config.json.

• Rollout: to use OTEL, set EVAL_TRACE_TRANSPORT=otel and configure a sink via BRAINTRUST_API_KEY or LANGSMITH_API_KEY with LANGSMITH_TRACING=true; optionally set BRAINTRUST_PROJECT_NAME/LANGSMITH_PROJECT or persist via evals config tracing.

^{Written for commit 0734457. Summary will update on new commits.}

[changeset-bot]

⚠️ No Changeset found

Latest commit: 0734457

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

[cubic-dev-ai]

2 issues found across 15 files

Confidence score: 3/5

• packages/evals/tui/commands/doctor.ts can suppress the no-sink warning when the Braintrust key is present only in the package .env, even though the OTEL provider reads only process.env; base the readiness check on the same environment source as the provider to avoid reporting tracing as ready when exports are unavailable.
• packages/evals/tui/commands/tracing.ts prints backslash-separated paths on Windows, making the new tracing path output inconsistent across platforms; normalize the display path to forward slashes before printing.

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="packages/evals/tui/commands/doctor.ts">

<violation number="1" location="packages/evals/tui/commands/doctor.ts:266">
P2: When the Braintrust key exists only in the package `.env`, this condition suppresses the no-sink warning even though the OTEL provider reads only `process.env` and exports nowhere. Base this readiness check on the same environment resolution used by the runner, or load the package value into `process.env` first.</violation>
</file>

<file name="packages/evals/tui/commands/tracing.ts">

<violation number="1" location="packages/evals/tui/commands/tracing.ts:54">
P3: On Windows, `resolveConfigPath` returns backslashes because it uses `path.join`, so the new tracing path output is not slash-stable. Normalize this display path to forward slashes before printing it.</violation>
</file>

Architecture diagram

sequenceDiagram
    participant User as CLI / TUI
    participant Config as evals.config.json
    participant Env as process.env
    participant Runner as Eval Runner
    participant Resolver as Braintrust Resolver
    participant OTEL as OTEL Framework (otel.ts)

    Note over User,Env: Phase 1: Configuration Resolution
    User->>User: evals run / doctor
    User->>Config: NEW: Read [tracing] section
    User->>Env: Read Tracing Env Vars
    
    rect rgb(240, 240, 240)
        Note right of User: resolveRunOptions() Precedence
        alt Env Var is NOT set
            User->>User: NEW: Map Config keys to Env Overrides
        else Env Var is set
            User->>User: Env value wins (Config ignored)
        end
    end

    Note over User,OTEL: Phase 2: Execution & Initialization
    User->>Runner: runEvals(options)
    
    Runner->>Resolver: CHANGED: resolveBraintrustProjectName(tier)
    Resolver->>Env: Check BRAINTRUST_PROJECT_NAME
    Resolver-->>Runner: Return project (Custom > Tier Default)

    alt transport == "otel"
        Runner->>OTEL: initializeTracerProvider(braintrustParent)
        
        rect rgb(230, 240, 255)
            Note right of OTEL: OTEL Sink Resolution Precedence
            OTEL->>Env: Check BRAINTRUST_OTEL_PARENT
            alt BRAINTRUST_OTEL_PARENT exists
                OTEL->>OTEL: Use raw parent override
            else braintrustParent passed by Runner
                OTEL->>OTEL: Use runner-computed project
            else Fallback
                OTEL->>Resolver: NEW: resolveBraintrustProjectName()
                Resolver-->>OTEL: Default project name
            end
        end

        OTEL->>Env: NEW: Check LANGSMITH_PROJECT
        OTEL->>OTEL: Initialize Spans for Sinks
    else transport == "native"
        Runner->>Runner: Native Braintrust Eval()
    end

    Note over User,Env: Diagnostic Flow (evals doctor)
    opt evals doctor
        User->>User: NEW: resolveTracingValue()
        User-->>User: Report source (env | config | none)
        alt transport=otel AND no API keys
            User->>User: NEW: Trigger 'warn' verdict (No Sink)
        end
    end

Loading

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[cubic-dev-ai]

P2: When the Braintrust key exists only in the package .env, this condition suppresses the no-sink warning even though the OTEL provider reads only process.env and exports nowhere. Base this readiness check on the same environment resolution used by the runner, or load the package value into process.env first.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At packages/evals/tui/commands/doctor.ts, line 266:

<comment>When the Braintrust key exists only in the package `.env`, this condition suppresses the no-sink warning even though the OTEL provider reads only `process.env` and exports nowhere. Base this readiness check on the same environment resolution used by the runner, or load the package value into `process.env` first.</comment>

<file context>
@@ -223,10 +255,23 @@ async function buildReport(entryDir: string): Promise<DoctorReport> {
+  const reasons = [...computed.reasons];
+  if (
+    tracing.transport.value === "otel" &&
+    keys.braintrust.state === "missing" &&
+    !tracing.langsmithEnabled
+  ) {
</file context>

[miguelg719]

Partially — the LangSmith side now uses the same resolveKey snapshot as the Braintrust/Keys rows (it was process.env-only, which was the actual inconsistency). The package-.env-vs-process.env gap itself is pre-existing and doctor-wide: every key row reads packages/evals/.env while cli.ts only loads cwd .env, and the snapshot already surfaces this as source: package-dotenv. This warning follows that established convention; reconciling the two loaders is a separate change.

[akeimach]

awesome!

[cubic-dev-ai]

1 issue found across 3 files (changes from recent commits).

Confidence score: 3/5

• packages/evals/tui/commands/doctor.ts propagates sensitive text from EVAL_TRACE_TRANSPORT into the doctor warning and JSON report, risking exposure through terminal output or persisted diagnostics; store only a fixed sanitized error message.

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="packages/evals/tui/commands/doctor.ts">

<violation number="1" location="packages/evals/tui/commands/doctor.ts:176">
P2: Custom agent: **Exception and error message sanitization**

When `EVAL_TRACE_TRANSPORT` contains sensitive text, `transport.invalid` copies it into both the doctor warning and JSON report. Store only a fixed sanitized marker or boolean for invalid values before rendering diagnostics.</violation>
</file>

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>