Export BenchmarkResult / UnifiedQuestion as EvalPort TestCase + ResultSet (portable eval format)

[adhabnr-ux]

Really like how cleanly separated the Benchmark interface and report.ts phase are here — the UnifiedQuestion / EvaluationResult / BenchmarkResult shapes in src/types/unified.ts map almost directly onto EvalPort, an open interchange format for eval data (TestCase / Grader / Result / ResultSet). Wanted to suggest a thin exporter so a memorybench run can be diffed/compared against runs from DeepEval, Promptfoo, Inspect AI, etc. without a bespoke converter each time — EvalPort ships a real TypeScript SDK (@evalport/sdk, sdk/typescript/src/) with validateSuite/validateResultSet/createResultSet, so this would be a straight dependency, not a spec you'd have to hand-roll.

Concrete mapping, no changes needed to Benchmark or the orchestrator:

• UnifiedQuestion (questionId, question, groundTruth, questionType) → TestCase (id, input, expected_output, tags: [questionType])
• getHaystackSessions(questionId) messages → TestCase.context: string[]
• EvaluationResult (score, label, explanation, hypothesis) → Result.grader_results[] (one GraderResult per judge) + Result.actual_output
• BenchmarkResult.byQuestionType / retrieval / memscore → ResultSet.metadata.openeval.memorybench (a namespaced extension key, same pattern EvalPort already uses for its own openeval.cost reserved key)

Sketch, as a standalone module (e.g. src/exporters/openeval.ts) called from report.ts after generateReport(), so it's opt-in and doesn't touch the pipeline:

import type { BenchmarkResult, EvaluationResult } from "../types/unified"
import type { EvalSuite, Result, ResultSet, TestCase } from "@evalport/sdk"

export function toEvalSuite(evals: EvaluationResult[], benchmark: string): EvalSuite {
  return {
    version: "1.0.0-rc.4",
    id: `memorybench_${benchmark}`,
    graders: [{ id: "gr_judge", type: "llm_judge" }],
    test_cases: evals.map((e): TestCase => ({
      id: e.questionId,
      input: e.question,
      expected_output: e.groundTruth,
      graders: ["gr_judge"],
      tags: [e.questionType],
    })),
  }
}

export function toResultSet(result: BenchmarkResult): ResultSet {
  return {
    version: "1.0.0-rc.4",
    suite_id: `memorybench_${result.benchmark}`,
    run_id: result.runId,
    started_at: result.timestamp,
    runner: { name: "memorybench", version: result.provider },
    results: result.evaluations.map((e): Result => ({
      test_case_id: e.questionId,
      actual_output: e.hypothesis,
      passed: e.label === "correct",
      duration_ms: e.totalDurationMs,
      grader_results: [{
        grader_id: "gr_judge",
        type: "llm_judge",
        score: e.score,
        passed: e.label === "correct",
        reason: e.explanation,
      }],
    })),
    summary: { total: result.summary.totalQuestions, passed: result.summary.correctCount, pass_rate: result.summary.accuracy },
    metadata: { openeval: { memorybench: { memscore: result.memscore, retrieval: result.retrieval } } },
  }
}

For precedent that this converter shape actually holds up against a real framework's output (not just a toy mapping), see adapters/ragas-openeval-adapter in the EvalPort repo — it does the same input/context/expected_output → TestCase normalization from Ragas's EvaluationResult.to_pandas() rows, which is structurally close to what getQuestions() + getHaystackSessions() would feed in here.

Happy to open a draft PR with src/exporters/openeval.ts + a --export-openeval flag on run/compare if this fits where you want the project to go — otherwise no worries either way, just flagging it since the internal types already line up unusually well.

— Sahi, independent contributor (not affiliated with this project)