Feature request: EvalPort import/export for eval sets and ResultSets

[adhabnr-ux]

Context on why I'm filing this

I maintain EvalPort, an open interchange spec (TestCase / Grader / EvalSuite / ResultSet — JSON Schemas under spec/schemas/, Python/TS SDKs) for moving eval datasets and results between frameworks without a bespoke converter per tool. I read through docs/eval-set-format.md, runner.py, output.py, and samples/eval_set_helm.json before writing this — not just the README — so this is grounded in what agentevals actually does today, not a guess.

Why agentevals is a genuinely interesting fit

Two things stood out that make this different from most eval tools I look at:

1. Your eval set format is Google ADK's EvalSet (EvalSet.model_validate() straight from google.adk.evaluation.eval_set) — you say as much in docs/eval-set-format.md: "eval sets are portable between agentevals and ADK tooling." An agentevals-openeval-adapter would therefore double as an ADK-EvalSet-to-EvalPort adapter for free, which is a bigger surface than agentevals alone.
2. Your whole premise — score pre-recorded OTel traces without re-execution — is the opposite of most eval frameworks I've adapted (DeepEval, Opik, etc. all re-run the app). That's a real, useful distinction to preserve rather than flatten away, and I want to be upfront that it's also where the mapping gets lossy (below).

What a converter would actually look like, based on the real shapes

Import side — ADK EvalCase → EvalPort TestCase (per Invocation, using the exact fields in your eval-set-format.md):

def invocation_to_testcase(eval_id: str, inv_idx: int, invocation: dict) -> dict:
    user_parts = invocation["user_content"]["parts"]
    input_text = " ".join(p["text"] for p in user_parts if "text" in p)

    final = invocation.get("final_response")
    expected_output = " ".join(p["text"] for p in final["parts"] if "text" in p) if final else None

    tool_uses = invocation.get("intermediate_data", {}).get("tool_uses", [])
    expected_tools = [t["name"] for t in tool_uses]

    return {
        "id": f"{eval_id}::{invocation.get('invocation_id', inv_idx)}",
        "input": input_text,
        "expected_output": expected_output,
        "expected_tools": expected_tools,
        # tool_uses carries args/id too — EvalPort's TestCase has no native
        # per-tool-call trajectory field, so the full tool_uses/tool_responses
        # pair goes into metadata rather than being silently dropped.
        "metadata": {"agentevals": {"tool_uses": tool_uses,
                                     "tool_responses": invocation.get("intermediate_data", {}).get("tool_responses", [])}},
        "graders": ["gr_tool_trajectory"],
    }

Export side — your RunResult (runner.py: MetricResult / TraceResult) → EvalPort ResultSet:

def metric_result_to_grader_result(mr) -> dict:
    return {
        "grader_id": mr.metric_name,          # e.g. "tool_trajectory_avg_score"
        "type": mr.metric_name,               # framework-native type, not one of
                                               # EvalPort's 11 well-known ones, so
                                               # the *Grader* definition needs
                                               # params.handler="agentevals" per spec
        "score": mr.score,
        "passed": mr.eval_status == "PASSED",
        "reason": mr.error,
        "metadata": mr.details or {},
    }

def trace_result_to_result(tr) -> dict:
    return {
        "test_case_id": tr.trace_id,          # see honest gap #2 below
        "grader_results": [metric_result_to_grader_result(mr) for mr in tr.metric_results],
        "passed": all(mr.eval_status == "PASSED" for mr in tr.metric_results),
        "metadata": {"num_invocations": tr.num_invocations,
                     "conversion_warnings": tr.conversion_warnings},
    }

Real gaps, not glossed over

1. TraceResult/MetricResult (in runner.py) don't carry the agent's actual output text or trajectory — only scores, status, and details. EvalPort's Result.actual_output would have to stay empty unless the adapter reaches into ConversionResult/Invocation data that isn't currently exposed outside the runner internals.
2. _find_expected_invocations in runner.py matches a trace to an EvalCase by comparing lowercased user-turn text when there's more than one eval case — there's no stable eval_id carried onto TraceResult. So Result.test_case_id in the exported ResultSet would need to fall back to trace_id, which isn't the same identifier space as TestCase.id. That's a real seam, not a five-minute fix.
3. A whole ADK EvalCase.conversation (multi-turn) doesn't collapse cleanly into one EvalPort TestCase — I'd emit one TestCase per Invocation (turn) rather than per EvalCase, which changes the unit of comparison. Worth discussing before committing to a shape.

What I'm asking

Not proposing any change inside agentevals core — per CONTRIBUTING.md this would be a standalone agentevals-openeval-adapter package (same pattern as the opik adapter), built against your existing public EvalSet/RunResult surface with zero changes to this repo. I'm mainly checking whether this is a direction that's actually useful to you before I sink time into it, given gap #2 above is a real modeling question I'd want your input on rather than guessing.

Spec: https://github.com/adhabnr-ux/evalport/blob/main/spec/SPEC.md

— Sahi, independent contributor (not affiliated with this project)