Interop idea: map EvaluationExample/EvaluationResult onto EvalPort's TestCase/Result for portable benchmark output

[adhabnr-ux]

Hi — I maintain EvalPort, an open, vendor-neutral schema (TestCase / Grader / Result / ResultSet / GraderResult) for making LLM/VLM eval results portable across frameworks. Read through rewardharness/domain.py and rewardharness/benchmark.py and think there's a clean, small interop win here — flagging per CONTRIBUTING.md before writing any code.

What I saw: EvaluationExample (source_img, edited_a, edited_b, prompt, group_id, ground_truth: Preference) and EvaluationResult (preference, scores: ScoreCard{a_instruction,a_quality,b_instruction,b_quality}, reasoning, chain) are already a clean, typed pair — and benchmark.py's pair_results (group_id, prompt, gt, prediction, correct, reasoning_chain, scores) is basically a ResultSet shape wearing different field names. Emitting benchmark_results.json as an EvalPort ResultSet alongside the current format would let anyone diff RewardHarness's K=2/3/4 runs against other reward-model / LLM-judge benchmarks without a bespoke parser.

Proposed mapping:

# rewardharness -> EvalPort  (pip install evalport-sdk; import openeval)
from openeval.types import TestCase, Grader, Result, GraderResult

def to_testcase(ex: EvaluationExample) -> TestCase:
    return TestCase(
        id=str(ex.group_id),
        input=ex.prompt,
        graders=["pairwise_preference"],
        expected_output=ex.ground_truth.value if ex.ground_truth else None,
        # SPEC.md's TestCase has no first-class image field yet, so the
        # candidates travel in metadata for now — flagging this in case
        # it's worth a schema issue on the EvalPort side too.
        metadata={"source_img": ex.source_img, "edited_a": ex.edited_a, "edited_b": ex.edited_b},
    )

def to_result(ex: EvaluationExample, out: EvaluationResult) -> Result:
    correct = ex.ground_truth is not None and out.preference == ex.ground_truth
    sc = out.scores
    norm = lambda v: (v - 1) / 3  # RewardHarness's 1-4 Likert -> EvalPort's [0,1] score range
    grader_result = GraderResult(
        grader_id="pairwise_preference",
        type="llm_judge",
        score=norm((sc.a_instruction + sc.a_quality + sc.b_instruction + sc.b_quality) / 4),
        passed=correct,
        reason=out.reasoning,
        metadata={"a_instruction": sc.a_instruction, "a_quality": sc.a_quality,
                  "b_instruction": sc.b_instruction, "b_quality": sc.b_quality},
    )
    return Result(
        test_case_id=str(ex.group_id),
        passed=correct,
        grader_results=[grader_result],
        actual_output=out.preference.value,
    )

run_benchmark() in benchmark.py already builds almost this exact dict per pair (group_id, gt→ex.ground_truth, prediction→out.preference, correct) — the K-group loop would just need a second writer next to benchmark_results.json.

Precedent: adapters/deepeval-openeval-adapter in the EvalPort repo does the same kind of "structured per-test-case metric output → Grader/GraderResult" mapping for DeepEval's metrics list, so there's a working pattern to crib the shape from rather than inventing one here.

Happy to open a PR for a standalone rewardharness/interop/evalport.py (zero new runtime deps — evalport-sdk is dependency-free) if this is something you'd want, or happy to drop it if it's out of scope for a COLM-camera-ready research codebase. No pressure either way — mostly wanted to flag the mapping while it was fresh, per the "open an issue first" note in CONTRIBUTING.md.

— Sahi, independent contributor (not affiliated with this project)