Repository navigation
Enhancement: Eval history observability with repo SHA provenance and judge drift markers #4
Description
Summary
Implement eval-history observability for multi-model comparisons so historical trends remain useful while clearly signaling comparability caveats.
Problem
Current comparison outputs are useful per run, but decision-makers need reliable trend visibility over time. With LLM-as-judge, score changes can come from model-under-test changes or judge/eval regime changes, which can make historical interpretation ambiguous.
Proposed enhancement (v1)
Capture and report score, cost, and duration over time with run provenance and event markers.
1) Persist run provenance metadata
Add these fields to persisted run artifacts (JSON and CSV):
run_at(ISO timestamp)repo(owner/repo)git_sha(commit evaluated)judge_providerjudge_modeljudge_version(when available)judge_prompt_hasheval_hash(or per-eval spec hash)comparison_fingerprint(stable hash of judge+prompt+eval config)
2) Capture decision metrics
For each model/eval run, persist:
- quality score (existing pass-rate metrics)
- total/normalized cost (if available from provider/runtime)
- total duration / wall-clock time
3) Historical trend reporting
Add/extend report generation to show time-series trends per model/eval with two modes:
- Strict comparable: same
comparison_fingerprint - All runs: includes all historical runs with caveat labeling
4) Regime-change markers
In trend outputs (charts/tables), show vertical event markers when any of these change:
- judge model/version
- judge prompt hash
- eval hash/spec
These markers should behave like release/event bars so jumps are interpretable.
Why
- Preserves all historical data (no loss of context)
- Improves reproducibility with repo+SHA provenance
- Makes drift explicit rather than hidden
- Supports practical model decisions using quality + cost + speed together
Acceptance criteria
- JSON/CSV artifacts include provenance metadata fields listed above
-
repoandgit_shaare captured for every run - score, cost, and duration are persisted for each model/eval run
- report supports strict-comparable and all-runs modes
- regime-change markers appear when judge/prompt/eval fingerprints change
- each trend point can be traced back to run timestamp, repo SHA, and judge config
Non-goals (v1)
- statistical normalization across different judge versions
- mandatory human re-judging workflow
- full BI/dashboard stack
References
docs/eval-comparison.md- Feedback: preserve historical numbers, record judge/date/SHA, and mark major judge/prompt changes as vertical bars.
Activity
coston commentedon Aug 31, 2026
Operator has opened a pull request.
PR: #5
Status: Awaiting review
coston commentedon Aug 31, 2026
Implementation Plan
Files to modify:
src/eval/types.ts— addRunProvenance,costUsd/totalCostUsdfields onTestResult/EvalRun,provenancefield onEvalComparisonsrc/eval/runner.ts— captureoutput:costevents fromrunAgentsorunPromptreturns{ output, costUsd }src/eval/index.ts— thread cost throughrunEval, build provenance once per eval-file run, attach it to single- and multi-modelEvalComparisonobjects, add--history <path>flag that appends each run to a JSONL history logsrc/eval/export.ts— add provenance +cost_usdcolumns to CSV, restorecostUsdinloadExistingResultssrc/eval/report.ts— addprintTrends()for the terminal trend view with regime-change marker rowsdocs/eval-comparison.md— document provenance fields, cost tracking,--history, and the neweval:trendcommandpackage.json— addeval:trendscript; haveeval:comparepass--history results/eval-history.jsonlsrc/tests/eval-comparison.test.ts,src/tests/eval.test.ts— update fixtures/assertions for the new CSV columns andrunPromptreturn shapeFiles to create:
src/eval/provenance.ts—getGitSha,getRepoSlug,getJudgeVersion,hashContent,getJudgePromptHash,getEvalHash,computeComparisonFingerprint,buildProvenancesrc/eval/history.ts—appendHistory/loadHistory(JSONL),buildTrends(strict-comparable vs all-runs modes, regime-change marker detection per group)src/eval/trend-index.ts—npm run eval:trendCLI entry pointsrc/tests/eval-provenance.test.ts,src/tests/eval-history.test.ts— unit tests for the new modulesSteps:
src/eval/provenance.ts(git SHA/repo viagitCLI, judge model/version, sha256 hashes of the judge prompt and the resolved eval spec, and a stablecomparisonFingerprintcombining them).runPrompt/runEval(Claude only — OpenCode has no cost signal, leftundefined) and roll it up intoEvalRun.totalCostUsd.provenanceto everyEvalComparisonbuilt inrunEvalFile(single- and multi-model paths) and extend JSON/CSV export with the provenance + cost fields.src/eval/history.tsto append one JSONL record per model/eval per run (score, cost, duration, provenance) behind an opt-in--history <path>flag, plusbuildTrendsto group records by eval+model, sort by time, support strict-comparable (matching the latestcomparisonFingerprint) vs all-runs modes, and flag regime-change points.printTrends(report.ts) and theeval:trendCLI (trend-index.ts) to render the time series with a marker line wherever the judge/prompt/eval fingerprint changes.docs/eval-comparison.mdandpackage.jsonscripts.Note on scope: the separate end-to-end workflow-eval harness (
src/eval/workflow.ts, git-worktree based) is left untouched — this issue anddocs/eval-comparison.mdare about the criterion-judge multi-model comparison system, so provenance/history land there.fix: enhancement eval history observability with repo sha provenance …
chore(release): 2.14.0 [skip ci]
github-actions commentedon Aug 31, 2026
🎉 This issue has been resolved in version 2.14.0 🎉
The release is available on GitHub release
Your semantic-release bot 📦🚀