Part of the §23 Driver Validation family (PRD). Prior art: LLM-as-a-judge evaluation practice (rubric scoring + pairwise comparison against a reference; deepeval/Monte-Carlo 2026 guidance) and offline eval pipelines whose primary objective is regression detection — catch quality drift before it ships, not after.
Gap. FR-VAL-01..04 prove the driver runs correctly (happy path, setup, visibility, crash recovery). Nothing measures whether the quality of what it ships degrades over time: a loop that opens green PRs whose diffs are increasingly shallow/tangential passes every existing check. The repo already has lessons scoring ('must-beat-best-so-far' + held-out tier, shipped for prompt ranking) — this extends the same ratchet discipline from prompts to artifacts.
Requirement.
devagent eval score (or a --rubric mode on the existing ledger/analysis surface) scores each shipped PR against a checked-in rubric (.devagent/eval/rubric.md): requirement coverage, test evidence, diff relevance, PRD currency, no-regression claim.- Judge = the configured worker model via the existing
prompt/worker-adapter plumbing; scores land as ledger rows (eval-score event: taskId, PR, rubric version, per-criterion scores, total) — no new event system.
- Pairwise ratchet: a new score below the trailing window's best (per goal class) raises a drift warning in
devagent ledger --clusters and the loop's next research prompt (the failure-class plumbing already exists).
- Nightly CI job scores the last N merged PRs; a drop beyond a configurable threshold fails the job (alert-only first release, gate after calibration).
- Rubric version is recorded with every score so rubric edits can't silently redefine the baseline.
Acceptance. Scoring a known-good and a deliberately shallow PR separates them by a clear margin; introduce a planted quality regression into a fixture PR and the nightly job reports drift with the offending criterion named.
Part of the §23 Driver Validation family (PRD). Prior art: LLM-as-a-judge evaluation practice (rubric scoring + pairwise comparison against a reference; deepeval/Monte-Carlo 2026 guidance) and offline eval pipelines whose primary objective is regression detection — catch quality drift before it ships, not after.
Gap. FR-VAL-01..04 prove the driver runs correctly (happy path, setup, visibility, crash recovery). Nothing measures whether the quality of what it ships degrades over time: a loop that opens green PRs whose diffs are increasingly shallow/tangential passes every existing check. The repo already has lessons scoring ('must-beat-best-so-far' + held-out tier, shipped for prompt ranking) — this extends the same ratchet discipline from prompts to artifacts.
Requirement.
devagent eval score(or a--rubricmode on the existing ledger/analysis surface) scores each shipped PR against a checked-in rubric (.devagent/eval/rubric.md): requirement coverage, test evidence, diff relevance, PRD currency, no-regression claim.prompt/worker-adapter plumbing; scores land as ledger rows (eval-scoreevent: taskId, PR, rubric version, per-criterion scores, total) — no new event system.devagent ledger --clustersand the loop's next research prompt (the failure-class plumbing already exists).Acceptance. Scoring a known-good and a deliberately shallow PR separates them by a clear margin; introduce a planted quality regression into a fixture PR and the nightly job reports drift with the offending criterion named.