Repository navigation
fix: regression on artifact rate; grader files that print while loading - #47
Conversation
Pass rates exclude no-artifact runs, so a skill change that stops the agent writing the artifact could pass a regression assertion: proposed at 1/1 plus 1 no-artifact did not count as a regression from 2/2. A regression assertion now also fails when proposed's share of runs that produced the artifact is below baseline's. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
There was a problem hiding this comment.
Nitpick review — approve
The change adds compare-mode artifact-rate gating with accompanying tests covering both the failing and safe directions, plus docs and CLI wiring. The logic is bounded and well-covered; no correctness, security, or supply-chain concerns remain. Approving.
0 inline comment(s).
Verdict was
approve, posted as a comment — this repo does not allow GitHub Actions to approve PRs.
Scenario load parsed all of `promptdiff grade --list` stdout as JSON, so a grader file with a top-level console.log failed to load with a bare SyntaxError. Parse the last line, where the listing always lands, and report unparseable output with an excerpt. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
queso
changed the title
There was a problem hiding this comment.
Nitpick review — approve
The change adds grader-listing validation to the grading pipeline, with the new unparseable-listing error path in grader-file.ts failing loudly with a truncated stdout excerpt for debugging. The logic is straightforward and bounded, and the accompanying test covers the primary success path. No blocking issues; approved.
0 inline comment(s).
Verdict was
approve, posted as a comment — this repo does not allow GitHub Actions to approve PRs.
1 participant
Follow-up to #45. Two fixes.
Regression check counts runs that produced no artifact
This resolves the open question from #45. Pass rates exclude no-artifact runs, so a skill change that stops the agent writing the artifact could pass a regression assertion. For example, proposed at
1/1 pass, 1 no-artifactdid not count as a regression from baseline2/2.A regression assertion now also fails when proposed produces the artifact in a smaller share of its runs than baseline:
Target assertions are unchanged.
Grader files that print while loading
This addresses the nitpick bot's late finding on #45. At scenario load, the whole stdout of
promptdiff grade --listwas parsed as JSON. A grader file with a top-levelconsole.logtherefore failed to load with a bareSyntaxError. Scenario load now parses only the last line, where the listing always lands. If that line is not valid JSON, the error includes an excerpt of the output.Testing
bun run check: 163 tests pass, typecheck clean.🤖 Generated with Claude Code