Skip to content

evaluator: preserve signed GSM8K ground-truth answers - #19

Merged
huangyz0918 merged 1 commit into
mainfrom
fix/gsm8k-signed-ground-truth
Sep 16, 2026
Merged

huangyz0918 merged 1 commit into
mainfrom
fix/gsm8k-signed-ground-truth

Conversation

@huangyz0918

Copy link
Copy Markdown
Member

Summary

Preserve optional +/- signs when extracting GSM8K ground-truth answers, both after #### and in the plain-number fallback.

Bug

During a live OpenRouter smoke test, Gemini 2.5 Flash Lite correctly returned -3 for a temperature changing from 2 to -3. With reference #### -3, ProEval stripped the minus sign, compared against 3, and returned error score 1.0.

When a signed final answer followed earlier numbers in the solution, the #### pattern failed entirely and the fallback could select an unrelated earlier number.

The fix changes only the two extraction regexes. Existing comma handling and final-answer precedence are preserved.

Verification

  • Real OpenRouter/Gemini calls after the fix returned -3 and -1234.5; both now receive error score 0.0, including a comma-formatted reference with earlier reasoning numbers.
  • Existing repository suite: 123 passed.
  • Local regression checks cover 31 extraction/precedence/scoring subcases.
  • Only the two-line production fix is included. Test scripts, logs, API credentials, and result artifacts remain outside the PR.

@huangyz0918
huangyz0918 merged commit 9d37b03 into main Sep 16, 2026
7 checks passed
@huangyz0918
huangyz0918 deleted the fix/gsm8k-signed-ground-truth branch September 16, 2026 00:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant