Skip to content

Add CAR evaluation metadata and improve structured answer parsing - #490

Merged
ShuxinLin merged 3 commits into
mainfrom
car-evaluation-metadata
Jul 29, 2026
Merged

Add CAR evaluation metadata and improve structured answer parsing#490
ShuxinLin merged 3 commits into
mainfrom
car-evaluation-metadata

Conversation

@ChathurangiShyalika

Copy link
Copy Markdown
Collaborator

Summary

  • Added support for per-scenario groundtruth_eval.json metadata.
  • Updated CAR scoring to check mode key, one-key format, and required term coverage.
  • Added CAR soft score and aggregate reporting fields.
  • Fixed structured answer parsing so final JSON is preferred over earlier parenthesized prose.
  • Added regression tests for CAR metadata loading, CAR scoring, and noisy JSON parsing.

Validation

.venv/bin/python -m pytest src/evaluation/tests/test_static_json_scorer.py src/evaluation/tests/test_loader.py

@ChathurangiShyalika
ChathurangiShyalika force-pushed the car-evaluation-metadata branch 2 times, most recently from 5ce8c15 to 980dd2a Compare July 29, 2026 20:07
Signed-off-by: Chathurangi Shyalika <chathurangishyalika@Chathurangis-MacBook-Pro.local>
Signed-off-by: Chathurangi Shyalika <chathurangijks@gmail.com>
@ShuxinLin
ShuxinLin merged commit 75f79a2 into main Jul 29, 2026
4 of 5 checks passed
@ShuxinLin
ShuxinLin deleted the car-evaluation-metadata branch July 29, 2026 20:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants