Skip to content

Pull requests: huggingface/lighteval

New pull request New
145 Open 679 Closed
145 Open 679 Closed
Author
Filter by author
Loading
Label
Filter by label
Loading
Use alt + click/return to exclude labels
or ⇧ + click/return for logical OR
Projects
Filter by project
Loading
Milestones
Filter by milestone
Loading
Reviews
Filter by reviews
No reviews Review required Approved review Changes requested
Assignee
Filter by who’s assigned
Assigned to nobody Loading
Sort
Sort by
Newest Oldest Most commented Least commented Recently updated Least recently updated Best match
Most reactions
👍 👎 😄 🎉 😕 ❤️ 🚀 👀

Pull requests list

Restore log-likelihood HellaSwag and PIQA variants
#1315 opened Jul 27, 2026 by kunwar-vikrant Loading…
Fix always-true conditional that made DEFAULT_FORMAT unreachable
#1314 opened Jul 25, 2026 by AbdullahRasheed45 Loading…
Fix litellm judge sending max_tokens as a 1-tuple
#1313 opened Jul 25, 2026 by AbdullahRasheed45 Loading…
1
Add support for Likelihood Accuracy metric in HellaSwag task
#1312 opened Jul 24, 2026 by SauravP97 Loading…
Fix balanced few-shot sampling: falsy label truncation and global RNG use
#1310 opened Jul 20, 2026 by ErenAta16 Loading…
1
Fix sequential few-shot sampling mutating the shared few-shot pool
#1308 opened Jul 20, 2026 by ErenAta16 Loading…
1
fix metric_class names in avg_at_n / avg_at_n_math / maj_at_n test cases
#1305 opened Jul 20, 2026 by jayzuccarelli Loading…
1
1
Fix unreachable response_format default in JudgeLM
#1303 opened Jul 17, 2026 by oraziooztas Loading…
Fix PEFT example and base-model config resolution for adapter models
#1301 opened Jul 17, 2026 by oraziooztas Loading…
1
fix: correct stop_sequence -> stop_sequences typo in _continuous_greedy_until
#1300 opened Jul 17, 2026 by RudrenduPaul Loading…
1
Fix asdiv prompt truncating multi-character gold answers
#1298 opened Jul 14, 2026 by vineethsaivs Loading…
Fix max_tokens sent as a list by the litellm judge backend
#1297 opened Jul 14, 2026 by AbhilakshSinghReen Loading…
1
1
fix: MixEval judge extractors use last-match (verdict-injection defense)
#1295 opened Jul 13, 2026 by AUTHENSOR Loading…
4 tasks done
Use ast.literal_eval instead of eval in hellaswag_arabic_pfn
#1294 opened Jul 11, 2026 by jayzuccarelli Loading…
1
1
Fix string-name normalization in SamplingMetric (inspect.getmembers returns a list, not a dict)
#1292 opened Jul 8, 2026 by teddytennant Loading…
1
Add a regression test for truncated-then-full sample-cache runs
#1291 opened Jul 8, 2026 by shipbehaves Contributor Loading…
1
fix(normalizations): exclude -1 padding from token-norm length + guard mismatch (#1170)
#1290 opened Jul 8, 2026 by WatchTree-19 Loading…
Add Armenian language evaluation suite (ArmBench-LLM)
#1289 opened Jul 8, 2026 by bugdaryan Loading…
1
Batch tokenization in the transformers backend's scoring loop
#1286 opened Jul 6, 2026 by ErenAta16 Loading…
1
Add sample count to the results summary table
#1285 opened Jul 6, 2026 by ErenAta16 Loading…
1
Add support for selecting few-shot examples by id
#1284 opened Jul 6, 2026 by ErenAta16 Loading…
1
1
Make doc builds faster
#1283 opened Jul 6, 2026 by mishig25 Loading…
1
add missing docstrings to three public functions in utils/imports.py
#1282 opened Jul 4, 2026 by RavSinghChandan Loading…
feat: add sample count column to results table
#1280 opened Jul 2, 2026 by SnehaDeshmukh28 Loading…
1
Honor the model's generation_config.eos_token_id in the transformers backend
#1279 opened Jul 2, 2026 by ranawaqas-ai Loading…
1
ProTip! Add no:assignee to see everything that’s not assigned.