Skip to content

evaluator: validate persisted runs before resuming - #14

Merged
huangyz0918 merged 2 commits into
mainfrom
fix/csv-resume-integrity
Sep 16, 2026
Merged

huangyz0918 merged 2 commits into
mainfrom
fix/csv-resume-integrity

Conversation

@huangyz0918

@huangyz0918 huangyz0918 commented Sep 15, 2026

Copy link
Copy Markdown
Member

Summary

  • Validate CSV dataset identity, value types, and row order before reusing predictions.
  • Fingerprint checkpoints by dataset, predictor model, config, and skip policy while retaining legacy-checkpoint compatibility.
  • Preserve structured and mixed scalar values across CSV round trips.
  • Preserve Boolean comparator scores, legacy JSON key representations, and exact-value legacy pandas numeric CSV formatting.
  • Report all-skipped and empty evaluations as N/A.
  • Accept numeric SVAMP JSON answers.

Why

A same-length reordered dataset or stale checkpoint could previously attach predictions to the wrong rows. Resumed runs could also mix scoring policies, and all-skipped runs could crash while formatting accuracy.

Verification

  • Unchanged repository suite: 123 passed, including PyTorch paths.
  • Additional local regression checks cover reordered datasets, CSV round trips, checkpoint identity and compatibility, empty/skipped results, and numeric SVAMP answers.
  • Only production bug fixes are included. Additional validation code stays local; no credentials or live API calls are involved.

@huangyz0918
huangyz0918 force-pushed the fix/csv-resume-integrity branch from 654d664 to c9997d3 Compare September 15, 2026 23:15
@huangyz0918
huangyz0918 merged commit ed10e9f into main Sep 16, 2026
7 checks passed
@huangyz0918
huangyz0918 deleted the fix/csv-resume-integrity branch September 16, 2026 00:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant