fix(benchmarks): plug three eq_bench_public data-integrity gaps

[TechNickAI]

Summary

Follow-up to #21. Three data-integrity / workflow defects flagged by Cursor and Codex bots:

• merge_model drops sources.eq_bench_public on refresh — weekly --refresh rebuilds sources from transform_model() and preserved artificial_analysis + eq_bench but not eq_bench_public. Public scores survived; their provenance flag silently vanished. Added eq_bench_public to the preservation loop.

• apply_public_eq leaves stale fields when upstream entry disappears — if a mapped leaderboard row is removed or renamed, public_eq_block() returns None and the old public_* fields + source flag were left untouched, publishing outdated data. Now clears all PUBLIC_EQ_FIELDS and removes the source flag when no valid block is returned.

• --discover skipped public EQ fetch — discovery is already gated on EQBENCH_PUBLIC_MAP (hand-verified), so every discovered model has a known mapping. Running --discover without --eq-public left new rows with empty public scores. Now auto-applies apply_public_eq scoped to the discovered IDs.

Declined: Codex comment 3653485341 ("render public EQ metric in the UI") — the product deliberately keeps public 17-trait data separate from the displayed local 22-trait v3 metric. The column shows v3_score only; this is by design.

Test plan

• All 47 existing tests pass (uv run --no-project python -m unittest discover -s model-benchmarks/tests)
• Manual: --refresh run on a model with sources.eq_bench_public: true — verify flag survives
• Manual: --discover on a clean dataset — verify public EQ scores are populated for discovered models without needing --eq-public

🤖 Generated with Claude Code

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: be8607a5cd

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Include dry-run discoveries before applying public EQ

When --discover is combined with --dry-run, the fetch loop only prints each transformed model and never merges it into data, so this call filters over a dataset containing none of the IDs in discovered. It therefore performs both EQ-Bench requests but always reports zero updates and shows no public fields for the new models, preventing the dry run from validating the enrichment this branch adds. Merge the temporary models into an in-memory copy for this step or skip/clearly report the enrichment during dry runs.

Useful? React with 👍 / 👎.

[TechNickAI]

Review from the benchmark steward pass (2026-07-28). One blocking hazard, one now-redundant hunk, one good catch worth keeping.

⚠️ Blocking: the stale-clearing hunk is a data-loss landmine

The new block is None branch in apply_public_eq deletes PUBLIC_EQ_FIELDS + the source flag. That is correct when a row genuinely disappears upstream, but public_eq_block() also returns None when the leaderboard fetch is empty or degraded — and it cannot distinguish the two cases.

Measured against the current dataset:

• Upstream healthy right now: 0 rows would be cleared (nothing is actually stale).
• Simulating one bad/empty fetch: 28 of 28 rows are wiped.

So the hunk buys nothing today and costs the entire public EQ dataset the first time eqbench.com 500s, times out, or ships a format change that breaks the parser. That is precisely the destructive-refresh class the charter forbids ("never let a refresh destroy existing data") — same shape as the AA wipe this pipeline already had to be fixed for once.

Suggested guard before this merges: only clear when the fetch is known-good, e.g. bail out if the leaderboard returned zero rows, and require the model's mapped key to be genuinely absent from a leaderboard with a plausible row count. Deleting verified data should require positive evidence of staleness, never the absence of evidence.

ℹ️ Redundant: the merge_model whitelist hunk

Fixed on main in d327614, generically. Rather than appending eq_bench_public to the tuple, that commit removed the whitelist entirely and preserves every truthy non-openrouter source key — because the whitelist pattern was the bug, and a three-item tuple fails again the next time a source is added.

It also restored the 28 flags that 72b7149 destroyed (set only where a real public_rubric_0_100 exists; flag set == score set, verified 28/28 against live upstream), and added 6 regression tests that were control-checked: they fail against the old whitelist code and pass against the fix. A real --refresh --no-aa run now preserves 28/28 flags where it previously preserved 0.

This hunk will conflict with main. Recommend dropping it and rebasing.

✅ Keep: the --discover auto-fetch

Good catch, and the reasoning holds — discovery is already gated on the hand-verified EQBENCH_PUBLIC_MAP, so a discovered model always has a verified mapping and leaving its public fields empty was just a gap. The discovered = [] initialization fixes a real UnboundLocalError too.

Net: add the fetch-health guard to hunk 1, drop hunk 2, keep hunk 3.

[TechNickAI]

Superseded by main. The fix in PR #22 (preserving eq_bench_public source flag) was incorporated more robustly in commits d327614 and ca3bfd8 on main: the hardcoded key list was replaced with a generic loop over all non-openrouter source keys. PR #22's change is already on main, in better form.