Repository navigation
ci: stop formatters fighting generated data files - #25
Conversation
The pre-commit gate has been red on `main` for six consecutive runs since 2026-07-25. The failures were not caused by the PRs they appeared on: a clean `main` reproduces them exactly. PR #23 sat blocked for five review passes on a check that had nothing to do with it. Root cause is two formatters disagreeing with two generators, permanently: - `scripts/fetch-model.py:559` writes model-data.json via `json.dump(..., indent=2)`, expanding arrays one element per line. Prettier collapses short arrays. Each pipeline run re-breaks the hook. - `generate_llms_txt()` re-emits upstream OpenRouter descriptions verbatim, which carry smart quotes and trailing whitespace. The trailing-whitespace and fix-smartquotes hooks rewrite them on every run. These files are machine-written and machine-read. Their formatting is decided by the generator, so a style hook there cannot catch a human mistake -- it only reports the generator being itself. Excludes the three generated data files from Prettier via a new .prettierignore, and excludes llms.txt from the two text-fixup hooks. Verified, including the counterfactuals -- an exclusion that silences a check is worse than the red it replaces: - `pre-commit run --all-files` now exits 0 and rewrites nothing. - Deliberately malformatting js/app.js still FAILS Prettier, so real code is still gated. - Appending garbage to model-data.json still FAILS the pipeline test, so the ignored files are still checked for content, just not for style. The security hook (forbid-bidi-controls) is deliberately left applying to everything, including llms.txt.
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: d4702c93ab
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| # by the generator, not by a human editing them. Prettier has no useful opinion | ||
| # here. | ||
| model-benchmarks/data/model-data.json | ||
| model-benchmarks/data/eqbench-upstream-keys.json |
There was a problem hiding this comment.
When eqbench-upstream-keys.json becomes malformed—for example after manual conflict resolution or an interrupted write—this exclusion removes the only configured hook that parses the file as JSON. The benchmark test hook only selects changes under model-benchmarks/scripts or model-benchmarks/tests, and its snapshot tests use temporary files rather than this checked-in snapshot, so a data-only commit can pass pre-commit and then make the weekly watcher fail in load_snapshot(). Add a non-formatting JSON syntax check while excluding these files from Prettier.
Useful? React with 👍 / 👎.
1 participant
Why
The
pre-commitgate has been red onmainfor six consecutive runs since 2026-07-25 (d327614, b0e3b89, f4881b9, b51e3d4, 9a6f687, 1c45350).Those failures were being attributed to whatever PR was open at the time. They were not caused by those PRs — a clean
mainreproduces them exactly. PR #23 (which makes 28 already-committed EQ-Bench scores visible on the site) has sat blocked for five review passes on a check that had nothing to do with it.A gate that fails identically on every branch carries no information. Fixing it is a prerequisite for trusting any other red check in this repo.
Root cause
Two formatters permanently disagreeing with two generators:
fetch-model.py:559model-data.jsonviajson.dump(indent=2)— arrays one element per linegenerate_llms_txt()llms.txtfrom upstream OpenRouter descriptionstrailing-whitespace,fix-smartquotesEvery pipeline run reintroduces the "problem." There was no
.prettierignore.What this does
.prettierignorecovering the three generated JSON data filesllms.txtfromtrailing-whitespaceandfix-smartquotesThese files are machine-written and machine-read. Their formatting is decided by the generator, so a style hook there cannot catch a human mistake — it only reports the generator being itself.
Verification, including counterfactuals
An exclusion that silences a check is worse than the red it replaces, so each was tested:
pre-commit run --all-filesexits 0 and rewrites nothingjs/app.jsstill FAILS Prettiermodel-data.jsonstill FAILS the pipeline testforbid-bidi-controls(security) deliberately left applying to everything, includingllms.txtNote
Style-only, no site output changes — hence
[no-deploy]. Once this merges, the red check on #23 and #24 should clear and become meaningful again.