[ZachFara]

What changed

tools/eval/github_tasks.py gets 1 more verified GitHub task, for 7 in all. Zach wants 20, so this is well short. The rest of the batch's leads failed the bar (see the rejected candidates).

Task	Repo (license)	Upstream release	upstream_change
gh-pydantic-settings-validate-by-name	pydantic/pydantic-settings (MIT)	pydantic 2.14.0, 2026-10-08	pydantic-2.14-model-config-not-resolved (new)

• Pairs: 0. The new key has one task.
  • The repository already has a task, gh-pydantic-settings-signature, but under a different key: that one is pydantic#13730, the private signature helper.
  • Batch 2 rejected this candidate as "the same upstream change". The issue itself cites a different change, pydantic#13825, so it is re-checked here at the coordinator's request.
• Timing:
  • The upstream change is Do not mutate model_config attribute pydantic/pydantic#13825, "Do not mutate model_config attribute". It merged 2026-09-25, and commit 5da36b5 was authored and committed that same day, after the June 2026 cutoff.
  • It is an ancestor of v2.14.0, which was released 2026-10-08.
  • Issue pydantic-settings#1021 was opened 2026-10-09.
  • Fix #1022 was squash-merged 2026-10-13 as 3b405d4.
• Base: the squash commit's parent, f0dd382.
• Fix: it changes code. A new _validate_by_name_enabled in sources/utils.py resolves the config itself: validate_by_name, then the populate_by_name fallback, then the validate_by_alias=False implication. main.py, the env and secrets sources, the Azure source and the CLI source use it.
• Laid over: only tests/test_settings.py.
  • tests/test_utils.py is left out, because it imports the fix's new helper by name.
  • tests/test_source_azure_key_vault.py is left out as well.
• Test command: the two new tests that fail at the base, test_validate_by_alias_false_when_using_name and test_validate_by_alias_false_with_secrets_when_using_name. The PR's third new test (init kwarg priority) passes at the base on 2.14.0, so it is left out.
• Setup: the same as the existing pydantic-settings task, uv pip install --exclude-newer=... -e . pytest==9.1.1, then pydantic==2.14.0.
• Hard-set: not claimed. The error doesn't name a fix, but I didn't try the obvious one.

Base failing output (containerized)

Run	Result
Base, pydantic 2.14.0	2 failed
Fix, pydantic 2.14.0	2 passed
Base, pydantic 2.13.5	2 passed (previous version, checked)

The key line, from both tests:

E       AssertionError: assert 'default' == 'honeycrisp'

The runs went through a .scratch/ helper that calls repos.materialize, repos.prepare and repos.run_tests with container.run, the same steps agent_eval.py check runs. Containers ran one at a time, none were left behind, and --on-host was never used.

Cause-word checks

• Cause words (after the review fix):
  • back into model_config, back to model_config
  • no longer mutates, no longer mutates model_config, mutate model_config, mutates model_config, mutating model_config
  • model_config as declared, model_config no longer, model_config unchanged, leaves model_config
  • no longer writes validate_by_name, no longer sets validate_by_name, stopped writing validate_by_name, stopped setting validate_by_name
  • 2.14
• Rule: every phrase carries its object, model_config or validate_by_name. The first version had bare verbs (written back, no longer writes, write back), which a symptom-only fix matches ("never written back to the field"), and write back also matches "write backend". Those were dropped, along with resolved validate_by_name. no longer mutates is still bare, as the review's list has it.
• What they name: pydantic's change. Since 2.14 it leaves model_config as declared, and no longer writes the resolved validate_by_name back into it.
• Version words: every word but 2.14 is a non-version word.
• Left out: implies and validate_by_name alone, because a fix that only treats the symptom says them ("validate_by_alias=False implies validate by name"). I also dropped 13825, since is_version_word would read it as a version.
• No hits. After the review fix I checked every word with grading v2's mentions against:
  • a fresh containerized base output, on 2.14.0 (2 failed) and on 2.13.5 (2 passed);
  • the whole laid-over tests/test_settings.py at the fix sha (5,388 lines);
  • the base pydantic_settings/ source (every file, from git archive of f0dd382);
  • the symptom words (validate_by_alias, alias, environment variable, default), in both directions;
  • the names of the traceback's test functions and of the functions the fix edits (_settings_restore_init_kwarg_names, _extract_field_info, _get_resolved_field_value, _nested_field_targets, _add_parser_args, __init__) and of the new helper;
  • three symptom-only sentences from the review: "no longer writes the value when validate_by_alias is False", "the env value is never written back to the field", "fall back to the resolved validate_by_name".
• test_no_example_names_a_scenarios_cause in server/tests/test_variants.py passes.
• Leak review: I read the laid-over file's 72 comments, 53 docstrings and 730 names at the fix sha, flagging anything about versions, model_config, resolving, writing, mutating or validate_by_name.
  • The hits are the existing test_populate_by_name_* and test_model_config_* tests, which are already in the base file, plus unrelated comments ("all resolve to the field", forward-reference docstrings).
  • The new tests have no comments or docstrings, and their names state only the symptom.
• New version words: none. 2.14 is already in the expected set of test_the_cause_words_grader_v2_skips_are_exactly_the_versions.

Rejected candidates (this batch)

• Two tests fail on pytest 9.2.0, turning the required pytest-latest check red mikelane/pytest-gremlins#862/#863 (pytest 9.2): the new test asserts the fix, its docstring states the cause, and the other change is a skipif.
• Keep moved-checkout regression compatible with pytest 9.2 xu-hao/pytest-telescope#70/feat(data): audited corrections and exclusions, test keys (T45) #71 (pytest 9.2 rewrite filenames): the fix changes only a test.
• PytestRemovedIn10Warning for base_url with pytest 9.2.0 and pytest-playwright 0.10.0 without pytest-base-url microsoft/playwright-pytest#341/#345 (pytest 9.2 fixture deprecation): the tests need a Chromium install.
• Fix compatibility with pytest 9.2 inmanta/pytest-inmanta#581 (pytest 9.2 option dest): no test change.
• fix: Compatibility with pytest 9.2 schemathesis/schemathesis#5657 (pytest 9.2): no test change; 139 MB repo.
• Apply the open Dependabot updates and fix ConfigProxy for pytest 9.2 Pappmichel/eve-trader#417 (monkeypatch undo): a test name in the laid-over file states the fix; a full app stack.
• pptx2svg.fonts: restore a deleted shared name (pytest 9.2 monkeypatch); 0.3.2 uvrt/pptx2svg#44 (monkeypatch undo): no license.
• Fixes for Pytest 9.2 openforcefield/tyff#220: a lock and pin change.
• fix(zscaler): tests fail with pytest 9.2.0 because of pytest-bdd OpenCTI-Platform/connectors#8070, Cap pytest below 9.2 for pytest-asyncio compatibility temporalio/sdk-python#2113, AttributeError collecting tests when running old version of pytest-asyncio cylc/cylc-flow#7581: pins only.
• pydantic 2.14 TypedDictExtraConfigWarning fires for every closed TypedDict nested under extra='allow' specs d-v-b/pydantic-zarr#15/docs(client): add the CLAUDE.md snippet for agents #16 (pydantic 2.14 TypedDict warning): the fix was never merged.
• Support Pydantic 2.14 fractal-analytics-platform/fractal-task-tools#181/#182: CI and lock only.
• [Bug]: ImportError on startup with pydantic 2.14.0 (eval_type_backport removed from pydantic._internal) dbt-labs/dbt-mcp#921/#923 (pydantic 2.14 removed eval_type_backport): the test docstring states the cause.
• Pinned mcp 1.4.1 fails to import with current pydantic jonemo/openpyxl-mcp-server#13, MCP extra floor mcp>=1.17.0 admits versions that cannot import with current pydantic Oaklight/toolregistry-server#95, mcp version floor is wrong; CI does not test the floor or run on pull requests AkashChatterjee/glin#49, CI: mcp-majors floor leg (mcp==1.17.0) broken by pydantic 2.14 — floor must move to 1.21.1 DustinTrap/kvm-pilot#271, [shared-ci] test failing across Danathar/goodreads-mcp Danathar/goodreads-mcp#326: mcp floor pins only.
• fix(response): nested models lose their aliases in direct responses yeongseon/azure-functions-validation-python#523, bug: ModelFactory with __by_name__ = True ignores field names when Field(alias=...) is used litestar-org/polyfactory#918, Fix cleaning a Quantity in QuantityFormFieldMixin CarliJoy/django-pint#142, Lone annotation like {rbc} is not converted to unity dalito/ucumvert#85: not caused by a release.
• Request schemas publish null for members that reject it vulnwatch/sentinel#1140/#1142: a broad refactor of a large app with a database.
• Pluggy 1.7.0 breaks scancode-toolkit pytest-dev/pluggy#794 (pluggy 1.7): only scancode is affected (far too large), and its fixes are pins.
• Bugfix: pint 0.26 broke the chemistry unit conversions molssi-seamm/seamm_util#77 (pint 0.26): a pin only, no license.
• fix(calc): pin the displayed micro sign instead of taking pint's Squishiba/refdes#47 (pint 0.26 Greek mu): the test docstrings state the cause.
• Fix failure related to the use of U+03BC micro symbol,  hyperspy/rosettasciio#549 (pint 0.26 Greek mu): the laid-over helper normalize_micro states the cause; 144 MB repo.
• Stabilize unit-string multiplication glyph across pint versions beautifulmachines/gpkit-core#241 (pint 0.26 dot operator): snapshot changes, so the base passes on 0.26.
• feat(units): adapt mento to pint 0.26 and lift the dependency cap mento-calc/mento#159 (pint 0.26 typing): a mypy-only change.
• fix: pint 0.26 import and s3fs floor (two items of #86) jmineau/lair#88 (pint 0.26 import): a pyrefly suppression mixed with an s3fs floor.
• Support pint 0.26, update boilerplate, and migrate the cuda platform ttsim-dev/ttsim#156 (pint 0.26 with beartype): no test change for the fix; mostly boilerplate.
• pytest 9.2 approx issues (pyreconstruct#1197, tengri): the approx change landed in June.
• SqlBaseRelation.window does not keep input row order for partitioned windows on DuckDB 2.0 mloda-ai/mloda#2393/#2395/#2406 (DuckDB 2.0): handed to the data-library coder unjudged after the area split.
• Candidates from PRs MIT-618: three verified GitHub tasks (first batch, partial) #164, MIT-618: one more verified GitHub task (second batch) #165, MIT-618: one more verified GitHub task (third batch) #167 and MIT-618: add the marklassian mistune 3.4 GitHub task (batch 4) #168 and batch 5's list were not re-checked, apart from pydantic-settings#1021, now accepted.

Why

MIT-618 (sixth batch), the GitHub-task eval: real post-cutoff tasks for the arms run.

How it was tested

• Containerized runs of the base and the fix on pydantic 2.14.0, and of the base on 2.13.5, as above.
• server/tests/test_eval_grader.py and server/tests/test_eval_repos.py pass (105 tests).
• make check passes after git fetch origin and merging origin/feature/eval-setup (already up to date).

Learned

• One repository can hold tasks for two upstream changes in one release. Check the issue's own cited upstream PR before calling a sibling "the same change": pydantic-settings#1021 cites pydantic#13825, not #13730.
• When a fix PR adds several tests, run each at the base on the new version. Here one of the three new tests already passed at the base, so it stays out of the test command.
• A test file that imports the fix's new helper by name (tests/test_utils.py here) can't be laid over, because the import names the fix. Lay over only the files that test the behaviour.
• is_version_word treats a bare issue number such as 13825 as a version, so an upstream PR number as a cause word would add a version word. Leave it out.
• pint 0.26.0 came out 2026-09-10, with two post-cutoff changes, both merged 2026-07-26: Greek mu for micro (pint#2365) and DOT OPERATOR in pretty printing (pint#2273). Their downstream fixes mostly state the cause in a test docstring or helper.
• pytest 9.2 plugin-API and monkeypatch-undo breakages mostly get pins or test-only fixes downstream. In the pytest, pydantic and pint area, fewer than 1 in 20 searched candidates qualify.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Qc1CQTb8r67J32HpkhS43y