Run BabelTest assertions embedded in GitHub issues as pytest tests - #103
Merged
Conversation
GitHubIssueTest / GitHubIssuesTestCases pull BabelTest assertions embedded
in GitHub issue bodies (wiki {{BabelTest|...}} markers or fenced YAML
blocks), resolve each to an assertion handler from the assertions package,
and evaluate it against NodeNorm/NameRes. Add the dependencies this needs:
pygithub, pyyaml, tqdm and python-dotenv.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Dynamically generate one pytest per GitHub issue (across the repos listed in targets.ini [DEFAULT] Repositories), each running the issue's BabelTest assertions as independent subtests. Open-issue failures are reported as xfail so they don't block CI. Add a --issue option (and selected_github_issues fixture) to restrict a run to specific issues, wipe the issues cache at session start, parallelize with pytest-xdist, and add a PR workflow that runs the offline `-m unit` suite. tests/github_issues/ test_system.py covers the parsing/dispatch logic with mocked GitHub data. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This was referenced Jun 26, 2026
4 tasks
# Conflicts: # tests/conftest.py
Arity was checked by each handler at the top of test_params_list(), which runs
after prepare_params_lists() has already warmed the NodeNorm cache. A
params_list of the wrong length can never pass, so that lookup was always
wasted: {{BabelTest|SearchByName|water|CHEBI:15377|unexpected}} normalized
CHEBI:15377 before rejecting the assertion for having three params.
Declare the bounds instead, as MIN_PARAMS/MAX_PARAMS alongside PARAMETERS, and
check them in _rejection() before CURIE validation. Arity now sits with the
rest of what the class says about its params rather than buried in a method
body, and five hand-rolled checks go away. Ordering it first also removes a
wart in SearchByName.curie_params(), which had to slice rather than index
specifically so a malformed params_list could survive validation long enough
for test_params_list() to report the arity problem.
Messages are generated from the bounds, preserving the "exactly two" and "at
least two" wording the existing tests pin. display_name() moves to
AssertionHandler so gen_docs.py and the failure messages share one definition.
Adds tests that a rejected params_list issues no NodeNorm request, and that a
well-formed one still gets pre-warmed. Disabling the new check fails four of
them.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two ways a well-meaning issue body could break a run:
- {{BabelTest| Resolves |CHEBI:15365}} kept the padding in the assertion
name, so it missed ASSERTION_HANDLERS and the test hard-failed as an
unknown assertion type — bypassing the open-issue xfail path entirely.
The name is now stripped, and a name that is only whitespace raises the
"Missing assertion name" error rather than sneaking through.
- YAML 1.1 resolves an unquoted `no`, `on` or `1.5` to a bool or a float,
so `HasLabel: [CHEBI:X, no]` reached prepare_params_lists and died on
param.strip() with an opaque AttributeError. _to_str_list now rejects a
non-string leaf where we can name the offending value and say to quote it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
get_issues_by_ids only caught UnknownObjectException in the bare-'N' branch, so `--issue org/repo#99999` let PyGitHub's 404 escape out of pytest_generate_tests as a raw collection error. The 'org/repo#N' and 'repo#N' branches now fall through to the "Could not resolve issue ID ..." ValueError that was written for exactly this case. The loop variable is renamed to raw_id, making room for the module-level issue_id() helper added next. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
For an issue that came out of the search API, PyGitHub's .repository getter first completes the issue (a full GET, since search payloads omit the repository) and then returns an incomplete Repository whose .full_name triggers a second GET — two extra REST calls per issue, on every collection and in every xdist worker. html_url already spells out owner and repo. Both call sites now share one issue_id() helper rather than formatting the ID by hand. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
get_all_issues() has had no callers since get_issues_with_tests() replaced it, and importing its progress bar was the only reason tqdm was a dependency. It also logged its "Found %d issues" summary from inside a generator, so the count never printed unless a caller exhausted it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
_fetched_issues_cache was read by the fixture but only ever written on the network path in _get_all_test_issue_ids. Whenever the JSON file cache was hit — i.e. every xdist worker after the first — each test refetched its own issue individually, and refetched it again for every --target. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…test pytest_configure unlinked babel_validation_issues_cache.lock along with the cache itself. A concurrent pytest run holding that lock while it fetched from GitHub would keep its now-unlinked inode, while this run created a fresh one — two processes inside "the" lock at once. The lock is left alone now; only the cache file is cleared. The two filenames were also duplicated as string literals in the root conftest, with nothing tying them to the definitions in the github_issues conftest, so renaming the cache there would have silently stopped the cleanup. Both paths now live in tests/_pytest_helpers.py. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
nodenorm-es has completely replaced the nodenorm.ci that [ci-redis] names, so [ci] tracks it as what CI actually runs. [ci-es] pairs the same NodeNorm with the Elasticsearch NameLookup this repository is validating, and will be folded back into [ci] once namelookup-es is signed off. Reading like a copy-paste slip, this had already been flagged once in review. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
gaurav
force-pushed
the
split/3-github-issues
branch
from
August 19, 2026 18:02
068608f to
b91805c
Compare
nodenorm.ci.transltr.io no longer exists — nodenorm-es has completely replaced it — so [ci-redis] only made `pytest --target all` hit a dead endpoint. The Vue validator offered the same dead URL under "NodeNorm-ITRB-ci" and now points at nodenorm-es too. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Declaring a permissions block sets every unlisted scope to none, so the job stops inheriting whatever the org's default-permissions setting happens to be. Only actions/checkout needs anything, and it needs contents:read. The GITHUB_TOKEN passed to the pytest step was never read: test_github_issue carries no `unit` marker, so `-m unit` deselects it and deselected_by_markexpr short-circuits the parametrization before conftest looks for a token, while the unit tests build their own parser with a dummy one. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Spells out what to add back, and the constraint that is not obvious from the workflow: github.token is scoped to this repository, so the other Repositories in targets.ini are reachable only because they are public. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Without GITHUB_TOKEN the GitHub issue tests skip at the module level, so dropping the -m unit filter without restoring the env: block would leave CI green having tested nothing. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The top-level README covered the Google Sheet tests but never mentioned the GitHub issue tests, the GITHUB_TOKEN they need, or why they need it. A token is not required for authentication — the scanned repositories are public and both the single-issue and search endpoints answer unauthenticated requests — but the unauthenticated budget of 60 core requests an hour is counted per IP, which a run cannot stay inside. Measured rather than assumed: discovery of all 96 issues costs zero core requests, because search results already carry the body and html_url the harness reads; core requests are spent re-hydrating issues one at a time when the cached ID list is reused. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Both behaviours changed in this branch without direct coverage. The issue_id tests pin the html_url parsing and assert the helper never touches .repository, which is the whole point of it. The get_issues_by_ids tests cover all three ID formats plus an unparseable one, since the 404 previously escaped from two of the three branches. Also records two things worth knowing before touching this code: the repo is not black-clean, so formatting the whole tree buries real changes; and `GITHUB_TOKEN=` (empty) is how to test the no-token path, because load_dotenv() will not override a key already in os.environ. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This was referenced Aug 20, 2026
gaurav
changed the title
Issue bodies are untrusted input: anyone can write one, nothing reviews it, and we parse it and turn it into live NodeNorm and NameRes calls. Two of the gaps were reproducible denial of service, not theory. The babel_tests block pattern ended `babel_tests:\s+.*?\s+```. Those three nested backtracking quantifiers made matching cubic on a body that opens a block and never closes the fence: 6.8s at 4KB, 53s at 8KB, hours at GitHub's 65536-character limit. It runs in issue_has_tests() during collection, which pytest-timeout does not cover, so one such issue hung the whole run before any test started. Anchoring on the newline that follows `babel_tests:` in a real fenced block removes the ambiguity: 0.0008s at 65536 characters, matching byte-identical text. It also stops matching a one-line ```yaml babel_tests: ``` written in prose while discussing the syntax, which the old pattern picked up and then failed the issue on (#100). safe_load blocks code execution but still resolves aliases, and PyYAML shares the aliased nodes rather than copying them, so the load looks cheap and the cost lands on whatever formats the result. A 337-byte body of chained anchors reached 25 MB in an error message. Aliases are now refused, which covers merge keys too. Duplicate keys go with them: YAML keeps the last silently, so the block a reviewer reads would not be the one that runs. Also: caps on body length, assertions, param sets and parameters per issue, counted across both syntaxes, since one test item runs all of it — exceeding one fails the issue rather than running part of it, and the fix is to split it up. Assertion names are shape-checked where both syntaxes meet. Issue text is repr()'d into log lines, which escapes exactly what str.isprintable() rejects, so an ANSI escape or bidi override cannot reach an operator's console. __str__ no longer dumps param_sets, which was repeated into every TestResult message. And --issue resolves only within the configured repositories, checked before get_repo() because the pattern's repo group admits slashes and would otherwise let 'org/repo/../../elsewhere#1' reach the API as a URL path.
…service AssertionHandler._rejection() is the one choke point every handler routes through, and the only check that sees every param: _CURIE_RE skips whatever curie_params() excludes — notably SearchByName's free-text query, the one value that reaches a URL query string — and handlers with VALIDATE_CURIES off skip it entirely. The checks run before the arity and CURIE checks, because both of those interpolate the params into their message. MAX_PARAM_LENGTH is 1000 rather than something tighter: CURIEs run under 100 characters and Biolink types under 60, but a chemical label can be long — IUPAC names run well past 255 — and rejecting those would be a false positive on real data. str.isprintable() covers ANSI escapes, C0/C1 controls, bidi overrides and zero-width characters in one call. A bad param fails only its own param set, so the rest of the issue still runs.
Both caches used fixed names in the world-writable temp directory, so on a multi-user machine or a CI runner anyone could pre-create one as a symlink and have us overwrite whatever it pointed at, or simply rewrite its contents. That second one matters more than it looks: the GitHub issue cache holds the issue IDs a later run fetches and executes assertions from, so being able to write it was close to being able to choose what the run tests. A directory under the user's own home, created 0700, has neither exposure — it removes the problem rather than mitigating it. BABEL_VALIDATION_CACHE_DIR overrides it for a runner without a writable home.
The load-bearing one is test_unterminated_yaml_block_is_fast: a 40KB hostile body must parse in under a second, where the old pattern took hours. The budget is deliberately generous, because what it catches is a difference between milliseconds and hours rather than a slow regression. The rest pin the asymmetry the guards are built around — structural caps fail the whole issue, a bad param fails only its own param set — and the two halves that are easy to lose: that a rejected param never reaches the service, and that it is escaped in the message rather than merely rejected. Also records something found while writing them: PyYAML's reader refuses control characters outright, so the YAML syntax never reaches our isprintable() check and the wiki syntax is the path that needs it.
The caps belong where issue authors will meet them, so they go in the generated assertion reference rather than only in the code. Regenerated with gen_docs.
It deletes whatever it is handed, and it runs from pytest_configure before anything else in the session. Both callers today build their paths from cache_dir(); the check is so a later one cannot quietly turn a cache sweep into a delete of something that matters. An OSError other than a missing file — a directory left where a cache file belongs, a permissions problem — now warns instead of failing the run before it starts. A stale cache is not worth that. Not a symlink fix: os.unlink removes the link rather than following it, so this was never an overwrite vector. It was the write side that could be redirected, and moving the caches out of the shared temp directory closed that.
gaurav
changed the title
The guards added across this branch are only useful if the next person writing a parser knows why they are there. Each entry is a failure mode that was real in this code, phrased as the shape to look for rather than as a description of the fix: adjacent backtracking quantifiers, safe_load still expanding aliases, %s where %r was needed, a guard placed after the value was already logged, an outside identifier choosing what we fetch, and a cache in the shared temp directory. Also states which inputs are trusted — targets.ini is checked-in config — since "validate everything" and "validate the right thing" are not the same advice.
An open issue whose assertions all pass is a strict XPASS meaning it looks closeable, and a closed issue with failing assertions means it looks reopenable. Both are findings about Babel rather than defects here, so the repair is to the issue, not to the assertion. Without this written down the obvious reading of 18 red tests is that something needs fixing in this repository.
test_cache_dir_is_not_world_writable_temp pointed BABEL_VALIDATION_CACHE_DIR at a tmp_path and then asserted the result was not under the temp directory, which contradicts itself. It passed locally only because pytest resolves tmp_path to /private/var on macOS while gettempdir() reports /var, so the parent check never matched; on Linux CI, where tmp_path really is under /tmp, it failed. Split into the two things actually worth asserting: the default location is under the user's home and not the shared temp directory, and a created cache directory carries no group or other permissions. Both fail if cache_dir() moves back to gettempdir().
Neither had a test. The log line matters because it runs before anything has validated the text, so it is the last place an escape sequence can be stopped before it reaches a terminal. The assertions read caplog.records rather than caplog.text: pytest does not carry a raw control character through caplog.text, so a test written against it passes whether the code escapes or not. Both tests were checked by reverting the behaviour and confirming they fail — worth doing here, because %r and %s are the same length, so restoring the file left a same-size source whose .pyc was not invalidated and the first attempt appeared to pass a mutation it had not seen.
An unset HOME is not the problem it looks like — Path.home() falls back to the pwd database — but a read-only home on a locked-down runner or in a container is, and the bare PermissionError names a path without hinting that BABEL_VALIDATION_CACHE_DIR exists, which is the only reason it does. Both cases are pinned, the second skipped under root, where the permission bits it relies on are ignored and it would fail rather than skip.
48b1c44 stopped unlinking the GitHub issue cache's .lock, because a concurrent pytest holding it while fetching would keep its now-unlinked inode while this run created a fresh one, putting two processes inside "the" lock. The Google Sheet sweep had the same shape — it derived `<name>.lock` from each `<name>.csv` and unlinked both — and was missed at the time. Only the .csv files are cleared now, which is all that has to be fresh.
Both cost real time this round and both leave a test that looks verified and is worthless: a same-length source edit whose .pyc is never invalidated, so the mutation is never actually run, and caplog.text, which does not carry control characters through and so passes whether the code escapes them or not.
At 605 lines it was the largest test file in the repo, and "system tests" named none of the three things it covered: what the two syntaxes mean, how issues are discovered and identified, and the guards on untrusted body content. Those are now test_syntax.py, test_discovery.py and test_untrusted_input.py. A subdirectory rather than three siblings, because of the fixture. The parent conftest defines a session-scoped github_issues_test_cases that needs a real GITHUB_TOKEN, and these tests override it with a dummy-token parser. Three flat files would each need their own copy of that override, and it cannot move up to the parent conftest without replacing the real fixture for the live test_github_issues.py as well. A conftest here scopes it to exactly these files. _mock_issue() sits in _helpers.py rather than conftest.py: it is a plain function, not a fixture, so it has to be imported rather than discovered. A pure move — every test body is verbatim and only the imports differ. Verified by diffing collected node IDs with the file path stripped, before and after: identical, 97 either way.
Every one of the 13 @pytest.mark.unit class decorators was already implied by the module-level pytestmark, and two classes declared a local `fixture` that rebuilt the dummy-token parser the shared github_issues_test_cases fixture already provides. Removing those orphaned a _REPOS constant and the GitHubIssuesTestCases import along with them. Kept separate from the move so that one stays reviewable as a pure move. The collected node IDs are unchanged either way.
This repository is in the scanned Repositories list, so a complete marker written into an issue body is collected and run — an issue describing an assertion becomes a test of it, and since a new issue is open, an assertion that passes reports as a strict XPASS failure. Issue #115 was filed during this work with a marker in its explanation of the size caps, and failed the live suite on the next run. The README warns about this for anyone discussing the syntax; CLAUDE.md is where an agent about to file an issue will actually look.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Test cases for Babel currently live in a Google Sheet, which is fine for bulk regression data and poor for anything tied to a specific bug: the sheet has no idea which issue a row came from, so nothing tells us when a fixed issue regresses or when an open one has quietly started working. This PR lets a test case live in the GitHub issue that motivated it, and runs those cases as pytest tests.
An issue body carries assertions in either of two syntaxes — a wiki-style
{{BabelTest|Resolves|CHEBI:15365}}marker, or a fenced YAML block whose top-level key isbabel_tests:— and the harness turns each issue into one pytest item, with each assertion a subtest. Open issues are expected to fail, so an open issue whose assertions all pass is reported as a strict XPASS: the tool is telling you the issue looks closeable. Closed issues are expected to pass, so one that starts failing is telling you to reopen it.Stack 3 of 4 splitting #67. #101 and #102 have merged, so this now bases on
main; #104 is the remaining piece. This leaves #67 open, which stays open for #104.What's here
Parsing —
src/babel_validation/sources/github/.GitHubIssuesTestCasesfinds assertion blocks in issue bodies and resolves each to a handler from the assertions framework that landed in #102. Issue discovery uses the GitHub search API rather than paginating every issue, and readsbodyandhtml_urlstraight off the search result, so scanning the five configured repositories costs two search requests per repo and no core requests at all.Harness —
tests/github_issues/. One pytest item per issue, parametrized by issue ID, across the repositories listed underRepositoriesintargets.ini's[DEFAULT]section.--issuetargets a specific issue byorg/repo#N,repo#NorN. Issue IDs and hydrated issues are cached and shared acrosspytest-xdistworkers behind aFileLock.Input validation. Issue bodies are untrusted: anyone with a GitHub account can write one, nothing reviews it, and we parse it and turn it into live NodeNorm and NameRes calls. That is tolerable while a human watches the run and can hit Ctrl-C; it is not tolerable for the unattended daily runs this is meant to enable. Two of the gaps were reproducible denial of service rather than theory:
babel_testsblock ended with\s+.*?\s+before the closing fence. Those three nested backtracking quantifiers made matching cubic on a body that opens a block and never closes the fence — 6.8s at 4KB, 53s at 8KB, and hours at GitHub's 65536-character body limit. It runs during collection, whichpytest-timeoutdoes not cover, so a single such issue hung the whole run before any test started. Anchoring on the newline that followsbabel_tests:in a real fenced block removes the ambiguity: 0.0008s at 65536 characters.yaml.safe_loadblocks code execution but still resolves aliases, and PyYAML shares the aliased nodes rather than copying them, so the load looks cheap and the cost lands on whatever formats the result afterwards. A 337-byte body of chained anchors reached 25MB in an error message. Aliases are refused outright, which covers merge keys too.On top of those: caps on body length, assertions, param sets and parameters per issue; per-parameter checks for empty, over-long and non-printable values; duplicate YAML keys rejected, because YAML keeps the last silently and the block a reviewer reads would not be the block that runs; issue text
repr()'d into log lines so an ANSI escape or bidi override cannot reach an operator's terminal; and--issueresolving only within the configured repositories.Structural caps fail the whole issue — the fix is to split it across several issues — while a bad parameter fails only its own param set, so the rest of the issue still runs. The caps sit far above anything the configured repositories currently contain, so they are a no-op for real issues today, and they are documented in the generated
assertions/README.mdwhere issue authors will meet them.Caches moved out of the shared temp directory. Both the issue-ID cache and the Google Sheet CSV cache used fixed names in the world-writable temp directory. The issue cache holds the IDs a later run fetches and executes assertions from, so being able to write it was close to being able to choose what the run tests. They now live in
~/.cache/babel-validation, created0700, overridable withBABEL_VALIDATION_CACHE_DIR— and failing to create that directory names the override, since a read-only home on a locked-down runner otherwise raises aPermissionErrorthat gives no hint an escape hatch exists.The cache sweep in
pytest_configurealso stopped deleting the Google Sheet.lockfiles. 48b1c44 had already removed that for the GitHub issue lock — a concurrent pytest holding the lock keeps its now-unlinked inode while this run creates a fresh one, so two processes end up inside "the" lock — and the Google Sheet path had the same shape.unlink_if_exists()now also refuses any path outside the cache directory, because it deletes whatever it is handed and runs before anything else in the session.Test layout. The offline tests live in
tests/github_issues/unit/—test_syntax.py(what the two syntaxes mean),test_discovery.py(finding issues, identifying them, resolving an ID) andtest_untrusted_input.py(the guards on body content). A subdirectory rather than three siblings because of the fixture:tests/github_issues/conftest.pydefines a session-scopedgithub_issues_test_casesthat needs a realGITHUB_TOKEN, and these tests want a dummy-token parser instead. Aconftest.pyin the subdirectory scopes that override to exactly the files that want it, where putting it in the parent would replace the real fixture for the livetest_github_issues.pyas well.Documentation.
CLAUDE.mdgains anUntrusted Inputsection: which inputs are hostile and which (targets.ini) are trusted config, and each failure mode above written as the shape to look for rather than as the fix that was applied — so the next parser added here starts from them. The generatedassertions/README.mdcarries the caps, where issue authors meet them. And it now says in as many words that a redpytest tests/github_issuesis the tool working, not a defect to be fixed by editing the assertions, because the obvious reading of eighteen red tests is otherwise the wrong one. And it warns against writing a complete{{BabelTest|...}}marker into an issue: this repository is itself in the scannedRepositorieslist, so an issue that merely describes an assertion gets collected and runs it.What a run produces
pytest -m unitis fully offline and needs no token: 97 passed, in about a second. This is what CI runs, and it is green.pytest tests/github_issues --target devcurrently reports 18 failing issues, and that is the tool working rather than a defect in it:ShouldNotHaveSynonym, in NCBIGene:828367 includes "''cytochrome P450" as a synonym NCATSTranslator/Babel#744. Unknown assertion names fail loudly by design rather than being skipped, so this stays a hard failure until Implement a ShouldNotHaveSynonym assertion type #110 lands. It is unrelated to any NodeNorm behaviour.What it deliberately does not do
pytest --timeout=Nalready exists and is the runtime knob for a slow issue.targets.ini, which is trusted config, not from issue bodies.-m unitonly, so nothing here exercises the live GitHub path on a PR. Enabling it needsissues: readand a token; the recipe is in a comment intests.yaml, and Add a tier of GitHub API tests that need a token but not a full issue crawl #114 covers building a tier of tests that need a token but not a full crawl. Worth knowing that without a token the issue tests skip rather than fail, so a run can go green having tested nothing.Follow-on work
Nothing is blocking this merge. Two pieces are tracked separately:
ShouldNotHaveSynonymassertion type, which is the one live failure above that is about this harness rather than about NodeNorm.GitHubIssueTesthas been built, so an oversized body still constructs a few thousand objects before being rejected. Bounded by GitHub's own body limit and off the network path, so it matters only if the caps are ever tightened much further.History — stack split, review rounds, and two false paths. Kept for anyone tracing why a particular line looks the way it does; the durable conclusions are in the code comments and docs above.
Originally part of #67, split into four stacked PRs. This one was rebased onto
mainonce #101 (library reorg) and #102 (assertions framework) merged; the description previously said it based onsplit/2-assertions.Review rounds on #67 produced the arity check on
ResolvesWith/DoesNotResolveWith, the coercion of YAML param values tostr(YAML 1.1 reads a bareNOas boolean, soHasLabel: - [CHEBI:16480, NO]used to crash), readingSearchByName's top-N fromtargets.inirather than hardcoding 5, and makingResolvesWithfailure messages deterministic instead of dict-ordered.get_all_issues()and thetqdmdependency were added and then dropped: once discovery moved to the search API, nothing paginated every issue any more.issue_id()was originally read fromissue.repository.full_name. Search results carry no repository, so that attribute cost two extra REST calls per issue; it is parsed out ofhtml_urlinstead. There is a test asserting.repositoryis never touched, because the regression is invisible — it costs requests, not correctness.Two false paths from the hardening pass, both caught by running the live suite:
```yaml\s+babel_tests:.*?```, which is linear but too loose: it started matching a one-line```yaml babel_tests: ```written in prose while discussing the syntax, and then failed that issue. Optionally factor BabelTest parsing out from GitHub fetching (futureproofing) #100 is exactly such an issue. Requiring the newline a real fenced block must have fixes both.MAX_PARAM_LENGTHwas first set to 255, which would have rejected legitimate data: IUPAC chemical names run well past that, andHasLabelcompares against NodeNorm labels. It is 1000.Issue #115, filed from this branch to track a deferred item, explained the size caps using a complete
{{BabelTest|Resolves|CHEBI:1}}marker. The harness collected it and ran it on the next live run, and because a new issue is open, the assertion passing reported as a strict XPASS failure. The issue body now quotes a partial marker, and the rule is inCLAUDE.md.Three verification false paths, all of which produced a test that looked fine and proved nothing:
test_cache_dir_is_not_world_writable_temppointedBABEL_VALIDATION_CACHE_DIRat atmp_pathand then asserted the result was not under the temp directory — a self-contradiction that passed locally only because pytest resolvestmp_pathto/private/varon macOS whilegettempdir()reports/var. Linux CI caught it. It now asserts against the default location instead.caplog.textcannot test control-character escaping: pytest does not carry a raw ESC through it, so the test passed whether the code used%ror%s. The assertions readcaplog.recordsandgetMessage().%r→%sappeared to pass because the two are the same length, so the source size was unchanged and the.pycwas never invalidated — the interpreter never ran the mutation. Both gotchas are now inCLAUDE.md.Running
blackon the touched files was also reverted. The repository is not black-clean, so it turned a 384-line diff into 703 lines and buried the change in reformatting;CLAUDE.mdnow records that.