csv-to-babeltests CLI for generating BabelTests YAML (4/4) - #104
Open
gaurav wants to merge 3 commits into
Open
Conversation
…AML. Translator collaborators sometimes circulate spreadsheets pairing a CURIE column with a label, equivalent CURIE, or Biolink type. This adds a click CLI that ingests such CSVs and emits a paste-ready YAML babel_tests block for a GitHub issue, optionally validating each row against a NodeNorm target from tests/targets.ini and reporting failures on stderr. The tool reuses ASSERTION_HANDLERS and CachedNodeNorm directly so the emitted YAML is guaranteed to match what GitHubIssuesTestCases parses, and per-assertion semantics never drift between the two code paths. Wires hatchling as the build backend (packaging src/ as-is so existing `from src.babel_validation.X` imports keep working) so the csv-to-babeltests console script can be exposed via [project.scripts]. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Importing the tool registered the _FlowList representer on the global yaml.SafeDumper, changing YAML output process-wide. Move it onto a local _BabelTestDumper subclass; emitter output is unchanged. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This was referenced Jun 26, 2026
gaurav
added a commit
that referenced
this pull request
Jun 26, 2026
The repo's `.gitignore` predates the Python rewrite — it only covered Scala/Giter8/IntelliJ artifacts, so `__pycache__/` and `*.pyc` showed up as untracked noise throughout the tree and were easy to `git add` by accident. Appends the standard [GitHub `Python.gitignore`](https://github.com/github/gitignore/blob/main/Python.gitignore) template (bytecode, `build/`/`dist/`, `.pytest_cache/`, `.venv`/`.env`, mypy/coverage caches, …) below the existing entries, plus a `.DS_Store` line for macOS. Existing entries are kept at the top, so this is purely additive. Independent of the #67 split stack (#101–#104) and based on `main`, so it can merge immediately. No files are currently tracked that the new patterns would retroactively ignore — `git ls-tree` shows no `.pyc`/`__pycache__` in the tree — so nothing needs `git rm --cached`. ### Verify ``` git check-ignore -v src/__pycache__/x.pyc .venv .env # all matched ``` 🤖 Generated with [Claude Code](https://claude.com/claude-code)
gaurav
added a commit
that referenced
this pull request
Aug 26, 2026
) Test cases for Babel currently live in a Google Sheet, which is fine for bulk regression data and poor for anything tied to a specific bug: the sheet has no idea which issue a row came from, so nothing tells us when a fixed issue regresses or when an open one has quietly started working. This PR lets a test case live in the GitHub issue that motivated it, and runs those cases as pytest tests. An issue body carries assertions in either of two syntaxes — a wiki-style `{{BabelTest|Resolves|CHEBI:15365}}` marker, or a fenced YAML block whose top-level key is `babel_tests:` — and the harness turns each issue into one pytest item, with each assertion a subtest. Open issues are expected to fail, so an open issue whose assertions all pass is reported as a strict XPASS: the tool is telling you the issue looks closeable. Closed issues are expected to pass, so one that starts failing is telling you to reopen it. **Stack 3 of 4** splitting #67. #101 and #102 have merged, so this now bases on `main`; #104 is the remaining piece. This leaves #67 open, which stays open for #104. ## What's here **Parsing — `src/babel_validation/sources/github/`.** `GitHubIssuesTestCases` finds assertion blocks in issue bodies and resolves each to a handler from the assertions framework that landed in #102. Issue discovery uses the GitHub search API rather than paginating every issue, and reads `body` and `html_url` straight off the search result, so scanning the five configured repositories costs two search requests per repo and no core requests at all. **Harness — `tests/github_issues/`.** One pytest item per issue, parametrized by issue ID, across the repositories listed under `Repositories` in `targets.ini`'s `[DEFAULT]` section. `--issue` targets a specific issue by `org/repo#N`, `repo#N` or `N`. Issue IDs and hydrated issues are cached and shared across `pytest-xdist` workers behind a `FileLock`. **Input validation.** Issue bodies are untrusted: anyone with a GitHub account can write one, nothing reviews it, and we parse it and turn it into live NodeNorm and NameRes calls. That is tolerable while a human watches the run and can hit Ctrl-C; it is not tolerable for the unattended daily runs this is meant to enable. Two of the gaps were reproducible denial of service rather than theory: - The pattern that finds a `babel_tests` block ended with `\s+.*?\s+` before the closing fence. Those three nested backtracking quantifiers made matching cubic on a body that opens a block and never closes the fence — 6.8s at 4KB, 53s at 8KB, and hours at GitHub's 65536-character body limit. It runs during collection, which `pytest-timeout` does not cover, so a single such issue hung the whole run before any test started. Anchoring on the newline that follows `babel_tests:` in a real fenced block removes the ambiguity: 0.0008s at 65536 characters. - `yaml.safe_load` blocks code execution but still resolves aliases, and PyYAML shares the aliased nodes rather than copying them, so the load looks cheap and the cost lands on whatever formats the result afterwards. A 337-byte body of chained anchors reached 25MB in an error message. Aliases are refused outright, which covers merge keys too. On top of those: caps on body length, assertions, param sets and parameters per issue; per-parameter checks for empty, over-long and non-printable values; duplicate YAML keys rejected, because YAML keeps the last silently and the block a reviewer reads would not be the block that runs; issue text `repr()`'d into log lines so an ANSI escape or bidi override cannot reach an operator's terminal; and `--issue` resolving only within the configured repositories. Structural caps fail the whole issue — the fix is to split it across several issues — while a bad parameter fails only its own param set, so the rest of the issue still runs. The caps sit far above anything the configured repositories currently contain, so they are a no-op for real issues today, and they are documented in the generated `assertions/README.md` where issue authors will meet them. **Caches moved out of the shared temp directory.** Both the issue-ID cache and the Google Sheet CSV cache used fixed names in the world-writable temp directory. The issue cache holds the IDs a later run fetches and executes assertions from, so being able to write it was close to being able to choose what the run tests. They now live in `~/.cache/babel-validation`, created `0700`, overridable with `BABEL_VALIDATION_CACHE_DIR` — and failing to create that directory names the override, since a read-only home on a locked-down runner otherwise raises a `PermissionError` that gives no hint an escape hatch exists. The cache sweep in `pytest_configure` also stopped deleting the Google Sheet `.lock` files. 48b1c44 had already removed that for the GitHub issue lock — a concurrent pytest holding the lock keeps its now-unlinked inode while this run creates a fresh one, so two processes end up inside "the" lock — and the Google Sheet path had the same shape. `unlink_if_exists()` now also refuses any path outside the cache directory, because it deletes whatever it is handed and runs before anything else in the session. **Test layout.** The offline tests live in `tests/github_issues/unit/` — `test_syntax.py` (what the two syntaxes mean), `test_discovery.py` (finding issues, identifying them, resolving an ID) and `test_untrusted_input.py` (the guards on body content). A subdirectory rather than three siblings because of the fixture: `tests/github_issues/conftest.py` defines a session-scoped `github_issues_test_cases` that needs a real `GITHUB_TOKEN`, and these tests want a dummy-token parser instead. A `conftest.py` in the subdirectory scopes that override to exactly the files that want it, where putting it in the parent would replace the real fixture for the live `test_github_issues.py` as well. **Documentation.** `CLAUDE.md` gains an `Untrusted Input` section: which inputs are hostile and which (`targets.ini`) are trusted config, and each failure mode above written as the shape to look for rather than as the fix that was applied — so the next parser added here starts from them. The generated `assertions/README.md` carries the caps, where issue authors meet them. And it now says in as many words that a red `pytest tests/github_issues` is the tool working, not a defect to be fixed by editing the assertions, because the obvious reading of eighteen red tests is otherwise the wrong one. And it warns against writing a complete `{{BabelTest|...}}` marker into an issue: this repository is itself in the scanned `Repositories` list, so an issue that merely *describes* an assertion gets collected and runs it. ## What a run produces `pytest -m unit` is fully offline and needs no token: **97 passed**, in about a second. This is what CI runs, and it is green. `pytest tests/github_issues --target dev` currently reports **18 failing issues**, and that is the tool working rather than a defect in it: - **10 open issues XPASS** — every assertion now passes, so they look closeable. - **7 closed issues have failing assertions** — #406, #552, #584, #711, #714, #723, #906 — so they look like they should be reopened. - **1 issue uses an assertion type we have not written yet**: `ShouldNotHaveSynonym`, in NCATSTranslator/Babel#744. Unknown assertion names fail loudly by design rather than being skipped, so this stays a hard failure until #110 lands. It is unrelated to any NodeNorm behaviour. ## What it deliberately does not do - **No new CLI options for the limits.** They are module constants. `pytest --timeout=N` already exists and is the runtime knob for a slow issue. - **No guards around the NodeNorm/NameRes responses.** Those URLs come from `targets.ini`, which is trusted config, not from issue bodies. - **CI runs `-m unit` only**, so nothing here exercises the live GitHub path on a PR. Enabling it needs `issues: read` and a token; the recipe is in a comment in `tests.yaml`, and #114 covers building a tier of tests that need a token but not a full crawl. Worth knowing that without a token the issue tests **skip rather than fail**, so a run can go green having tested nothing. ## Follow-on work Nothing is blocking this merge. Two pieces are tracked separately: - #110 — implement the `ShouldNotHaveSynonym` assertion type, which is the one live failure above that is about this harness rather than about NodeNorm. - #114 — a tier of GitHub API tests that need a token but not a full issue crawl, so CI can cover the integration itself. - #115 — the per-issue size caps are checked after every `GitHubIssueTest` has been built, so an oversized body still constructs a few thousand objects before being rejected. Bounded by GitHub's own body limit and off the network path, so it matters only if the caps are ever tightened much further. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stack 4 of 4 splitting #67. Base:
split/2-assertions(PR #102) — independent of the GitHub-issue PR #103; only needs the assertions framework.Cherry-picked (with authorship preserved) from #67: a
clickCLI that converts a CSV of test cases into a BabelTests YAML block, reusing the assertions handlers from PR #102. Addsclick+pyyaml; registers thecsv-to-babeltestsscript entry point.Verify
uv run pytest tests/tools/test_csv_to_babeltests.py -q→ 13 passed.uv run csv-to-babeltests --help, or run it on a small sample CSV and eyeball the YAML.🤖 Generated with Claude Code