Assertions framework for BabelTest expectations

[gaurav]

Adds src/babel_validation/assertions: the engine that turns a named BabelTest assertion plus its parameters into a check evaluated against NodeNorm/NameRes. Nothing consumes it yet — the GitHub issue parser that produces the parameters arrives later in the stack — so this PR is the engine, its generated documentation, and its offline tests.

Stack 2 of 4 splitting #67. Base: main (#101 is merged).

The assertion types

AssertionHandler is the base class; NodeNormTest and NameResTest specialize it per service. Each concrete handler declares its parameters and yields TestResults. Registered: Resolves, DoesNotResolve, ResolvesWith, DoesNotResolveWith, HasLabel, ResolvesWithType, SearchByName, and Needed (a placeholder that always fails, marking an issue as still needing a real test).

Both wiki ({{BabelTest|Resolves|CHEBI:15365}}) and YAML syntax are supported, and an assertion can carry several independent params lists, each evaluated separately so one bad one doesn't sink the rest.

Shared parameter handling

AssertionHandler.prepare_params_lists() strips whitespace from every param, rejects params lists whose CURIEs are malformed, and warms the NodeNorm cache for the survivors in a single batched request. Both test_with_nodenorm() and test_with_nameres() route through it, so the NameRes path gets the same validation and the same one-request warming rather than a NodeNorm round-trip per params list.

Two escape hatches keep that uniform treatment from being wrong for particular assertions:

• curie_params() narrows which params are CURIEs — HasLabel to the first, ResolvesWithType to everything after the Biolink type, SearchByName to the expected CURIE only (its first param is a free-text query).
• VALIDATE_CURIES = False opts an assertion out of format validation entirely. DoesNotResolve sets it: an identifier that isn't even a well-formed CURIE trivially doesn't resolve, which is exactly what that assertion exists to state, so rejecting it up front would leave the assertion unable to express its own purpose.

A NodeNorm bulk-normalization fix

CachedNodeNorm.normalize_curies() built its return value from response.json(), so a CURIE that NodeNorm silently omitted from its response was absent from the returned dict rather than present with a None value. A caller iterating the results would never see it and would report success for a CURIE it never tested. The warm-cache path happened to re-add the missing key, so the hole only opened on a cold lookup.

It now builds the result from the requested CURIEs: exactly one entry per request, in request order. The ordering guarantee matters independently — "first CURIE that resolved" logic previously depended on the server's JSON ordering when cold and on set iteration order (per-process string hash randomization) when warm, so which CURIE got blamed in a failure message could vary between runs of the same test.

Types and naming

Parameters are named rather than left as nested list[str]:

• ParamsList — one assertion invocation's parameters. Position is significant (ResolvesWithType takes its Biolink type first, HasLabel is [curie, label]), which is why this is a list.
• PreparedParamsList — a frozen (params, failure) record. prepare_params_lists() returns a list of these rather than a (stripped, failures_by_index) tuple, so callers don't re-zip two structures by hand.

Service parameters are annotated with the NodeNormService / NameResService Protocols that services/ already defines for the purpose, rather than the concrete CachedNodeNorm / CachedNameRes, so a future drop-in replacement needs no changes here.

Guardrails

• Registration. _register() replaces the {h.NAME: h for h in [...]} comprehension and raises on a NAME that isn't lowercase or one that's already taken. Lowercase is load-bearing — the README promises users that assertion names are matched case-insensitively, which only holds if every registry key is lowercase — and a duplicate NAME would otherwise silently drop a handler. Both are mistakes only made while adding an assertion, so they fail loudly at import.
• Missing Biolink types. When NodeNorm returns a node with no type, messages show NO TYPE RETURNED. The earlier placeholder, unknown type, had the shape of a real type — the older Biolink vocabulary was lowercase prose like chemical entity — so it could be misread as something Babel actually returned.

Documentation

gen_docs.py renders assertions/README.md from the handler class attributes, grouping handlers by the service they test rather than by their order in ASSERTION_HANDLERS — otherwise a handler registered in the wrong place lands under the wrong heading, and the sync test can't catch it because it regenerates the same wrong output. tests/test_environment/test_assertions_docs.py asserts the checked-in README stays in sync.

The "Adding a New Assertion Type" instructions live in that generated README and nowhere else. There had been a second copy in the package docstring and the two had already drifted; the docstring now points at the README and describes the module layout instead.

Tests

tests/test_environment/test_assertions.py stubs requests.post rather than the service, so handlers run against the real CachedNodeNorm and exercise its contract instead of a fake restating it. The fixture DB drops one CURIE from the response entirely, reproducing what NodeNorm does for some unknown identifiers. 13 unit tests, all offline.

Also registers the unit pytest marker, first used by these tests.

Outcomes

• The assertion vocabulary is fixed, documented, and enforced, so the issue parser in the next part of the stack has a stable target.
• Every check here runs offline — no network, no GitHub API, no Google Sheet.
• Deliberately not here: parsing assertions out of issue bodies, and any wiring into the existing pytest suites.

Notes for review

gh pr diff shows hunks in .gitignore, services/nameres.py, sources/google_sheets/, and tests/conftest.py that came from #101 and are already on main — they merge as no-ops. The net change against main is the assertions/ package, the normalize_curies() fix, the two test files, and the pytest marker.

result: dict is left untyped for NodeNorm response entries. A TypedDict would be more precise, but the response shape is Babel's to change and a wrong one is worse than an honest dict, so the docstrings say what the dict is instead.

Verify

• uv run pytest -m unit -q → 13 assertion tests pass offline.
• uv run python -m src.babel_validation.assertions.gen_docs reproduces the committed README.

🤖 Generated with Claude Code

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[Copilot]

Pull request overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated 3 comments.

[Copilot]

Pull request overview

Copilot reviewed 10 out of 10 changed files in this pull request and generated no new comments.

Suppressed comments (6)

src/babel_validation/assertions/nodenorm.py:205

• No unit test exercises HasLabelHandler; the exact-match, mismatch, unresolved-node, and missing-label branches are all currently uncovered. Add offline cases for these outcomes using the existing NodeNorm fixture so this registered assertion's contract is verified.

        if 'label' not in result['id']:
            yield self.failed(
                f"CURIE {curie} has no label but expected '{expected_label}' on {nodenorm}"
            )
            return

src/babel_validation/assertions/nodenorm.py:122

• The public contract omits the implementation's requirement that every CURIE resolve. As written, users can reasonably expect one unresolved CURIE to satisfy “must NOT all resolve to the same result,” but lines 137–144 reject it as a configuration error. Document that all CURIEs must resolve in DESCRIPTION/PARAMETERS, then regenerate the README.

This issue also appears on line 201 of the same file.

    PARAMETERS = "Two or more CURIEs per param_set. They must not all resolve to the same result."

src/babel_validation/assertions/nodenorm.py:5

• This module types every handler against the concrete cache class even though services/nodenorm.py:28-33 explicitly requires callers to depend on NodeNormService. Use that protocol in the import and annotations so alternative service implementations remain type-compatible.

from src.babel_validation.services.nodenorm import CachedNodeNorm

src/babel_validation/assertions/nameres.py:8

• These concrete client imports conflict with the explicit interface convention in services/nameres.py:34-39 and services/nodenorm.py:28-33. Annotate the handler with NameResService and NodeNormService instead, preserving compatibility with drop-in implementations and test doubles.

from src.babel_validation.services.nameres import CachedNameRes
from src.babel_validation.services.nodenorm import CachedNodeNorm

src/babel_validation/assertions/nameres.py:52

• The only SearchByName test supplies a malformed CURIE and nameres=None, so it exits during prevalidation and never exercises this lookup or the canonical-ID/top-N matching. Add offline found, not-found, and rank-boundary cases using a fake NameRes client; otherwise the only NameRes assertion's core behavior can regress unnoticed.

        results = nameres.lookup(search_query, autocomplete='false', limit=pass_if_found_in_top)

src/babel_validation/assertions/nodenorm.py:252

• ResolvesWithTypeHandler has no behavioral coverage, including the newly guarded missing/empty type response. Add offline resolved, wrong-type, missing-type, and unresolved-CURIE cases using the existing NodeNorm fixture.

            if not node:
                yield self.failed(f"Could not resolve {curie} with NodeNormalization service {nodenorm}")
                continue
            biolink_types = node.get('type') or []
            if expected_biolink_type in biolink_types: