tests/project/test_release_gates.py pins the release gates by asserting over the text of each job's if: expression. Four rounds of cross-review on #961 produced twenty mutations of one clause; the assertions were strengthened three times, and each strengthening was defeated by a mutation the previous one did not anticipate.
The current state after #961. test_version_check_must_strictly_succeed carries three assertions over a whitespace-stripped condition: the clause appears as a top-level && conjunct, the set of comparisons against version_check.result is exactly {('==', 'success')}, and .result is referenced exactly once. That catches all twenty mutations except one: prefixing the whole expression with true || short-circuits it from outside while leaving the inner conjunction intact. Closing that by anchoring the expression's opening was considered and rejected — it would fail on any legitimate reordering of conjuncts, and a test people route around is worse than a known residual.
The structural fix. Evaluate the expressions instead of matching them. For each of tag, pypi and conda, walk the four values needs.<job>.result can take — success, failure, cancelled, skipped — and assert the gate is false for every value but success. That is immune to spelling: reversed operands, contains(fromJSON(...)), a negation, a true || prefix and a &&→|| slip all change the truth table, which is what gets checked.
This module already does exactly that, for exactly this reason. TestReleaseStatusScriptExecutesCorrectly extracts the release_status shell script and runs it over a table of scenarios, and its own docstring says the earlier tests were "being a string assertion over the YAML, structurally unable to catch" the two #905 defects. The same argument now applies one layer up, to the if: expressions themselves.
Scope. A full GitHub Actions expression parser is not wanted. What the gates use is a small subset — !cancelled(), &&, ||, parentheses, ==/!= against single-quoted literals, startsWith(...), contains(fromJSON(...), ...), and needs.<job>.result / needs.<job>.outputs.<name> lookups. An evaluator over that subset, with the lookups supplied from a scenario table, is a bounded piece of work.
Two things to settle in doing it: whether the text assertions stay as a cheap first line of defence or are replaced, and what the scenario table covers beyond version_check — the github/tag equality pairs and the startsWith(github.ref_name, 'v') short-circuit have the same structural blind spot.
Found by the cross-review on #961, which built the twenty mutations and identified this as the principled endpoint. Not a gate on that pull request.
tests/project/test_release_gates.pypins the release gates by asserting over the text of each job'sif:expression. Four rounds of cross-review on #961 produced twenty mutations of one clause; the assertions were strengthened three times, and each strengthening was defeated by a mutation the previous one did not anticipate.The current state after #961.
test_version_check_must_strictly_succeedcarries three assertions over a whitespace-stripped condition: the clause appears as a top-level&&conjunct, the set of comparisons againstversion_check.resultis exactly{('==', 'success')}, and.resultis referenced exactly once. That catches all twenty mutations except one: prefixing the whole expression withtrue ||short-circuits it from outside while leaving the inner conjunction intact. Closing that by anchoring the expression's opening was considered and rejected — it would fail on any legitimate reordering of conjuncts, and a test people route around is worse than a known residual.The structural fix. Evaluate the expressions instead of matching them. For each of
tag,pypiandconda, walk the four valuesneeds.<job>.resultcan take —success,failure,cancelled,skipped— and assert the gate is false for every value butsuccess. That is immune to spelling: reversed operands,contains(fromJSON(...)), a negation, atrue ||prefix and a&&→||slip all change the truth table, which is what gets checked.This module already does exactly that, for exactly this reason.
TestReleaseStatusScriptExecutesCorrectlyextracts therelease_statusshell script and runs it over a table of scenarios, and its own docstring says the earlier tests were "being a string assertion over the YAML, structurally unable to catch" the two #905 defects. The same argument now applies one layer up, to theif:expressions themselves.Scope. A full GitHub Actions expression parser is not wanted. What the gates use is a small subset —
!cancelled(),&&,||, parentheses,==/!=against single-quoted literals,startsWith(...),contains(fromJSON(...), ...), andneeds.<job>.result/needs.<job>.outputs.<name>lookups. An evaluator over that subset, with the lookups supplied from a scenario table, is a bounded piece of work.Two things to settle in doing it: whether the text assertions stay as a cheap first line of defence or are replaced, and what the scenario table covers beyond
version_check— thegithub/tagequality pairs and thestartsWith(github.ref_name, 'v')short-circuit have the same structural blind spot.Found by the cross-review on #961, which built the twenty mutations and identified this as the principled endpoint. Not a gate on that pull request.