Repository navigation
test(project): evaluate the release gates' if: expressions (#962) - #966
Conversation
`tests/project/test_release_gates.py` pinned the release gates by matching the text of each `if:`. Four review rounds on #961 produced twenty mutations of one clause, each defeating the previous round's assertions, because matching the text that produces an answer is not checking the answer. * Add a GitHub Actions expression evaluator over the subset the four gates use -- `!`, `&&`, `||`, parentheses, `==`/`!=` against single-quoted literals, `startsWith`, `contains`, `fromJSON`, the status-check functions, and `needs.*`/`github.*` lookups. Anything outside it raises, and `test_every_declared_if_is_evaluable` makes that a loud failure. * Evaluate `tag`, `pypi` and `conda` over 17/17/21 scenario rows: all four values of `needs.version_check.result`, all four of each publishing predecessor's, a cancelled run, and all four corners of the `startsWith(ref_name, 'v') || <evidence> == 'false'` disjunction. `github` and `unit-tests` get their own smaller tables. * Add the 14 mutations as a test that applies each to a scratch copy of the real condition and asserts which layer rejects it. The text assertions stay unchanged: the evaluator catches the `true ||` prefix #961 left open, and the text layer catches reversed operands, which change no truth table. 35 -> 58 tests, 188 subtests; `pytest` and plain `unittest` agree.
|
GOOD TO GO at Re-derived on my side rather than relayed: 58 tests pass under plain It found a fifteenth mutation that survives both layers, and I confirmed it. One correction to the review's own reasoning: it called this unreachable because Not blocking: it is the same character as the four exclusions the module already |
Please follow the guide below
make pylint,make mypy,make isort)make testpasses, and a test case covers the changedocs/source/changelog/and regeneratedCHANGELOG.md, if the change is user-visible — N/A — centralised in docs(changelog): shared 1.5.0 changelog — long-lived, merges last (#610, #616, #617, #618, #620) #657What is the purpose of your pull request?
fix— corrects a defectfeat— adds a featureperf— changes performance, not behaviourrefactor— changes neither behaviour nor performancetest— tests onlydocs— documentation onlyci— workflows or build toolingchore— anything elseDescription of your pull request and other information
Closes #962.
.github/workflows/is untouched — the workflow is the fixture.What is now established.
tag/pypi/condaare evaluated over 17/17/21scenario rows: all four values of
needs.version_check.result(twice — outputspresent, which is what catches the clause being deleted, and outputs absent, the
skipped shape), all four of each publishing predecessor's, a cancelled run, and
all four corners of
startsWith(ref_name, 'v') || <evidence> == 'false'with theother three evidence outputs inverted so a gate reading the wrong one fails.
githubandunit-testsget their own tables. One axis per row, everything elseheld at a running value — a row holding two clauses false proves less than it looks.
Mutations. 14 applied to a scratch copy of each real condition, asserting
which layer rejects each. Caught by both layers:
&&→||(either side),== 'success'→!= 'failure',|| == 'skipped'added,contains(fromJSON(…)),the conjunct dropped, the predecessor pair weakened to
!= 'failure',|| == 'skipped'removed, the evidence output swapped. Evaluator only:
true ||prefix (#961'srecorded residual),
!cancelled()→always(),startsWithprefix emptied. Textonly, and correctly so: reversed operands — same truth table, so the evaluator has
nothing to say and the retained text assertion is what notices. Nothing survives both.
Residuals. The implicit
success()Actions prepends togithub'sif:, and thecascade-skip, are runner behaviour and stay pinned textually. Case-only mutations are
invisible to the evaluator by construction (Actions compares case-insensitively).
35 → 58 tests, 188 subtests;
pytest -qand plainunittestagree.