Repository navigation
test(project): tell == 'false' from != 'true' on an evidence output (#967) - #970
Conversation
…967) `tests/project/test_release_gates.py` could not distinguish `<evidence> == 'false'` from `<evidence> != 'true'`. Every row holding `version_check` at `success` also held the evidence outputs at a literal `'true'`/`'false'`, where the two spellings agree, and `text_gate_violations` only checks that the output's *name* appears in the condition, never which comparison is made against it. Measured against the module's own helpers on e1262ed, the mutation was rejected by neither layer: 0 evaluator and 0 text violations for each of `tag`, `pypi` and `conda`. The spellings diverge only on an output that is neither -- `null` -- and for a publishing job that is the dangerous direction: `!= 'true'` runs where `== 'false'` skips, so evidence nothing established becomes a publish attempt. So `release_context` grows an `unknown=` axis holding named outputs at the module's existing `NULL`, and `gate_rows` grows one row per gated job holding that job's own evidence at `null` on a non-`v` ref and expecting a skip. The mutation joins `MUTATIONS` as rejected by the evaluator alone, and it fails on exactly that one row: 1 evaluator, 0 text per job. `github` does not have the hole. Its `version_check published no outputs at all` row holds every output at `null` while leaving `version_check.result` at `success`, so that gate's evidence clause alone decides the row -- a property of the row rather than of the gate, and now pinned by a test of its own, since rewriting the row to fail `version_check` too would take the discrimination with it. The new row guards a future change rather than a present bug, and `gate_rows`' docstring says so: neither curl-based check can emit an empty output under `set -euo pipefail`, but `PCAPKIT_TAG_EXISTS` and `PCAPKIT_CONDA_TAG_EXISTS` come from `mukunku/tag-exists-action@v1.7.0` and `tag`'s gate reads the second. Nothing is weakened: the text assertions and every existing row stay, and the row counts in `test_the_tables_are_not_empty` go 17/17/21 -> 18/18/22. The module passes 59 tests / 197 subtests under pytest, and `Ran 59 tests ... OK` under plain unittest.
|
GOOD TO GO at It found a second weakening the row closes, and I verified both sides. Beyond So this is a family of absent-evidence weakenings, not a single mutation. Four mutants do survive both layers — Also confirmed independently: no existing row was weakened — the first 17/17/21 One incidental confirmation: a malformed mutant of mine made the evaluator raise |
make pylint,make mypy,make isort)make testpasses, and a test case covers the changedocs/source/changelog/and regeneratedCHANGELOG.md, if the change is user-visible — N/A — centralised in docs(changelog): shared 1.5.0 changelog — long-lived, merges last (#610, #616, #617, #618, #620) #657What is the purpose of your pull request?
fix— corrects a defectfeat— adds a featureperf— changes performance, not behaviourrefactor— changes neither behaviour nor performancetest— tests onlydocs— documentation onlyci— workflows or build toolingchore— anything elseDescription of your pull request and other information
Closes #967.
gate_rowsgrows one row per gated job holding that job's own evidence output at themodule's existing
NULL, on a non-vref withversion_checkatsuccess, and asserts the gateskips — the only shape on which
== 'false'and!= 'true'disagree, and!= 'true'is thedangerous direction: it runs where
== 'false'skips, so unknown evidence becomes a publish attempt.Applying
== 'false'→!= 'true'to a scratch copy of each real condition (the workflow file isuntouched), violations before → after this PR:
tagPCAPKIT_CONDA_TAG_EXISTSpypiPCAPKIT_PYPI_COMPLETEcondaPCAPKIT_CONDA_COMPLETEOne row per job, resolved from
GATE_EVIDENCE[job]; the mutation joinsMUTATIONSas evaluator-onlyand fails on exactly that new row.
githubdoes not have the hole —version_check published no outputs at allholds every output atnullwithversion_check.resultstillsuccess, so itsevidence clause alone decides that row, now pinned by a test since it is a property of the row.
Reachability is stated rather than overclaimed: the row guards a future change, not a present bug —
neither curl-based check can emit an empty output under
set -euo pipefail, butPCAPKIT_TAG_EXISTS/PCAPKIT_CONDA_TAG_EXISTScome frommukunku/tag-exists-action@v1.7.0andtag's gate reads one. Nothing is weakened; row counts go 17/17/21 → 18/18/22.