Skip to content

test(project): evaluate the release gates' if: expressions (#962) - #966

Merged
JarryShaw merged 1 commit into
mainfrom
test/962-evaluate-release-gate-expressions
Oct 1, 2026
Merged

JarryShaw merged 1 commit into
mainfrom
test/962-evaluate-release-gate-expressions

Conversation

@JarryShaw

Copy link
Copy Markdown
Owner

Please follow the guide below

What is the purpose of your pull request?

  • fix — corrects a defect
  • feat — adds a feature
  • perf — changes performance, not behaviour
  • refactor — changes neither behaviour nor performance
  • test — tests only
  • docs — documentation only
  • ci — workflows or build tooling
  • chore — anything else

Description of your pull request and other information

Closes #962. .github/workflows/ is untouched — the workflow is the fixture.

What is now established. tag/pypi/conda are evaluated over 17/17/21
scenario rows: all four values of needs.version_check.result (twice — outputs
present, which is what catches the clause being deleted, and outputs absent, the
skipped shape), all four of each publishing predecessor's, a cancelled run, and
all four corners of startsWith(ref_name, 'v') || <evidence> == 'false' with the
other three evidence outputs inverted so a gate reading the wrong one fails.
github and unit-tests get their own tables. One axis per row, everything else
held at a running value — a row holding two clauses false proves less than it looks.

Mutations. 14 applied to a scratch copy of each real condition, asserting
which layer rejects each. Caught by both layers: &&→|| (either side),
== 'success'→!= 'failure', || == 'skipped' added, contains(fromJSON(…)),
the conjunct dropped, the predecessor pair weakened to != 'failure', || == 'skipped'
removed, the evidence output swapped. Evaluator only: true || prefix (#961's
recorded residual), !cancelled()→always(), startsWith prefix emptied. Text
only, and correctly so: reversed operands — same truth table, so the evaluator has
nothing to say and the retained text assertion is what notices. Nothing survives both.

Residuals. The implicit success() Actions prepends to github's if:, and the
cascade-skip, are runner behaviour and stay pinned textually. Case-only mutations are
invisible to the evaluator by construction (Actions compares case-insensitively).

35 → 58 tests, 188 subtests; pytest -q and plain unittest agree.

`tests/project/test_release_gates.py` pinned the release gates by matching
the text of each `if:`. Four review rounds on #961 produced twenty mutations
of one clause, each defeating the previous round's assertions, because
matching the text that produces an answer is not checking the answer.

* Add a GitHub Actions expression evaluator over the subset the four gates
  use -- `!`, `&&`, `||`, parentheses, `==`/`!=` against single-quoted
  literals, `startsWith`, `contains`, `fromJSON`, the status-check functions,
  and `needs.*`/`github.*` lookups. Anything outside it raises, and
  `test_every_declared_if_is_evaluable` makes that a loud failure.
* Evaluate `tag`, `pypi` and `conda` over 17/17/21 scenario rows: all four
  values of `needs.version_check.result`, all four of each publishing
  predecessor's, a cancelled run, and all four corners of the
  `startsWith(ref_name, 'v') || <evidence> == 'false'` disjunction.
  `github` and `unit-tests` get their own smaller tables.
* Add the 14 mutations as a test that applies each to a scratch copy of the
  real condition and asserts which layer rejects it. The text assertions stay
  unchanged: the evaluator catches the `true ||` prefix #961 left open, and
  the text layer catches reversed operands, which change no truth table.

35 -> 58 tests, 188 subtests; `pytest` and plain `unittest` agree.
@JarryShaw JarryShaw added ci Pull requests that change CI or workflow configuration (ci: subject prefix) review: pending No verdict for the current head - never reviewed, or the head moved since the last one test Pull requests that add or correct tests (test: subject prefix) labels Oct 1, 2026
@JarryShaw

Copy link
Copy Markdown
Owner Author

GOOD TO GO at e403c9166 — sonnet cross-review, first round. Different model
from the author (opus).

Re-derived on my side rather than relayed: 58 tests pass under plain unittest
in the PR's own tree (pcapkit.__file__ asserted into that worktree), 35 → 58 is
23 new, and the only 8 deleted lines are the stale "Knowingly not closed"
docstring paragraph — the hole this PR closes. No test removed or weakened.

It found a fifteenth mutation that survives both layers, and I confirmed it.
Rewriting an evidence comparison from == 'false' to != 'true' yields zero
violations from the evaluator and zero from the text mirror, on all three jobs
(baseline condition clean as a control). The text layer only checks the evidence
name appears; every modelled row holds the outputs at literal 'true'/'false',
so the two spellings never diverge on a modelled row.

One correction to the review's own reasoning: it called this unreachable because
version_check's steps always emit a literal. True of the two curl checks, which
default complete=false then set true. Not true of PCAPKIT_TAG_EXISTS and
PCAPKIT_CONDA_TAG_EXISTS, which come from mukunku/tag-exists-action@v1.7.0 —
a third-party output this repo does not control, and tag's gate reads one. So
the gap is narrower than a defect but realer than unreachable.

Not blocking: it is the same character as the four exclusions the module already
discloses, and worth less than the 23 tests it would hold up. Filed as a
follow-up.

@JarryShaw JarryShaw added review: good-to-go Cross-review at the current head says ready; CI state is separate and removed review: pending No verdict for the current head - never reviewed, or the head moved since the last one labels Oct 1, 2026
@JarryShaw
JarryShaw merged commit d8f6959 into main Oct 1, 2026
63 checks passed
@JarryShaw
JarryShaw deleted the test/962-evaluate-release-gate-expressions branch October 1, 2026 14:58
@JarryShaw JarryShaw removed the review: good-to-go Cross-review at the current head says ready; CI state is separate label Oct 1, 2026
@JarryShaw JarryShaw added this to the 1.5 milestone Oct 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci Pull requests that change CI or workflow configuration (ci: subject prefix) test Pull requests that add or correct tests (test: subject prefix)

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

test: evaluate the release gates' if: expressions instead of matching their text

1 participant