Skip to content

test(ci): pin the supply-chain scanners the compliant tree never exercises - #1247

Merged
mrbobbytables merged 1 commit into
mainfrom
quality/ci-supply-chain-scanners
Oct 9, 2026
Merged

mrbobbytables merged 1 commit into
mainfrom
quality/ci-supply-chain-scanners

Conversation

@hivecommons-hive

Copy link
Copy Markdown
Contributor

Test Improvement

tests/ci-supply-chain.test.mjs asserts the workflow supply-chain contract —
SHA pinning, version comments, permissions:, persist-credentials,
timeout-minutes, runner-image pinning — over the real .github/workflows/**
tree. That tree is fully compliant and all three retiring baselines
(KNOWN_PERSISTING_CHECKOUTS, KNOWN_UNBOUNDED_JOBS, KNOWN_FLOATING_RUNNERS)
are new Set([]), so nothing in the file ever reached a scanner's offence arm
or a baseline guard's body.

  • Extracted the scanners and the three baseline guards into
    tests/helpers-ci-supply-chain.mjs, each returning its offender list instead
    of asserting inline. Behaviour and every assertion message are unchanged.
  • Imported them back into tests/ci-supply-chain.test.mjs, which still runs
    the same 15 contract tests over the same tree.
  • Added tests/ci-supply-chain-helpers.test.mjs — 21 tests driving each
    scanner over synthetic workflow documents that do offend, and each baseline
    guard over a non-empty baseline, covering the "entry names a workflow that no
    longer exists", "entry names a job that no longer exists" and "entry is now
    compliant" arms. The fixtures are written as YAML and parsed with the same
    yaml reader the contract uses, so they cannot drift from the real shape.

Why it mattered

With unpinnedActions mutated to skip every step — a scanner that detects
nothing — the contract still reports 15 pass / 0 fail, while the new file
reports 20 pass / 1 fail. That is the hole, measured: before this change a
silently-broken scanner was a green check over an unverified contract.

The three "baseline retires itself" tests are the guards that force an
exception to be deleted in the same change that fixes the workflow. They had
never executed a line, so the first contributor to re-add a baseline entry
would have been the first person to run that code.

Measured

npm run test:unit:coverage (TZ=UTC node tests/tools/coverage-report.mjs,
node v26.10.0) at 59ba83c:

lines regions
tests/ci-supply-chain.test.mjs before 86.12 84.94
tests/ci-supply-chain.test.mjs after 100.00 100.00
tests/helpers-ci-supply-chain.mjs after 100.00 100.00
tests/ci-supply-chain-helpers.test.mjs after 100.00 100.00

All files 99.47 | 95.51 → 99.59 | 95.80. src files stay at
100.00 | 100.00. Full unit suite 2167/2167 pass;
npm run check and npm run test:unit:coverage:check both exit 0; Prettier
clean on all three files.

No e2e evidence is claimed or needed: these files read .github/workflows/**
off disk and never run in a browser, so no end-to-end suite can cover them.

Scope / overlap

Touches tests/ci-supply-chain.test.mjs, tests/helpers-ci-supply-chain.mjs
(new) and tests/ci-supply-chain-helpers.test.mjs (new) — and no workflow
file, so nothing here needs workflow-write permission. None of the open
hold-gated PRs touch any of the three: #1225 is the closest neighbour and is
confined to tests/ci-pipeline-exit-codes.test.mjs; #1231 applies this same
helper-extraction pattern to tests/docs-contract.test.mjs, a disjoint file.
Branch cut from a fresh origin/main.

Related Issue

Closes #1246


Filed by quality agent (hold-gated mode). Human review required.

— hive: agent=quality backend=copilot model=claude-opus-5 copilot=1.0.88

…cises

Extract the workflow scanners and the three retiring-baseline guards from
tests/ci-supply-chain.test.mjs into tests/helpers-ci-supply-chain.mjs, and
drive them from tests/ci-supply-chain-helpers.test.mjs over synthetic
workflow documents that do offend.

Every workflow in .github/workflows/ is compliant and all three KNOWN_*
baselines are empty, so the contract never reached a scanner's offence arm or
a baseline guard's body: 86.12% lines / 84.94% regions, with the three
baseline-guard bodies uncovered in full. A scanner that stopped detecting
would have left the contract green over a tree it was no longer checking.

Closes #1246

Signed-off-by: quality <quality@hive.kubestellar.io>
@hivecommons-hive hivecommons-hive Bot added the hold label Oct 9, 2026
@hivecommons-hive

Copy link
Copy Markdown
Contributor Author

Important

Held for human review by the hive's ACMM level gate.

This PR was opened by the "quality" agent while Hive policy required a human checkpoint for that agent. Non-outreach agents are held at ACMM L3–L5; the outreach agent is always held because it publishes project-facing communication.

Hive will keep the hold label until a human removes it. Operators can make a deliberate one-off release during an ACMM level change with release_level_holds=true, but level changes never release this hold automatically.

@mrbobbytables
mrbobbytables added this pull request to the merge queue Oct 9, 2026
Merged via the queue into main with commit 154f2c5 Oct 9, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[quality] the CI supply-chain contract's scanners and retiring baselines are never exercised

1 participant