fix(quality): enable-axe went GREEN having measured nothing; the needs-deletion suspicion was a cancelled run

[rubenvdlinde]

What was suspected, and what the measurement says

ConductionNL/openregister #2185 showed quality / Hydra Gates twice in its check list: once skipping (0s) and once fail with 0 seconds duration. A zero-duration "fail" is not a verdict, and #148 had just wired needs: [playwright] into that job — so the obvious suspicion was that needs: had deleted the job and !cancelled() was not enough to stop it.

It had not, and it is. Both halves of the observation are explained without the guard being at fault:

what the check list showed	what it actually is
fail, 0s — job 91986924166, run 30907440099	conclusion cancelled. A push run superseded by concurrency. gh pr checks renders CANCELLED as "fail"; a cancelled job carries no verdict.
skipping, 0s — job 91988619080, run 30907444439	conclusion skipped, and correct: openregister's caller never sets enable-hydra-gates, which defaults to false. Playwright in that same run succeeded (3m45s), so the needs: edge was satisfied and cannot be the cause.

The guard, measured in all three producer states

Run 30909727531 reproduced this exact needs:/if: pair three times in one run and drove the producer into each state. The shipped guard and the always() alternative behaved identically and correctly:

producer	inputs.enable-hydra-gates && !cancelled()	always() && inputs.enable-hydra-gates
succeeded	RAN (needs.playwright.result=success)	RAN
failed	RAN (needs.playwright.result=failure)	RAN
skipped	RAN (needs.playwright.result=skipped)	RAN

Confirmed a fourth time end-to-end on the real quality.yml (run 30910234160): Playwright skipped, Hydra Gates ran. !cancelled() is sufficient, always() is not needed, and the comment above the guard now records this so nobody re-derives it. No change to the guard.

What the measurement DID find

Keeping the job alive when Playwright is skipped or crashed is right for the other sixty gates — and it is exactly what made gate-33 dangerous. With enable-axe: true, no artifact could arrive, so:

• gate-33 reported SKIPPED,
• the step printed a ::notice,
• the job went green.

Measured, not argued — run 30910234160, Hydra Gates SUCCESS:

##[notice]No axe report was downloaded. ... accessibility runtime checking is UNVERIFIED for this run.
[gate-33] axe-core: SKIPPED - no src/ and no tests/axe/report.json

A caller that explicitly opted into axe enforcement got a green tick over zero accessibility measurement. The absence of the check was indistinguishable from the check succeeding — the shape this file exists to eliminate. The same held for a report that arrived and was rejected by #148's provenance check: deleted, gate-33 skipped, green.

The fix

enable-axe: true is an opt-in to enforcement, not to a best effort. No usable report is now a failure with a named reason, quoting the producer's resolved state. Not enforcing axe is enable-axe: false — one line in a caller, visible in a diff, attributable to a person.

The verdict is recorded at the check and raised at the end of the job, and that detail was itself measured. A plain exit 1 at the check (run 30910345294) aborted the job before Run the Hydra gates, so a missing axe artifact took the other sixty gates down with it. A gate that suppresses sixty other gates to report itself is its own kind of outage. The gates now run and print in full, then the job fails.

Also: the Quality Report gained a Hydra gates row. The gates were in report's needs: and in its failure gate, so a red gates job did red the report — but the artefact people actually read never mentioned sixty gates in any state, which read identically to a repo that has none.

Proof

	run	Hydra Gates
before — quality.yml@main, gates on, axe on, producer off	30910234160	SUCCESS, gate-33 SKIPPED, nothing measured
after — same caller, same inputs, this branch	30910908474	FAILURE at Fail when axe was requested but never measured; Run the Hydra gates still success and reported in full
no-regression — same inputs, enable-axe: false	30911069404	SUCCESS, both axe steps SKIPPED

The arms differ only in which quality.yml they call — same caller, same inputs, one A/B arm per run so the two could not collide on the result-hydra-gates artifact name and let the collision, rather than the change, decide the verdict.

Positive control on the validator itself, run directly against two fixtures: a well-formed report exits 0 (so unmeasured is never written and the new terminal step is skipped — the green path is genuinely still green), {} exits 1.

Blast radius

enable-hydra-gates is set in exactly one repo fleet-wide (openbuild), and no repo anywhere sets enable-axe: true. The no-regression arm reproduces openbuild's live configuration and stays green. Nothing that passes today turns red.

The temporary measurement harnesses were removed in the last two commits; their run IDs are recorded in quality.yml itself.

The producer-failed state, and the one thing not measured end to end

The guard was measured directly against a genuinely failing producer (run 30909727531, needs.playwright.result=failure, gates RAN). The fix was not, and the attempt is recorded rather than hidden: run 30911499894 turned enable-playwright on to force a failing producer, but this repo is not a Nextcloud app, so the security jobs failed first and playwright's own needs.security.result != 'failure' guard turned it back into a skip. The gates job failed correctly there regardless — a second independent absent-report failure — because this step branches on whether a usable report exists, not on how the producer resolved. The producer's state appears only in the error text, which read resolved 'skipped' as expected.