Skip to content

ci: add stable consumer and compatibility gates - #39

Merged
codeforester merged 3 commits into
mainfrom
ci/27-20260918-ci-require-consumer-and-compatibility-checks-before-merging
Sep 19, 2026
Merged

codeforester merged 3 commits into
mainfrom
ci/27-20260918-ci-require-consumer-and-compatibility-checks-before-merging

Conversation

@codeforester

@codeforester codeforester commented Sep 18, 2026

Copy link
Copy Markdown
Contributor

Summary

Add stable aggregate checks that fail if consumer validation or the supported Base-CLI compatibility matrix fails, plus documentation for the exact ruleset check names and narrow single-maintainer review exception.

Issue

Refs #27

Validation

  • uv run --extra dev pytest -q — 34 passed, 2 skipped.
  • Parsed both changed workflow YAML files.

Notes

The PR does not mutate repository ruleset settings. After merge, require the exact job checks required-consumer and required-compatibility in the default-branch ruleset and read the settings back; #27 remains open until that external step is verified.

@codeforester

Copy link
Copy Markdown
Contributor Author

Multi-angle review of this PR (ci: add stable consumer and compatibility gates, addresses #27). Five findings, ranked by severity — the first is the core one:

1. This PR doesn't actually make anything required yet, and that's not verified anywhere. I checked the live ruleset directly: gh api repos/basefoundry/base-cli-demo/rulesets/21786832 ("Base default branch protection") currently has zero required_status_checks rules — only a PR-review rule (0 required approvals), deletion, and non_fast_forward. This PR only adds the CI job names (required-consumer, required-compatibility); actually gating merges requires a separate, undiffed, manual GitHub Settings action that docs/branch-protection.md describes but this PR doesn't perform or verify. Issue #27's own acceptance criteria explicitly asks to "read back the effective rules and demonstrate a failed required check prevents merge in a safe fixture/PR" — that demonstration doesn't exist here. If #27 gets closed on the strength of this PR alone, the repo is left in exactly its pre-#27 state (merges ungated) while the tracking issue looks resolved.

2. Enabling the ruleset right after merge (as the doc instructs) will permanently block already-open PRs. GitHub Actions runs a PR's workflow YAML from the PR's head ref, not the updated base branch. Any PR opened/last-pushed before this change lands on main has a head branch whose tests.yml/compatibility.yml still lack the required-consumer/required-compatibility jobs. If the ruleset is switched on immediately after this merges (per docs/branch-protection.md's own instruction), those PRs will show "Expected — waiting for status to be reported" indefinitely and can't merge until rebased — a rollout hazard the doc doesn't mention.

3. The required checks can run twice per commit and race each other. tests.yml's trigger (unchanged, but now load-bearing) is push with no branch filter plus pull_request — a push to an open PR branch fires both events for the same SHA, each independently computing and posting its own required-consumer status, in different concurrency groups that don't cancel each other. If the two runs disagree (flakiness, timing), branch protection shows whichever status was posted last — a later-finishing push-triggered success can silently overwrite a real PR-run failure, or vice versa. This also means required-consumer (wrapping a single non-matrix validate job with no aggregation benefit, unlike required-compatibility's genuine matrix) is billed twice per push for no reason.

4. The required-check identity has no in-repo protection against drift. docs/branch-protection.md tells a human to select the exact job-name strings required-consumer/required-compatibility in the external ruleset UI, and required-compatibility's needs: [supported] / required-consumer's needs: [validate] reference job ids directly. A future rename or restructuring of either dependency job breaks the needs: reference or silently desyncs the reported check name from what the ruleset expects — nothing in CI cross-checks the doc's claimed names against the live ruleset or the workflow's actual job ids.

5. The repo already has a more robust pattern for exactly this problem, unused here. .github/workflows/issue-branch-policy.yml (pre-existing) publishes its result via the Statuses API with a fixed custom context string, completely decoupled from job-graph structure — renaming jobs there can't change the reported check name. This PR instead hand-rolls two near-identical if: always() + needs: [...] + test "$X" = success blocks (one per workflow file) tying check identity directly to job ids, which is exactly the fragility issue #4 describes. With three other CI PRs in flight in this repo (#26, #28, #29), the next required-gate is as likely to copy this weaker pattern as the sturdier existing one.

@codeforester
codeforester merged commit b461b5a into main Sep 19, 2026
14 checks passed
@codeforester
codeforester deleted the ci/27-20260918-ci-require-consumer-and-compatibility-checks-before-merging branch September 19, 2026 11:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant