Skip to content

ci(sdk-pr-review-gate): green when the agent RAN on the latest commit, not only when it passes #202

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
kamal-kaur04 merged 2 commits into from
Sep 16, 2026
+33 −27
Merged

ci(sdk-pr-review-gate): green when the agent RAN on the latest commit, not only when it passes #202

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
d3083c7
ci(sdk-pr-review-gate): green when the agent RAN on the latest commit…
rounak610 Sep 16, 2026
13b5f31
chore(changeset): auto-generate from PR template (patch)
github-actions[bot] Sep 16, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .changeset/pr-202.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
"@wdio/browserstack-service": patch
---

- N/A — CI/workflow-only change; no customer-facing or package impact.
55 changes: 28 additions & 27 deletions .github/workflows/sdk-pr-review-gate.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,12 +1,15 @@
name: SDK PR Review Gate

# Required check for the SDK PR Review Agent's GTG signal. Green only when BOTH:
# Required check that confirms the SDK PR Review Agent has RUN on the PR's latest commit.
# Green only when BOTH:
# 1. the `ready-for-review` label is present, and
# 2. the SDK PR Review Agent's latest verdict marker reports `state=success` on the
# PR's current head SHA. The agent posts a single signed verdict comment carrying
# `<!-- sdk-pr-review:verdict sha=<HEAD> state=<success|failure|pending> -->`; this
# workflow reads that marker (there is no separate `sdk-pr-review` commit status, so
# the agent's outcome shows as this ONE check, never a raw status + derived check pair).
# 2. the SDK PR Review Agent has posted a verdict marker for the PR's CURRENT head SHA —
# ANY verdict (success | failure | pending). The gate requires only that a review RAN
# on the latest commit; the verdict itself is ADVISORY (read the findings, use your
# judgement) and does NOT block merge. The agent posts a single signed verdict comment
# carrying `<!-- sdk-pr-review:verdict sha=<HEAD> state=<success|failure|pending> -->`;
# this workflow reads that marker (there is no separate `sdk-pr-review` commit status,
# so the outcome shows as this ONE check, never a raw status + derived check pair).
# Native PR-review approval is enforced separately (branch-protection "Require
# approvals"), not by this check.
#
Expand Down Expand Up @@ -81,8 +84,10 @@ jobs:
// Condition 1: ready-for-review label present.
const hasLabel = pr.data.labels.map(l => l.name).includes('ready-for-review');

// Condition 2: the SDK PR Review Agent's verdict marker reports `success` on the
// CURRENT head SHA only — a stale marker from a prior commit must not count.
// Condition 2: the SDK PR Review Agent has RUN on the CURRENT head SHA — i.e. it
// posted a verdict marker (ANY state: success | failure | pending) for this exact
// commit. A stale marker from a prior commit must not count. The verdict itself is
// ADVISORY; the gate only requires that a review ran on the latest commit.
const comments = await github.paginate(github.rest.issues.listComments, {
owner, repo, issue_number: prNumber, per_page: 100
});
Expand All @@ -96,20 +101,14 @@ jobs:
if (m[1] === headSha) verdictState = m[2]; // latest matching marker wins
}
}
const reviewOk = verdictState === 'success';
const reviewRan = verdictState !== null;

const green = hasLabel && reviewOk;
const green = hasLabel && reviewRan;

const reasons = [];
if (!hasLabel) reasons.push('the `ready-for-review` label is not present.');
if (!reviewOk) {
if (verdictState === 'failure') {
reasons.push('the SDK PR Review Agent reported blocking findings (🔴) on the current head commit — fix them and re-run the agent.');
} else if (verdictState === 'pending') {
reasons.push('the SDK PR Review Agent flagged findings that need human review (⚠️) on the current head commit — a reviewer must resolve them before this can go green.');
} else {
reasons.push('the SDK PR Review Agent has not reviewed the current head commit yet — run the SDK PR Review Agent.');
}
if (!reviewRan) {
reasons.push('the SDK PR Review Agent has not been run on the current head commit yet — run the SDK PR Review Agent (its verdict is advisory; this gate only requires that it ran on the latest commit).');
}

core.setOutput('green', green ? 'true' : 'false');
Expand All @@ -126,18 +125,20 @@ jobs:
let body;
if (green) {
body = marker + '\n' +
'🟢 **SDK PR Review gate is green** — the SDK PR Review Agent has given a GTG for this PR ' +
'(the `ready-for-review` label is present and the latest SDK PR Review Agent run ' +
'reports `success` on the current head commit).\n\n' +
'🟢 **SDK PR Review gate is green** — the SDK PR Review Agent has run on the current ' +
'head commit (verdict: `' + verdictState + '`).\n\n' +
'This gate confirms a review **ran** on the latest commit. The verdict itself is ' +
'**advisory** — read the findings and use your judgement; it does not block merge. ' +
'A native GitHub reviewer approval is still separately required by branch protection ' +
'before this PR can merge — this check does not substitute for that.';
'before this PR can merge.';
} else {
const pending = reasons.map(r => '- ' + r.charAt(0).toUpperCase() + r.slice(1)).join('\n');
body = marker + '\n' +
'🔴 **SDK PR Review gate is red.** Pending:\n\n' +
pending + '\n\n' +
'It turns green once the latest SDK PR Review Agent run reports GTG on the current ' +
'head commit. A native reviewer approval is separately required by branch protection before merge.';
'It turns green once the SDK PR Review Agent has run on the current head commit ' +
'(any verdict — the gate only requires that the review ran). A native reviewer ' +
'approval is separately required by branch protection before merge.';
}
await github.rest.issues.createComment({ owner, repo, issue_number: prNumber, body });
}
Expand Down Expand Up @@ -165,9 +166,9 @@ jobs:
|| echo "Slack notify failed (non-fatal)"

- name: Require both gate conditions
# Hard gate: fails the required check whenever either condition (label, SDK
# PR Review Agent GTG verdict) is unmet on a real PR. No job-level `if` above
# can skip it — a skipped required check reads as passing on GitHub.
# Hard gate: fails the required check whenever either condition (the label, or the SDK
# PR Review Agent having RUN on the current head commit) is unmet on a real PR. No
# job-level `if` above can skip it — a skipped required check reads as passing on GitHub.
if: steps.gate.outputs.applicable == 'true' && steps.gate.outputs.green != 'true'
run: |
echo "::error::sdk-pr-review-gate is red: ${{ steps.gate.outputs.reason }}"
Expand Down
Loading