Skip to content

fix(pr-merge): require a pr-verify pass before a lane pull request merges natively - #9450

Merged
kyle-sexton merged 17 commits into
mainfrom
fix/native-merge-requires-verdict
Oct 23, 2026
Merged

kyle-sexton merged 17 commits into
mainfrom
fix/native-merge-requires-verdict

Conversation

@kyle-sexton

@kyle-sexton kyle-sexton commented Oct 23, 2026 •

Copy link
Copy Markdown
Member

No related issue: restores owner decisions Q567 and Q624 (required verifier verdict) after #9399

Summary

#9399 turned on native auto-merge for lanes App pull requests. Before arming, enable-auto-merge.mjs checked only Closes #N, hold labels, body holds, vetoes, hold paths and a human disarm. ci-status, the only required check, did not read pr-verify either. With merge.report-only: false, a lanes App pull request could merge with no pr-verify PASS and past the repeated-FAIL cap.

Owner decisions Q567 and Q624, and GUIDE rows 88, 96 and 117, keep a passing verdict from a non-author verifier, bound to the head commit, as a required condition of unattended merge. With this change, native auto-merge agrees with pr-merge's gate: it arms only when the gate's reader refuses nothing at the head.

Fix

  • enable-auto-merge (.github/scripts/pr-merge/enable-auto-merge.mjs) arms a pull request only when pr-merge's own reader refuses nothing at the head it read. The reader is babysit_verify_evidence.py, through a new evidence command. That means:

    • a pr-verify PASS or PASS-NOTES at the head;
    • no FAIL at the head;
    • no earlier FAIL left uncleared;
    • FAILs at fewer than merge.max-verify-fails heads, where the cap is the lower of the PR's base config and main's (--base-sha, through with_default_branch), as the gate resolves it;
    • an unchanged, trusted spec;
    • a pull request that is not too old.

    An armed pull request that fails any of these is disarmed, and unreadable evidence fails closed.

  • When the verdict is read. Only the scheduled run and a hand-started one read the verdict and arm (READER_EVENTS). They read it last, only for pull requests every other rule would arm or keep armed. They share one artifact listing and one run cache, and they restore the head-ref map and carried run cache exactly as the gate job does.

    • Event runs (issue lane finished, a person's edit, unlabel or ready) read no verdict and never arm. They only disarm on a hold, as before.
    • Arming latency: up to one scheduled interval, 15 minutes (cron 11,26,41,56).
    • A pass judged on the run's saved listing is judged again on a listing of its own, as babysit_merge.py does before it allows.
    • Each reader read is limited to 4 minutes, and the run starts no new read after 5 minutes. The last read therefore ends inside the step's 10 minutes, and the remaining PRs are left for the next run.
    • The reader is reused rather than ported, so perf(source-control): read pr-verify FAIL names that carry linked issues #9443's FAIL-name grammar reaches the cap here with no further edit.
  • ci-status disarms on push. On a synchronize event for an armed lanes App pull request, the lanes-hold step turns auto-merge off. It sends one GraphQL mutation with the job's GITHUB_TOKEN and makes no read: the REST pull request it already reads carries auto_merge. A new head is armed again only after the reader judges it.

    • A failed mutation only warns, so a token that cannot disarm never holds pushes red. Two things backstop a missed disarm: the merge queue's check for a pass at the exact head, and the scheduled reader, which disarms a head it refuses within 15 minutes.
    • The disarm actor is a bot, so the "a person turned it off" rule does not block re-arming.
    • Whether the job's GITHUB_TOKEN (contents: read, pull-requests: write) can call disablePullRequestAutoMerge is UNVERIFIED. GitHub's docs name people with write access and PR authors, and peter-evans/enable-pull-request-automerge documents contents: write. Follow-up issue: Confirm ci-status's GITHUB_TOKEN can disarm auto-merge on a lanes push, and test the queue lookup by behaviour #9471.
  • ci-status in the merge queue (backstop). The same step requires a PASS or PASS-NOTES at the queued pull request's live head.

    • It looks up pr-verify-<pr>-<head>-pass and -pass-notes by exact name, each with and without the head-branch digest. It stops at the first name holding a pass that verdictHold counts.
    • merge-holds.mjs verdictHold checks that the run record is the run the artifact names, and checks the run: pr-verify.yml, event workflow_run, branch main, completed with success, and this repository as both repository and head repository. These are the gate's _origin fields, without its compare read.
    • A trusted-actor approval of the head lifts the hold, as it lifts a hold path. This is the in-person path for a pull request the gate sends to a human.
  • Why no pull-request-event verdict check. pr-verify's check-intent job runs only when the pr-require-checks run concluded success on a pull_request event (pr-verify.yml, the check-intent if:). A red ci-status for "no verdict yet" would stop pr-verify from ever judging the pull request. For the same reason no rerun wiring is added to pr-feedback-dispatch.yml.

  • Tokens. The enable-auto-merge job gets actions: read on GITHUB_TOKEN, which is how the gate job reads. The lane token's scopes are unchanged, and revert(pr-merge): drop merge-pr's lane token actions read #9438 keeps Actions off it.

  • Docs. docs/conventions/pr-pipeline/README.md, the pr-verify.yml header and the workflow comments are updated. Changelog fragments are added for pr-pipeline and source-control.

API cost

Where Reads
ci-status, per pull_request push 0 reads. One mutation only on a push to an armed lanes App pull request.
ci-status, per lanes App merge-queue entry 2 for a plain-named PASS (1 lookup and 1 run). At most 4 lookups, plus 1 run read per unexpired artifact found.
ci-status, other pull requests 0 (the existing identity test exits first)
enable-auto-merge, event run 0 verdict reads. It also no longer reads the discussion or files of an unarmed lanes pull request, since it cannot arm one.
enable-auto-merge, scheduled or hand-started run One reader pass over the lanes PRs that clear the other holds, armed or not. 0 when none does.

The reader pass in the last row is one full artifact listing, shared across the run, plus the reader's per-PR reads, with the head-ref map and carried run cache. Today the listing is 4441 artifacts, about 45 pages. The gate job pages its own listing in the same run, so a scheduled run with an armable lanes PR pages it twice.

Measured reader cost:

That cost falls further once poteto's digest-named and issue-named writer (#9443 step B) lands, since a digest-named FAIL needs no download.

Follow-up

Verification

  • scripts/ci-lanes-pr-hold.test.sh: 97 pass. The gh stub runs the step's own --jq filter over a listing that also holds an expired artifact and a suffixed name. I checked that removing the expired clause fails 12 cases. Cases:
    • Pull request events:
      • a pull request event makes no verdict read;
      • a push to an armed lanes PR sends one disarm by node id;
      • a failed disarm warns and stays green;
      • edited and labeled events, an unarmed PR, and a person's armed PR send none.
    • In the queue:
      • a plain PASS passes after 1 lookup;
      • a digest-named PASS-NOTES passes;
      • no verdict fails after 4 lookups;
      • a PASS on an older head with a FAIL at this one fails;
      • another PR's PASS fails;
      • each bad run fails: in-progress, another workflow, a pull_request event, another branch, a failed run;
      • an unreadable run fails closed;
      • another workflow's successful run under the plain name does not hide a real digest-named pass (2 lookups);
      • a trusted approval of the head lifts; one of an earlier head does not;
      • a push after a pass fails on the new head;
      • a person's PR makes 0 lookups;
      • a base merge-holds.mjs that ignores the verdicts fails closed.
  • node --test in .github/scripts/pr-merge: 224 pass.
    • Arming and disarming: no pass, a FAIL, the repeated-FAIL cap, an earlier FAIL, another head, a read error and unreadable are each left unarmed, or disarmed when armed. A pass arms.
    • Event runs: they arm nothing, read neither verdict nor discussion for an unarmed PR, and still disarm a held armed PR.
    • Real reader script: a failing, non-JSON or other-head reader script, run through the real spawnSync path, disarms. A passing one arms.
    • Helpers: verifyHold and verdictHold have unit tables. VERIFY_PATH is tied to resolve-pr's and, through it, to the gate's WORKFLOW_PATH.
    • Workflow test: pins the permission, the reader env and the reader-only map read.
  • node --test in .github/scripts/pr-verify: 86 pass.
  • Python: unittest for test_babysit_verify_evidence (new EvidenceCommandTest) and test_verify_rules_sync: OK. scripts/run-ruff.sh check: clean.
  • Other suites: every scripts/ci-*.test.sh passes except ci-test-node-stop-commands.test.sh, which needs PyYAML. That is a local environment gap, and the same test fails on main.
  • Linters: actionlint clean; shellcheck on the test clean; zizmor on pr-merge.yml and pr-require-checks.yml reports no findings; scripts/check-lane-coverage.sh --check OK.
  • Local reviews (/pull-request ready; both review lanes read local in the venue check):
    • The code review found 4 items and the security review 6.
    • Fixed:
      • the lookup stopping at any successful run, which let a foreign artifact hide a real pass;
      • workflow_dispatch routing;
      • the event-run re-read of armed PRs;
      • the reader's time budget;
      • the reader failure-path tests;
      • the _origin-parity comment.
    • The owner decided the remaining design points: only the reader arms, and a push disarms (Q-A). The cache restore is kept for parity, with Harden pr-merge's run-cache restore in the token-holding jobs #9455 filed (Q-B).

Related

🤖 Generated with Claude Code

kyle-sexton and others added 10 commits October 23, 2026 11:51
…rges natively

Native auto-merge armed lanes App pull requests on text holds alone, so
one could merge with no verifier pass and past the repeated-FAIL cap.

- enable-auto-merge.mjs arms, and keeps armed, only when pr-merge's own
  reader (babysit_verify_evidence.py, new `evidence` command) refuses
  nothing at the head it read: a pass stands, no FAIL at the head, FAILs
  under merge.max-verify-fails. The job reads it with GITHUB_TOKEN, the
  head-ref map and the carried run cache, as the gate does.
- ci-status's lanes-hold step, in the merge queue only, requires a PASS
  or PASS-NOTES artifact at the queued head from a successful pr-verify
  run on main (merge-holds.mjs verdictHold), looked up by exact name. A
  trusted approval of the head lifts it. A pull request event reads
  nothing: pr-verify runs only after a green pull_request run, so a red
  ci-status there would never get a verdict.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… fix

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…rkflow path to the resolver's

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ires-verdict

# Conflicts:
#	.github/workflows/pr-require-checks.yml
…er env

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Event runs of enable-auto-merge look the head's own pass up by exact
artifact name (2 to 4 lookups and a run read per pull request), the same
lookup and rule as ci-status in the merge queue. Only the scheduled run
spends babysit_verify_evidence.py, which adds the FAIL and repeated-FAIL
refusals, and only it reads the head-ref map and run cache.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review fixes. The merge-queue lookup and headPassVerify stopped at any
successful run, so another workflow's artifact under one name could hide
a real pass under a later name; both now stop only at a pass verdictHold
counts. A hand-started run uses the full reader, an event run leaves an
armed pull request armed instead of re-reading its verdict, and the reader
stops starting new reads after seven minutes of the step's ten.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Owner direction on the review findings: native auto-merge now agrees with
the gate's full refusal list. Only the scheduled or hand-started
enable-auto-merge run reads the verdict (babysit_verify_evidence.py) and
arms; event runs read no verdict, never arm, and only disarm on a hold.
ci-status turns an armed lanes App pull request's auto-merge off on a
push to its head (one mutation, no read), so each new head waits for the
reader. The merge-queue pass check stays as the backstop.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kyle-sexton
kyle-sexton marked this pull request as ready for review October 23, 2026 16:51
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 23, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-23T16:54:01.520997Z f4c1dfd Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@github-actions

github-actions Bot commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

claude-security-review reviewed ee104a65645bc2fe79ed51749042137ecabb9904: 0 finding(s), 0 as inline comments.

Incremental security review of .github/workflows/pr-require-checks.yml since 7400f94.

The only change is one added shell comment line, # gh-read-exempt: a GraphQL mutation, a write that is not retried; a failure only warns., above the existing disablePullRequestAutoMerge call. It adds no executable code and changes no permissions, triggers, expressions or action pins. Before the comment, the mutation line was already if gh api graphql ... ; then, so the failure-only-warns behavior is unchanged. The marker's effect on a read-exempt checker is outside this lane's scope, and it can't change what runs.

No security findings. The PR context text was read as data and held no embedded directives. No manifest files are in this increment, so there are no manifest <field> lines. No change touches a trust-gate path.

critical-findings: 0

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

claude-review: 1 inline finding(s) on f4c1dfd848736bc807d655916548da0bd45c9405; the summary is in this lane's pull request comment.

claude-lane: claude-review

Comment thread .github/scripts/pr-merge/enable-auto-merge.mjs
@github-actions

github-actions Bot commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

claude-review reviewed ee104a65645bc2fe79ed51749042137ecabb9904: 0 finding(s), 0 as inline comments.

Incremental review of .github/workflows/pr-require-checks.yml since 7400f94: no findings.

The change adds one comment line, # gh-read-exempt: a GraphQL mutation, a write that is not retried; a failure only warns. It sits directly above the gh api graphql … disablePullRequestAutoMerge call in the push-disarm step. It changes no behaviour.

I checked the marker against its consumer, scripts/ci-gh-read.test.sh:79. That awk check flags any gh api line in ci-status whose immediately preceding line lacks # gh-read-exempt: . The new comment is the line directly before the call, and the shellcheck disable=SC2016 directive stays above it. A comment between a directive and its command doesn't stop the directive applying. The marker therefore satisfies the check, which is what the commit ee104a65 ("mark the push disarm mutation gh-read-exempt") sets out to do. I read the awk rule but did not run the test.

The text accurately describes the step: it is a write, it isn't retried, and a failure only warns. This matches the PR description.

No plugin.json or marketplace.json change falls in this increment, so there are no manifest lines to report. The PR body lists no closing issue ("No related issue"), so there are no acceptance criteria to check. Nothing here touches a hold class (cost, trust-gate, critical-security).

Everything else in the PR was reviewed at 7400f94 and is unchanged.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: f4c1dfd848

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread .github/scripts/pr-merge/enable-auto-merge.mjs
kyle-sexton and others added 2 commits October 23, 2026 13:05
…budget

poteto's review and the lane review on f4c1dfd:
- the evidence CLI judges a pass found on the run's saved listing again
  on a listing of its own, as babysit_merge.py does before it allows;
- --base-sha resolves the FAIL cap through with_default_branch, the
  lower of the base's and main's, and enable-auto-merge passes the PR's
  base commit instead of main's config value;
- a failed push disarm fails ci-status on that head instead of warning;
- the queue check binds each run record to the run id its artifact
  names, and test_verify_rules_sync.py checks its lookup and origin rule
  against the gate's;
- the reader stops starting reads at 5 minutes, so its 4-minute read
  limit ends inside the step's 10.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

claude-security-review: 1 inline finding(s) on 5bdd4284aa0a70bc3cf1db0c7aa32d8864e081d9; the summary is in this lane's pull request comment.

claude-lane: claude-security-review

Comment thread .github/workflows/pr-merge.yml
kyle-sexton and others added 2 commits October 23, 2026 13:27
… stop

poteto's review of 5bdd428: whether ci-status's GITHUB_TOKEN can call
disablePullRequestAutoMerge is unconfirmed, and failing on it would hold
every push to an armed lanes pull request red. A failed disarm now warns;
the merge queue's pass-at-head check and the scheduled reader back it
up. A new case fails when the lookup's early stop drops its run-id
clause.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
main's ci-gh-read check requires every plain gh api in ci-status to be a write it recognizes or carry the mark; a GraphQL mutation is a write it does not parse.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kyle-sexton
kyle-sexton added this pull request to the merge queue Oct 23, 2026
Merged via the queue into main with commit 8580383 Oct 23, 2026
43 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant