Skip to content

fix(pr-pipeline): arm lane pull requests reliably and say on each why it stays unarmed - #9545

Merged
kyle-sexton merged 10 commits into
mainfrom
agent-ab0dff723afdb6f4c
Oct 24, 2026
Merged

kyle-sexton merged 10 commits into
mainfrom
agent-ab0dff723afdb6f4c

Conversation

@kyle-sexton

@kyle-sexton kyle-sexton commented Oct 24, 2026 •

Copy link
Copy Markdown
Member

No related issue: transient arm failures seen in runs 40051924577 and 40023195885

Summary

pr-merge's enable-auto-merge job logged #9540 arm: fetch failed in scheduled run 40051924577 and #9489 arm: fetch failed in run 40023195885. Quota was not the cause. Each failure followed a verifier read, which runs as a python3 subprocess. The arm mutation then went out on undici's pooled keep-alive connection, which had sat idle past GitHub's keep-alive window. "fetch failed" is undici's TypeError for a network or socket error, and it keeps the cause in err.cause, which the log dropped. One such error left a ready lane pull request unarmed until the next scheduled run, about an hour later.

Fix

  • Fresh connection. Every request the script sends carries Connection: close. Each socket closes after its answer, so no request goes out on an idle pooled connection. A local probe on Node v24.21.0 confirmed that undici forwards the header and that the next request opens a new socket.
  • Retry. A POST that fails is attempted at most three times when the failure is a TypeError "fetch failed", a cause code of ECONNRESET, ETIMEDOUT, UND_ERR_SOCKET or UND_ERR_CONNECT_TIMEOUT, or a cause of "other side closed". The waits use full jitter, capped at 4 s and then 8 s. An HTTP error answer (4xx included) and a GraphQL error response come back as a GitHubError with no cause, so neither is retried.
  • One PR fails, the rest still run. A pull request whose arm or disarm still fails is left as it was. Its ::error line names the PR number and the error's cause code and message. The run moves on to the remaining pull requests and exits 1 at the end.
  • Reuse check. The shared helper retryServerErrors in .github/actions/check-trusted-trigger/check-trusted-trigger.mjs resends only GETs, by design ("Only GETs are idempotent"). GETs stay with it, and the script's own wrapper withRetry covers its GraphQL POSTs.
  • Idempotent arm. A resent arm sends the same expectedHeadOid, so it can arm only the head the run read. GitHub's GraphQL reference (pulls) does not say what a second enablePullRequestAutoMerge on an already-armed pull request returns (UNVERIFIED). There are two possible outcomes, and both are harmless: the call succeeds, or it returns a GraphQL error that the run reports while the pull request stays armed. A redundant disarm behaves the same way.
  • Refusals shown on the pull request. Under merge.lane-prs: native the gate never lists a lanes App pull request. Until now, a refusal from enable-auto-merge was only an Actions log line, and the pull request sat with no visible reason. Now a reader run (the schedule or a hand-started run) writes the refusal on the pull request whenever it leaves the PR unarmed or disarms it for a hold path, a cost path, the gate's verifier verdict, or its risk score.
    • It writes one comment, using the pre-feat(pr-pipeline): merge lanes App pull requests through native auto-merge #9508 gate's reasons marker <!-- babysit-merge-risk --> and the shape of its risk_comment_body: the marker, a line naming at head <sha>, and the reason in a text fence.
    • The comment is edited in place when the reason or head changes. A run where the reason is unchanged writes nothing.
    • A refusal a person must clear also gets the human-gated label, and its comment carries the attended queue's work-items:escalation marker first. These refusals are: a hold or cost path, an issue link the lane does not take, a verdict the gate marks needsHuman, and a risk hold whose comment has stood unchanged for merge.risk-escalation-minutes (the gate's escalation rule; 60 when unset).
    • A refusal the lanes clear on their own, such as a missing or failed verdict, gets the comment only.
    • The label is added when the write turns the comment into a person's. The run also adds it while that person comment stands without the label, so a failed label write is retried. In that state only, the run first reads the pull request's label removals, and it leaves the label off if a person removed it at or after the comment's last edit, or if the timeline did not read whole. Under native, clear-escalations never removes this label, so the comment asks the person to fix the reason and then remove the label.
    • Nothing is written for an armed pull request, on event runs, for holds a person already set (labels, a body hold, a veto), for the lane room, or for failed reads. Report-only logs the write instead.
    • The job's lane token already has issues: write and pull-requests: write (effect merge, .github/actions/resolve-config/effect-grants.json).
  • Logging. Every catch site logs the error's cause through describe, which first scrubs token-shaped text (Bearer …, gh?_…, github_pat_…). The resend ::warning:: line names the arm or disarm and its pull request, but gives only the error's name and cause code (errorLabel), as the shared client does.

Verification

  • node --test in .github/scripts/pr-merge: 294 tests, all pass, after merging main. Refusal-note cases, with the expected bodies written out by hand:
    • A hold path posts the person comment and the needs-human label.
    • A needsHuman verdict is treated as a person's refusal; verify:missing is a lane's and gets the comment only.
    • An unchanged lane comment writes nothing. A person comment with a person's later removal of the label writes nothing.
    • A person comment standing without its label gets the label back: with no removal, a bot's removal, a person's removal before the comment's last edit, or a removal of another label.
    • A timeline that did not read whole puts no label back.
    • A changed reason edits comment 77 in place.
    • A risk hold gets the lane comment at first, nothing at 59 minutes, then an edit to the person comment plus the label at 60.
    • An armed pull request, an event run and report-only write nothing.
    • A disarm is followed by its note.
    • Mutation check: dropping the unchanged check, the reader-run guard, or the risk window each turns its test red.
  • Earlier cases in enable-auto-merge.test.mjs, for the retry change:
    • ECONNRESET, then "other side closed", then success: arms on the third send, with two waits of at most 8 s.
    • Three failures fail the run, and the error line names the cause ECONNRESET read ECONNRESET.
    • PR 1's arm fails three times, PR 2 still arms, and the run exits 1.
    • Each network trigger is retried on its own, and an unlisted cause code is not.
    • A 422 and a GraphQL error are each sent once.
    • Every request carries Connection: close.
    • The warning names the arm and its pull request.
    • A token in a cause message is scrubbed.
  • Mutation check: dropping the header, cutting the attempts to two, and rethrowing on a failed arm each turn the matching new tests red.
  • nice bash scripts/check-affected.sh --uncommitted: 9 passed, 4 skipped.
  • Left out: a live duplicate enablePullRequestAutoMerge call, because arming a real pull request would merge it.

Related

N/A

🤖 Generated with Claude Code

kyle-sexton and others added 3 commits October 23, 2026 22:13
…ork once

enable-auto-merge's GraphQL calls (the listing, discussion reads, arm and
disarm) are POSTs, which createGitHub's retryServerErrors never resends. A
"fetch failed" TypeError or an ECONNRESET, ETIMEDOUT, UND_ERR_SOCKET or
UND_ERR_CONNECT_TIMEOUT cause is now sent once more after 3 s; an HTTP
error answer or a GraphQL error is not. A retried arm passes the same
expectedHeadOid. Failure lines name the cause's code and message, tokens
scrubbed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…st up to twice

The arm failures came after the verifier's subprocess: the mutation went out
on undici's pooled keep-alive socket, which had sat idle past GitHub's
keep-alive window. Every request now sends Connection: close, so each one
opens its own connection. A POST that still fails on the network is sent at
most three times in all, with capped full-jitter waits (4 s, then 8 s). The
retry warning names the arm or disarm and its pull request. The remaining
catch sites log the cause, and the scrub also covers github_pat_ tokens.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…rning

The ::error line keeps the scrubbed cause message for diagnosis; the
warning follows the shared client's errorLabel convention, since undici can
put a header value in a message.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kyle-sexton kyle-sexton changed the title fix(pr-pipeline): resend an auto-merge request that fails on the network once fix(pr-pipeline): arm on a fresh connection and resend a failed request up to twice Oct 24, 2026
…r forwarding

Review findings: the jitter caps, the GET guard and createGitHub's
forwarding of the Connection header had no test that fails when broken,
and the github_pat_ scrub had none.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kyle-sexton
kyle-sexton marked this pull request as ready for review October 24, 2026 02:33
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 24, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-24T02:34:54.912038Z aab020a Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@github-actions github-actions Bot added the needs-human Human-in-the-loop required; autonomous sessions must not resolve items carrying this. label Oct 24, 2026
@github-actions

Copy link
Copy Markdown
Contributor

claude-review holds this pull request for a person (trust-gate) at aab020ab5823017809b8d2500b6e894ebf46dc67.

Changed files match hold-paths: .github/actions/check-trusted-trigger/check-trusted-trigger.test.mjs.

claude-lane: claude-review

@github-actions

Copy link
Copy Markdown
Contributor

claude-security-review holds this pull request for a person (trust-gate) at aab020ab5823017809b8d2500b6e894ebf46dc67.

Changed files match hold-paths: .github/actions/check-trusted-trigger/check-trusted-trigger.test.mjs.

claude-lane: claude-security-review

@github-actions

Copy link
Copy Markdown
Contributor

claude-security-review holds this pull request for a person (trust-gate) at c6d90e6129821593f0fc5e255aa74f7802726699.

Changed files match hold-paths: .github/actions/check-trusted-trigger/check-trusted-trigger.test.mjs.

claude-lane: claude-security-review

@github-actions

github-actions Bot commented Oct 24, 2026 •

Copy link
Copy Markdown
Contributor

claude-review reviewed cd7b9dea61ec6ead98884cae0eac24173c582f2c: 0 finding(s), 0 as inline comments.

Incremental review of the three files changed since b9f5483. I found nothing to flag.

  • Retry scope (enable-auto-merge.mjs:209): path !== "/graphql" replaces method === "GET", so REST writes such as the refusal comment, the PATCH and the label POST are no longer resent. A lost answer can no longer post a comment twice. GETs are still left to createGitHub's own retry. Every GraphQL call goes through the /graphql path, so the arm, disarm and read queries keep their retry. A repeated read query is safe.
  • continue after a failed arm or disarm (enable-auto-merge.mjs:733): for a failed arm this changes nothing, because only the disarm branch wrote a note afterwards. For a failed disarm it stops the run writing "unarmed" on a pull request that is still armed. failed is still incremented, so the run still exits 1.
  • Tests: the two new tests cover a failed disarm and a refusal comment that fails on the network. The failed-disarm test checks that no note is written, the exit code is 1 and the error line names the disarm. The comment test checks that exactly one POST is sent and that nothing sleeps. Both exercise the real withRetry, because main wraps the mocked github. I did not run the tests. The 294-test pass in the PR description is the author's claim, and I have not confirmed it.
  • Changelog fragment: the wording now says a REST write is never resent, which matches the code.
  • Closing issue: none. The PR says "No related issue".
  • Hold classes: none apply. The change adds no model calls and touches no trusted-actors, CODEOWNERS or .gitmodules files.

@github-actions

Copy link
Copy Markdown
Contributor

claude-review holds this pull request for a person (trust-gate) at c6d90e6129821593f0fc5e255aa74f7802726699.

The PR adds a test to .github/actions/check-trusted-trigger/check-trusted-trigger.test.mjs, a file under .github/actions/check-trusted-trigger/**. It is test-only and the action's source is unchanged, but this path is a trust-gate class that needs a person's sign-off.

Changed files match hold-paths: .github/actions/check-trusted-trigger/check-trusted-trigger.test.mjs.

claude-lane: claude-review

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
kyle-sexton and others added 2 commits October 23, 2026 23:06
…s it unarmed

Under merge.lane-prs native the gate never lists a lanes App pull request,
so a refusal enable-auto-merge read left only a log line. A reader run now
writes one babysit-merge-risk comment naming the head and reason, edited in
place and left alone while unchanged, and adds the human-gated label for a
refusal a person must clear: a hold or cost path, a link the lane does not
take, a needsHuman verdict, or a risk hold that stood
merge.risk-escalation-minutes, as the gate escalates it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
# Conflicts:
#	.github/scripts/pr-merge/enable-auto-merge.test.mjs
@kyle-sexton kyle-sexton changed the title fix(pr-pipeline): arm on a fresh connection and resend a failed request up to twice fix(pr-pipeline): arm lane pull requests reliably and say on each why it stays unarmed Oct 24, 2026
@github-actions

github-actions Bot commented Oct 24, 2026 •

Copy link
Copy Markdown
Contributor

claude-security-review reviewed cd7b9dea61ec6ead98884cae0eac24173c582f2c: 0 finding(s), 0 as inline comments.

Incremental security review of the three files changed since b9f5483. I found no security issues.

critical-findings: 0

  • Retry narrowed to GraphQL (.github/scripts/pr-merge/enable-auto-merge.mjs, withRetry). The retry condition changed from method === "GET" to path !== "/graphql". REST writes such as the refusal comment, the label write and the comment PATCH are no longer resent, so a lost answer cannot post a duplicate. This narrows the behaviour. A resent arm still carries the same expectedHeadOid, so the head it arms does not change.
  • Failed arm or disarm now skips the note (enable-auto-merge.mjs, new continue in the catch block). A pull request still armed after a failed disarm no longer gets a comment or label saying it is unarmed. The continue skips only the post-action note call. failed is still incremented and the run still exits 1. No guard, gate or label write that should run after a failure is bypassed.
  • Tests and changelog. The new tests cover a failed disarm writing no note and a refusal comment not being resent. The .changes entry only describes the change. None of these touch a trust boundary, a token path or a manifest.
  • Not consulted. The instruction-surface lens did not apply, because no standing instruction was deleted or softened. The diff has no manifest changes, so there are no manifest <field> lines. No trust-gate paths are touched, so I set no hold.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

claude-review: 2 inline finding(s) on b667b314d97c1e612617f325ca5b7d63355daf0c; the summary is in this lane's pull request comment.

claude-lane: claude-review

Comment thread .github/scripts/pr-merge/enable-auto-merge.mjs Outdated
Comment thread .github/scripts/pr-merge/enable-auto-merge.mjs
kyle-sexton added a commit to melodic-software/standards that referenced this pull request Oct 24, 2026
…contract (#864)

No related issue: follow-up to
[ci-workflows#874](melodic-software/ci-workflows#874),
which fixed a review-lane failure on
[claude-code-plugins#9545](melodic-software/claude-code-plugins#9545).

## Summary

Registers `pr-review-cli.yml@b49ac366bb3426039363a6927c41512e44533808`
(ci-workflows
[v0.65.1](https://github.com/melodic-software/ci-workflows/releases/tag/v0.65.1)),
so claude-code-plugins' two review callers can re-pin to the release
that passes the review between jobs as an artifact.

## Fix

v0.65.1 changes only `pr-review-cli.yml`'s jobs and its tests: the
review and the last marker's summary now travel as a checked artifact
instead of a job output, which the runner dropped whenever the text
contained a registered secret's value. The workflow's `on.workflow_call`
inputs, secrets and runner did not change (`git diff d02b8697..b49ac366
-- .github/workflows/pr-review-cli.yml` touches the header comment and
`jobs:` only), so the contract copies its v0.65.0 entry forward
verbatim.

## Verification

- `node --test runner-policy.test.mjs` in `components/runner-policy`:
343 pass, 0 fail. That includes the new copy-forward test, which checks
the new entry deep-equals the v0.65.0 entry and that exactly one
contract names the new SHA.
- `npx --no-install @biomejs/biome check` (2.5.15) on both changed
files: clean.

## Related

- Same shape as
[#856](#856) (v0.65.0)
and [#853](#853)
(v0.64.2).
- Next: the standards sync into claude-code-plugins, then that
repository's review-caller re-pin.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…fusal comment

A label write that failed after its person comment was never retried: every
later run saw the comment unchanged and wrote nothing. When the lanes App's
person comment stands and the label is off, the run now reads the pull
request's label removals and adds the label back unless a person removed it
at or after the comment's last edit. A timeline not read whole adds nothing.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions github-actions Bot added needs-human Human-in-the-loop required; autonomous sessions must not resolve items carrying this. labels Oct 24, 2026
@kyle-sexton kyle-sexton removed the needs-human Human-in-the-loop required; autonomous sessions must not resolve items carrying this. label Oct 24, 2026
…y once it succeeds

The refusal note's comment POST is not idempotent, so a lost answer could
post a second marker comment; only /graphql requests are resent now. A
disarm that threw left the pull request armed, yet the note followed and
said it was unarmed; the note now runs only after a disarm that succeeded.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kyle-sexton
kyle-sexton added this pull request to the merge queue Oct 24, 2026
Merged via the queue into main with commit eda837d Oct 24, 2026
34 checks passed
@kyle-sexton
kyle-sexton deleted the agent-ab0dff723afdb6f4c branch October 24, 2026 03:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant