Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions .changes/pr-pipeline/fix-lane-room-recompute-9577a1c3.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
---
bump: patch
---

### Fixed

- A pending lane merge no longer makes `enable-auto-merge` disarm every armed lanes App pull request. The post-merge step takes about as long as the gate's two-minute reuse age, so the lane's room read the report as too old. It now keeps the report while nothing it was read from has moved, and otherwise checks again as the gate does, with the lane records the last run kept, so that check reads them without an artifact or run read each. When the room is held, by a refusal or because it cannot be read, the run raises one warning annotation and step-summary line naming the reason and each lanes App pull request it left unarmed or, in report-only mode, that a disarm would have turned off. A pull request still armed because its disarm failed gets an error annotation and summary line instead. It posts no comment on the pull requests.
7 changes: 7 additions & 0 deletions .changes/source-control/fix-lane-room-recompute-9577a1c3.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
---
bump: patch
---

### Fixed

- `babysit_verify_evidence.py lane-room` no longer reports the lane's room unreadable when the run's post-merge answer is past the gate's two-minute reuse age only because a lane merge is still unverified (`unverified-stale`). It keeps the answer when nothing it was read from has moved: main's head, its config commit, the demotion records, and the unverified merges' full and canary runs (`babysit_postmerge.pending_runs_unchanged`, one run listing: none reached a conclusion since, none is past the dispatch deadline, and no quarantine that excused a failing suite has expired). Otherwise it runs the post-merge check again as the gate does (`babysit_merge_policy.lane_room` with the gate's `post_merge`), so a failure found since still refuses. A check that fails or cannot be read leaves the room unreadable, and every other refusal reads as before. `babysit_postmerge.gate_reader` is the one post-merge reader the gate and the room share. The post-merge report now carries `excusesExpireAt`, when the earliest quarantine that excused a failing suite expires. `SINCE_CLOCK_MARGIN` is now 60 seconds, which covers clock skew only, because `computedAt` is stamped before the reads it covers. It applies to the demotion re-check and to the run check.
60 changes: 52 additions & 8 deletions .github/scripts/pr-merge/enable-auto-merge.mjs

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The pr-verify verifier failed head d6fbf9bcb72b2c760c5d469b7e26dbe535f57535. What it wrote, as data:

Summary: Issue #9577: the PR delivers what the issue asks for. On `unverified-stale`, `lane_room` now recomputes the post-merge answer as the gate does, and a failure found since `computedAt` still refuses. A new test shows a pending merge more than 2 minutes old giving a counted room. The scope-addition comment is also met: one warning annotation plus a step-summary line naming the reason and each pull request left unarmed. It fails for one reason. To make its cheap reuse path work, the PR cuts the shared `SINCE_CLOCK_MARGIN` from 5 minutes to 60 seconds. That constant also drives the gate's demotion re-check, so a merge-safety guard is weakened outside the issue's scope.
- PR body: its claims (tests failing on earlier heads, measured timings, a full suite run) cannot be checked from the diff, and the body itself labels the post-change wall time of the enable-auto-merge full check UNVERIFIED. I found no text trying to steer a reviewer.
- .github/scripts/pr-merge/enable-auto-merge.mjs:794: the per-pull-request `const held` inside the loop shadows the later `const held = (state) => ...` helper in the enclosing block. It works, but renaming one of them (for example `heldList`) would make it clearer.

Original file line number Diff line number Diff line change
Expand Up @@ -53,7 +53,13 @@
// lane's unverified merges and the armed pull requests not yet queued, it
// arms; at the cap it arms nothing more. A room that cannot be read (no
// post-merge report, a failed read) holds as a refusal does: nothing is
// armed and every armed lanes App pull request is disarmed.
// armed and every armed lanes App pull request is disarmed. A report past
// its reuse age only for a lane merge still unverified is kept while
// nothing it was read from has moved, else checked again as the gate
// does. A held room raises one warning annotation and step-summary line
// naming why and each lane pull request left unarmed or, report-only,
// that a disarm would have turned off, and one error and summary line
// naming each one still armed because its disarm failed.
// - pr-merge's gate would not hold it on its risk score: `evidence
// --lane-reviews --rung <the room's rung>` runs the gate's scorer
// (babysit_risk.score, holding) over the inputs the gate builds, the
Expand Down Expand Up @@ -108,7 +114,7 @@
// run cache, as the gate's
// GITHUB_API_URL
import { spawnSync } from "node:child_process";
import { readFileSync } from "node:fs";
import { appendFileSync, readFileSync } from "node:fs";
import process from "node:process";
import { pathToFileURL } from "node:url";

Expand Down Expand Up @@ -619,22 +625,26 @@ export async function main({ env = process.env, github, verify, room, sleep = (m
let inFlight = lane === null ? 0 : lane.count + pulls.filter((pull) => lanes(pull) && pull.autoMergeRequest != null && !lane.queued.includes(pull.number)).length;
// Why the room keeps `pull` from arming, with `disarm` when it also turns
// an armed one off; null when it has room.
// An unknown room holds as a refusal does: unknown holds.
const laneHeld =
lane?.refusal != null ? `pr-merge's gate holds its lane off (${lane.refusal})` : lane?.unreadable != null ? `the lane's room could not be read (${lane.unreadable})` : null;
const roomHold = (pull) => {
if (lane === null || !lanes(pull)) {
return null;
}
if (lane.refusal !== null) {
return { disarm: true, reason: `pr-merge's gate holds its lane off (${lane.refusal})` };
}
// An unknown room holds as a refusal does: unknown holds.
if (lane.unreadable !== null) {
return { disarm: true, reason: `the lane's room could not be read (${lane.unreadable})` };
if (laneHeld !== null) {
return { disarm: true, reason: laneHeld };
}
if (pull.autoMergeRequest != null) {
return null;
}
return inFlight >= lane.cap ? { disarm: false, reason: `merge.max-in-flight is ${lane.cap} and ${inFlight} merges are in flight` } : null;
};
// Each lane pull request the room's refusal or unread room holds, by
// what became of it: unarmed (never armed, or disarmed), armed (its
// disarm failed, so the hold is not enforced) or report-only (a disarm
// would have run).
const roomHeld = new Map();
let failed = 0;
const minutes = riskEscalationMinutes(env.RESOLVED_CONFIG ?? "");
// A reader run's per pull request refusal (a hold path or cost hold, or
Expand Down Expand Up @@ -684,6 +694,9 @@ export async function main({ env = process.env, github, verify, room, sleep = (m
let detail = null;
// Checked before the reads, so a held lane spends none.
const held = ["read", "verify", "arm"].includes(action) ? roomHold(pull) : null;
if (held?.disarm) {
roomHeld.set(pull.number, pull.autoMergeRequest != null ? "armed" : "unarmed");
}
if (held !== null) {
({ action, reason } = { action: held.disarm && pull.autoMergeRequest != null ? "disarm" : "none", reason: held.reason });
}
Expand Down Expand Up @@ -736,6 +749,9 @@ export async function main({ env = process.env, github, verify, room, sleep = (m
continue;
}
if (!write) {
if (roomHeld.get(pull.number) === "armed") {
roomHeld.set(pull.number, "report-only");
}
console.log(`enable-auto-merge: #${pull.number} would ${action} (report-only): ${escapeCommand(reason)}`);
await note(pull, detail, { clears, reason });
continue;
Expand All @@ -757,6 +773,9 @@ export async function main({ env = process.env, github, verify, room, sleep = (m
console.log(`enable-auto-merge: #${pull.number} armed at ${pull.headRefOid}`);
} else {
await github("POST", "/graphql", { query: DISARM_MUTATION, variables: { id: pull.id } });
if (roomHeld.has(pull.number)) {
roomHeld.set(pull.number, "unarmed");
}
console.log(`enable-auto-merge: #${pull.number} disarmed: ${escapeCommand(reason)}`);
}
} catch (error) {
Expand All @@ -769,6 +788,31 @@ export async function main({ env = process.env, github, verify, room, sleep = (m
await note(pull, detail, { clears, reason });
}
}
// One warning (and an error while a disarm failed) and their summary
// lines per run, not a comment per pull request: the room is read again
// every scheduled run.
const held = (state) => [...roomHeld].filter(([, outcome]) => outcome === state).map(([number]) => `#${number}`).join(", ");
const lines = [];
const warned = [
["left unarmed", held("unarmed")],
["report-only, a disarm would have run", held("report-only")],
].filter(([, numbers]) => numbers);
if (warned.length > 0) {
lines.push(["warning", escapeCommand(`${laneHeld}; ${warned.map(([what, numbers]) => `${what}: ${numbers}`).join("; ")}`)]);
}
if (held("armed")) {
lines.push(["error", escapeCommand(`${laneHeld}; still armed, its disarm failed, so the hold is not enforced: ${held("armed")}`)]);
}
for (const [level, text] of lines) {
console.log(`::${level} title=enable-auto-merge::${text}`);
}
if (lines.length > 0 && env.GITHUB_STEP_SUMMARY) {
try {
appendFileSync(env.GITHUB_STEP_SUMMARY, lines.map(([, text]) => `enable-auto-merge: ${text}\n`).join(""));
} catch (error) {
console.log(`enable-auto-merge: the step summary was not written: ${describe(error)}`);
}
}
return failed === 0 ? 0 : 1;
} catch (error) {
console.log(`::error title=enable-auto-merge::${describe(error)}`);
Expand Down
67 changes: 66 additions & 1 deletion .github/scripts/pr-merge/enable-auto-merge.test.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@
// the parity tests read their verdicts from Python.
import assert from "node:assert/strict";
import { spawnSync } from "node:child_process";
import { mkdtempSync, writeFileSync } from "node:fs";
import { mkdtempSync, readFileSync, writeFileSync } from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";
import process from "node:process";
Expand Down Expand Up @@ -1114,6 +1114,71 @@ test("a room that cannot be read arms nothing and disarms every armed lane pull
assert.deepEqual(writes(), [{ mutation: "disarm", id: "PR_2" }]);
});

// A held room that logs only to the console leaves nothing on the run
// saying why nothing armed. One annotation and one summary
// line per run, naming the reason and every lane pull request left unarmed.
test("a held room raises one warning and one step-summary line naming why and each pull request left unarmed", async () => {
const armed = { enabledAt: "2026-10-23T10:00:00Z" };
const pulls = [pull(1), pull(2, { autoMergeRequest: armed }), pull(3, { author: { __typename: "User", login: "kyle-sexton" } }), pull(4, { isDraft: true })];
const cases = [
[{ ...ROOM, unreadable: "post-merge answer unverified-stale" }, "the lane's room could not be read (post-merge answer unverified-stale); left unarmed: #1, #2"],
[{ ...ROOM, refusal: "rung-off:demotion" }, "pr-merge's gate holds its lane off (rung-off:demotion); left unarmed: #1, #2"],
];
for (const [lane, expected] of cases) {
const summary = join(mkdtempSync(join(tmpdir(), "enable-auto-merge-summary-")), "summary.md");
writeFileSync(summary, "");
const { lines } = await run(pulls, { GITHUB_STEP_SUMMARY: summary }, {}, lane);
assert.deepEqual(
lines.filter((line) => line.startsWith("::warning")),
[`::warning title=enable-auto-merge::${expected}`],
);
assert.equal(readFileSync(summary, "utf8"), `enable-auto-merge: ${expected}\n`);
}
// A lane with room raises neither.
const summary = join(mkdtempSync(join(tmpdir(), "enable-auto-merge-summary-")), "summary.md");
writeFileSync(summary, "");
const { lines, writes } = await run(pulls, { GITHUB_STEP_SUMMARY: summary });
assert.deepEqual(writes, [{ mutation: "arm", id: "PR_1", head: HEAD }]);
assert.deepEqual(
lines.filter((line) => line.startsWith("::warning")),
[],
);
assert.equal(readFileSync(summary, "utf8"), "");
});

// A pull request whose disarm failed, or that report-only left armed, is
// still armed: naming it "left unarmed" would say the hold was enforced when
// it was not.
test("a held room names a pull request whose disarm failed as still armed, at error level", async () => {
const armed = { enabledAt: "2026-10-23T10:00:00Z" };
const summary = join(mkdtempSync(join(tmpdir(), "enable-auto-merge-summary-")), "summary.md");
writeFileSync(summary, "");
const lane = { ...ROOM, refusal: "rung-off:demotion" };
const { lines, code } = await run([pull(1), pull(2, { autoMergeRequest: armed })], { GITHUB_STEP_SUMMARY: summary }, { disarmError: new Error("POST /graphql failed with status 502") }, lane);
assert.equal(code, 1);
const why = "pr-merge's gate holds its lane off (rung-off:demotion)";
assert.deepEqual(
lines.filter((line) => /^::(warning|error) title=enable-auto-merge::pr-merge/.test(line)),
[`::warning title=enable-auto-merge::${why}; left unarmed: #1`, `::error title=enable-auto-merge::${why}; still armed, its disarm failed, so the hold is not enforced: #2`],
);
assert.equal(readFileSync(summary, "utf8"), `enable-auto-merge: ${why}; left unarmed: #1\nenable-auto-merge: ${why}; still armed, its disarm failed, so the hold is not enforced: #2\n`);
});

test("a held room in report-only names the armed pull request a disarm would have turned off, not as unarmed", async () => {
const armed = { enabledAt: "2026-10-23T10:00:00Z" };
const summary = join(mkdtempSync(join(tmpdir(), "enable-auto-merge-summary-")), "summary.md");
writeFileSync(summary, "");
const lane = { ...ROOM, unreadable: "post-merge answer missing or unreadable" };
const { lines, writes } = await run([pull(1), pull(2, { autoMergeRequest: armed })], { GITHUB_STEP_SUMMARY: summary, REPORT_ONLY: "true" }, {}, lane);
assert.deepEqual(writes, []);
const expected = "the lane's room could not be read (post-merge answer missing or unreadable); left unarmed: #1; report-only, a disarm would have run: #2";
assert.deepEqual(
lines.filter((line) => line.startsWith("::warning") || line.startsWith("::error")),
[`::warning title=enable-auto-merge::${expected}`],
);
assert.equal(readFileSync(summary, "utf8"), `enable-auto-merge: ${expected}\n`);
});

// An event run reads no room, so it never disarms on one.
test("an event run leaves an armed lane pull request armed whatever the room", async () => {
const armed = { enabledAt: "2026-10-23T10:00:00Z" };
Expand Down
19 changes: 19 additions & 0 deletions .github/scripts/pr-merge/workflow.test.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -218,6 +218,25 @@ test("enable-auto-merge reads the lane's room from this run's post-merge-dispatc
assert.ok(names.indexOf("Read this run's post-merge report") < names.indexOf("Arm and disarm auto-merge"));
});

// A room that checks the post-merge answer again reads the lane records the
// last run kept, as post-merge-dispatch's check does, so the check answers
// them with no artifact or run read each; without them the check can run
// past the room's VERIFY_TIMEOUT_MS.
test("enable-auto-merge hands the room's post-merge check the lane records the last run kept", () => {
const arm = job("enable-auto-merge");
const records = /- name: Read the lane records the last run kept\n([\s\S]*?)(?=\n {6}- name: )/.exec(arm)?.[1];
assert.ok(records, "a lane-records read");
assert.match(records, /\n {8}continue-on-error: true\n/);
assert.match(records, /\n {10}ACTIONS_TOKEN: \$\{\{ github\.token \}\}\n {10}LANE_RECORDS_PATH: \$\{\{ runner\.temp \}\}\/lane-records\.json\n/);
assert.match(records, /\n {8}run: node \.base\/\.github\/scripts\/pr-merge\/lane-records\.mjs read\n/);
const step = /- name: Arm and disarm auto-merge\n([\s\S]*?)(?=\n {6}- name: )/.exec(arm)[1];
assert.match(step, /\n {10}BABYSIT_POSTMERGE_LANE_RECORDS: \$\{\{ runner\.temp \}\}\/lane-records\.json\n/);
const names = [...arm.matchAll(/\n {6}- name: (.*)/g)].map((m) => m[1]);
assert.ok(names.indexOf("Read the lane records the last run kept") < names.indexOf("Arm and disarm auto-merge"));
// Within the job's own permissions: the read needs actions: read alone.
assert.match(arm, /\n {6}actions: read #/);
});

test("enable-auto-merge's step timeouts fit inside its job's", () => {
const arm = job("enable-auto-merge");
const total = Number(/\n {4}timeout-minutes: ([0-9]+)\n/.exec(arm)[1]);
Expand Down
15 changes: 15 additions & 0 deletions .github/workflows/pr-merge.yml
Original file line number Diff line number Diff line change
Expand Up @@ -1496,6 +1496,19 @@ jobs:
name: pr-merge-post-merge-report
path: ${{ runner.temp }}/post-merge

# The lane records earlier checks proved, as post-merge-dispatch reads
# them, for a room whose post-merge answer must be checked again: the
# check answers those records with no read. Any error leaves no file,
# and the check reads each record. Rules: lane-records.mjs.
- name: Read the lane records the last run kept
if: steps.lane-token.outputs.token != '' && (github.event_name == 'schedule' || github.event_name == 'workflow_dispatch')
continue-on-error: true
timeout-minutes: 2
env:
ACTIONS_TOKEN: ${{ github.token }}
LANE_RECORDS_PATH: ${{ runner.temp }}/lane-records.json
run: node .base/.github/scripts/pr-merge/lane-records.mjs read

# While merge.report-only is anything but false it reports each
# decision and writes nothing.
- name: Arm and disarm auto-merge
Expand All @@ -1514,6 +1527,8 @@ jobs:
# rung cap as the merge job passes it, empty meaning no cap.
POSTMERGE_RESULT: ${{ runner.temp }}/post-merge/post-merge.json
RUNG_CAP: ${{ vars.CLAUDE_LANES_MERGE_RUNG_CAP }}
# The lane records read above, for the room's post-merge check.
BABYSIT_POSTMERGE_LANE_RECORDS: ${{ runner.temp }}/lane-records.json
# Name the two tokens' rate-limit readings for the reader's gh calls
# (api-budget.bash): the lane token by default, the job token for
# the Actions reads that carry it.
Expand Down
12 changes: 10 additions & 2 deletions docs/conventions/pr-pipeline/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -1113,12 +1113,20 @@ review, the trust gate and the cost review all hold. Each such run also
waits for the same run's `post-merge-dispatch` job and reads the lane's room from the post-merge
report that job keeps as an artifact (`babysit_verify_evidence.py lane-room`; with no report, or
one the gate would not reuse (`shared_refusal`: too old, a demotion record changed since, or a lane
merge landed since that it does not count), it holds as a refusal does): while the gate holds its lane off (a demotion record,
merge landed since that it does not count), it holds as a refusal does; one past the gate's
two-minute reuse age only because a lane merge is still unverified is kept while main's head, its
config commit, the demotion records and the unverified merges' full and canary runs are as they
were (no run reached a conclusion since, no merge is past the six-hour dispatch deadline, and no
quarantine that excused a failing suite has expired), and otherwise checked again as the gate does,
with the lane records the last run kept, holding only when that check cannot be read): while the gate holds its lane off (a demotion record,
`merge.rung`, the loop setting or the rung cap off, a lane merge that failed its post-merge check)
nothing is armed and every armed lanes App pull request is disarmed; at `merge.max-in-flight`,
counting the queue's lane entries, the lane's unverified merges and armed pull requests the queue
has not taken, nothing more is armed; a room that cannot be read (no report, a failed read) holds
as a refusal does.
as a refusal does. A held room raises one warning annotation and step-summary line naming why and
each lanes App pull request it left unarmed or, report-only, that a disarm would have turned off,
and one error annotation and summary line naming each one still armed because its disarm failed;
it posts no comment on the pull requests.
Only the scheduled run and a hand-started one read
them and arm, last, for a pull request every other rule would arm or keep armed, with the gate's
saved listing, head-ref map and run cache; an event run reads no link or verdict, never arms, and
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -3285,16 +3285,9 @@ def assess_unattended(
the review evidence and the score. Raises `PolicyReadError` when the
pull request or its pipeline config cannot be read."""

def post_merge(floor: datetime, identity: str) -> merge_policy.PostMergeGate:
verdict = postmerge.check(
repo,
agent_identity=identity,
is_human=postmerge._default_is_human(repo, identity, self_logins),
merged_since=floor,
pinned_sha=postmerge.pin_reader(repo, identity),
lane_merges_only=authority == merge_policy.PR_MERGE,
)
return postmerge.gate_from_report(verdict.report())
post_merge = postmerge.gate_reader(
repo, self_logins, lane_merges_only=authority == merge_policy.PR_MERGE
)

def promotion(window_start: datetime | None) -> merge_policy.Promotion:
return merge_policy.read_promotion(
Expand Down
Loading
Loading