Skip to content

ci: skip the merge queue's checks for trees Pull CI already passed - #1288

Merged
zibi-pplx merged 1 commit into
mainfrom
zibi/merge-queue-reuse-pull-ci
Nov 2, 2026
Merged

zibi-pplx merged 1 commit into
mainfrom
zibi/merge-queue-reuse-pull-ci

Conversation

@zibi-pplx

@zibi-pplx zibi-pplx commented Nov 2, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

The merge queue no longer reruns Pull CI's checks on a tree that Pull CI already passed. An entry whose trees match such a run takes two short jobs instead of about six minutes. That is rare: 3 of the 35 successful entries since 2026-10-31 03:00 would have qualified.

Cost to every other entry. Its checks now start after the lookup job: a standard runner's pickup (2-8 s in recent queue runs), a one-file checkout, and three API calls capped at 15 s each. The job is capped at two minutes, after which the checks run anyway. So the queue's median and p90 rise by about that job's duration, while the entries that qualify save about six minutes and their 16-core runner time.

Why. A merge-queue entry runs Pull CI's rust_static and common jobs again on the merge group's commit. Successful Merge Queue CI runs have a median of 5.8 minutes over the last 22. The rerun is worth it when main moved after the PR's last Pull CI run: two PRs that each pass Clippy can still fail it together.

When main has not moved, the merge group's commit has exactly the tree Pull CI tested. For example, #1267's squash commit on main, 0456537, has the tree of the synthetic merge its last Pull CI run tested (c8efea9…). Enqueuing a stack makes this worse: GitHub builds one cumulative merge group per PR, each with its own run, so a two-PR stack ran these checks twice more.

What changes.

  • Pull CI records each pass. kind_paths, which has the synthetic merge checked out, names the record after the base tree and the merge tree (ci/scripts/pull-ci-reuse.py name). space-pull-ci uploads it as an artifact once every check passed, kept for one day.
    • A failure to upload the record never fails the required check, and the upload is capped at two minutes; without a record, the queue runs its checks.
    • The record steps run only when rust_static and common passed.
    • A rerun overwrites it.
  • Merge Queue CI looks it up. A new reuse job takes the trees of the entry's base and merge-group commit and looks for a record (pull-ci-reuse.py lookup). It trusts a record only when all of these hold:
    • it came from .github/workflows/pull-ci.yml on a pull_request event;
    • the run belongs to this repository, and its head is in this repository, not a fork;
    • the run's latest attempt passed space-pull-ci.
  • What happens next. With a trusted record, rust_static and common are skipped, and space-pull-ci requires them to be skipped. Otherwise, including any API error or malformed answer, they run and must pass, as today.
  • Base trees must match too: common's migration guard compares against the base.

When it applies.

  • A PR enqueued before main moves past its last Pull CI run.
  • In a stack whose bottom PR was on main's head when CI ran, each entry whose last Pull CI ran on the current head of the PR below it. A push to a lower PR does not rerun Pull CI on the PR above, so that entry runs its checks.
  • When main moved first, nothing changes.

How often. Of the 35 successful entries since 2026-10-31 03:00, main had moved between the PR's last Pull CI run and the entry in 32. In all 32, the commits in between changed Rust inputs, so these checks had new inputs to test. The merge queue's slowest runs before that date waited 3-17 minutes for a 16-core runner, and raising the runner limit removed that; none since took over 7.6 minutes.

Validation

  • scripts/tests/test_pull_ci_reuse.py drives the script against a scripted gh. It passes 15 tests:
    • a trusted pass is reused, and the log names its run, even after an untrusted record listed first;
    • each of these runs the checks instead: no record, an expired record, a record with another name, a fork's run, another repository's run, another workflow or event, a latest attempt whose space-pull-ci failed while another job passed, an unreadable or malformed API answer;
    • lookup without GITHUB_REPOSITORY exits 2;
    • name gives a synthetic merge's base and merge trees in a git fixture.
    • Removing the required-job, repository or record-name check makes a test fail.
  • I checked the API fields the lookup reads, and the commit-to-tree lookup, against this repository's real runs.
  • Both workflows parse, and the merge-queue jobs depend on reuse as intended.
  • Not yet observed end to end: an entry enqueued after this merges, with main unchanged since its PR's CI, should show Reuse a passing Pull CI run finding the record and both checks skipped.

Severity and risk

Severity: does not apply; this cuts merge-queue time and runner use.

Risk:

  • Checks that depend on time. A reused entry does not rerun checks that read live data, such as the cargo deny advisory database, within the day a record lives. Main CI runs them after the merge, as it does today when an advisory appears after the merge queue.
  • Trust. A record carries no more trust than the PR's own space-pull-ci check, which comes from the same run. Fork runs are ignored.
  • Rollback. Revert this PR. Records then expire unused.

AI assistance: Claude Code drafted the change, its tests and this description.

🤖 Generated with Claude Code

@ac-pplx ac-pplx left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The problem. Every merge-queue entry reruns Pull CI's rust_static and common workflows on the merge-group commit, which takes about six minutes. When nothing has landed on main between the PR's last Pull CI run and the enqueue, that commit has the same tree as the synthetic merge Pull CI already passed, on a base with the same tree. The rerun then repeats work on identical inputs. This PR has Pull CI record each pass as a one-day artifact named after the two trees. The queue skips both workflows when it finds a trusted record for the entry's trees.

Example.

10:00  PR A pushed, main at X. Pull CI tests M = merge(X, A), passes, and uploads
       pull-ci-passed-<tree(X)>-<tree(M)>
10:40  A enqueued, main still at X. Merge group G = squash(A) on X, so tree(G) == tree(M)
before: rust_static and common rerun on G (~6 min)
after:  reuse finds the record, both are skipped, and space-pull-ci, which requires both
        to be skipped, passes
If main has moved to Y first: tree(Y) != tree(X), so no record matches and the checks run as today.

Verdict: approved; 2 threads to resolve before merge.

The tree comparison is sound:

  • A tree includes .github/ and the scripts, so equal trees run the same jobs.
  • Pull CI runs a superset of the queue's steps, because archcheck runs only on pull_request.
  • Both sides resolve the migration guards' base to the same tree. resolve-migration-base.sh uses the synthetic merge's first parent, and the queue passes merge_group.base_sha.
  • Every error, unknown output, and failed or cancelled lookup I traced runs the checks.
  • The new tests are collected, and they ran in this PR's Python tests job.

Is it worth it? By the description's own count, 3 of 35 entries would each skip about six minutes, and every other entry now starts its checks one short job later (item 2). That is still a net saving, mostly in 16-core runner minutes and for stacks, and I think the mechanism is worth keeping.

Before merge (inline threads)

  1. Three trust checks have no test that fails without them
  2. Every entry that does not reuse now waits for the lookup; say so and bound it

Nits

  1. A failing name is silent. .github/workflows/pull-ci.yml:77. A failed command substitution inside printf's arguments does not trigger set -e. The step stays green, writes name=, and records silently stop. That is safe, because the queue then runs the checks. Assign the name first, name="$(python3 -I ci/scripts/pull-ci-reuse.py name)", then printf it.
  2. Gate the record on the results it certifies. .github/workflows/pull-ci.yml:126. The record steps run because the step above them exited 0. Today that implies rust_static and common passed. Adding needs.rust_static.result == 'success' && needs.common.result == 'success' to both steps' if: keeps that meaning if the step above ever gains another way to pass.
  3. The stack claim is broader than the mechanism. .github/workflows/merge-queue-ci.yml:23 and the description. A push to a lower PR does not rerun Pull CI on the PR above it, so that PR's entry finds no record and runs its checks. That is safe, but the claim is broader than what happens. Suggested wording: "for a stack whose bottom PR was on main's head, each entry whose last Pull CI ran on the current head of the PR below it".
  4. The upload step has no timeout. .github/workflows/pull-ci.yml:132. continue-on-error covers a failure but not a stall. Before this PR, the required space-pull-ci job made no network calls that could hang. Add timeout-minutes: 2 to the step; it took 1.5 s in this PR's run.

Comment thread scripts/tests/test_pull_ci_reuse.py Resolved
Comment thread .github/workflows/merge-queue-ci.yml Resolved
A merge-queue entry reruns Pull CI's rust_static and common jobs on the
merge group's commit. When main has not moved since the PR's last Pull
CI run, the merge group's tree is the synthetic merge's tree that run
tested, against a base with the same tree, so the queue reran the same
jobs on identical inputs. A stack enqueued after its CI started one
such run per PR.

Pull CI now records each pass as an artifact named after the base and
synthetic-merge trees, kept for a day. A new reuse job in Merge Queue
CI looks the entry's trees up and skips rust_static and common when it
finds a record from a Pull CI run of this repository's own branches,
not a fork's, whose latest attempt passed space-pull-ci. A missing,
untrusted or unreadable record runs the checks as before. Main CI still
runs every check after the merge.

Every other entry now starts its checks after that lookup: a standard
runner's pickup, a one-file checkout and three API calls, each capped
at 15 s, with the job capped at two minutes.
@zibi-pplx
zibi-pplx force-pushed the zibi/merge-queue-reuse-pull-ci branch from 53a3b8a to 6ae64e0 Compare November 2, 2026 20:59
@zibi-pplx

zibi-pplx commented Nov 2, 2026 •

Copy link
Copy Markdown
Collaborator Author

Thanks for the review. Both threads are fixed in 6ae64e0, along with the nits:

  1. Silent name failure: the step assigns name first, then prints it, so a failing pull-ci-reuse.py name fails the step.
  2. Gate the record: both record steps now also require needs.rust_static.result == 'success' && needs.common.result == 'success'.
  3. Stack wording: the workflow comment and the description now say "each entry whose last Pull CI ran on the current head of the PR below it".
  4. Upload timeout: the upload step has timeout-minutes: 2.

@zibi-pplx
zibi-pplx added this pull request to the merge queue Nov 2, 2026
Merged via the queue into main with commit 1c9974a Nov 2, 2026
33 checks passed
@zibi-pplx
zibi-pplx deleted the zibi/merge-queue-reuse-pull-ci branch November 2, 2026 21:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants