Skip to content

quest: close the parallel-frontend measurement; the queue keeps stable rustc - #7900

Merged
kixelated merged 3 commits into
mainfrom
quest/m0/ci/nightly-levers
Nov 5, 2026
Merged

kixelated merged 3 commits into
mainfrom
quest/m0/ci/nightly-levers

Conversation

@kixelated

@kixelated kixelated commented Nov 4, 2026 •

Copy link
Copy Markdown
Collaborator

Closes quest/m0/ci/nightly-levers.md.

Problem

The quest asked for a report: does building the merge queue's tests on a pinned nightly with the parallel frontend save enough wall time to justify the risks? It was to be measured once the structural queue changes had landed. Cranelift was ruled out on 2026-10-27.

That precondition holds. The structural baseline was the landing quest, which was Required until moq-dev/moq#7828 closed it. Parallel modules (moq-dev/moq#6617) and one test build (moq-dev/moq#6616) had landed before that. The maximumEntriesToBuild: 2 trial (queue-concurrency) was not kept as a blocker, and it does not change what a single entry compiles.

Approach

Verdict: don't adopt. The queue keeps stable rustc. The quest is deleted, and the measurement is recorded in the CI epic.

1. moq-dev/rust2's build benchmark (first, as the Plan says)

The jobs-frontend variant is pinned nightly + -Zunstable-options --jobs-frontend=8, measured against the same nightly on 4-CPU hosted runners. It covers moq-relay only, with no test builds. Over the last 8 scheduled runs, 2026-10-27 to 11-03 (from run 41657425644 back):

scenario wall vs nightly CPU vs nightly A/A control (wall)
clean build +1.2% to +7.1% (one run -1.4%) +8.7% to +15% -0.8% to +3.9%
edit-hub (moq-net edit) -1.6% to -5.7% +7.7% to +11.7% -1.3% to +1%

Its moq nightly job has failed since 11-04 because the disk fills up (cache-proc-macros hits ENOSPC). That is a follow-up for rust2.

2. Where the queue spends time (27 successful merge-group runs, 2026-11-03 to 11-04)

Source: the queue's cargo-timings artifacts, matched to each job's Check step.

  • The Rust test build takes a median of 172 s of a 345 s job, which is 37% of summed job time.
  • The queue host runs CARGO_BUILD_JOBS=8 in a container with a 16-CPU quota (kixelated/nixos modules/ci-runners.nix).
  • On average 6.7 of the 8 cargo slots are busy. Cargo runs fewer than 8 units for 27% of the build, and 2 or fewer for 16%.
  • That under-filled time is the only place spare frontend threads could help. So the ceiling is roughly 10% of job time, and only if those units' frontend parallelized perfectly.

3. The queue's test build, measured directly

Setup:

  • Build: cargo nextest run --no-run with the queue's exact selection for a whole-workspace diff (features, --benches --examples, clippy-driver as the workspace wrapper).
  • Flags: CARGO_BUILD_JOBS=8, CARGO_INCREMENTAL=0.
  • Toolchains: stable 1.98.1, and the flake's pinned nightly 2026-10-18 with clippy.
  • Each edit run rewrites a marker in moq-net's crate root (rust2's test-edit). Every unit that doesn't depend on it stays fresh, as mbx serves those units in the queue.
  • Variants have separate target directories, run in 5 interleaved rounds with rotating order, plus an A/A nightly-control.

Host: nixos2, 12 CPUs, load about 3-5 before the run. desktop was at load 26-74 with other agents and nixos is the live queue, so neither was used. A first attempt on desktop was discarded for load.

variant trimmed wall s vs base CPU s vs base peak cargo RSS walls
stable (today's queue) 186.0 base 1290 base 2.9 GiB 174 178 185 195 231
nightly 191.5 +2.9% vs stable 1243 -3.7% 3.1 GiB 178 180 195 199 222
nightly-control (A/A) 203.6 +6.4% vs nightly 1293 +4.0% 3.1 GiB 169 174 199 239 253
nightly + --jobs-frontend=8 220.4 +15.1% vs nightly 1505 +21.1% 3.8 GiB 210 214 216 231 234
  • Every --jobs-frontend=8 run (210-234 s) was slower than the median of the 10 plain-nightly runs (196 s).
  • The critical unit, moq_net lib (test), took 103-125 s with the flag and 95-137 s without. It got no faster, while total unit-seconds rose.
  • The clean warm-up builds show the same shape: 365 s stable, 367 s nightly, 377 s frontend.

Risks the Plan listed

  • The gate on the toolchain that ships: confirmed, and it already bites. The pinned nightly deprecates AtomicUsize::fetch_update (renamed try_update), and the gate's build.warnings = deny fails on rs/kio/src/consumer.rs. The timed runs therefore allowed warnings for every variant. Adopting a nightly means chasing deprecations that stable doesn't have yet, or splitting the lint toolchain.
  • Incremental races: don't apply. The queue builds without incremental compilation. Upstream still has open parallel-frontend bugs, such as a query-feeding race (rust-lang/rust#162316) and diagnostic ordering (rust-lang/rust#142063). The default of 2 frontend threads on nightly (rust-lang/rust#162848) is still unmerged. --jobs-frontend replaced -Zthreads in rust-lang/rust#160697.
  • Cycle errors: none seen in 6 frontend builds.
  • mbx key split: moot for the queue. It uses a local-only mbx store, so a nightly would only warm keys of its own and never reads the hosted Nightly's R2 writes.

Impact

None. Quest docs only: quest/m0/ci/nightly-levers.md is deleted, and quest/m0/ci/README.md drops its Required link and records the measurement.

Public API: none. Wire: none.

Alternatives

  • Adopt the parallel frontend with fewer threads (--jobs-frontend=2, the planned nightly default). Not measured. With 8 threads the critical unit did not get faster, and the extra CPU slowed the rest of the build. A smaller count can at best close part of that loss, and it would still carry the nightly gate breakage.
  • Measure on the queue host itself. Rejected for now. nixos is live and a sibling is measuring merge-queue throughput there. nixos2's 12 CPUs leave fewer spare cores than the queue's 16-CPU quota, which could understate the gain. But rust2's 4-CPU runs and these 12-CPU runs agree, and the under-filled share of the queue's own builds caps the gain at a few percent of job time.

Choices made without asking

  • Measured on nixos2, since desktop was too loaded and nixos runs the live queue.
  • Allowed warnings in every timed variant, because the nightly fails the gate's deny on a deprecation.
  • Deleted the quest rather than handing adoption to the maintainer, since the measurement does not justify adopting.

Follow-ups

  • moq-dev/rust2: the scheduled Bench (moq nightly) job runs out of disk on cache-proc-macros (since run 41853203857), and Bench (moq nix) has failed every scheduled run since 10-27. Worth a rust2 quest so its numbers stay current.

Validation

  • just ci origin/main passes (quest, markdown, text, gh, and the rest the diff selects).
  • quest check --base origin/main passes.

Review

  • Reviewer: Codex (quest review), gpt-6.1-sol, medium effort, round 1.
  • Reviewed head: 3d8f238, against origin/main at 56b28d2.
  • Findings: none.

(Written by Claude Opus 5.5)

🤖 Generated with Claude Code

@kixelated kixelated changed the title Measure the parallel frontend in the queue quest: close the parallel-frontend measurement; the queue keeps stable rustc Nov 4, 2026
@kixelated
kixelated marked this pull request as ready for review November 4, 2026 22:06
@coderabbitai

coderabbitai Bot commented Nov 4, 2026

Copy link
Copy Markdown
Contributor

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 3 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used all 4 included reviews currently available.

Learn how review limits work.

Review configuration:

Run configuration
  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 1cbbddae-030c-4a83-a485-8ae92f982a8d
Commits

Reviewing files that changed from the base of the PR and between 4ecc744 and 3d8f238.

Files selected for processing: 2
  • quest/m0/ci/README.md
  • quest/m0/ci/nightly-levers.md
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@kixelated kixelated left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated review by review (OpenAI)

Reviewed head 3d8f23822d082cd2675ce5780b47abbe3a641f24 against base 4ecc744d381abb889dcafc0421b28f6cbb5ad313. No actionable finding in the inspected measurement and quest closure. The report addresses the original conditional adoption question, states the host and warning-policy differences and A/A spread, and distinguishes the unmeasured smaller frontend count. The checked rust2 nightly job and sampled clean/edit timings are consistent with the direction reported; the frontend worker pool also participates in Cargo's jobserver rather than freely using every physical CPU.

Verification: GitHub source and documentary review only. I did not reproduce the benchmarks, independently recalculate all 27 timing artifacts, or verify the local five-round raw measurements. No CI configuration changes are made by this PR.

(Written by review (OpenAI))

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ No new issues found.

Reviewed changes — this run reviewed the complete 2-file diff closing the parallel-frontend measurement quest.

  • Quest doc deleted — quest/m0/ci/nightly-levers.md is removed; its goal, plan, and Cranelift decision no longer live as a standalone document.
  • CI epic updated — quest/m0/ci/README.md drops the quest's Required link and records the measured result in the Plan section: the pinned nightly's --jobs-frontend=8 ran 15% slower and used 21% more CPU against a 6% A/A spread, without shortening the critical moq-net test unit, and the nightly fails the gate on a fetch_update deprecation.
  • No stale references — a repo-wide search finds no remaining links to nightly-levers.md, and the README summary's numbers match the evidence in the PR body (queue build share, rust2 benchmark ranges, risk notes).

Pullfrog  | View workflow run | Using deepseek-v4.1-flash-1017 (free via Pullfrog for OSS) | 𝕏

@kixelated
kixelated added this pull request to the merge queue Nov 5, 2026
Merged via the queue into main with commit c150456 Nov 5, 2026
14 checks passed
@kixelated
kixelated deleted the quest/m0/ci/nightly-levers branch November 5, 2026 06:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant