Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 0 additions & 2 deletions .github/workflows/bench.yml
Original file line number Diff line number Diff line change
Expand Up @@ -117,8 +117,6 @@ jobs:
args: --corpus moq --variant nightly --variant jobs-frontend --variant cranelift --variant cache-proc-macros --variant nightly-control
- name: moq ra-jit
args: --corpus moq --variant ra-jit-baseline --variant ra-jit --no-attribute
- name: moq pure-crypto
args: --corpus moq --corpus moq-pure-crypto --variant baseline
- name: pure
args: --corpus regex --corpus wasmparser

Expand Down
17 changes: 8 additions & 9 deletions bench/build/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,8 +18,7 @@ The run also ignores the caller's `CARGO*`, `RUSTC*`, `RUSTDOC*`, `RUSTUP*`, and

## Corpus

- **moq** builds `moq-relay`, which pulls in moq-net, the QUIC stack, and C code through aws-lc-sys.
- **moq-pure-crypto** builds the same `moq-relay` with its crypto on graviola and RustCrypto, so it compiles no C: the run fails if any crate in its build gains a `links` key or pulls in cc or cmake, while pure-Rust build scripts stay allowed.
- **moq** builds `moq-relay`, which pulls in moq-net and the QUIC stack. Its crypto is on rind, which is pure Rust, so it compiles no C: the run fails if any crate in its build gains a `links` key or pulls in cc or cmake, while pure-Rust build scripts stay allowed.
- **regex** and **wasmparser** are pure Rust: the run fails if any crate in their build gains a build script or a `links` key.

Each corpus pins a git revision (and a lockfile when the project commits none).
Expand All @@ -32,7 +31,7 @@ Bump a pin in its own change, since numbers on either side of it do not compare.
- `edit-leaf`: rebuild after a body edit in a crate nothing else in the build depends on.
- `edit-hub`: the same edit in a crate most of the build depends on (moq-net in moq).
- `edit-sync`: rebuild after an edit to the body of a sync function the build calls, in a module of the hub (moq: `goaway::Producer::send`, which moq-relay calls to drain a session).
- `edit-async`: the same in an `async fn` (moq: `Session::closed`, which moq-relay awaits).
- `edit-async`: the same in an `async fn` (moq: `Session::closed`, which moq-relay awaits, a wrapper over the sync `poll_closed`).
- `test-build`: a clean build of the corpus's tests with `cargo nextest run --no-run`, as CI builds them with a cold cache.
- `test-run`: run those tests, already built.
- `test-edit`: rebuild the tests after the hub edit, as CI does when its compilation cache serves every dependency.
Expand All @@ -50,25 +49,24 @@ Over 21 CI runs since the sync, the controls' change had an RMS of 1.3% on `edit

## Variants

A variant is a toolchain plus rustflags.
A variant is a toolchain plus rustflags and cargo config.
Each is compared against its `base`, run in the same job: the levers on stable against `baseline`, the nightly ones against `nightly`.
The `control` variants repeat their base, so their change is the run's noise floor: an A/A control.
Timed runs interleave the variants and rotate which goes first each round, so drift over a long job spreads across all of them.
To measure a new build-speed change, add a variant whose base is the configuration it improves on.

The `ra-jit` variant builds with `cargo ra-jit` ([ra-jit](https://github.com/indexable-inc/ra-jit), pinned in `flake.nix` and compiled from source on a run that selects it), a daemon that compiles only the functions an edit changed and splices them into the program cargo linked, or hands the build back to cargo.
ra-jit works with one rustc only, so its base, `ra-jit-baseline`, is plain cargo on that toolchain.
Both run on the moq corpora only, since ra-jit needs a program to load, and skip `check`, which `cargo ra-jit` fails on moq.
Both run on the moq corpus only, since ra-jit needs a program to load, and skip `check`, which `cargo ra-jit` fails on moq.
The daemon keeps working after a build returns, so each ra-jit run waits for it to go idle before and after, and none of its work overlaps another run.
Its CPU time adds the daemon's, its peak RSS adds what the daemon holds, and its instructions read `n/a`; the report also says how many of its runs the daemon built, and why cargo built the rest.
The leaf and hub markers are functions nothing calls: rustc recompiles their crate and every crate that depends on it, while ra-jit, when it builds the edit itself, compiles nothing, so its times there are a lower bound.
The body edits are functions the program runs, which ra-jit compiles.

ra-jit bugs seen on moq at the pinned rev, not reported upstream:
ra-jit bugs seen on moq, not reported upstream:

- `cargo ra-jit check` with no build loaded reads every crate of the workspace, takes minutes and GBs, and fails on errors rustc does not report (`unresolved macro moq_clock::test` in moq-net's integration tests).
- Async body edits in moq-net go to cargo (every `edit-async` run): "`new` is not compiled by ra-jit and rustc's copy is from before the edit: an instance of it does not compile". The daemon then loads the workspace again: on CI such an edit took 55.7 s and 222 CPU-s, against plain cargo's 9.1 s and 12.3 CPU-s ([run](https://github.com/moq-dev/rust2/actions/runs/40285528419)).
- A fallback logs `Duplicate definition of identifier ... <cc::Build>::cargo_metadata` while compiling aws-lc-sys's build script.
- Async body edits in moq-net go to cargo (every `edit-async` run): "`parsed_values_given` is not compiled by ra-jit and rustc's copy is from before the edit: an instance of it does not compile", a generic of usage-argv's. The daemon then loads the workspace again: on CI such an edit took 50.6 s and 196 CPU-s, against plain cargo's 10.5 s and 14.5 CPU-s ([run](https://github.com/moq-dev/rust2/actions/runs/42351623273)).
- At the previous pin (`f36c848`), `cargo ra-jit check` with no build loaded read every crate of the workspace, took minutes and GBs, and failed on errors rustc does not report (`unresolved macro moq_clock::test` in moq-net's integration tests).
- On a library-only build such as regex, `cargo ra-jit build` returns after `cargo clean` without building anything.

The report shows wall time (trimmed mean, and mean ± standard deviation), CPU time and its change against the base, retired instructions, and peak RSS, all from hyperfine.
Expand Down Expand Up @@ -134,6 +132,7 @@ Recommended on 2026-10-25: moq's CI keeps building its tests with LLVM, since 42
- 19 abort on an x86 SIMD intrinsic it does not implement, `llvm.x86.sse2.pmulhu.w` ([rustc_codegen_cranelift#171](https://github.com/rust-lang/rustc_codegen_cranelift/issues/171)). moq's CI runs on aarch64, whose intrinsics this x86 benchmark does not try.

The `cranelift` variant's `build` numbers are for that abort build too.
Since moq moved its crypto onto rind, the variant builds rind with LLVM: its `naked_asm!` takes `sym` operands, which Cranelift cannot compile yet ([rustc_codegen_cranelift#1620](https://github.com/rust-lang/rustc_codegen_cranelift/pull/1620)).

Even passing, it would save little where CI spends its time.
CPU seconds from the [PR run](https://github.com/moq-dev/rust2/actions/runs/40235946635) (4 CPUs; an A/A control on that run stayed within 3%):
Expand Down
44 changes: 20 additions & 24 deletions bench/build/corpus.toml
Original file line number Diff line number Diff line change
Expand Up @@ -14,24 +14,28 @@
[[corpus]]
name = "moq"
git = "https://github.com/moq-dev/moq"
rev = "f36c8482066bb5704d02d7cf7ea5350f5f78e025"
rev = "142b03bd5c8360444750872223b03db04b106978"
args = ["-p", "moq-relay"]
leaf = "rs/moq-relay/src/lib.rs"
hub = "rs/moq-net/src/lib.rs"
# What moq-relay calls to drain a session at shutdown, and awaits to see it close.
no-c = true
# What moq-relay calls to drain a session at shutdown, and awaits to see it close:
# a wrapper over `poll_closed`, as moq writes an async fn.
sync = ["rs/moq-net/src/goaway.rs", "pub fn send(self, goaway: Goaway) -> Result<()> {"]
async = ["rs/moq-net/src/session.rs", "pub async fn closed(&self) -> Error {"]
# The tests of every workspace crate moq-relay builds, under the profile moq's CI
# uses. Not moq-auth's: its tests depend on its published self, so no package
# spec names it alone. The versions pick the workspace crate over a published one.
# uses. Not moq-auth's or moq-pattern's: moq-auth's tests depend on its published
# self, which brings a published moq-pattern, and both share their workspace
# twin's version, so no package spec names either alone. qmux's version picks the
# workspace crate over a published one.
test = [
"--profile=ci",
"--package=kio@0.6.2",
"--package=moq-clock",
"--package=moq-flate",
"--package=kio",
"--package=kio-macros",
"--package=moq-congestion",
"--package=moq-crypto",
"--package=moq-json",
"--package=moq-net",
"--package=moq-pattern@0.1.2",
"--package=moq-quic",
"--package=moq-relay",
"--package=moq-sock",
Expand All @@ -40,17 +44,6 @@ test = [
"--package=qmux@0.7.0",
]

# The moq corpus's rev plus moq-dev/moq's spike/pure-crypto branch, which moves
# moq-relay's crypto off aws-lc-rs onto graviola and RustCrypto. Compare it with moq.
[[corpus]]
name = "moq-pure-crypto"
git = "https://github.com/moq-dev/moq"
rev = "fd53b86f6d3fd2b3d40be5e0c8c00f77f464c149"
args = ["-p", "moq-relay"]
leaf = "rs/moq-relay/src/lib.rs"
hub = "rs/moq-net/src/lib.rs"
no-c = true

[[corpus]]
name = "regex"
git = "https://github.com/rust-lang/regex"
Expand All @@ -74,8 +67,8 @@ pure = true

# Each variant is one build configuration. `base` names the variant it is
# compared against; a variant without one is a base. `rustflags` replaces any
# host rustflags the corpus configures (none do today). `subcommand` goes between
# cargo and `build` or `check`. `corpora` and `scenarios` limit a variant to those
# host rustflags the corpus configures (none do today). `config` holds cargo
# `--config` values, and `subcommand` goes between cargo and `build` or `check`. `corpora` and `scenarios` limit a variant to those
# they list, which its base must run too. A variant without `scenarios` runs
# `check`, `build`, `edit-leaf`, and `edit-hub`; only the levers worth their cost
# list the body edits and the test scenarios.
Expand Down Expand Up @@ -113,11 +106,14 @@ base = "nightly"
toolchain = "nightly"
rustflags = "-Z unstable-options --jobs-frontend=8"

# rind, moq's crypto, builds with LLVM: its naked_asm! takes sym operands, which
# Cranelift cannot compile yet (rust-lang/rustc_codegen_cranelift#1620). A profile
# picks the backend, since cargo passes RUSTFLAGS after a package's own flags.
[[variant]]
name = "cranelift"
base = "nightly"
toolchain = "nightly"
rustflags = "-Z codegen-backend=cranelift"
config = ["unstable.codegen-backend=true", 'profile.dev.codegen-backend="cranelift"', 'profile.dev.package.rind.codegen-backend="llvm"']
scenarios = ["check", "build", "edit-leaf", "edit-hub", "test-build", "test-run", "test-edit"]

[[variant]]
Expand All @@ -142,13 +138,13 @@ toolchain = "nightly"
[[variant]]
name = "ra-jit-baseline"
toolchain = "ra-jit"
corpora = ["moq", "moq-pure-crypto"]
corpora = ["moq"]
scenarios = ["build", "edit-leaf", "edit-hub", "edit-sync", "edit-async"]

[[variant]]
name = "ra-jit"
base = "ra-jit-baseline"
toolchain = "ra-jit"
subcommand = "ra-jit"
corpora = ["moq", "moq-pure-crypto"]
corpora = ["moq"]
scenarios = ["build", "edit-leaf", "edit-hub", "edit-sync", "edit-async"]
11 changes: 7 additions & 4 deletions bench/build/run.sh
Original file line number Diff line number Diff line change
Expand Up @@ -235,7 +235,9 @@ check_graph() { # corpus rule cargo-args...
}

# Enters a variant's toolchain, flags, and target directory for the cargo calls that
# follow, and sets `sub` to what goes between cargo and its subcommand.
# follow, sets `sub` to what goes between cargo and its subcommand, and `cfg` to
# what follows the command. Cargo does not forward a `--config` before an external
# subcommand such as nextest, so the variant's config goes last.
use_variant() { # corpus variant
case $(field variant "$2" .toolchain) in
stable) PATH=$stable_path ;;
Expand All @@ -248,6 +250,7 @@ use_variant() { # corpus variant
ra-jit) sub=(ra-jit) ;;
*) die "variant $2 has no valid subcommand" ;;
esac
mapfile -t cfg < <(field variant "$2" '(.config // [])[] | "--config=" + .')
RUSTFLAGS=$(field variant "$2" .rustflags)
CARGO_TARGET_DIR=$work/target/$1/$2
export PATH RUSTFLAGS CARGO_TARGET_DIR
Expand Down Expand Up @@ -321,7 +324,7 @@ time_once() { # corpus dir scenario variant prepare cargo-args...
fi
# hyperfine sends only stdout to the log; nextest and cargo report on stderr.
hyperfine --style none --runs 1 --prepare "$prepare && sync" --export-json "$json" --output "$log" "${failure[@]}" \
--shell bash "$(printf '%q ' cargo "${sub[@]}" "$@") 2>&1" ||
--shell bash "$(printf '%q ' cargo "${sub[@]}" "$@" "${cfg[@]}") 2>&1" ||
{ tail -40 "$log" >&2 && die "$corpus $scenario failed with $variant"; }
measurement=$(jq -c '.results[0].measurements[0]' "$json")
if ((${#failure[@]})); then
Expand Down Expand Up @@ -410,7 +413,7 @@ bench_corpus() { # corpus dir
rm -rf "$CARGO_TARGET_DIR"
cd "$dir"
# The `||` below turns errexit off in here, so a failed build has to exit itself.
cargo "${sub[@]}" build --locked --offline "${args[@]}" >"$out/logs/$corpus.$variant.log" 2>&1 || exit
cargo "${sub[@]}" build --locked --offline "${args[@]}" "${cfg[@]}" >"$out/logs/$corpus.$variant.log" 2>&1 || exit
((${#sub[@]} == 0)) || rajit_settle "$dir" >/dev/null
) || { tail -40 "$out/logs/$corpus.$variant.log" >&2 && die "$corpus failed to build with $variant"; }
done
Expand Down Expand Up @@ -458,7 +461,7 @@ bench_corpus() { # corpus dir
(
use_variant "$corpus" "$variant"
cd "$dir"
cargo "${sub[@]}" "${cmd[@]}"
cargo "${sub[@]}" "${cmd[@]}" "${cfg[@]}"
((${#sub[@]} == 0)) || rajit_settle "$dir" >/dev/null
)
done
Expand Down
14 changes: 10 additions & 4 deletions quest/m3/cranelift-unwind.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,16 +2,22 @@

## Goal

Waits on upstream. 42 of moq's tests fail under the Cranelift backend that
ships with nightly, so moq's CI cannot build its tests with it:
Waits on upstream. 58 of moq's 6379 tests fail under the Cranelift backend
that ships with nightly, so moq's CI cannot build its tests with it:

- It builds every panic as an abort, so a test that catches one fails.
- It builds every panic as an abort, so the 27 tests that catch one fail.
Unwinding on Unix is a flag on cg_clif's own build, off by default over its
compile-time cost
([rustc_codegen_cranelift#1567](https://github.com/rust-lang/rustc_codegen_cranelift/issues/1567)).
- It lacks the x86 intrinsic `llvm.x86.sse2.pmulhu.w`
- It lacks the x86 intrinsic `llvm.x86.sse2.pmulhu.w`, which 31 hit
([rustc_codegen_cranelift#171](https://github.com/rust-lang/rustc_codegen_cranelift/issues/171)).

It also cannot compile rind, moq's crypto, whose `naked_asm!` takes `sym`
operands
([rustc_codegen_cranelift#1620](https://github.com/rust-lang/rustc_codegen_cranelift/pull/1620)),
so a Cranelift build of moq picks LLVM for rind by profile, as the benchmark
does.

Check the issues, or the [build benchmark](/bench/build/README.md)'s moq
report after a nightly pin bump: once `test-run` cranelift fails no tests,
decide again from the test scenarios whether moq's CI builds its tests with
Expand Down
1 change: 0 additions & 1 deletion quest/m4/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,6 @@ build's retired instructions by 16%.

## Required

- [Bump the build benchmark's moq pin past poll-first](/quest/m4/bench-moq-pin.md) - the benchmark builds moq after poll-first, so its hub edit rebuilds the 10 crates main does
- [Bench TPDE on moq](/quest/m4/tpde.md) - what TPDE cuts from moq's clean build and non-incremental edit; waits on `-Zllvm-tpde` reaching a nightly
- [Waiting: a rustc and cargo fork](/quest/m4/fork.md) - the evidence ledger for carrying a fork; condition: enough measured reasons to pay for one
- [moq in pure Go](/quest/m4/go/README.md) - rust2-go translating moq-net, with no cgo, fast enough to replace the cgo bindings for Go users
27 changes: 0 additions & 27 deletions quest/m4/bench-moq-pin.md

This file was deleted.