Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion nix/overlay.nix
Original file line number Diff line number Diff line change
Expand Up @@ -179,7 +179,7 @@ let
# Resolve the paths the way a consumer's pkg-config will, so a template
# whose libdir or includedir misses what this prefix ships fails here.
for check in libdir:libmoq.a includedir:moq.h; do
dir=$(PKG_CONFIG_PATH=$out/lib/pkgconfig pkg-config --variable="''${check%%:*}" moq-c)
dir=$(pkg-config --variable="''${check%%:*}" "$pc")
if [ ! -f "$dir/''${check#*:}" ]; then
echo "moq-c.pc ''${check%%:*} $dir has no ''${check#*:}" >&2
exit 1
Expand Down
6 changes: 4 additions & 2 deletions nix/runner/userns-probe.nix
Original file line number Diff line number Diff line change
@@ -1,7 +1,9 @@
{ nonce }:
let
flake = builtins.getFlake (toString ../..);
pkgs = import flake.inputs.nixpkgs { system = builtins.currentSystem; };
# The capability probe needs only the pinned nixpkgs, independent of build output.
lock = builtins.fromJSON (builtins.readFile ../../flake.lock);
nixpkgs = builtins.fetchTree lock.nodes.${lock.nodes.root.inputs.nixpkgs}.locked;
pkgs = import nixpkgs.outPath { system = builtins.currentSystem; };
in
pkgs.runCommand "moq-userns-probe-${nonce}"
{
Expand Down
2 changes: 1 addition & 1 deletion quest/m0/ci/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -155,7 +155,7 @@ The historical samples locate candidates; they are not controlled benchmarks.
- [Land main through GitHub's merge queue](/quest/m0/ci/land.md) - one PR at a time on the self-hosted runner; the host's Codex fixer fixes and re-enqueues every dropped PR
- [A cache error is a miss](/quest/m0/ci/nix-cache-5xx.md) - a transient `nix.moq.dev` 5xx no longer fails `just ci`
- [Cheap checks skip the queue](/quest/m0/ci/gate-quick.md) - condition: quest ships `quest gate --quick`; then cheap modules wait at most a minute
- [Gate each diff in under five minutes](/quest/m0/ci/five-minutes.md) - `just check` and `just test` each finish within a worker's 5-minute prompt cache, admission wait included
- [Gate each diff in under five minutes](/quest/m0/ci/five-minutes.md) - `just check`, test phases included, finishes within a worker's 5-minute prompt cache, admission wait included
- [Measure the parallel frontend](/quest/m0/ci/nightly-levers.md) - a pinned nightly's parallel frontend, adopted only if the numbers beat its risks; Cranelift is ruled out
- [Verify the R2 Rust cache](/quest/m0/ci/mbx-r2-verify.md) - confirm the hosted Nightly jobs reuse the previous Nightly, then measure restore time and expiry
- [Verify the Nix cache](/quest/m0/ci/nix-cache-verify.md) - confirm releases use `nix.moq.dev`, check its bucket sizes, and retire Cachix
Expand Down
127 changes: 96 additions & 31 deletions quest/m0/ci/five-minutes.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,15 +2,22 @@

## Goal

`just check` and `just test` each finish in under 5 minutes of wall time for
every diff-scoped run on the desktop with warm caches, counting time queued in
`quest gate` admission. A worker agent then runs its own gate inside its
`just check`, including its selected modules' test phases, finishes in under
5 minutes of wall time for every diff-scoped run on the desktop with warm
caches, counting time queued in `quest gate` admission. A worker agent then runs its own gate inside its
5-minute prompt-cache window. Explicit full runs (`--all`, nightly, release)
are exempt; work moved out of the diff-scoped gate goes to the merge queue or
nightly with its coverage kept.

## Plan

Decided 2026-11-03 (maintainer): the bound covers "`just check`, including its
selected modules' test phases" ✅, not bare `just test` or a separately named
set of cross-language suites. `just check` is the diff-scoped gate and already
runs each selected module's tests; bare `just test` only lists the
cross-language suites. The historical `just test` baseline row below is not
compared with new measurements.

Decided 2026-10-29 (user): sub-agents run with a 5-minute prompt cache and the
coordinating session with a 1-hour one
([kixelated/quest#276](https://github.com/kixelated/quest/pull/276)). A worker
Expand Down Expand Up @@ -50,41 +57,99 @@ dependent at both feature extremes in 981 s (52%); tokio-features and
media-features, selected only as dependents, took 294 s; `cargo doc` 159 s;
the release loom compile 112 s.

Landed in [moq-dev/moq#7282](https://github.com/moq-dev/moq/pull/7282): the
local `just check` runs `feature-shapes` and the net, QUIC, tokio, and media
The first reduction landed in [moq-dev/moq#7282](https://github.com/moq-dev/moq/pull/7282): the
local `just check` ran `feature-shapes` and the net, QUIC, tokio, and media
feature contracts for the crates the diff changed, while the landing worker's `just ci`
keeps their dependents; the gate compiles the loom models in the dev profile
kept their dependents; the gate compiled the loom models in the dev profile
(46 s against release's 146 s after a moq-net edit). The same lint then took
319 s at load near 30, with `feature-shapes` at 29 s. The whole `just check`
for a moq-net edit took 547 s with load falling from 24 to 8: `check rs`
230 s and `test rs` 317 s, run one after the other in the rs lane. The test
phase is a 206 s build of moq-net's dependents and a 110 s run of 9,049 tests.

What remains for a moq-net edit, largest first:

- The test build recompiles every dependent after a moq-net edit. Candidates
are the epic's [hakari](/quest/m0/ci/hakari.md) lever (the BTree thin-generics
lever was measured at about 30 ms of CPU per build and dropped), or running the rs lint beside the
test build in a second target directory, which trades CPU for wall time
against the epic's CPU measure. Decided 2026-10-30 (user): defer that
parallel lint; measure it after the hakari quest lands (slow-tests landed in
[moq-dev/moq#6987](https://github.com/moq-dev/moq/pull/6987)).
- The lint's remaining 230 s: the dev loom compile and `cargo doc`. Decided
2026-10-31 (maintainer, see the [epic](/quest/m0/ci/README.md)): the gate
drops the loom compile, the Send-depth check (now its own Nightly job), the
feature shapes, and the feature contracts (net-features's 62 s among them),
and tests the relay's io_uring runtime in the one test build instead of a
separate clippy pass.
- Measure the other routes the same way: a moq-ffi edit, which selects every
binding's lane under `CHECK_JOBS=3`, and a JS-only edit.
- `just test` has no root recipe: bare `just test` lists the cross-language
suites. Name which command the `just test` baseline above counts before
measuring it.
- Prefer the epic's levers, such as [hakari](/quest/m0/ci/hakari.md) and cache
hits, and tighter diff selection,
over moving coverage. Move a check to the queue or nightly only under the
epic's rule: evidence and an explicit coverage tradeoff for each.
- Split any large single lever into its own quest.
The settled gate changes have since landed:

- [moq-dev/moq#7472](https://github.com/moq-dev/moq/pull/7472) removed the
feature shapes and contracts from the shared local and landing gate. The
Features workflow retains the matrix nightly and on release PRs. The
relay's io_uring tests join the default test build on Linux.
- [moq-dev/moq#7473](https://github.com/moq-dev/moq/pull/7473) removed the loom
compile and moved Send-depth to its own Nightly job.

Measured 2026-11-02 on the desktop (x86_64, Rust 1.98.1, Bun 1.4.2,
mbx 1.25.0), with other agents running. Each row is one observation after a
successful preparation run, not a controlled comparison with the historical
samples or proof of the bound for every diff. A comment-only source edit
invalidated the package again after preparation. The temporary edits were
restored after measuring.

`bench/local.py --checkout . --workers 4 --jobs 3 --samples 1 --cache ... --
just check HEAD` admits the whole command once, separates admission from work,
and records child CPU time. `HEAD` makes the edited source the entire measured
diff. The net and JS runs used
[196e4f8126](https://github.com/moq-dev/moq/commit/196e4f81262f71d1477c7b2c5e14e0e241c12125);
the FFI run used
[e6afdcca32](https://github.com/moq-dev/moq/commit/e6afdcca3210962b7ee34896fc0cff87cfeec186)
with the installed pkg-config lookup fix. Logs remain in the checkout's
`.tmp/local-bench/`.

| Edited source | Admission (s) | Work (s) | Total (s) | CPU (s) | Starting load (1 min) |
| --- | ---: | ---: | ---: | ---: | ---: |
| `rs/moq-net/src/lib.rs` | 88.2 | 227.1 | 315.3 | 1,048.0 | 31.4 |
| `js/net/src/index.ts` | 0.2 | 228.9 | 229.0 | 352.4 | 12.5 |
| `rs/moq-ffi/src/lib.rs`, explicit Gradle executable | 0.2 | 101.1 | 101.2 | 219.8 | 9.7 |

The net lane was 34 s of lint and docs, then 193 s of tests: a 68 s build
and 122 s running 9,529 tests (all passed). The JS lane was 115 s checking,
54 s testing, then the shared Markdown and tooling checks. The FFI route ran
the Rust, Python, C, C++, Kotlin, Go, OBS compile, and Dart lanes with
`CHECK_JOBS=3`; Swift skipped because this Linux host has no Swift toolchain.

The first FFI attempt exposed two base-branch failures:

- C and C++ installed and ran their CMake probes, then failed to find their
pkg-config packages. Nix's pkg-config wrapper replaces `PKG_CONFIG_PATH`
with its target-specific value. Passing the installed `.pc` file directly
fixes the lookup without changing the wrapper's dependency search. The
same explicit-prefix queries in OBS and Nix packaging now use their `.pc`
file too. Published smoke keeps its name-based lookup to test consumer
resolution. C and C++ passed in both subsequent
configured FFI runs.
- Kotlin recursively called its own Gradle wrapper because `command -v gradle`
selected Nix's exported function. The attempt was cancelled and its child
processes stopped. [moq-dev/moq#7632](https://github.com/moq-dev/moq/pull/7632)
already owns this fix. Both subsequent FFI runs used the supported
`GRADLE_CMD` override set to the dev shell's absolute Gradle executable;
the table is conditional on that configuration, not a passing default run.

Final local validation also reproduced a namespace-probe race: a raw path
flake copied ignored CMake output while another lane replaced a generated
`render` file. The probe now reads `flake.lock` and fetches its pinned nixpkgs
directly, without copying the checkout. It still reports the host builder's
namespace failure, allowing the existing local skip; strict CI continues to
require a capable builder.

What remains:

- Admission still makes the measured net route exceed 5 minutes despite its
command taking 227 s. The 2026-10-29 admission decision above remains with
moq-dev/quest.
- Repeat the default FFI route after the Gradle fix lands, without the override.
- Keep the dependency-build and cache work with
[hakari](/quest/m0/ci/hakari.md). Decided 2026-10-30 (user): defer parallel
Rust lint until that quest lands, then measure it. The slow-tests reduction
landed in [moq-dev/moq#6987](https://github.com/moq-dev/moq/pull/6987); the
BTree thin-generics lever measured about 30 ms of CPU per build and was dropped.
- Prefer the epic's levers and tighter diff selection over moving coverage.
Moving a check to the queue or nightly still needs evidence and an explicit
coverage tradeoff. Split a large independent lever into its own quest.

## Waiting

- moq-dev/quest: admission must keep the whole gate inside the worker's cache
window; repeat the net measurement after its admission work lands.
- moq-dev/moq: [moq-dev/moq#7632](https://github.com/moq-dev/moq/pull/7632)
must land before repeating the default FFI measurement without `GRADLE_CMD`.

## Related

Expand Down
3 changes: 1 addition & 2 deletions sh/c/check.sh
Original file line number Diff line number Diff line change
Expand Up @@ -114,9 +114,8 @@ if [[ "$(uname -s)" == MINGW* || "$(uname -s)" == MSYS* ]]; then
fi

if command -v pkg-config >/dev/null 2>&1; then
export PKG_CONFIG_PATH="$prefix/$libdir/pkgconfig"
# Assigned first: set -e ignores a failing substitution inside a command's arguments.
flags=$(pkg-config --cflags --libs moq-c)
flags=$(pkg-config --cflags --libs "$prefix/$libdir/pkgconfig/moq-c.pc")
mkdir -p "$build/pkg-config"
# shellcheck disable=SC2086 # flags is a flag list.
"${CC:-cc}" -std=c11 -Wall -Wextra -Werror -pedantic-errors -o "$build/pkg-config/probe" moq/test/probe.c $flags
Expand Down
6 changes: 3 additions & 3 deletions sh/cpp/check.sh
Original file line number Diff line number Diff line change
Expand Up @@ -96,10 +96,10 @@ bash ../doc/lib/samples.sh cpp ../doc/lib/cpp/index.md >"$build/doc-samples.cpp"
"${CXX:-c++}" -std=c++20 -fsyntax-only -Wall -Werror -I"$prefix/include" moq/bench/main.cpp

if command -v pkg-config >/dev/null 2>&1; then
export PKG_CONFIG_PATH="$prefix/lib/pkgconfig"
pc="$prefix/lib/pkgconfig/moq-cpp.pc"
# Assigned first: set -e ignores a failing substitution inside a command's arguments.
sources=$(pkg-config --variable=sources moq-cpp)
flags=$(pkg-config --cflags --libs moq-cpp)
sources=$(pkg-config --variable=sources "$pc")
flags=$(pkg-config --cflags --libs "$pc")
# Those sources already compiled at C++17 with these flags in test-17, so the probe links
# that library instead of compiling them again.
if [[ ! -f "$sources" ]]; then
Expand Down
6 changes: 3 additions & 3 deletions sh/obs/qlog.sh
Original file line number Diff line number Diff line change
Expand Up @@ -10,10 +10,10 @@ fi
here=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)

moq_prefix=$("$here/moq.sh" qlog)
export PKG_CONFIG_PATH="$moq_prefix/lib/pkgconfig"
moq_pc="$moq_prefix/lib/pkgconfig/moq-cpp.pc"
# Assigned first: set -e ignores a failing substitution inside a command's arguments.
moq_sources=$(pkg-config --variable=sources moq-cpp)
moq_flags=$(pkg-config --cflags --libs moq-cpp)
moq_sources=$(pkg-config --variable=sources "$moq_pc")
moq_flags=$(pkg-config --cflags --libs "$moq_pc")

out=$(mktemp -d)
trap 'rm -rf "$out"' EXIT
Expand Down
6 changes: 3 additions & 3 deletions sh/obs/unit.sh
Original file line number Diff line number Diff line change
Expand Up @@ -35,9 +35,9 @@ ffmpeg=$(pkg-config --cflags libavcodec libavutil libswscale)
# The output and source tests run against the real moq-ffi, over a server in
# the same process. The bindings compile here, with the tests' flags.
moq_prefix=$("$here/moq.sh")
export PKG_CONFIG_PATH="$moq_prefix/lib/pkgconfig"
moq_sources=$(pkg-config --variable=sources moq-cpp)
moq_libs=$(pkg-config --libs moq-cpp)
moq_pc="$moq_prefix/lib/pkgconfig/moq-cpp.pc"
moq_sources=$(pkg-config --variable=sources "$moq_pc")
moq_libs=$(pkg-config --libs "$moq_pc")

flags=(-std=c++17 -g -O0 -pthread -Werror)
if [ "$mode" = "tsan" ]; then
Expand Down