Skip to content

chore(quest): find and plan the gaps on MoQ's QUIC packet path - #387

Merged
kixelated merged 4 commits into
mainfrom
quest/m1/moq/quic-path
Nov 6, 2026
Merged

kixelated merged 4 commits into
mainfrom
quest/m1/moq/quic-path

Conversation

@kixelated

@kixelated kixelated commented Nov 6, 2026 •

Copy link
Copy Markdown
Collaborator

Claims quest/m1/moq/quic-path.md.

This PR investigates MoQ's QUIC packet path across rind, rind-rustls, and moq-dev/moq's moq-quic, and plans a quest for each gap the maintainer approved below. It changes only quests:

  • Adds vaes-short-open [S], header-protection-block [XS], and aarch64-short-gcm [S] under the MoQ epic's ## Required.
  • Rewrites quic-path.md to what remains, an [XS] that waits on zshenker's Zen 2 rerun to pick Vale or by8 for AMD CPUs without VAES, and records the decisions in its Plan. It is not deleted, since that rerun is still open.
  • Records in the MoQ epic's Plan that rind adds no QUIC-specific public API.

Summary

  • On current main, rind is faster than aws-lc-rs at every packet size on every CPU measured, except for three small gaps. The x86 gaps are in AES-128 header protection on AMD and AES-GCM opens under 256 B. The aarch64 gaps are ACK-sized sends and the per-call DIT toggle.
  • In moq-uring's echo_quic, crypto is about 20% of moq's CPU: AEAD kernels 16%, header protection 1.9%, and the rustls and moq-quic dispatch layers 1.7%. The kernel takes 18% and memcpy 11%.
  • The biggest win is the first gap. On VAES CPUs, rind's by8 loop opens sub-256 B messages, and Vale's kernel, already in rind and now proven, opens them 13-61% faster.
  • Neither new QUIC API measures a clear end-to-end win. A fused seal-plus-header-protection key could save at most the 3.6% that header protection and dispatch take, and only if rustls exposed QUIC secrets. A GSO batch seal would only amortize per-call cost, which at 1200 B is small.

Method

The bench is zshenker's from moq-dev/moq#7873, extended with 32 and 1452 B and with the rustls QUIC traits moq-quic calls. A send is PacketKey::encrypt_in_place plus HeaderProtectionKey::encrypt_in_place on a short-header datagram of the given total size (13 B header, 16 B tag). A receive is the reverse. It also covers AES-256-GCM and ChaCha20-Poly1305. Each binary runs rind and aws-lc-rs 1.19.0 back to back on one pinned core, with rustls 0.23.46. Every cell is rind's throughput over aws-lc-rs's, the median of two interleaved rounds, so above 1 means rind is faster.

Builds: rind and rind-rustls 0.1.12, 0.1.13 (the current release), and main at 0ad76c6. A fourth build of main has --cfg rind_cpu_disable, to force tiers at runtime:

  • vale: VAES denied, so the CPU runs Vale's kernel at every length, as Zen 2 and Intel through Cascade Lake do.
  • by8: VAES and MOVBE denied, so it runs rind's SSE by8 loop, as before moq-dev/rind#329.
  • no-dit: DIT denied on aarch64.

Hosts:

  • ssh desktop: Arrow Lake, Core Ultra 7 270K Plus, on a P-core. Quiet (load ~5 of 24), so this is the cleanest data.
  • ssh nixos: Zen 3, Ryzen 9 5900XT. It ran quest-runner builds throughout (load 40-70 of 32), so its absolute rates are noisy; its ratios agree with GitHub's Zen 3.
  • GitHub runners, run 42140284319, grouped by their CPU: line: EPYC 7763 (Zen 3), EPYC 9V74 (Zen 4, AVX-512 hidden), two EPYC 9V45 (Zen 5, AVX-512) that agree, with the second's numbers shown, and four Neoverse N2 (Azure Cobalt 100) that agree.

No Zen 2 host was available, so zshenker's 0.1.12 numbers stand for Zen 2; the vale and by8 columns approximate it on other cores.

zshenker's rows, 0.1.13 → main

AES-128-GCM, 20 B AAD:

Arrow Lake Zen 3 (GH) Zen 4 (GH) Zen 5 (GH) Neoverse N2 Zen 2, 0.1.12 (zshenker)
seal 64 B 1.71 → 1.78 1.96 → 2.02 1.93 → 2.04 1.57 → 1.65 1.05 → 1.06 1.54-1.56
open 64 B 1.47 → 1.32 1.83 → 1.92 1.90 → 1.87 1.53 → 1.49 1.07 → 1.04 1.40-1.42
seal 1200 B 1.75 → 1.78 1.95 → 1.98 1.95 → 1.99 1.35 → 1.39 1.19 → 1.18 1.08-1.09
open 1200 B 1.48 → 1.51 1.90 → 1.95 1.97 → 1.98 1.46 → 1.50 1.13 → 1.12 1.06
seal 1280 B 1.79 → 1.82 2.07 → 2.06 2.07 → 2.11 1.35 → 1.38 1.21 → 1.20 1.09
open 1280 B 1.57 → 1.57 1.94 → 1.97 2.04 → 2.03 1.47 → 1.57 1.15 → 1.14 1.06-1.07
seal 8192 B 1.65 → 1.66 1.95 → 1.96 2.11 → 2.12 1.06 → 1.11 1.25 → 1.24 1.01-1.02
HP mask, AES-128 2.53 → 2.54 0.83 → 0.83 0.89 → 0.89 0.90 → 0.90 1.01 → 1.01 0.99-1.00
TLS 1.3 handshake 1.86 → 1.89 1.65 → 1.66 1.62 → 1.62 1.61 → 1.60 1.45 → 1.42 1.65-1.66

main matches 0.1.13 within noise everywhere except DIT on Neoverse N2 (gap 3). nixos's Zen 3 rows agree with GitHub's: seal 1.44-1.96, open 1.33-1.80, HP 0.84, handshakes 1.80.

Through the rustls QUIC traits, main:

Arrow Lake Zen 3 (GH) Zen 4 (GH) Zen 5 (GH) Neoverse N2
AES-128 send 32 B 1.16 1.47 1.40 1.27 0.91
AES-128 recv 32 B 1.04 1.52 1.45 1.44 0.99
AES-128 send 1200 B 1.49 1.77 1.80 1.40 1.14
AES-128 recv 1200 B 1.30 1.72 1.75 1.62 1.11
AES-128 header protection 1.19 0.98 1.08 1.10 0.87
ChaCha20 send 1200 B 1.03 1.05 1.04 1.00 1.02
ChaCha20 recv 1200 B 1.01 1.02 1.03 1.01 1.01

AES-256-GCM tracks AES-128 within a few percent on every host.

Vale's tier on the CPUs that run it: with VAES denied, main at 1200-1452 B is 1.11-1.21x aws-lc-rs on Arrow Lake and 1.19-1.23x on Zen 3, against zshenker's 1.06-1.09x for by8 on Zen 2.

Where moq's time goes

moq-uring's echo_quic/udp/all-on echoes 1 MiB through moq-quic on rind 0.1.13, at 495 MiB/s on desktop. perf ran on the bench's four pinned cores (io_uring's rings leave too little of the 8 MiB RLIMIT_MEMLOCK for a whole-system perf buffer). Shares of the echo process's samples:

share
AEAD (aes_gcm_{enc,dec}_update_vaes_avx2, GHASH, glue) 16.0%
Header protection (aes128_block, xor_in_place, masks) 1.9%
Dispatch (rind-rustls PacketKey, moq-quic's Box<dyn rustls::quic::PacketKey> wrapper) 1.7%
Kernel 18%
memcpy 11%

moq-quic's in-memory streams bench needs --features fuzz, which swaps in plaintext keys (fuzz::plaintext::Clear). So it measures the stack with no crypto: 1.43 GiB/s for one 16 KB-write stream on nixos. No moq bench runs a relay fan-out over QUIC with real keys: moq-relay's benches cover auth and HTTP, and moq-net's run without QUIC crypto.

Gaps

1. AES-GCM opens under 256 B on VAES CPUs (rind)

On CPUs with VAES, rind runs its VAES kernels from 256 B and its SSE by8 loop below that. by8's decrypt hashes the whole sub-128 B tail through the generic ClmulTable::add, while its encrypt hashes each block inline, so opens trail seals at small sizes: 36 M/s against 46 M/s at 32 B on Arrow Lake. Vale's kernel, which rind already ships and has proven, has no such split. Main over its own vale column, as ratios to aws-lc-rs:

Arrow Lake Zen 3 (GH) Zen 4 (GH) Zen 5 (GH)
open 32 B 1.51 → 2.29 (+52%) 2.08 → 2.40 (+15%) 1.98 → 2.36 (+19%) 1.39 → 1.98 (+42%)
open 64 B 1.32 → 2.13 (+61%) 1.92 → 2.16 (+13%) 1.87 → 2.17 (+16%) 1.49 → 1.97 (+32%)
seal 32 B 1.96 → 2.03 (+4%) 2.28 → 2.17 (-5%) 2.17 → 1.99 (-8%) 1.73 → 1.79 (+3%)
seal 64 B 1.78 → 1.83 (+3%) 2.02 → 2.03 (0%) 2.04 → 1.85 (-9%) 1.65 → 1.65 (0%)
rustls recv 32 B 1.04 → 1.50 (+44%) 1.52 → 1.97 (+30%) 1.45 → 2.01 (+39%) 1.44 → 1.90 (+32%)
rustls recv 64 B 1.16 → 1.53 (+32%) 1.72 → 2.23 (+30%) 1.77 → 2.13 (+20%) 1.27 → 1.52 (+20%)
  • Expected win: ACK-sized receives 20-44% faster on every VAES CPU. A relay opens about one ACK per two data packets it seals; on Arrow Lake that cuts about 7% of its packet-crypto time, about 1.4% of its CPU.
  • Fix lives in rind: give the VAES and AVX-512 GHASH tables Vale's key powers and route opens under 256 B to Vale. Seals stay on by8, which wins or ties there on AMD. Every VAES CPU has the AVX and MOVBE that Vale needs.
  • Recommendation: approve as an S quest in rind. Its benches should add 32-64 B opens with a 13 B AAD, which rind-bench's 32 B row and 20 B-AAD rows don't separate.

2. AES-128 header protection on AMD (rind)

rind's AES-128 mask is 0.83x aws-lc-rs on Zen 3, 0.89x on Zen 4, and 0.90x on Zen 5, about 4.6 ns slower per mask on the EPYC 7763. On Arrow Lake it is 2.5x faster, and on N2 level. perf shows the cause: cipher::AesKey::encrypt_block, which every mask and by8's tag block go through, reserves a 3.8 KB stack frame because the no-AES-NI variant's key schedule inlines into it. It also decodes a 128-bit niche discriminant and makes two more non-inlined calls before reaching aes128_block.

  • Expected win: parity or better with aws-lc-rs on AMD, about 4-5 ns per packet per direction. That is under 1% of a relay's CPU, plus sub-256 B seals on by8, which share the path.
  • Fix lives in rind: give the header-protection Masker the hardware key and its block function directly, and keep the software variants' code out of line.
  • Recommendation: approve as an XS quest in rind. It is the one place on x86 where rind trails aws-lc-rs.

3. DIT on every call on aarch64 (rind)

rind reads, sets, and clears PSTATE.DIT around every secret operation, as Graviola does. aws-lc-rs never does: aws-lc's SET_DIT_AUTO_RESET compiles only with ENABLE_DATA_INDEPENDENT_TIMING, which defaults off and aws-lc-sys's builder never sets. Since moq-dev/rind#370, main detects DIT on Azure Cobalt 100 too. On Neoverse N2, through rustls:

0.1.13 main main, no-dit
AES-128 header protection 0.99 0.87 0.98
AES-256 header protection 1.03 0.90 1.02
AES-128-GCM open 32 B 1.07 1.04 1.07

At 1200 B the cost is within noise. Real Graviton 3 and 4 report DIT in HWCAP, so 0.1.13 already pays this there. Hyper-V may trap DIT writes on Cobalt, which would make this cost larger than on bare metal; no Graviton was available to check.

  • Expected win from dropping the toggle: about 1.6 ns per call, so 11% of a header-protection call and about 3% of an ACK-sized AEAD call on N2.
  • Fix lives in rind. It is a constant-time policy question, so it's the maintainer's call (decision 3 below).

4. ACK-sized sends on aarch64 (rind)

On Neoverse N2, sending a 32 B datagram (3 B payload) through rustls is 0.91x aws-lc-rs, and 64 B (35 B payload) is 0.89x. With DIT denied they are still 0.93x and 0.91x, while a raw 32 B seal with 20 B AAD is 1.03x. rind's aarch64 AES-GCM hashes the AAD, the sub-block tail, and the lengths block in separate GHASH calls and encrypts the tail and the tag block one at a time.

  • Expected win: parity or better at ACK-sized sends, 7-10% on N2.
  • Fix lives in rind: a one-pass path for messages of a few blocks, computing the counter blocks and E(Y0) together and aggregating AAD, data, and lengths into one GHASH reduction.
  • Recommendation: approve as an S quest in rind, after gap 3's decision, since DIT changes what it measures.

No quest proposed

  • Fused seal and header protection, a new rind API (rind and moq-quic). On Arrow Lake a 32 B send costs about 62 ns through rustls, against about 22 ns for the seal and 5 ns for the mask alone. aws-lc-rs pays the same overhead (72 ns). A fused key would keep the header-protection sample in a register and drop two dyn hops. But header protection and dispatch together are 3.6% of moq's CPU in echo_quic. And moq-quic can only reach such a key by deriving its own QUIC keys: rustls keeps quic::Secrets private and hands out only Box<dyn PacketKey>, so it would need a rustls change or a fork. Not a clear end-to-end win at that cost.
  • GSO batch seal API. At 1200-1452 B the AEAD kernels are bound by AES and GHASH throughput and already run 1.4-2.1x aws-lc-rs; batching amortizes only per-call cost, a few ns of about 100-270 ns.
  • moq-quic's second dyn layer. moq-quic wraps Box<dyn rustls::quic::PacketKey> in its own Box<dyn crypto::PacketKey> and asks for sample_size() and tag_len() through vtables on every packet. All dispatch is 1.7% of CPU, so removing moq-quic's half is under 1%.
  • ChaCha20-Poly1305. 1.00-1.10x everywhere on 0.1.13 and main. 0.1.12 was 0.74-0.93x on Zen 5, already fixed.
  • Handshakes and key updates. Handshakes run 1.4-1.9x faster. A connection sets up a handful of packet keys, and AES-GCM's confidentiality limit triggers one key update per 2^23 packets, so key setup is off the hot path.

Decisions

The maintainer answered on 2026-11-05 (✅ chosen):

  1. Route AES-GCM opens under 256 B to Vale on VAES and AVX-512 CPUs (rind, S): approve ✅ / drop. Planned as quest/m1/moq/vaes-short-open.md.
  2. Call the hardware AES block directly for header protection (rind, XS): approve ✅ / drop. Planned as quest/m1/moq/header-protection-block.md.
  3. DIT on aarch64:
    • a. Keep toggling DIT per call ✅. Constant time stays a Required rule. Where an API already processes a batch, it toggles once per batch, never per block. Recorded in aarch64-short-gcm.md, the quest it affects.
    • b. Never set DIT, as aws-lc-rs.
    • c. Leave DIT set once a thread first enters rind.
  4. One-pass short-message AES-GCM on aarch64 (rind, S), with DIT kept per call: approve ✅ / drop. Planned as quest/m1/moq/aarch64-short-gcm.md.
  5. Ask zshenker to re-run on Zen 2 with 0.1.13: ask ✅ / skip. The maintainer's session posts that comment on moq-dev/moq#7873; quic-path.md waits on the reply.

Settled: no new public QUIC API, neither a key that seals and applies header protection in one call nor a GSO batch seal, since neither showed an end-to-end win. No moq-quic quest is proposed, so nothing goes to moq-dev/moq's perf epic.

Per moq-dev/rind#386, bulk encryption takes the fastest source outright, so the new quests carry no ~10% margin for verified code.

Choices made without asking

  • Kept quic-path.md as an [XS] waiting on the Zen 2 rerun rather than deleting it, since that measurement is the investigation's one open question and has no other home.
  • Put the three quests directly under the MoQ epic rather than nesting them under quic-path.md, since each is independent of the Zen 2 wait.
  • Measured GitHub runners through a scratch branch, probe/quic-path, whose build.yml held only the bench job, dispatched once and deleted afterwards.
  • Treated nixos's absolute rates as noise because quest-runner builds kept it at load 40-70, and relied on its ratios, which agree with GitHub's Zen 3.
  • Used --cfg rind_cpu_disable tier toggles on VAES hosts to approximate Zen 2 and Cascade Lake, since neither was available.
  • Profiled echo_quic on desktop's four pinned cores, since perf's buffers and io_uring's rings together exceed the 8 MiB memlock limit on both hosts.

Public API and wire impact

None. This PR changes only quests.

Validation

  • quest check and quest check --base origin/main pass.
  • CARGO_BUILD_JOBS=3 nix develop --command just check passes on d52246c.
  • The bench source is in the first comment.

Review

Codex (gpt-6.1-sol, medium) via quest review --base origin/main, round 1 on d52246c: no findings.

(Written by Claude Opus 5.5)

🤖 Generated with Claude Code

kixelated and others added 2 commits November 5, 2026 16:55
Quest-Claim: cc6d6a8a14bac221
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated kixelated changed the title MoQ's QUIC packet path is as fast as rind can make it chore(quest): find and plan the gaps on MoQ's QUIC packet path Nov 6, 2026
@kixelated

Copy link
Copy Markdown
Collaborator Author

Bench source for the numbers in the PR body. Each v-<version>/Cargo.toml points [[bin]] path at ../src/main.rs and depends on that rind and rind-rustls (path deps for main), aws-lc-rs = "1", rustls 0.23 with aws-lc-rs, and rcgen 0.14. run.sh runs every build interleaved on one pinned core, plus main's --cfg rind_cpu_disable build with the tier toggles. analyze.py takes medians.

src/main.rs
//! zshenker's rows from moq-dev/moq#7873, plus the rustls QUIC traits moq-quic calls.

use std::{
    hint::black_box,
    sync::Arc,
    time::{Duration, Instant},
};

use rustls::quic::{DirectionalKeys, Keys, Version};

const SIZES: [usize; 6] = [32, 64, 1200, 1280, 1452, 8192];
const AAD: [u8; 20] = [0x5a; 20];

fn run() -> Duration {
    Duration::from_millis(std::env::var("RUN_MS").ok().and_then(|s| s.parse().ok()).unwrap_or(1000))
}

fn rate(mut f: impl FnMut()) -> f64 {
    let warm = Instant::now();
    while warm.elapsed() < Duration::from_millis(200) {
        f();
    }
    let run = run();
    let start = Instant::now();
    let mut n = 0u64;
    while start.elapsed() < run {
        for _ in 0..64 {
            f();
        }
        n += 64;
    }
    n as f64 / start.elapsed().as_secs_f64()
}

fn nonce(i: u64) -> [u8; 12] {
    let mut n = [0u8; 12];
    n[4..].copy_from_slice(&i.to_be_bytes());
    n
}

/// One machine-readable row: group, op, size, rind ops/s, aws ops/s, rind/aws.
fn row(group: &str, op: &str, size: usize, r: f64, a: f64) {
    println!("ROW\t{group}\t{op}\t{size}\t{r:.0}\t{a:.0}\t{:.3}", r / a);
}

fn aead() {
    use aws_lc_rs::aead as aws;
    let key = [7u8; 16];
    let rind_key = rind::aead::Key::new(&rind::aead::AES_128_GCM, &key).unwrap();
    let aws_key = aws::LessSafeKey::new(aws::UnboundKey::new(&aws::AES_128_GCM, &key).unwrap());
    let sizes: Vec<usize> = match std::env::var("SIZES") {
        Ok(v) => v.split(',').map(|s| s.parse().unwrap()).collect(),
        Err(_) => SIZES.to_vec(),
    };
    for size in sizes {
        let mut buf = vec![0x11u8; size];
        let mut i = 0u64;
        let r = rate(|| {
            i += 1;
            let n = rind::aead::Nonce::assume_unique_for_key(nonce(i));
            black_box(rind_key.seal_in_place_separate_tag(n, &AAD, &mut buf).unwrap());
        });
        let a = rate(|| {
            i += 1;
            let n = aws::Nonce::assume_unique_for_key(nonce(i));
            black_box(aws_key.seal_in_place_separate_tag(n, aws::Aad::from(&AAD), &mut buf).unwrap());
        });
        row("aead128", "seal", size, r, a);

        let plain = vec![0x22u8; size];
        let mut sealed_r = plain.clone();
        let tag_r = rind_key
            .seal_in_place_separate_tag(rind::aead::Nonce::assume_unique_for_key(nonce(1)), &AAD, &mut sealed_r)
            .unwrap();
        sealed_r.extend_from_slice(tag_r.as_ref());
        let mut sealed_a = plain.clone();
        aws_key
            .seal_in_place_append_tag(aws::Nonce::assume_unique_for_key(nonce(1)), aws::Aad::from(&AAD), &mut sealed_a)
            .unwrap();
        let mut work = vec![0u8; size + 16];
        let r = rate(|| {
            work.copy_from_slice(&sealed_r);
            let n = rind::aead::Nonce::assume_unique_for_key(nonce(1));
            black_box(rind_key.open_in_place(n, &AAD, &mut work).unwrap());
        });
        let a = rate(|| {
            work.copy_from_slice(&sealed_a);
            let n = aws::Nonce::assume_unique_for_key(nonce(1));
            black_box(aws_key.open_in_place(n, aws::Aad::from(&AAD), &mut work).unwrap());
        });
        row("aead128", "open", size, r, a);
    }
}

fn header_protection() {
    let key = [9u8; 16];
    let rind_hp = rind::aead::quic::HeaderProtectionKey::new(&rind::aead::quic::AES_128, &key).unwrap();
    let aws_hp = aws_lc_rs::aead::quic::HeaderProtectionKey::new(&aws_lc_rs::aead::quic::AES_128, &key).unwrap();
    let mut sample = [3u8; 16];
    let r = rate(|| {
        sample[0] = sample[0].wrapping_add(1);
        black_box(rind_hp.mask(&sample));
    });
    let a = rate(|| {
        sample[0] = sample[0].wrapping_add(1);
        black_box(aws_hp.new_mask(&sample).unwrap());
    });
    row("hp128", "mask", 16, r, a);
}

/// 1-RTT-like keys for `suite` from each provider, through the rustls traits.
fn quic_keys(suite: rustls::CipherSuite) -> (Keys, Keys) {
    let dcid = [0x83u8, 0x94, 0xc8, 0xf0, 0x3e, 0x51, 0x57, 0x08];
    let find = |p: &rustls::crypto::CryptoProvider| {
        let s = p.cipher_suites.iter().find(|s| s.suite() == suite).copied().unwrap();
        s.tls13().unwrap().quic_suite().unwrap().keys(&dcid, rustls::Side::Client, Version::V1)
    };
    (
        find(&rind_rustls::default_provider().unwrap()),
        find(&rustls::crypto::aws_lc_rs::default_provider()),
    )
}

/// moq-quic's `PacketKey::encrypt` and `decrypt`, and `HeaderKey`, over a provider's keys.
fn seal_packet(k: &DirectionalKeys, pn: u64, buf: &mut [u8], header_len: usize) {
    let (header, payload_tag) = buf.split_at_mut(header_len);
    let (payload, tag_storage) = payload_tag.split_at_mut(payload_tag.len() - 16);
    let tag = k.packet.encrypt_in_place(pn, &*header, payload).unwrap();
    tag_storage.copy_from_slice(tag.as_ref());
}

fn hp(k: &DirectionalKeys, pn_offset: usize, packet: &mut [u8], protect: bool) {
    let (header, sample) = packet.split_at_mut(pn_offset + 4);
    let (first, rest) = header.split_at_mut(1);
    let pn_end = Ord::min(pn_offset + 3, rest.len());
    let s = &sample[..k.header.sample_len()];
    let pn = &mut rest[pn_offset - 1..pn_end];
    match protect {
        true => k.header.encrypt_in_place(s, &mut first[0], pn),
        false => k.header.decrypt_in_place(s, &mut first[0], pn),
    }
    .unwrap();
}

fn rustls_quic() {
    use rustls::CipherSuite::*;
    // A short header: first byte, 8-byte CID, 4-byte packet number.
    const HDR: usize = 13;
    const PN_OFFSET: usize = 9;
    for (name, suite) in [
        ("q128", TLS13_AES_128_GCM_SHA256),
        ("q256", TLS13_AES_256_GCM_SHA384),
        ("qchacha", TLS13_CHACHA20_POLY1305_SHA256),
    ] {
        if std::env::var("QSUITE").is_ok_and(|q| q != name) {
            continue;
        }
        let (rk, ak) = quic_keys(suite);
        let qsizes: Vec<usize> = match std::env::var("QSIZES") {
            Ok(v) => v.split(',').map(|s| s.parse().unwrap()).collect(),
            Err(_) => vec![32, 48, 64, 1200, 1452],
        };
        for size in qsizes {
            // `size` is the whole datagram: header, payload, tag.
            let mut buf = vec![0x40u8; size];
            let mut pn = 0u64;
            let mut send = |k: &DirectionalKeys, buf: &mut [u8], pn: &mut u64| {
                *pn += 1;
                seal_packet(k, *pn, buf, HDR);
                hp(k, PN_OFFSET, buf, true);
            };
            let r = rate(|| send(&rk.local, &mut buf, &mut pn));
            let a = rate(|| send(&ak.local, &mut buf, &mut pn));
            row(name, "send", size, r, a);

            // Receive: unprotect the header, then open, from a fixed sealed packet.
            let sealed = |k: &Keys| {
                let mut p = vec![0x40u8; size];
                p[0] = 0x43;
                seal_packet(&k.local, 7, &mut p, HDR);
                hp(&k.local, PN_OFFSET, &mut p, true);
                p
            };
            let (sr, sa) = (sealed(&rk), sealed(&ak));
            // The peer's remote keys are our local ones for a symmetric test; reuse local.
            let mut work = vec![0u8; size];
            let mut recv = |k: &DirectionalKeys, src: &[u8], work: &mut [u8]| {
                work.copy_from_slice(src);
                hp(k, PN_OFFSET, work, false);
                let (h, p) = work.split_at_mut(HDR);
                black_box(k.packet.decrypt_in_place(7, h, p).unwrap());
            };
            let r = rate(|| recv(&rk.local, &sr, &mut work));
            let a = rate(|| recv(&ak.local, &sa, &mut work));
            row(name, "recv", size, r, a);
        }
        // Header protection alone, through the trait.
        let mut pkt = vec![0x40u8; 64];
        let r = rate(|| hp(&rk.local, PN_OFFSET, black_box(&mut pkt), true));
        let a = rate(|| hp(&ak.local, PN_OFFSET, black_box(&mut pkt), true));
        row(name, "hp", 64, r, a);
    }
}

fn handshake(client: &Arc<rustls::ClientConfig>, server: &Arc<rustls::ServerConfig>) {
    let mut c = rustls::ClientConnection::new(client.clone(), "localhost".try_into().unwrap()).unwrap();
    let mut s = rustls::ServerConnection::new(server.clone()).unwrap();
    let mut buf = Vec::with_capacity(8192);
    while c.is_handshaking() || s.is_handshaking() {
        buf.clear();
        while c.wants_write() {
            c.write_tls(&mut buf).unwrap();
        }
        let mut rd = &buf[..];
        while !rd.is_empty() {
            s.read_tls(&mut rd).unwrap();
        }
        s.process_new_packets().unwrap();
        buf.clear();
        while s.wants_write() {
            s.write_tls(&mut buf).unwrap();
        }
        let mut rd = &buf[..];
        while !rd.is_empty() {
            c.read_tls(&mut rd).unwrap();
        }
        c.process_new_packets().unwrap();
    }
}

fn configs(mut provider: rustls::crypto::CryptoProvider) -> (Arc<rustls::ClientConfig>, Arc<rustls::ServerConfig>) {
    provider.kx_groups.retain(|g| g.name() == rustls::NamedGroup::X25519);
    let provider = Arc::new(provider);
    let key = rcgen::KeyPair::generate_for(&rcgen::PKCS_ECDSA_P256_SHA256).unwrap();
    let cert = rcgen::CertificateParams::new(vec!["localhost".into()]).unwrap().self_signed(&key).unwrap();
    let der = cert.der().clone();
    let key_der = rustls::pki_types::PrivateKeyDer::Pkcs8(key.serialize_der().into());
    let server = rustls::ServerConfig::builder_with_provider(provider.clone())
        .with_protocol_versions(&[&rustls::version::TLS13])
        .unwrap()
        .with_no_client_auth()
        .with_single_cert(vec![der.clone()], key_der)
        .unwrap();
    let mut roots = rustls::RootCertStore::empty();
    roots.add(der).unwrap();
    let client = rustls::ClientConfig::builder_with_provider(provider)
        .with_protocol_versions(&[&rustls::version::TLS13])
        .unwrap()
        .with_root_certificates(roots)
        .with_no_client_auth();
    (Arc::new(client), Arc::new(server))
}

fn handshakes() {
    let (rc, rs) = configs(rind_rustls::default_provider().unwrap());
    let (ac, asv) = configs(rustls::crypto::aws_lc_rs::default_provider());
    let r = rate(|| handshake(&rc, &rs));
    let a = rate(|| handshake(&ac, &asv));
    row("tls13", "handshake", 0, r, a);
}

fn main() {
    println!("rind CPU check: {:?}", rind::cpu::check());
    let only = std::env::var("ONLY").unwrap_or_default();
    let want = |s: &str| only.is_empty() || only.split(',').any(|o| o == s);
    if want("aead") {
        aead();
    }
    if want("hp") {
        header_protection();
    }
    if want("quic") {
        rustls_quic();
    }
    if want("hs") {
        handshakes();
    }
}
run.sh
#!/usr/bin/env bash
# Runs every quicbench variant in bin/, interleaved over $REPEATS rounds, pinned to one core.
# Each output line is "<variant>\t<round>\tROW\t<group>\t<op>\t<size>\t<rind>\t<aws>\t<ratio>".
set -u
cd "$(dirname "$0")"
REPEATS=${REPEATS:-2}
CORE=${CORE:-2}
arch=$(uname -m)
model=$(lscpu | sed -n 's/^Model name: *//p' | head -n1)
echo "CPU: $model; flags: $(grep -m1 -E '^(Features|flags)' /proc/cpuinfo | cut -d: -f2 | tr ' ' '\n' | grep -xE 'aes|vaes|pclmulqdq|vpclmulqdq|avx|avx2|avx512f|movbe|pmull|dit|sha3' | tr '\n' ' ')"
variants=(0.1.12 0.1.13 main)
for r in $(seq 1 "$REPEATS"); do
  for v in "${variants[@]}"; do
    taskset -c "$CORE" ./bin/quicbench-$v | sed -n "s/^ROW/$v\t$r\tROW/p"
  done
  # Tier toggles on main's rind_cpu_disable build: AEAD and the QUIC traits only.
  if [ "$arch" = x86_64 ]; then
    toggles=("vale:RIND_CPU_DISABLE_vaes=1" "clmul:RIND_CPU_DISABLE_vaes=1 RIND_CPU_DISABLE_movbe=1")
    grep -qw avx512f /proc/cpuinfo && toggles+=("noavx512:RIND_CPU_DISABLE_avx512f=1")
  else
    toggles=("nodit:RIND_CPU_DISABLE_dit=1" "dis-baseline:")
  fi
  for t in "${toggles[@]}"; do
    name=${t%%:*}; vars=${t#*:}
    env $vars ONLY=aead,hp,quic taskset -c "$CORE" ./bin/quicbench-main-dis | sed -n "s/^ROW/main-$name\t$r\tROW/p"
  done
done
analyze.py
"""Median rind/aws ratio and rind ops/s per variant, group, op, size, from run.sh output."""
import sys, statistics, collections
rows = collections.defaultdict(list)
cpu = None
for line in open(sys.argv[1]):
    line = line.rstrip("\n")
    if "CPU:" in line and cpu is None:
        cpu = line[line.index("CPU:"):]
    p = line.split("\t")
    if len(p) >= 9 and p[2] == "ROW":
        v, _, _, g, op, size, r, a, ratio = p[:9]
        rows[(g, op, int(size))].append((v, float(r), float(a), float(ratio)))
print(cpu)
variants = []
for k in rows:
    for v, *_ in rows[k]:
        if v not in variants:
            variants.append(v)
print("group\top\tsize\t" + "\t".join(variants) + "\t(rind Mops/s main | aws Mops/s)")
for k in sorted(rows, key=lambda k: (k[0], k[1], k[2])):
    cells = []
    for v in variants:
        rs = [x[3] for x in rows[k] if x[0] == v]
        cells.append(f"{statistics.median(rs):.2f}" if rs else "-")
    m = [x for x in rows[k] if x[0] == "main"]
    extra = f"{statistics.median(x[1] for x in m)/1e6:.2f} | {statistics.median(x[2] for x in m)/1e6:.2f}" if m else ""
    print(f"{k[0]}\t{k[1]}\t{k[2]}\t" + "\t".join(cells) + "\t" + extra)

(Written by Claude Opus 5.5)

@kixelated
kixelated marked this pull request as ready for review November 6, 2026 06:30
@pullfrog

pullfrog Bot commented Nov 6, 2026 •

Copy link
Copy Markdown

Your OpenCode Go account is out of credit. Top it up, then re-trigger Pullfrog.

OpenCode Go's response
provider error: Upstream request failed: Insufficient account funds

Top up OpenCode Go →

Pullfrog  | Rerun failed job ➔ | View workflow run | via Pullfrog | Using glm-5.3

@kixelated
kixelated added this pull request to the merge queue Nov 6, 2026
@coderabbitai

coderabbitai Bot commented Nov 6, 2026

Copy link
Copy Markdown

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 50 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

Learn how review limits work.

Review configuration:

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 6046bd49-5558-4fcb-a0f2-37738fe3ae8b
Commits

Reviewing files that changed from the base of the PR and between d0308e7 and d52246c.

Files selected for processing: 5

  • quest/m1/moq/README.md
  • quest/m1/moq/aarch64-short-gcm.md
  • quest/m1/moq/header-protection-block.md
  • quest/m1/moq/quic-path.md
  • quest/m1/moq/vaes-short-open.md
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant