[kixelated]

Finishes quest/m2/performance-survey/datagram-delivery.md.

Problem

Session-level datagram delivery had no complete-delivery benchmark or real-QUIC congestion measurements.

Approach

Adds session-level datagram measurement. The measurements show no datagram-specific per-object work worth removing yet, so no optimization lands.

• In-memory session bench (rs/moq-net/benches/session/mod.rs): session_delivery_datagram writes cadence datagrams per watched track each round and reads every one back from every viewer. It runs on lite-06, lite-07-wip and moq-transport-22. Each datagram must arrive in order with its sequence and one of 16 distinct payloads, so a lost or crossed datagram fails the run. The sweep covers publishers {1, 16} × viewers {1, 16, 64} at 64 B and cadence 1, payload 1024 B, and cadence {16, 64}. Throughput counts delivered datagrams. A priming group binds the moq-transport alias before the first datagram. The relay target registers the same group as relay_session_delivery_datagram.
• Deterministic counts: SESSION_ALLOCS=1 now also reports allocated bytes. On Linux it also reports user instructions from a per-thread perf counter (perf_event_open) read around each timed round. Results are given per delivered datagram or per viewer-group. Every task runs on the bench's single thread, so the count covers all protocol work, and an A/A rerun repeats within ±0.2%.
• Real QUIC under bufferbloat (rs/moq-tokio/tests/integration/datagram_delivery.rs): a stock server publishes 1000 B datagrams for 8 s to two viewers. One is on loopback. The other sits behind a 2 Mbit/s, 250 ms-queue shaper (10 ms delay, no random loss) at 1 Mbit/s ("under") and 4 Mbit/s ("over"). Per viewer it reports offered, received, lost, shaper queue drops, QUIC DATAGRAM frames, reordered arrivals and append-to-receive latency. It fails on a duplicate or corrupted datagram, or if the loopback viewer loses any. This is the slow-viewer isolation check.
• Nightly: the model and relay groups run in the existing rust-bench / rust-relay legs (--bench '*' and the relay session target). benchmark_datagram_delivery runs separately after the receive-window benchmark in the rust-test leg in sh/gh/nightly.sh, with a nextest slow-timeout override like benchmark_receive_windows. sh/gh/test-inventory.py passes.
• Docs: bench/README.md. The quest file and its link in the survey epic are deleted. The performance audit no longer lists session datagrams as unbenchmarked.

Impact

• Public API and wire formats: unchanged. Only bench, test, nightly and doc files change.

Measurements

The original report below is retained for its measured comparisons. Its allocation counts included cache ticks; the final head corrects that scope to match instruction counts. Current measurements are recorded under Validation. Neither set is a production optimization.

Host: Intel Core Ultra 7 270K Plus, 24 CPUs, Linux 7.0.0, rustc 1.98.1, bench profile (fat LTO). The load average was 10–14 on a shared host. Instruction counts are per thread and repeat regardless of load. The process was pinned with taskset -c 7. Baseline: 17e00c8148 plus this bench. After merging main 14e9973dc8 (including lite-07 padding datagrams, moq-dev/moq#7435), every row moved by −0.2% to −0.5%.

SESSION_ALLOCS=1 taskset -c 7 target/release/deps/session-* --bench session_delivery, model target, 256 rounds per shape:

shape (payload 64 B unless noted)	lite-06 instr	lite-07-wip instr	mt-22 instr	lite-06 allocs	mt-22 allocs
1 pub × 1 viewer	312,604	341,243	415,253	91.4	106.4
1 × 16	65,701	70,656	54,470	28.9	26.9
1 × 64	62,779	67,507	52,259	27.6	25.6
16 × 1	126,427	134,919	105,027	68.2	65.3
16 × 16	122,776	131,504	101,347	50.5	47.6
16 × 64	77,069	82,719	64,055	33.2	31.0
16 × 16, 1024 B	123,584	132,337	102,585	50.5	47.6
16 × 16, cadence 16	45,402	46,081	31,649	24.3	32.5
16 × 16, cadence 64	42,465	42,942	28,376	23.2	31.9

All counts are per delivered datagram. A 1024 B payload costs the same instructions as a 64 B one; only the allocated bytes grow (1,793 → 3,713 per datagram on lite-06), which is the per-subscriber encode copy. The relay target (relay_session_delivery_datagram) is within a few percent of the model except at 1×1, where per-round stats and admission work is not amortized.

CPU time from Criterion (SESSION_CPU=1 … --warm-up-time 1 --measurement-time 2 session_delivery_datagram_cpu): lite-06 is 7.3 µs per datagram at 1×16, 14.0 µs at 16×16 cadence 1 and 4.2 µs at cadence 64. moq-transport-22 is 6.3, 12.4 and 2.7 µs.

Attribution

perf record -e instructions:u --call-graph fp on the profiling build, lite-06, 16×16, cadence 1 (inclusive shares):

• SubscribeServe::poll 56%. Inside it, track::Subscriber::poll_drop is 29% (Cursor::probe 19%, Misses::judge 14%, RangeMap::remove 8%), poll_finished 13%, poll_group 11%, and resume::Subscriber::poll_recv_datagram 12% (Reader::sync/learn_end/let_go, WaiterList::register).
• Datagram-specific work is small: serve_datagram (encode + send) 2.4%, insert_wire_datagram 3.9%, push_datagram 2.9%, Ledger::account 2.6%. Allocator time is about 3%.

So most of a datagram's cost is the subscription's per-wake polling, which every group frame pays as well. Each served datagram returns Continue from TrackRun::poll_step, and the next step re-polls drop forwarding, group readiness and the boundary before the next datagram. The cadence sweep shows the split: about 41k instructions per datagram marginal versus about 81k per wake at 16×16.

Alternatives

Draining every buffered datagram in one lite TrackRun step, as the IETF publisher's poll_datagrams already does, gave these paired, deterministic results against the same baseline:

lite-06, 16×16	baseline	drain	Δ
cadence 1	122,776 instr, 50.5 allocs	127,262 instr, 54.5 allocs	+3.7%
cadence 16	45,402, 24.3	30,958, 17.0	−31.8%
cadence 64	42,465, 23.2	27,163, 15.4	−36.0%

lite-07-wip matched: +3.3–3.4% at cadence 1 and −33–37% at cadence 16–64. moq-transport-22 was unchanged (−0.3%). The extra poll that finds the buffer empty costs more than the drain saves when each wake carries one datagram. That is the case for live audio, the main datagram consumer. The prototype also changes the documented one-item-per-step contract that lets the caller watch the requester between items.

Real QUIC (benchmark_datagram_delivery)

Two runs (seeds printed) gave matching results; the second run:

version	cell	viewer	offered	received	lost	shaper drops	reordered	p50 ms	p99 ms	max ms
lite-06	under	loopback	1000	1000	0	0	0	0.2	0.6	1.0
lite-06	under	bufferbloat	1000	1000	0	0	0	11.3	12.4	12.6
lite-06	over	loopback	4000	4000	0	0	291	0.2	1.3	4.2
lite-06	over	bufferbloat	4000	2984	1016	0	1321	51.9	8204.5	8463.5
lite-07-wip	over	bufferbloat	4000	2962	1038	0	1306	51.0	8110.4	8347.2
mt-22	over	bufferbloat	4000	2967	1033	0	1232	48.7	8147.6	8367.7

(The lite-07-wip and moq-transport-22 "under" rows and loopback rows match lite-06.)

• Isolation holds: the loopback viewer received every datagram while the other viewer was congested.
• Nothing expires a datagram by age. Congestion control keeps the shaper's queue from overflowing, so no drops happen there. The sender's QUIC datagram buffer (1 MiB) absorbs the overload instead and drops only on overflow, about 25% here. Datagrams are prioritized newest-first within the subscription. Half arrive fresh (p50 about 50 ms) and the oldest survivors wait up to about 8 s. At a shorter 4 s run nothing was dropped and p99 was 8.1 s. So under sustained overload, about a quarter of the bottleneck carries datagrams that a real-time consumer would discard. Changing that is an expiry and scheduling policy change, outside this quest ("keep … expiry … behavior intact").

Choices made without asking

• Landed no optimization. The plan accepts a measured no-win and lands an optimization only when consumer benefit justifies it. The drain prototype regresses the one-datagram-per-wake case real audio uses. The remaining cost is per-wake subscription polling shared with groups, which is a model-wide redesign rather than a datagram fix.
• "Expired" is reported through latency and loss columns, not a separate count. Nothing in moq-net or moq-quic expires a sent datagram by age, so that count would always be 0. The test documents this.
• Real-QUIC CPU is not reported. That process mixes QUIC, the shaper, the kernel and both clients, so CPU comes from the in-memory bench and delivery and latency from the QUIC run, as the quest asks.
• The QUIC run lasts 8 s per cell so the sender's 1 MiB buffer overflows and the drop path shows. It needed a nextest slow-timeout override, mirroring benchmark_receive_windows.
• Shared the native QUIC dial helper and rand_seed from receive_credit.rs; the dial helper also uses its shared AnyCert.

Follow-ups

1. Stale datagrams under congestion, now planned in quic-deadline: measured above, with p99 at 8 s and 25% of the bottleneck spent on late datagrams. Options: a send deadline from the subscription's max delay, a smaller datagram send buffer, or oldest-first within a subscription. Each changes expiry or scheduling policy, so it needs a maintainer decision with this benchmark as evidence.
2. Per-wake subscription polling: poll_drop/Cursor::probe/Misses::judge and resume::Reader::sync dominate both datagram and paced-group delivery. Waking only the facet whose state changed would cut the per-wake cost for datagrams and groups alike. The datagram drain becomes worth revisiting once an empty poll is cheap.
3. Each kio::WaiterList::register that initializes its OnceLock<Arc<Waker>> allocates (about 5% of instructions at 1×1). Ask Perf whether that is already owned.

Validation

Current head: 432ea3ba9549bb425c3e4355125de3e56c0a2e8a. Base merged: 693a0ec2292ed6e28a7f36dcab5af293bfc52383.

• quest gate nix develop --ignore-environment --keep HOME --keep USER --keep SSH_AUTH_SOCK --keep QUEST_GATE_ADMITTED --command bash -c 'export PATH="/nix/var/nix/profiles/default/bin:$PATH"; exec just ci origin/main': passed end to end, 11,333 Rust tests passed, 128 skipped, plus selected lint, docs, benchmark compile and tooling checks.
• backend::quic_client_close_drains_finished_track passed in both complete test runs. No failing test was retried. The separate root repair moq-dev/moq#7634 was neither duplicated nor merged into this branch; a pass here does not repair that race.
• quest check --base origin/main --branch quest/m2/performance-survey/datagram-delivery and python3 sh/gh/test-inventory.py: passed.
• SESSION_ALLOCS=1 taskset -c 7 cargo bench --locked -p moq-net -p moq-relay --features moq-net/fuzz,moq-relay/test-support --bench session -- session_delivery: passed twice, with 27 datagram shapes per backend per run. Every delivery was checked. The largest instruction A/A difference was 0.96%; this is measurement spread, not a production optimization. All accepted counts had equal enabled and running counter times. An earlier unpinned report was rejected by the new multiplexing guard (272,467,773 ns enabled vs 271,320,840 ns running).
• just rs test -p moq-tokio --release -E 'test(benchmark_datagram_delivery)' --run-ignored only --success-output immediate: passed on the final fixture, with the finite queue, overflow accounting, strict loopback delivery and post-drain snapshot intact. Full output rows and replay seeds follow.

Current CPU-7 counted delivery sweep (second A/A run)

All values are per delivered datagram; requested bytes include complete reallocation requests. Cache ticks are excluded. Each row delivered and checked every datagram.

backend	version	publishers	viewers	payload B	cadence	allocations	requested B	instructions
model	moq-lite-06	1	1	64	1	52.46	46295	355,094
model	moq-lite-06	1	16	64	1	8.77	401	55,975
model	moq-lite-06	1	64	64	1	8.28	387	53,408
model	moq-lite-06	16	1	64	1	15.20	673	107,310
model	moq-lite-06	16	16	64	1	15.16	672	104,688
model	moq-lite-06	16	64	64	1	10.35	455	65,662
model	moq-lite-06	16	16	1024	1	15.16	2592	105,446
model	moq-lite-06	16	16	64	16	7.99	330	35,021
model	moq-lite-06	16	16	64	64	7.76	323	32,388
model	moq-lite-07-wip	1	1	64	1	58.54	46435	378,998
model	moq-lite-07-wip	1	16	64	1	9.83	436	60,073
model	moq-lite-07-wip	1	64	64	1	9.29	421	57,366
model	moq-lite-07-wip	16	1	64	1	17.20	739	115,116
model	moq-lite-07-wip	16	16	64	1	17.16	738	112,403
model	moq-lite-07-wip	16	64	64	1	11.60	496	70,536
model	moq-lite-07-wip	16	16	1024	1	17.16	2658	113,175
model	moq-lite-07-wip	16	16	64	16	8.12	336	35,639
model	moq-lite-07-wip	16	16	64	64	7.82	327	32,802
model	moq-transport-22	1	1	64	1	85.21	62128	470,250
model	moq-transport-22	1	16	64	1	16.17	781	48,984
model	moq-transport-22	1	64	64	1	15.36	750	46,421
model	moq-transport-22	16	1	64	1	29.14	1388	93,956
model	moq-transport-22	16	16	64	1	29.10	1387	90,741
model	moq-transport-22	16	64	64	1	19.06	902	57,144
model	moq-transport-22	16	16	1024	1	29.10	5227	92,123
model	moq-transport-22	16	16	64	16	25.68	1165	28,683
model	moq-transport-22	16	16	64	64	25.60	1163	25,822
relay	moq-lite-06	1	1	64	1	64.34	59849	428,985
relay	moq-lite-06	1	16	64	1	8.85	404	55,056
relay	moq-lite-06	1	64	64	1	8.35	392	52,400
relay	moq-lite-06	16	1	64	1	15.38	642	108,199
relay	moq-lite-06	16	16	64	1	15.34	641	105,658
relay	moq-lite-06	16	64	64	1	10.47	451	65,128
relay	moq-lite-06	16	16	1024	1	15.34	2561	106,454
relay	moq-lite-06	16	16	64	16	8.00	338	35,424
relay	moq-lite-06	16	16	64	64	7.77	329	32,687
relay	moq-lite-07-wip	1	1	64	1	73.64	60081	468,999
relay	moq-lite-07-wip	1	16	64	1	9.92	439	59,239
relay	moq-lite-07-wip	1	64	64	1	9.37	426	56,412
relay	moq-lite-07-wip	16	1	64	1	17.44	709	116,547
relay	moq-lite-07-wip	16	16	64	1	17.40	708	113,640
relay	moq-lite-07-wip	16	64	64	1	11.73	493	70,095
relay	moq-lite-07-wip	16	16	1024	1	17.40	2629	114,462
relay	moq-lite-07-wip	16	16	64	16	8.13	344	36,063
relay	moq-lite-07-wip	16	16	64	64	7.83	333	33,126
relay	moq-transport-22	1	1	64	1	109.42	81248	615,762
relay	moq-transport-22	1	16	64	1	16.19	782	48,110
relay	moq-transport-22	1	64	64	1	15.36	754	45,357
relay	moq-transport-22	16	1	64	1	29.20	1353	96,701
relay	moq-transport-22	16	16	64	1	29.16	1352	91,759
relay	moq-transport-22	16	64	64	1	19.10	896	56,547
relay	moq-transport-22	16	16	1024	1	29.16	5193	93,663
relay	moq-transport-22	16	16	64	16	25.69	1174	29,395
relay	moq-transport-22	16	16	64	64	25.61	1169	25,927

Current real-QUIC delivery measurements

version,cell,viewer,offered,received,lost,overflowed,quic_frames,reordered,p50_ms,p99_ms,max_ms
moq-lite-06,under,loopback,1000,1000,0,0,1000,0,0.2,0.5,0.6
moq-lite-06,under,bufferbloat,1000,1000,0,0,1000,0,11.3,12.3,12.7
moq-lite-06,over,loopback,4000,4000,0,0,4000,58,0.2,0.5,0.8
moq-lite-06,over,bufferbloat,4000,2982,1018,0,2982,1274,50.9,8179.2,8449.6
moq-lite-07-wip,under,loopback,1000,1000,0,0,1000,0,0.2,0.6,0.8
moq-lite-07-wip,under,bufferbloat,1000,1000,0,0,1000,0,11.3,12.3,12.5
moq-lite-07-wip,over,loopback,4000,4000,0,0,4000,78,0.1,0.5,2.4
moq-lite-07-wip,over,bufferbloat,4000,2974,1026,0,2974,1276,50.9,8157.9,8376.3
moq-transport-22,under,loopback,1000,1000,0,0,1000,0,0.3,1.4,4.1
moq-transport-22,under,bufferbloat,1000,1000,0,0,1000,0,11.5,13.9,15.8
moq-transport-22,over,loopback,4000,4000,0,0,4000,665,0.3,1.7,3.9
moq-transport-22,over,bufferbloat,4000,2968,1032,0,2968,1352,56.0,8141.9,8396.5

Replay seeds in version/load order (lite-06 under/over, lite-07 under/over, mt-22 under/over):

MOQ_SHAPER_SEED=1793639245506563141
MOQ_SHAPER_SEED=1793639254606751197
MOQ_SHAPER_SEED=1793639268119134120
MOQ_SHAPER_SEED=1793639277250092685
MOQ_SHAPER_SEED=1793639290765426078
MOQ_SHAPER_SEED=1793639299891904502

Earlier validation

• just ci origin/main (inside nix develop, through quest gate) passed on head 9234af7d87: 10852 tests passed.
• benchmark_datagram_delivery passed on 9234af7d87 (68.0 s). Its rows match the earlier runs: 2972–2982 of 4000 received behind the bottleneck, p99 8.2 s, and 0 shaper queue drops now counted over the full drain window.
• Earlier, on 5633ec5051: nix develop --command python3 sh/gh/test-inventory.py and quest check passed, and the datagram report reran on the merged head (table deltas above).

Review

Claude Code, Claude Opus 5.5, medium effort, quest review --base origin/main --harness claude, session fe2b77db-7e12-4db0-ba24-fa5d012b3ab6, round 2, head 432ea3ba9549bb425c3e4355125de3e56c0a2e8a against ea1822d6ca0d638fcdab73bd6d359798a3635c29: no findings.

Round 1 dispositions, accepted in round 2:

• Fixed nightly contention by running the two real-time benchmarks in separate synchronous nextest invocations.
• Retained the strict loss-free loopback control. A scheduling-compromised control must fail the measurement; relaxing it would hide loss without establishing its cause. All six final loopback rows delivered every offered datagram.
• Fixed allocation and requested-byte counting to exclude cache ticks, matching the instruction window. Historical allocation figures above used the older scope; use the current counted sweep for delivery-only counts.
• Clarified that requested bytes sum complete allocation/reallocation requests, including growth in place. They are neither live bytes nor copied bytes.
• Fixed silent PMU multiplexing undercounts by reading enabled/running times and refusing unequal times, following the Linux perf read format.
• Shared the native TLS/ALPN/endpoint dial helper. The one-client credit fixture and two-viewer datagram fixture retain their different server/shaper lifecycles.

The prior online snapshot finding remains fixed: both receivers finish before the final shaper verification and drop snapshot.

(Written by GPT-6.1 Sol)

[coderabbitai]

No actionable comments were generated in the recent review.

Recent review info

Run configuration

• Source mode: Auto
• Repository: Unavailable
• Branch: Unavailable
• File: No configuration file from this repository applied
• Commit: Unavailable
• Layers: Organization UI
• Review profile: CHILL
• Plan: Advanced
• Run ID: ab1e5577-5ee3-4704-bf0c-9da03d002cc6

Commits

Reviewing files that changed from the base of the PR and between d52d55a and dec0914.

Files selected for processing: 12

• .config/nextest.toml
• bench/README.md
• quest/m2/audit-perf.md
• quest/m2/performance-survey/README.md
• quest/m2/performance-survey/datagram-delivery.md
• rs/moq-net/benches/session.rs
• rs/moq-net/benches/session/mod.rs
• rs/moq-relay/benches/session.rs
• rs/moq-tokio/tests/integration/datagram_delivery.rs
• rs/moq-tokio/tests/integration/main.rs
• rs/moq-tokio/tests/integration/receive_credit.rs
• sh/gh/nightly.sh

Files with no reviewable changes: 2

• quest/m2/performance-survey/README.md
• quest/m2/performance-survey/datagram-delivery.md

Included review availability: This review used your included allowance. Your plan provides up to 4 included reviews per hour; 2 remain after this review.

Walkthrough

The changes add datagram delivery scenarios to the in-memory session benchmarks and add an ignored QUIC integration benchmark for loopback and rate-limited viewers. The benchmarks validate delivered datagrams and report throughput and delivery measurements. Allocation mode also reports allocated bytes and optional Linux instruction counts. The changes update benchmark documentation, nightly test configuration, and performance-survey tracking.

Merge Risk: Minimal · up to dec09

No identified issue prevents merging after normal checks.

Pre-merge checks: 4 · 1

Failed checks: 1 warning

Check name	Status	Explanation	Resolution
Docstring Coverage		Docstring coverage is 53.33% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documenta…	Write docstrings for the functions missing them to satisfy the coverage threshold.

Passed checks: 4 passed

Check name	Status	Explanation
Title check		The title "bench(net): measure complete datagram delivery" matches the main change: new session-level datagram benchmarks in the moq-net bench suite, plus a real-QUIC datagram delivery test. The title…
Linked Issues check		Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check		Check skipped because no linked issues were found for this pull request.
Description check		The pull request description directly matches the changeset. It describes the new session datagram benchmarks, real-QUIC bufferbloat test, nightly integration, documentation updates, measured results,…

Full details: Title check

Explanation

The title "bench(net): measure complete datagram delivery" matches the main change: new session-level datagram benchmarks in the moq-net bench suite, plus a real-QUIC datagram delivery test. The title is short, specific, and follows the conventional prefix style. It does not cover the nightly wiring, docs, or the audit update, which is expected for a title.

Full details: Description check

Explanation

The pull request description directly matches the changeset. It describes the new session datagram benchmarks, real-QUIC bufferbloat test, nightly integration, documentation updates, measured results, and validation.

Full details: Docstring Coverage

Explanation

Docstring coverage is 53.33% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentation-only edits and deletions within surviving functions are included; unchanged diff context and fully deleted functions are excluded. Analyzed 15 functions across 7 files. (3 skipped: 3 unsupported.)

Resolution

Write docstrings for the functions missing them to satisfy the coverage threshold.

Finishing Touches

Simplify code

• Commit to this branch
• Create a new PR

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

• X
• Mastodon
• Reddit
• LinkedIn

[kixelated]

Automated review by review (OpenAI)

Reviewed commit: 5633ec505114c25797587180ab66ef5b04219e02.

[P2] Include the drain in shaper verification and drop accounting — rs/moq-tokio/tests/integration/datagram_delivery.rs:292-297. Shaper::verify() returns a snapshot, but this is taken immediately after the last append, before the receivers finish draining the congested sender. If queued traffic overflows the shaper during that drain, its loss appears in the final lost column but is omitted from overflowed; a forwarding failure during the same interval also escapes verification and can look like permitted best-effort loss. Await both receiver results while retaining their viewer handles, then verify/snapshot the shaper before reporting, so all columns cover the same observation interval.

Overall direction is sound: reuse the model/relay harness for protocol cost and keep real-QUIC congestion measurements separate. Leaving scheduling unchanged is simpler than landing the workload-sensitive drain optimization. No public API or wire change identified.

Verification: static GitHub review of the changed paths and relevant model/shaper code only; I did not run tests, benchmarks, or reproduce the reported performance figures.

(Written by OpenAI)

[pullfrog]

ℹ️ No critical issues — one measurement-accuracy nit inline.

Reviewed changes

• In-memory session bench: session_delivery_datagram writes burst datagrams per watched track and reads every one from every viewer with sequence/payload assertions, over a publishers × viewers × payload × cadence sweep. The new Room::next bookkeeping correctly relies on groups and datagrams sharing the model's sequence namespace, and the extracted group_round is behavior-preserving for the existing group benches.
• Alloc/instruction report: SESSION_ALLOCS=1 now also reports allocated bytes and, on Linux little-endian with perf access, user instructions via a hand-rolled perf_event_open.
• Real QUIC test: benchmark_datagram_delivery publishes to a loopback viewer and a bufferbloated one, reporting offered/received/lost/overflowed/reordered/latency and failing on duplicate, corrupt, or loopback loss. Reuses AnyCert/rand_seed from receive_credit.rs.
• Nightly and docs: nextest slow-timeout override, combined rust-test invocation, bench/README.md; deletes the completed quest and renumbers the audit.

  ^{｜ Fix all ➔ ｜ Fix 👍s ➔ ｜ View workflow run ｜ Using deepseek-v4.1-flash (free via Pullfrog for OSS) ｜ 𝕏}

[kixelated]

Automated review by review (OpenAI)

Reviewed commit: 9234af7d8755d6aa6c779451c5743592590799b1 (incremental review from 5633ec505114c25797587180ab66ef5b04219e02; one direct-child fix, not a rebase).

Prior P2 fixed: rs/moq-tokio/tests/integration/datagram_delivery.rs:292–300 now awaits every receiver while retaining its Viewer, then verifies/snapshots the shaper before reporting. This addresses the drain-window accounting and forwarding-check issue in the prior review, also identified in the existing thread.

No new actionable bugs found in this update. The small two-phase collect-then-report change is appropriately scoped; receiver tasks still drain concurrently, and no scheduling or wire redesign is needed.

Verification: static GitHub inspection of the complete incremental diff, receiver/report lifecycle, and shaper implementation. I did not run tests or benchmarks or independently reproduce the reported measurements.

[pullfrog]

✅ No new issues found.

Reviewed changes

Since the prior pullfrog review, one commit changes the real-QUIC test's tail.

• Drain the viewers before snapshotting the shaper: measure now awaits every receiver, keeping each Viewer, before a single shaper.verify(), so overflowed and the forwarding-failure check cover the same interval as received and lost. This resolves the prior P2.

  ^{｜ View workflow run ｜ Using deepseek-v4.1-flash (free via Pullfrog for OSS) ｜ 𝕏}

[moq-bot]

codex / Codex land fixer note:

Merged the current base and resolved the documentation conflict in quest/m2/audit-perf.md, preserving the removal of completed datagram work and the base’s updated JWT benchmark status. No behavior changes.

Validation passed: 4,230 package tests through Nextest with four test threads, both datagram benchmark targets in test mode, the ignored datagram integration benchmark across all three protocol versions, formatting, and 14 nightly script tests.

Follow-up test observations: plain cargo test failed setup::tests::credentials_stay_out_of_traces because another test had installed a global tracing subscriber; it passed in isolation and with Nextest. A high-concurrency Nextest run failed web::tests::an_unanswered_http_admission_is_refused_at_the_timeout and cluster::tests::a_critical_failure_closes_the_dials in moq-relay; both passed in the complete four-thread run.

(written by codex / Codex)

[moq-bot]

codex / Codex land fixer note:

The batch's unchanged test backend::quic_client_close_drains_finished_track passed on this pinned branch. The initial local suite had 4,229 passes and one failure in the unchanged relay test shutdown_signal::a_reconnecting_client_rides_out_a_restart: the publisher gave up during the restart: Err(Reconnect("reconnect timed out after 333.333333ms: cancelled")). A complete rerun with four test workers passed all 4,230 tests, including both tests. Both modified session benchmark targets compiled. Requeue unchanged; follow up both intermittent tests separately.

(written by codex / Codex)

[moq-bot]

codex / Codex land fixer: This maintainer pick already used its mechanical fi·x attempt. Run /quest-iterate to diagnose the repeated failure, then pick it again. Recommendation: run /quest-iterate before picking it again.

		.expect("relay task panicked")
		.expect("relay exited with an error");
	assert!(
		signalled.elapsed() >= DRAIN_TIMEOUT,
		"relay exited after {:?}, short of the {DRAIN_TIMEOUT:?} drain window",
		signalled.elapsed()
	);
}

async fn a_signal_right_after_ready_drains_inner() {
	let _ = rustls::crypto::aws_lc_rs::default_provider().install_default();

	let relay = Relay::load(relay_config()).await.expect("load relay");
	let port = relay.tcp_addr().expect("TCP listener bound").port();
	let ready = relay.ready();
	let run = tokio::spawn(relay.run());
	ready.wait().await.expect("relay ready");
    Checking moq-tokio v0.19.23 (/work/rs/moq-tokio)
    Checking moq-stats v0.2.12 (/work/rs/moq-stats)
    Checking moq-relay v0.17.2 (/work/rs/moq-relay)
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 4m 04s

codex
The run finished with 4,229 passes and one different failure: `shutdown_signal::a_reconnecting_client_rides_out_a_restart` exceeded its reconnect timeout. That test is unchanged by #7517. Both session benchmark checks passed. I’ll rerun the suite with the repository’s local limit of four test workers to see whether the timeout was transient.

exec
/bin/bash -lc 'cargo nextest run --locked --tests -p moq-net -p moq-relay -p moq-tokio --features moq-net/fuzz,moq-relay/test-support --profile ci --no-fail-fast -j 4 > /tmp/land-7517-retest.log 2>&1' in /work
exec
/bin/bash -lc "tail -5 /tmp/land-7517-retest.log; rg 'FAIL|TIMEOUT|quic_client_close_drains_finished_track|a_reconnecting_client_rides_out_a_restart' /tmp/land-7517-retest.log" in /work
 exited 1 in 0ms:
        PASS [   0.023s] (2966/4230) moq-net::integration lazy_discovery::requesting_an_unwatched_path_discovers_its_instance
        PASS [   0.020s] (2967/4230) moq-net::integration lazy_discovery::serving_the_peer_asks_nothing
        PASS [   0.026s] (2968/4230) moq-net::integration lazy_discovery::waiting_on_a_path_asks_about_it
        PASS [   0.105s] (2969/4230) moq-net::integration mesh_withdraw::cross_node_close_retracts_once_lite06
        PASS [   0.103s] (2970/4230) moq-net::integration mesh_withdraw::cross_node_close_retracts_once_lite07

exec
/bin/bash -lc "tail -6 /tmp/land-7517-retest.log; rg 'FAIL|TIMEOUT|quic_client_close_drains_finished_track|a_reconnecting_client_rides_out_a_restart' /tmp/land-7517-retest.log" in /work
 succeeded in 0ms:
        PASS [   0.026s] (3952/4230) moq-tokio server::tests::serving_nothing_is_an_error
        PASS [   0.091s] (3953/4230) moq-tokio server::tests::close_releases_quic_socket
        PASS [   0.029s] (3954/4230) moq-tokio server::tests::stalled_setup_reply_is_closed_at_the_deadline
        PASS [   0.029s] (3955/4230) moq-tokio server::tests::stalled_setup_is_closed_at_the_deadline
        PASS [   0.019s] (3956/4230) moq-tokio server::tests::stream_listener_config_parses
        PASS [   0.021s] (3957/4230) moq-tokio server::tests::stream_only_config_has_no_quic
        PASS [   1.157s] (3624/4230) moq-relay::integration shutdown_signal::a_reconnecting_client_rides_out_a_restart

codex
The relay restart test passed on the rerun, which has reached more than 3,900 passing tests with no failures. If the remaining tests pass, I’ll record both PRs for unchanged requeue and name the observed flaky tests in the result.

 succeeded in 88875ms:
exec
/bin/bash -lc "tail -5 /tmp/land-7517-retest.log; rg 'quic_client_close_drains_finished_track' /tmp/land-7517-retest.log; git rev-parse pr-7520 pr-7517; git status --short" in /work
 succeeded in 0ms:
        PASS [   0.060s] (4228/4230) moq-tokio::integration worker::the_other_wildcard_spelling_is_refused
        PASS [   0.191s] (4229/4230) moq-tokio::integration worker::spawner_contains_a_factory_panic
        PASS [   3.106s] (4230/4230) moq-tokio::integration reconnect::two_peers_redirecting_to_each_other_escalate_the_backoff
────────────
     Summary [  85.873s] 4230 tests run: 4230 passed, 38 skipped
        PASS [   0.088s] (4070/4230) moq-tokio::integration backend::quic_client_close_drains_finished_track
f19b868b7407552e18a2453129e065028da07e6b
dec0914ef77034a134adc091c6947226faad8b25

exec
/bin/bash -lc "git switch base; python3 - <<'PY'
import json
from pathlib import Path
result = [
    {
        \"number\": 7520,
        \"action\": \"requeue\",
        \"reason\": \"Healthy unchanged; batch flake moq-tokio::integration backend::quic_client_close_drains_finished_track passed. Passed: cargo nextest run --locked --tests -p moq-audio -p moq-tokio --profile ci --no-fail-fast (774 tests); cargo check --locked -p moq-audio --bench batches; bash -n sh/gh/nightly.sh.\",
        \"comment\": \"The batch failed in the unchanged test "'`backend::quic_client_close_drains_finished_track` with `subscribe failed: Unroutable`. Neither PR changes that test or its production path. All 774 audio and transport tests passed on this pinned branch, including that test, and the new audio benchmark target compiled. Requeue unchanged; follow up the intermittent QUIC test separately."
    },
    {
        "number": 7517,
        "action": "requeue",
        "reason": "Healthy unchanged; batch flake moq-tokio::integration backend::quic_client_close_drains_finished_track passed. Local flake moq-relay::integration shutdown_signal::a_reconnecting_client_rides_out_a_restart passed on full rerun. Passed: cargo nextest run --locked --tests -p moq-net -p moq-relay -p moq-tokio --features moq-net/fuzz,moq-relay/test-support --profile ci --no-fail-fast -j 4 (4230 tests); cargo check --locked -p moq-net -p moq-relay --bench session --features moq-net/fuzz,moq-relay/test-support; bash -n sh/gh/nightly.sh.",
        "comment": "The batch'"'s unchanged test "'`backend::quic_client_close_drains_finished_track` passed on this pinned branch. The initial local suite had 4,229 passes and one failure in the unchanged relay test `shutdown_signal::a_reconnecting_client_rides_out_a_restart`: `the publisher gave up during the restart: Err(Reconnect('"\\\"reconnect timed out after 333.333333ms: cancelled\\\"))"'`. A complete rerun with four test workers passed all 4,230 tests, including both tests. Both modified session benchmark targets compiled. Requeue unchanged; follow up both intermittent tests separately."
    }
]
Path('"'quest-land-result.json').write_text(json.dumps(result, indent=2) + '\\n')
assert len(result) == 2
assert {x['number'] for x in result} == {7520, 7517}
PY
 git status --short" in /work
 succeeded in 145ms:
Switched to branch 'base'
?? quest-land-result.json

codex
Wrote [quest-land-result.json](/work/quest-land-result.json): both PRs requeue unchanged, with no blame assigned.

All 774 audio/transport tests and 4,230 network/relay/transport tests passed, along with benchmark compilation checks. Recorded both observed flaky tests. Branches remain unchanged; the result file is uncommitted.
tokens used
70,645
Wrote [quest-land-result.json](/work/quest-land-result.json): both PRs requeue unchanged, with no blame assigned.

All 774 audio/transport tests and 4,230 network/relay/transport tests passed, along with benchmark compilation checks. Recorded both observed flaky tests. Branches remain unchanged; the result file is uncommitted.
fixer result written

(written by codex / Codex)

[kixelated]

Automated review by review (OpenAI)

Reviewed commit: 432ea3ba9549bb425c3e4355125de3e56c0a2e8a, incrementally from 9234af7d8755d6aa6c779451c5743592590799b1, separating intervening main merges from the benchmark changes.

No new actionable bugs found. The allocation/requested-byte window now excludes cache ticks like the instruction window; checking perf enabled/running times avoids silently accepting multiplexed undercounts. Separate synchronous nightly invocations avoid these two real-time measurements competing with each other. Sharing the test-only QUIC dial helper is a simpler, appropriately scoped refactor; production scheduling and wire behavior remain unchanged.

The prior drain-window accounting finding remains fixed: receivers finish while their viewer handles are retained, before the shaper snapshot (see previous review).

Verification: static GitHub inspection of the incremental commits, surrounding counter/runtime and receiver lifecycle code, and nightly wiring. I did not run tests or benchmarks or independently reproduce the reported measurements.

[pullfrog]

✅ No new issues found.

Reviewed changes

Since the prior pullfrog review, three commits land: a merge of newer main, the counter/nightly changes, and a one-line visibility fix.

• Fail on multiplexed perf counts: Instructions::read requests PERF_FORMAT_TOTAL_TIME_ENABLED | RUNNING and asserts time_enabled == time_running, so a multiplexed counter fails the run instead of silently undercounting.
• Scope allocation counting to the timed round: measure toggles COUNTING around round(), excluding the cache tick and matching the instruction window; allocations no longer sets the flag. ALLOCATED's doc now states it sums requested allocation/reallocation sizes.
• Share the QUIC dial helper: dial is extracted in receive_credit.rs; both datagram_delivery and setup_through call it.
• Keep AnyCert visible: 432ea3b restores pub(crate) for the cross-module use in receive_ts.rs; the 9607e2f narrowing would not have compiled, and the net change vs. base is nil.
• Isolate the real-time nightly runs: benchmark_receive_windows and benchmark_datagram_delivery run in separate just rs test invocations; bench/README.md documents the counter scope and the multiplexing guard.

  ^{｜ View workflow run ｜ Using deepseek-v4.1-flash (free via Pullfrog for OSS) ｜ 𝕏}