Repository navigation
feat!: lease qmux receive credit, one-segment floor, smaller windows, sized readiness - #7624
Conversation
Quest-Claim: 646b3953d7f54510
…oups/fastest-reader
… leases Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…nder over WebSocket; migration notes Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ions Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…d rate Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…oups/fastest-reader
…ic] to every qmux accept GROUP streams and FETCH answers waiting unread on TRACK_INFO (and ietf data streams waiting on their alias) rank below every other stream, so a full window evicts them before control streams. They watch poll_reset meanwhile, which now reports an eviction: moq-quic's received_reset fails with Evicted until the stream is stopped, through moq-tokio, moq-uring, and qmux, so an evicted parked or paused stream stops and eviction resumes. Every qmux session a server accepts starts at its resolved [quic] windows: the tcp, tls, unix, and WebSocket listeners, the tls acceptor, and websocket::Options, which the relay's own WebSocket path fills from [quic]. A WebRTC session keeps its fixed receive window at ok(). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…emains Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…INFO Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…on eviction Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
# Conflicts: # rs/moq-relay/MIGRATE.md
kixelated
left a comment
There was a problem hiding this comment.
Automated review by review (OpenAI)
Reviewed commit: 7b1b2627db2294a97792f17848c1f7ec7e719e37.
No actionable bugs found in the reviewed changes. The direction is sound: reuse moq-quic's existing lease/eviction machinery rather than add a separate qmux buffering policy, share the sized-readiness admission gate, and let parked streams observe eviction so they release their credit. The smaller windows are an explicit, documented throughput/memory tradeoff with configuration overrides.
The earlier local-review finding recorded in the PR body, the WebTransport reset wait panicking on eviction, is addressed by propagating ReadError::Evicted; it is not a new finding.
Verification limits: static GitHub source/diff inspection, including an independent moq-net review. I did not run builds, tests, interop, or benchmarks and have not independently validated the reported throughput figures.
(Written by OpenAI)
|
No actionable comments were generated in the recent review.
|
There was a problem hiding this comment.
✅ No new issues found.
Reviewed changes
Reviewed the full diff at 7b1b2627 (53 files): the qmux receive-lease rewrite, the one-segment floor, the smaller window defaults, sized readiness, the RECV_PARKED priority, and the qmux accept-path plumbing. I traced the eviction/credit bookkeeping end to end and tested the two highest-risk subsystems (qmux lease state machine; lite parked-stream lifecycle) with independent passes, both negative.
- qmux native receive leases —
RecvStream/RecvLeasedrive moq-quic's mux lease API directly; credit is held until release, a stuck window evicts the lowest-priority stream (Error::Evicted), andpoll_resetreports evictions. Verifiedheld/evicted_held/discard_lease/release_creditbalance acrosspoll_lease/poll_reset/stop/Drop/eviction, and that every in-tree caller drops an evicted reader soDropstops the stream and frees the withheld credit. - One-segment floor — lite-01/02
ANNOUNCE_INITcaps atSEGMENT - 16, soMIN_RECEIVE_WINDOWresolves to one 1 MiB segment; oversized legacy sets fail loudly on both ends. - Smaller windows — clients default to 4/1/4 MiB in moq-tokio, moq-uring, and qmux;
[limits.peer.quic]drops to 16 MiB; peers keep the[quic]1 MiB stream window. - qmux client/peer split — every qmux accept path takes the server's
[quic];with_quic_limitswidens peers atok()(WebRTC keeps its read-queue-sized window); a rate cap qmux cannot pace fails admission withError::QmuxLimits. - Parked-stream priority — streams waiting unread on TRACK_INFO rank at
RECV_PARKEDand watchpoll_reset; a failed parked stream withdraws its group. Lock order is uniformlysubscribes -> parked, no inversion. - Sized readiness —
group::Producer::poll_ready/readytake&frame::Infoand reusepoll_paced, whole-frame exception intact.
ℹ️ Sized readiness has no in-tree production consumer yet
group::Producer::poll_ready(&info) / ready(&info) now take the next frame's size, but the only callers in the tree are rs/moq-net/benches/group.rs and the pace.rs unit tests; no production local writer calls ready (nor did it before this PR). The whole-batch fill work moved to the flights quest, so I read this as deliberate API prep, but the local-writer shedding the PR describes only triggers once a writer waits on ready, so it may be worth naming the planned consumer in the quest before calling it landed.
Technical details
# Sized readiness consumer
## Affected sites
- `rs/moq-net/src/model/group.rs:2230` — `poll_ready(waiter, &frame::Info)`; the new 2-arg signature.
- `rs/moq-net/src/model/group.rs:2237` — `ready(&frame::Info)`.
- `rs/moq-net/src/model/pace.rs:2252` — `poll_paced(waiter, partial)` is the shared gate with ingest's `poll_admit`.
## Required outcome
- Either confirm the consumer is planned (flights / local writer) and note it where the quest marks sized readiness "landed", or wire the local writer in a follow-up.
## Open questions for the human
- Which local writer is expected to call `ready(&info)` first, and is it in scope for this PR or a later one?deepseek-v4.1-flash (free via Pullfrog for OSS) | 𝕏
Carries the waits onto main's sealed rustls crypto (#7028), freed receive state (#7528), and qmux's leased receive credit (#7624): - qmux's RecvStream leases and resets through moq-quic's poll_lease and poll_received_reset, with no readable map or driver kicks. - moq-tokio's and moq-uring's received_reset wait on poll_received_reset and keep main's Evicted result; moq-tokio's poll form holds the stream's park. - mux::Connection::set_limits wakes the transmit waiter, as QUIC's does. - Doc links to the deleted handshake events point at the waits. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Problem
Fastest-reader pacing paused ingest only on native moq-quic. Several gaps remained:
tcp://,tls://,unix://, WebRTC) read throughpoll::Buffered, which returns credit as it reads. Nothing bounded incomplete messages in aggregate (audit NET-3), and their streams never paused.ANNOUNCE_INIT. That forced 128 MiB client windows and 640 MiB peer windows.group::Producer::ready()ignored the size of the next frame, so a large local frame still shed what the fastest reader had yet to take.Approach
This implements the 2026-11-02 grill decisions below:
group::Producer::poll_ready(waiter, &frame::Info)/ready(&info)wait until that frame fits within 16 MB of the fastest reader. They use the samepoll_pacedgate as ingest's admission, so the whole-frame exception carries over: a reader at the edge never stalls a frame larger than the window.ANNOUNCE_INITis capped at 1 MiB less its 16 bytes of prefixes, soMIN_RECEIVE_WINDOWis 1 MiB. An oversized legacy set fails the announce withEncodeError::TooLargeand is refused on decode.[quic]defaults are now 4 MiB receive, 1 MiB stream and 4 MiB send in moq-tokio and moq-uring. They were 128 MiB, 4 MiB and 16 MiB.[limits.peer.quic]defaults are now 16 MiB receive and send, down from 640 MiB.ReceiveWindowTooSmall, the relay, uring) follow the new minimum.RecvStreamdrives moq-quic's mux lease API directly: credit is held until release, and a stuck window evicts the lowest-priority stream (Error::Evicted,CloseKind::Evicted). It also reportsreceive_windowand answerspoll_resetwithout reading.credit_stats, andset_limits. qmux gainedSession::set_limitsand refuses areceive_windowbelow the floor.Bufferedis now used only by iroh, which moq-dev/moq#7345 removes, and by test fakes.[quic]windows and stream count.[quic], so links the relay dials take[limits.peer.quic].server::Request::with_quic_limitsalso widens an accepted qmux session atok(), so peers over WebSocket or TCP get[limits.peer.quic].priority::RECV_PARKED(below every group band) until its group opens, and a lite FETCH answer does while it waits unread on TRACK_INFO, then ranks as the group it fills instead of as a control stream. An ietf data stream waiting on its alias does too. A full window then evicts them before control streams, so TRACK_INFO still arrives. They reserve nothing from the paused budget.poll_resetmeanwhile. moq-quic'srecv::Stream::received_resetnow fails withEvictedfor an evicted stream until it is stopped, carried through moq-tokio, moq-uring, and qmux, so an evicted parked or paused stream stops at once. Before, an evicted stream whose reader held leased bytes kept further eviction off for good, so a parked stream could hold the window stuck.[quic].Serverresolves[quic]once and gives it to itstcp://,tls://,unix://, and standalone WebSocket listeners and totls_acceptor.websocket::Options::quiccarries it forwebsocket::accept, which the relay fills from its[quic]throughWeb::with_quic. A WebRTC session keeps its fixed receive window atok(), since that window sizes its read queue.quest/m2/qmux-receive-lease.mdwithquest rmand folded its Goal and Plan in.Measured bandwidth-delay needs
moq-quic sim (
rs/moq-quictest harness, no congestion control, 10 Gbps link), 64 MiB through flow control (Mbps):moq-quic delivers about 80-90% of window/RTT. The old 16 MiB send window already capped the old 128 MiB client window.
Paused-stream budget at 4 MiB
Pacing still works at these windows. Fewer streams pause at once:
[quic]) pauses 8.Decisions
Settled in the 2026-11-02 /quest-grill interview, and recorded in the quest Plan:
quest/m2/qmux-receive-lease.md.ANNOUNCE_INITcapped at one 1 MiB segment, soMIN_RECEIVE_WINDOWis 1 MiB everywhere. Exactly 1 MiB needs the cap at 1 MiB minus 16 bytes of prefixes.frame::Info.poll_ready/ready.Settled by the maintainer on 2026-11-02, from the items this PR listed as needing a decision:
[limits.peer.quic], with the 1 MiB stream window from[quic].send_window[quic]windows into every accept path, including the relay's own WebSocket listener.Choices made without asking, for review:
received_reset/poll_reset, which the parked wait needs and which also lets a paused stream that gets evicted stop.ok()rather than widening towith_quic_limits.Impact
group::Producer::poll_ready/readytake&frame::Info.ANNOUNCE_INITcap andMIN_RECEIVE_WINDOWdrop to one segment.ANNOUNCE_INITpast about 1 MiB is refused on both ends, down from 64 MiB. Migrate: serve such an origin over lite-03+. No other format change: QMux MAX_DATA and STOP_SENDING only.ReceiveWindowTooSmallis at 1 MiB.[quic]windows and can fail streams as evicted.with_quic_limitsapplies to qmux.Error::QmuxLimits.[quic]is 4 / 1 / 4 MiB and[limits.peer.quic]is 16 / 16 MiB. Peers over WebSocket or TCP widen. Migrate: set the old values explicitly to keep them.RecvStreamleases natively, withRecvLease.Error::Evicted,LeaseTooLarge,ReceiveWindowTooSmall, andSession::set_limits.receive_windowbelow 1 MiB is refused.recv::Stream::received_resetfails withEvictedfor an evicted stream until it is stopped. Additivemux::Connection::{receive_window, stream_receive_window, credit_stats, set_limits}.transport::poll::RecvStream::poll_resetalso reports an eviction, which a leasing transport must surface. Newpriority::RECV_PARKED.websocket::Options::quic,tcp::Listener::with_quic,websocket::Listener::with_quic; driverreceived_resetfails withEvicted.[quic]; embedders buildingweb::Webpass it withWeb::with_quic.Alternatives
Grow-on-demand MAX_DATA for qmux was dropped: the one-segment floor makes a small static window cheap. Bounding
Bufferedwith a refusing cap gives no backpressure.Validation
just checkpasses on the pushed head (10216 tests), andjust test receive-creditpasses, including the qmux and uring Bun arms.announce_init_past_the_cap_failscovers the oversized legacy set.a_local_writer_waits_for_room_for_its_framecovers sized readiness.just test receive-credit(moq-tokio arm) passes, including the new Bun sender over WebSocket against a qmux receiver at the 1 MiB floor (receive_credit::javascript_partial_frame_gives_way_qmux).set_limitsand a reset seen through a lease.receive_credit::groups_waiting_for_track_info_give_way_first(moq-tokio, native QUIC, lite-07): GROUP streams for a SUBSCRIBE whose TRACK_INFO the publisher holds back fill the window beside an urgent track's partial frame. The waiting groups give way, the urgent frame completes, the TRACK_INFO arrives, and a waiting group is then read. It fails without the parked rank (the urgent frame is evicted) and without thepoll_resetwatch (the window stays stuck). 15 of 15 repeat runs passed.a_failed_parked_stream_withdraws_its_groupcovers the withdrawal, and moq-quic'sevicted_taken_bytes_keep_credit_until_stopnow checksreceived_resetbefore and after the stop.an_accepted_qmux_session_starts_at_the_configured_windows(atcp://session through aServerwith a 2 MiB / 256 KiB[quic]) andan_upgraded_session_starts_at_the_options_windows(websocket::accept, the relay's path) each fail without the change.Review
gpt-6.1-sol, medium effort, throughquest review --base origin/main, session01a19e03-2c8b-7ea2-b870-795b8b2b651c. It is a new session, since the base's instruction files changed after the previous session (01a19d92-5e87-7ee2-9636-f8734f346f47, rounds 1-4, design checkpoint at round 4, last head9bf847ca9fwith no findings).7b1b2627db2294a97792f17848c1f7ec7e719e37, round 3, no findings.968d07bb74) found one issue, now fixed:rs/moq-tokio/src/quic/web/recv.rs: the public reset wait panicked on an eviction. Once the driver reportedEvicted,RecvStream::received_reset(andweb_transport_trait::RecvStream::closed) reachedunreachable!. It now fails withReadError,ReadError::Evictedincluded, andstuck_window_evicts_the_lower_priorityasserts the driver'sreceived_resetreturnsEvicted.fda2c9b59c): no findings.mainwith one hand-resolved conflict inrs/moq-relay/MIGRATE.md, which kept both Unreleased sections: no findings.Follow-ups
(Written by Claude Opus 5.5)