Repository navigation
quest: plan root-cause redesigns for the invariant audit findings #8151
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
4a64513
db5c734
915a613
46276c9
857c00d
147f032
e129768
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,16 +1,15 @@ | ||
| # [S] Peer and config values never panic or wrap moq-net arithmetic | ||
| # [M] kio's Instant has no panicking arithmetic, and peer values never wrap | ||
|
|
||
| ## Goal | ||
|
|
||
| No value a peer sends or an application configures panics a moq-net task or silently wraps: | ||
| No value a peer sends or an application configures can panic a moq task or silently wrap: | ||
|
|
||
| - `rs/moq-net/src/ietf/publisher.rs` `refusal`: a REQUEST_ERROR Retry Interval near `u64::MAX` panics through kio's `Instant + Duration`. Any peer we send PUBLISH_NAMESPACE to can trigger it. | ||
| - `rs/moq-net/src/ietf/publisher.rs` `save_join`: `largest.object + 1` on an upstream's LARGEST_OBJECT of `u64::MAX`. | ||
| - `rs/moq-net/src/reconnect.rs`: `now + duration` and `mul_f64` on `Goaway.handover`, `Config::timeout` and `Backoff` panic for huge configured values once a GOAWAY arrives; `Server::deadline` already treats overflow as no deadline. | ||
| - `rs/moq-net/src/lite/subscriber.rs` and `subscriber/fills.rs`: SUBSCRIBE_DROP's inclusive end `u64::MAX` becomes `..u64::MAX` through `saturating_add(1)`, so the drop misses group `u64::MAX`. | ||
| - `rs/moq-net/src/lite/publisher.rs` `DatagramSend::send`: a sequence of 2^62 or more relayed from lite-07 to a lite-05/06 subscriber fails to encode and is dropped silently, and the subscription ends in a reset instead of a finish. | ||
| - `rs/moq-net/src/route/mod.rs` `requests`: `fd[&id].0 + 1` on an advertised seqno `u64::MAX` (test-only today; becomes reachable when lite-07 drives the router). | ||
| - `kio::clock::Instant` drops `Add`, `AddAssign`, `Sub` and `SubAssign` with `Duration` (`rs/kio/src/clock/instant.rs`), which panic past the clock's range. Every call site picks `checked_add`/`checked_sub` with an explicit meaning for overflow (no deadline, refused) or a saturating form. This removes the class behind, among others, the IETF publisher's REQUEST_ERROR retry interval (`rs/moq-net/src/ietf/publisher.rs` `refusal`, which any peer can panic) and `rs/moq-net/src/reconnect.rs` deadlines from huge configured durations. | ||
| - Peer-supplied counters never wrap with `+ 1`: the ietf `save_join` `largest.object + 1`, the lite SUBSCRIBE_DROP end `u64::MAX` (`saturating_add(1)` that misses that group in `lite/subscriber.rs` and `subscriber/fills.rs`), and the router's advertised seqno in `route/mod.rs`. | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
|
||
| - A datagram sequence of 2^62 or more relayed to a lite-05/06 subscriber is handled explicitly in `lite/publisher.rs` `DatagramSend::send` instead of dropped silently. | ||
|
|
||
| ## Plan | ||
|
|
||
| Found by the 2026-11-07 invariant audit (ietf-1, ietf-2, lite-3, lite-4, lite-9, model-3). Each site gets `checked_add` with an explicit meaning for overflow (no deadline, unsupported, refused) and a test at the extreme value. Prefer refusing the impossible value at decode or at `Reconnect::new` where that is simpler than handling it downstream. | ||
| Found by the 2026-11-07 invariant audit (ietf-1, ietf-2, lite-3, lite-4, lite-9, model-3). | ||
|
|
||
| Decided 2026-11-07 (maintainer, invariant-audit root causes): remove the operators rather than make them saturate, so each of the roughly 600 sites states what overflow means; most edits are mechanical. Breaking for the published `kio` crate; it stays in m0 because a remote peer can panic the publisher today. Removing `Sub` alongside `Add` is the same class (underflow before the clock's start). Each peer-reachable site gets a test at the extreme value. | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,19 +1,21 @@ | ||
| # [S] JS qmux keeps credit and final sizes consistent with Rust | ||
| # [M] JS qmux streams follow QUIC's per-stream state machine | ||
|
|
||
| ## Goal | ||
|
|
||
| A JS qmux session (the WebSocket fallback) never stalls or gets closed by a Rust peer because of its own flow-control bookkeeping, and a peer cannot grow its memory without bound: | ||
| `js/qmux` tracks each stream's send and receive halves through the states of [RFC 9000 §3](https://www.rfc-editor.org/rfc/rfc9000#section-3), as `rs/moq-quic/src/connection/streams/state.rs` does, instead of credit bookkeeping spread across `session.ts` handlers. A JS qmux session (the WebSocket fallback) then cannot leak or double-count credit by construction: | ||
|
|
||
| - Bytes that arrive on a stream we already stopped (late STREAM frames, and the final-size gap of a later RESET_STREAM) count toward connection receive credit. Today `js/qmux/src/session.ts` (`#onStream`, `#onResetStream`) returns early once the receive half is deleted on cancel, so each cancelled stream permanently shrinks the window until MAX_DATA is never sent again. | ||
| - A RESET_STREAM's final size is at least every byte already handed to the socket. Today `sendOffset` advances only after `scheduler.enqueueStream` resolves, so a STOP_SENDING handled while a frame is mid-`sink.write` replies with a short final size, which Rust's `rs/moq-quic/src/connection/streams/recv.rs` refuses with FINAL_SIZE_ERROR. | ||
| - MAX_DATA, MAX_STREAMS_{BIDI,UNI} and MAX_STREAM_DATA (per stream) replace any queued copy, as Rust does with its pending flags, instead of each taking a new entry on the uncapped control lane. | ||
| - An empty binary WebSocket message is a no-op on every qmux version, matching `rs/qmux/src/transport/websocket.rs`; qmux-00 currently throws. | ||
| - WebSocket messages larger than the record size are refused before they are buffered where the platform allows it: Rust `rs/qmux/src/ws.rs` `Upgraded` validates or sets the tungstenite `max_message_size` itself instead of trusting the caller, and JS documents the `ws` `maxPayload` it needs. | ||
| - A stream's flow record lives until both halves are terminal on the wire (all data or a reset received and consumed; FIN or RESET sent), not until the app releases its handle. Today `#maybeDeleteStreamFlow` drops it on cancel, so late STREAM bytes and a later RESET's final-size gap are never counted and the connection window shrinks until MAX_DATA stops (an honest Rust peer stalls the session). | ||
| - Connection and stream credit change only inside state transitions, so no handler can return early past the accounting (today twelve call sites do it by hand). | ||
| - The send half's offset counts bytes when the scheduler takes them, so a RESET_STREAM's final size covers a frame mid-`sink.write`. Today it does not, and Rust closes the connection with FINAL_SIZE_ERROR. | ||
| - Window updates (MAX_DATA, MAX_STREAMS_{BIDI,UNI}, MAX_STREAM_DATA) are pending flags or per-stream latest values drained by the scheduler, as Rust's `pending` flags are, not entries on the uncapped control lane. | ||
| - An empty binary WebSocket message is a no-op on every version, and messages over the record size are refused before buffering where the platform allows it (Rust `rs/qmux/src/ws.rs` `Upgraded` validates its own `max_message_size`; JS documents `ws` `maxPayload`). | ||
|
|
||
| ## Plan | ||
|
|
||
| Found by the 2026-11-07 invariant audit (qmux-1 to qmux-5). qmux-1 and qmux-2 are the urgent ones: an honest Rust relay triggers both whenever a JS app cancels group streams over WebSocket. | ||
| Found by the 2026-11-07 invariant audit (qmux-1 to qmux-5); qmux-1 and qmux-2 are reproduced as tests before the redesign. | ||
|
|
||
| - Keep each stopped peer stream's receive offset (and final size once known) until FIN or RESET so late bytes are accounted, and port Rust's `credit_returns_for_unread_and_stopped_streams` (`rs/moq-quic/src/connection/mux/tests.rs`). | ||
| - Count bytes as sent when the scheduler takes the frame from its queue, or track the in-flight frame per stream and include it in the RESET final size. Test with a sink whose `write` is held open while STOP_SENDING arrives. | ||
| - Send the MAX_* frames through `replaceControl` keyed by type (and stream id for MAX_STREAM_DATA); fix the "bounded" claim in `scheduler.ts`. | ||
| Decided 2026-11-07 (maintainer, invariant-audit root causes): fix the cause, the hand-rolled flow accounting, rather than patching each early return. Mirror Rust's state shapes and port its credit tests (`credit_returns_for_unread_and_stopped_streams` in `rs/moq-quic/src/connection/mux/tests.rs` among them) so both implementations pin the same invariants. A single implementation for both languages is the longer-term route, planned in [Evaluate WASM-to-JS for @moq/net](/quest/m5/rs2ts-wasm2js.md); this does not wait for it. | ||
|
|
||
| ## Related | ||
|
|
||
| - [js/qmux close](/quest/m1/explicit-state/qmux-close.md) - the session close phase and `recv.ts` stream-end sweep; land the per-stream state machine so both share one stream state |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,15 +1,16 @@ | ||
| # [S] Cluster peers have one identity each and fresh inputs | ||
| # [S] A cluster peer's identity is a type, not a string | ||
|
|
||
| ## Goal | ||
|
|
||
| A relay mesh fails loud on identity mistakes instead of silently partitioning or looping, and its peer inputs stay current and bounded (`rs/moq-relay/src/cluster.rs`, `relay.rs`): | ||
| Relay mesh identity mistakes are unrepresentable or fail loud (`rs/moq-relay/src/cluster.rs`, `session.rs`): | ||
|
|
||
| - A peer session whose declared hop (`Request::peer_hop`) equals our `cluster.id` is refused with an error naming the duplicate, instead of every route from it being dropped as reflected. | ||
| - One relay has one dial key: `DialTarget` keys on `canonicalize_peer_key` (which also strips URL userinfo, so credentials stop reaching logs and the `remote` span), and the self filter applies to static `cluster.connect` entries too. A test covers the self filter. | ||
| - `--cluster-lan` on a relay without a QUIC listener is refused at startup instead of advertising SRV port 0. | ||
| - A `PeerKey` type, constructible only through `canonicalize_peer_key`, keys the dial map, the self filter, LAN discovery and `/nodes`. Today `DialTarget` keys on the raw URL with only the query cleared, so one relay can get two links, a node can dial itself, and URL userinfo reaches logs and the `remote` span. The self filter applies to static `cluster.connect` entries too, and a test covers it. | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
|
||
| - A peer session whose declared hop (`Request::peer_hop`) equals our `cluster.id` is refused with an error naming the duplicate, instead of every route from it being dropped as reflected and the mesh silently partitioning. | ||
|
Comment on lines
+7
to
+8
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
|
||
| - The `cluster.token` file is re-read on redial (or watched like the API file), so a rotated or expiring token does not break every redial until restart. | ||
| - The `cluster.api.url` body and file are read with a size cap; an oversized answer keeps the last good peer list. | ||
| - A `session::Registry` registration removes only its own entry, so a reused `Request` id cannot unlist another live session. | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
|
||
|
|
||
| ## Plan | ||
|
|
||
| Found by the 2026-11-07 invariant audit (relay-2, relay-4, relay-5, relay-7, relay-9, relay-10). | ||
| Found by the 2026-11-07 invariant audit (relay-2, relay-4, relay-5, relay-9, relay-13). | ||
|
|
||
| Decided 2026-11-07 (maintainer, invariant-audit root causes): make the canonical key a type so no code path can key on a raw URL again. The audit's suggestion of a byte cap on the `cluster.api.url` body is dropped: that endpoint is operator-trusted, and a constant cap is not a design bound. | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Script executed:
Repository: moq-dev/moq
Length of output: 16658
Require a bound on distinct keys.
A peer can introduce new identifiers indefinitely while sending multiple updates for each. Keeping only the latest state per key prevents per-event growth but does not bound the map. Require an independent key-cardinality bound or removal mechanism.
Script executed:
Repository: moq-dev/moq
Length of output: 17615
Script executed:
Repository: moq-dev/moq
Length of output: 35020
Script executed:
Repository: moq-dev/moq
Length of output: 38235
Script executed:
Repository: moq-dev/moq
Length of output: 30790
Script executed:
Repository: moq-dev/moq
Length of output: 27208
Require a bound that survives credit replenishment.
Spent credit can increase indefinitely as the peer processes new cycles. A map can retain state from each cycle even after that credit returns. Require a bounded key set, non-replenishing credit, or credit held until the corresponding state is removed.
Script executed:
Repository: moq-dev/moq
Length of output: 16498
Script executed:
Repository: moq-dev/moq
Length of output: 27749
Script executed:
Repository: moq-dev/moq
Length of output: 19677
Allow credit-bounded per-event queues.
An incoming-stream queue can hold one entry per stream while each stream consumes unreplenished credit. Replacing entries with the latest state would lose streams. Limit the prohibition to unbounded per-event retention.
Suggested fix
Committable suggestion