Skip to content

fix(quic): close state lives in one phase per transport - #8208

Merged
kixelated merged 20 commits into
mainfrom
quest/m1/explicit-state/transport-close
Nov 8, 2026
Merged

kixelated merged 20 commits into
mainfrom
quest/m1/explicit-state/transport-close

Conversation

@kixelated

@kixelated kixelated commented Nov 7, 2026 •

Copy link
Copy Markdown
Collaborator

Finishes quest/m1/explicit-state/transport-close.md, a child of the explicit-state epic, and deletes it.

Problem

The Rust connection and QUIC transport code kept close state spread across flags and options beside each other, the epic's implicit-state shapes. That hid real bugs:

  • Trailing Disconnected. Once moq-tokio's reconnecting Connection gave up, status() yielded Disconnected before the terminal error: Shared::end wrote Disconnected in the same revision that set ended, contradicting poll_status's doc.
  • rebind after close. moq-tokio's QUIC Endpoint::rebind checked neither a lost driver nor a close, and refilled a released socket slot, binding a port nothing reads.
  • closed() disagreed across backends. For a local close, moq-tokio resolved closed() at once while moq-uring and qmux waited until the CONNECTION_CLOSE was staged. A moq-tokio caller that stopped its runtime the moment closed() resolved could leave the close unsent, so the peer waited for its idle timeout.
  • qmux leaked stream credit. A STOP_SENDING that beat finish() set fin, so the drop skipped the RESET_STREAM that RFC 9000 §3.5 requires. The peer never learned the final size and never returned the credit, so every such stream leaked a slot of the peer's MAX_STREAMS.
  • Relay double close. Serving::closed() moved a session from open to ending on every call, so a second call underflowed the drain tally.
  • PATH_CHALLENGE after close (found in review). moq-quic sent the PATH_CHALLENGE a migration owed the previous path before checking for a close, so a connection closed right after migrating sent the probe ahead of its CONNECTION_CLOSE. RFC 9000 §10.2.1 limits a closing endpoint to CONNECTION_CLOSE.

Approach

Each piece of close state now lives in one enum, and every bug above is fixed with a regression test.

  • moq-tokio reconnect loop (connection.rs): Lifecycle { Connecting { failure }, Live, Offline, Closed } replaces the status, ended flag, session, error, and failure. Status and Observation derive from it, so ending the loop goes straight to Closed. The draining predecessor is boxed (clippy's large_enum_variant).
  • moq-tokio QUIC endpoint: Lifecycle { Running, Closed { error_code, reason }, DriverLost } replaces driver_lost beside a close option, and every entry point checks it. rebind is crate-private, as the maintainer decided, and test-only.
  • moq-tokio QUIC connection: the driver latches the terminal error as moq-uring and qmux do. A local close publishes once its CONNECTION_CLOSE is on the socket, and a buffered transmit records whether it is the close. A close moq-quic never sent (the anti-amplification limit can withhold it) publishes once the connection drains, and a driver that stops publishes why. moq-net's lite revoke guards count a close this side sent as the session ending, rather than waiting for a backend to report it delivered.
  • moq-tokio WebTransport session: Mode { Raw { alpn }, Web(Arc<Web>) } replaces the WebTransport-only fields, kept as Options or empty values that were all set or all unset together.
  • moq-uring: End { Open, Closed, Dead } replaces the closed latch, the dead flag, the cached local close, and close_staged. Only the first publish wakes anyone. The WebTransport request and response move into Web, and the raw ALPN is read from the connection.
  • qmux: SendEnd { Open, Finished, Reset } and RecvEnd { Open, Reset(code), Stopped } replace the fin and reset flags. A stopped stream's drop echoes the stop's code in a RESET_STREAM, like moq-tokio and moq-uring. The session no longer caches a local close beside the connection, and the driver's Phase { Open, Flushing, Closing, Done } holds the close deadline that Timer.closing duplicated.
  • moq-quic QMux connection (mux): End { Open, Closing { error, frame }, Closed } replaces error, close, close_sent, and a borrowed QUIC State that claimed Established while handshaking. Streams take the one closed bit they read.
  • moq-quic connection:
    • Closed and Draining carry send_close, replacing a close: bool that every move into Draining inherited silently.
    • A path's challenge is Option<Challenge { token, pending }>.
    • A receive stream's Reader { Reading, Evicted, Stopped } replaces stopped and the lease's evicted.
    • fin_pending moves into SendState::DataSent.
    • transmit skips a pending PATH_CHALLENGE once closed, which fixes both drivers at once.
  • moq-relay: Count { Uncounted, Open, Ending(Instant) } replaces counted beside closed, so only the first close counts. The cluster's DialersPhase is an enum of its open state and its shutdown.

Impact

  • moq-tokio QUIC Connection::closed() / poll_closed: for a local close, these now resolve once the CONNECTION_CLOSE is on the socket rather than at once, matching moq-uring and qmux. A close held behind a blocked socket still resolves by the end of the closing period. Stream and datagram operations still fail as soon as close() is called.
  • moq-tokio Connection::status() / poll_status / state_changed, and moq-ffi's MoqConnectionState derived from them: a loop that gives up reports Closed and its error next, never a trailing Disconnected.
  • moq-tokio QUIC Endpoint: rebind is test-only. It was added after the last release, so just api-diff rust shows no break. A closed or driverless endpoint still refuses connect and accept, and a second close keeps the first close's code and reason.
  • qmux SendStream: dropping a stream the peer stopped now sends RESET_STREAM with the stop's code, which returns the peer's stream credit. finish() after a stop still returns StreamStop.
  • moq-relay: closing a session twice no longer corrupts the drain tally.
  • moq-quic: a connection closed right after migrating no longer sends the previous path's PATH_CHALLENGE.
  • API and wire: no published API break per just api-diff rust, and no wire format change.

Alternatives

  • Resolve closed() at once on every backend (moq-tokio's old behavior). Simpler, but a caller that exits on resolve can drop the CONNECTION_CLOSE, so the peer waits out its idle timeout.
  • Refuse rebind after close. The maintainer chose making it test-only instead (2026-11-07).
  • The quest's QMux Phase { Handshake, Established(Params), Closing, Closed }. The peer's parameters outlive the end: the record budget and max_datagram_size() read them during a close. Folding them into Established would change moq-net's datagram size mid-close.
  • The quest's endpoint Lifecycle { Running, Closing, Released, DriverLost }. The socket slot already records its own release, and with rebind test-only nothing refills it, so one Closed covers both closing and released.
  • Fix the PATH_CHALLENGE in each driver, for example by reading the encoded frames. moq-quic is where the RFC rule belongs, and one gate there fixes both drivers.

Choices made without asking

  • closed() latches on every backend: a local close resolves once the close is sent or staged. The quest asked to settle on one behavior, and this is the one that never drops the close.
  • rebind stays in this PR rather than splitting into an m2 quest. It is unreleased (absent from just api-diff rust), so it breaks no published surface. The epic's decision bullet now says so.
  • Fields kept beside the new enums, because they vary independently of the phase:
    • QMux: peer: Option<Params> beside End, and params_sent, which only records whether our own parameters went out.
    • moq-quic draining_reported: a server's first Initial can carry a CONNECTION_CLOSE and reach Draining without the report, so the state cannot stand in for it.
    • Send streams' stop_reason and released beside SendState: either can arrive in any of our states, and the stop outlives our reset so End::Stopped can report it.
    • A path's validated beside its challenge: a path is validated by the handshake or a PATH_RESPONSE, and stays validated after its challenge clears.
  • The reconnect enum is named Lifecycle, not the quest's Phase, since Phase is already the public observation enum.
  • BBR's congestion fields are untouched, as the quest preferred.
  • No benchmark run: the enums replace flags of the same size, and no per-stream or per-packet state grew.
  • The m5 quic-egress-profile quest's rebind mention is unchanged: it describes a fork prototype, not this API.

Caused by

  • Trailing Disconnected: moq-dev/moq#6487, whose Shared::end called state.offline() in the ending revision.
  • rebind refilling a released slot: moq-dev/moq#7045, which made the endpoint own a releasable socket slot. rebind itself dates from moq-dev/moq#5664.
  • closed() disagreement: moq-dev/moq#5664, which gave moq-uring a staged latch while moq-tokio resolved at once.
  • qmux credit leak: moq-dev/moq#6231, whose finish set fin on a stop.
  • Relay double close: moq-dev/moq#7858, which added Serving::closed.
  • PATH_CHALLENGE after close: inherited from the quinn-proto import in moq-dev/moq#4879.

Validation

  • just check origin/main passed at 4f0838c: 12750 tests, clippy, fmt, and the text and TOML checks. The one commit after it (8150dc6) only adds a test; moq-tokio's 579 tests and clippy pass on it.
  • New tests:
    • moq-tokio: a_stopped_loop_reports_its_error_next, closed_waits_for_the_close_to_be_sent, closed_waits_past_a_datagram_held_from_before_the_close (sim), and a_closed_or_driverless_endpoint_stays_closed.
    • moq-uring: an_end_published_again_wakes_nobody.
    • qmux: a_stopped_stream_resets_even_after_finish.
    • moq-relay: a_session_closed_twice_ends_once.
    • moq-quic: a_close_drops_a_pending_path_challenge.
  • Each test that pins a bug fails without its fix. a_closed_or_driverless_endpoint_stays_closed pins the quest's third goal through the public entry points.
  • Sol's local review rounds 1 and 2 found two regressions in this PR's own latch code (moq-uring's republish woke its driver through a whole drain; moq-tokio published closed() after sending a datagram held from before the close) and the PATH_CHALLENGE bug. All three are fixed, each with a test above that fails without its fix. Round 3 found nothing.
  • Unrelated flake: moq-uring quic::qlog::tests::an_open_trace_accounts_for_staging_and_close failed once while moq-tokio and moq-uring ran in parallel, then passed 5 of 5 alone. This PR does not touch it; see Follow-ups.

Follow-ups

  • De-flake an_open_trace_accounts_for_staging_and_close in rs/moq-uring/src/quic/qlog.rs. It spins a fixed 10,000 yield_now calls waiting for a background thread's memory accounting, so a loaded machine can run out of spins first. Recommendation: plan it (XS). Wait on the accounting itself instead of a spin count.

Review

Rewritten by quest review after each round.

  • Reviewer: Codex, gpt-6.1-sol, xhigh effort
  • Session: 01a1b8df-c564-74b1-9e01-e2d734b7d1c1, round 4
  • Head: 8150dc6
  • Base: origin/main at f8d30a1
  • Score: 10/10, The incremental change adds a focused regression test, and source inspection found no issues; builds and tests were not run.
  • Findings: none

Models

Size: L
Implementer: Claude Code, claude-opus-5-5, xhigh effort

(Written by Claude Opus 5.5 with xhigh)

🤖 Generated with Claude Code

kixelated and others added 15 commits November 7, 2026 14:38
The reconnect state keeps one Lifecycle enum instead of a status, an ended
flag, a session, an error, and a failure, so ending the loop no longer leaves
a trailing Disconnected for status() to report before the terminal error.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The endpoint keeps one Lifecycle (Running, Closed, DriverLost) instead of a
driver_lost flag beside a close option, and every entry point checks it.
Endpoint::rebind is test-only: after a close it bound a port nothing read.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
moq-tokio's driver latches the terminal error like moq-uring and qmux: a
local close resolves Connection::closed only once its CONNECTION_CLOSE is on
the socket, and a driver that stops publishes why. moq-net's lite revoke
guards count a close this side sent as the session ending, rather than
waiting for a backend to report its close delivered.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The closed latch, the dead flag, the cached local close, and the driver's
close_staged flag become one End enum (Open, Closed, Dead); a staged
CONNECTION_CLOSE publishes at once and the reason is read from moq-quic.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A STOP_SENDING that beat `finish` left `SendStream.fin` set, so the drop
skipped the RESET_STREAM that RFC 9000 section 3.5 requires. The peer
never learned the final size and never returned the stream's credit:
every such stream leaked one slot of the peer's MAX_STREAMS. The drop now
echoes a stop's code like moq-tokio and moq-uring do, and the send and
receive handles each keep one end enum instead of two flags.

The session no longer caches a local close beside the connection, which
already keeps its first reason: the driver publishes `conn.error()` once
the close is delivered. The close deadline lives in the driver's phase,
not a timer field beside it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The QMux connection kept its end in `error`, `close`, `close_sent`, and a
borrowed QUIC `State` it set to `Established` while still handshaking,
only so streams could ask whether it closed. `End { Open, Closing {
error, frame }, Closed }` replaces the four. The peer's parameters stay
an `Option` beside it: they outlive the end, which still reads them for
the record budget and the datagram size.

Streams now take the one bit they read, `closed`, rather than a
reference to a connection state.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`Serving` kept `counted` beside `closed`, and `closed()` moved the session
from open to ending on every call, so a second call underflowed the open
tally. One `Count { Uncounted, Open, Ending(Instant) }` replaces both, and
only the first close counts.

The cluster's `DialersPhase` kept its open state and its shutdown in two
options, exactly one of them set; it is now an enum of the two.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
moq-tokio's WebTransport `Session` kept the session ID, the accept state,
the stream and datagram headers, the settings, the CONNECT stream, the
request, and the response as `Option`s or empty values, all set for
WebTransport and all unset for raw QUIC, with the raw ALPN beside them.
`Mode { Raw { alpn }, Web(Arc<Web>) }` replaces them.

moq-uring already grouped most of it in `Web`; the request and response
move in too, and the raw ALPN is read from the connection, which keeps it.

The web `poll_closed` doc no longer contrasts with `Connection::closed`,
which now also waits for a local close to be sent.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…together

`PathData` kept `challenge: Option<u64>` beside `challenge_pending`, and
sending a previous path's challenge expected the token whenever the bit
was set. `Option<Challenge { token, pending }>` makes a pending packet
without a token unrepresentable.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The connection kept `close: bool` beside its `State`, meaningful only in
`Closed` and `Draining`, so every move into `Draining` inherited it
silently. `Closed` and `Draining` now carry `send_close`, and each move
says whether a close is due: the reply to a peer's close, the one left
over from closing, or none.

`draining_reported` stays: a server's first Initial can carry a
CONNECTION_CLOSE and reach `Draining` without the report, so the state
cannot stand in for it. Its doc now says so.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A receive stream kept `stopped` on the stream and `evicted` on its lease,
an eviction setting both. One `Reader { Reading, Evicted, Stopped }` on
the stream replaces them, and the lease's credit accounting takes the
eviction from the caller.

A send stream kept `fin_pending` beside `SendState`, set only while
`DataSent`; it is now a field of `DataSent`. The peer's `stop_reason` and
the application's `released` stay beside the state: either can come in
any of our states, and the stop outlives our reset to report
`End::Stopped`.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
# Conflicts:
#	rs/moq-quic/src/fuzz.rs
kixelated and others added 2 commits November 7, 2026 17:45
Clippy's large_enum_variant: `Offline` held the predecessor session inline.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
kixelated and others added 3 commits November 7, 2026 18:09
moq-tokio inferred that a sent datagram was the CONNECTION_CLOSE from the
connection's error, so retrying a datagram the socket held from before a local
close resolved closed() while the close itself was still unsent. The buffered
transmit now records whether it carries the close.

moq-uring's publish woke the driver even for an end already published, and the
driver republishes on every poll, so a remote close spun it through the drain.
Only the first publish wakes anyone now.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
transmit sent the PATH_CHALLENGE a migration owed the previous path before it
checked for a close, so a connection closed right after migrating sent the
probe first. RFC 9000 §10.2.1 limits a closing endpoint to CONNECTION_CLOSE,
and the drivers mark a transmit made after close as the close itself, so
moq-tokio and moq-uring resolved closed() on the probe while the close could
still sit unsent behind a blocked socket.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Pins the quest's goal through the public entry points: once closed, or
once its driver is gone, an endpoint neither dials nor accepts again.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated kixelated changed the title Transport close fix(quic): close state lives in one phase per transport Nov 8, 2026
@kixelated
kixelated marked this pull request as ready for review November 8, 2026 02:47
@coderabbitai

coderabbitai Bot commented Nov 8, 2026

Copy link
Copy Markdown
Contributor

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 26 seconds.

Check out review usage here.

View limit details

Limit details: You’ve used all 4 included reviews currently available.

Learn how review limits work.

Review configuration:

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 2daae5eb-f3bc-43d4-b782-f12835c39237
Commits

Reviewing files that changed from the base of the PR and between 4a86602 and 8150dc6.

Files selected for processing: 30

  • quest/m1/explicit-state/README.md
  • quest/m1/explicit-state/transport-close.md
  • rs/moq-net/src/lite/subscriber.rs
  • rs/moq-net/src/session.rs
  • rs/moq-net/src/transport/boxed.rs
  • rs/moq-quic/src/connection/mod.rs
  • rs/moq-quic/src/connection/mux/mod.rs
  • rs/moq-quic/src/connection/paths.rs
  • rs/moq-quic/src/connection/streams/lease.rs
  • rs/moq-quic/src/connection/streams/mod.rs
  • rs/moq-quic/src/connection/streams/recv.rs
  • rs/moq-quic/src/connection/streams/saturated.rs
  • rs/moq-quic/src/connection/streams/send.rs
  • rs/moq-quic/src/connection/streams/state.rs
  • rs/moq-quic/src/fuzz.rs
  • rs/moq-quic/src/tests/mod.rs
  • rs/moq-relay/src/cluster.rs
  • rs/moq-relay/src/shutdown.rs
  • rs/moq-tokio/src/connection.rs
  • rs/moq-tokio/src/quic/driver/connection.rs
  • rs/moq-tokio/src/quic/driver/endpoint.rs
  • rs/moq-tokio/src/quic/driver/mod.rs
  • rs/moq-tokio/src/quic/driver/runtime/mod.rs
  • rs/moq-tokio/src/quic/driver/tests.rs
  • rs/moq-tokio/src/quic/web/session.rs
  • rs/moq-uring/src/quic/driver/connection.rs
  • rs/moq-uring/src/quic/web.rs
  • rs/qmux/src/driver.rs
  • rs/qmux/src/session.rs
  • rs/qmux/src/session/tests.rs
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@kixelated kixelated left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated review by review (OpenAI)

Reviewed head 8150dc6 against base 4a86602.

No new actionable findings from static review of the lifecycle transitions, delayed local-close publication, buffered-transmit tagging, stopped-stream reset/credit handling, and relay close accounting. The added regression tests cover the identified close-ordering and double-close cases.

Verification limits: GitHub source/diff review only; I did not run builds, tests, simulations, or benchmarks. The PR reports an unrelated qlog accounting-test flake, which this review does not independently resolve or verify.

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ No new issues found.

Reviewed changes

  • Reconnect loop (rs/moq-tokio/src/connection.rs): Lifecycle { Connecting, Live, Offline, Closed } replaces the status/ended/error/session/version/transport/predecessor cluster, so a loop that gives up publishes Closed and its error without a trailing Disconnected; Status and Observation now derive from it.
  • QUIC endpoint (rs/moq-tokio/src/quic/driver/endpoint.rs): Lifecycle { Running, Closed, DriverLost } gates accept, connect, rebind, and close_now; rebind and its Slot/event plumbing are test-only, which is where the released-port bug came from.
  • QUIC connection close latch (rs/moq-tokio/src/quic/driver/connection.rs): a driver-published closed latch resolves poll_closed only once the CONNECTION_CLOSE is on the socket (or the drain ends for a withheld close), publishes why a stopped driver ended, and keeps a buffered pre-close datagram from counting as the close.
  • moq-quic connection and streams: Closed/Draining carry send_close, a path's challenge carries token/pending, a receive stream has one Reader phase, and transmit drops a pending PATH_CHALLENGE once closing (RFC 9000 §10.2.1), fixing both drivers at the source.
  • QMux mux connection and rs/qmux: End replaces error/close/close_sent/the borrowed ConnectionState; SendEnd/RecvEnd replace the FIN/reset flags, and dropping a peer-stopped stream now echoes its code in RESET_STREAM so the peer returns the credit; the driver's Phase absorbs Timer.closing.
  • moq-uring, moq-tokio web sessions, moq-relay, moq-net: End { Open, Closed, Dead }; one Mode for raw vs WebTransport; Count so a second close no longer underflows the drain tally; and poll_ending so a local close this side sent reads as ending before a backend reports delivery.
  • Quest: transport-close.md is deleted, its ## Required line is gone from the epic README, and the rebind decision now records test-only as done (consistent with just api-diff rust).

I traced the close paths across the three backends (sent vs staged vs drained), the evicted/stopped lease accounting, the QMux stop-then-finish credit path, and the migration/predecessor transitions; the new tests pin each fixed bug, and the mocked-clock/real-socket split matches REVIEW.md.

Pullfrog  | View workflow run | Using deepseek-v4.1-flash-1017 (free via Pullfrog for OSS) | 𝕏

@kixelated
kixelated added this pull request to the merge queue Nov 8, 2026
kixelated added a commit that referenced this pull request Nov 8, 2026
Follow-ups of #8208 and #8226.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Merged via the queue into main with commit 08a1f8a Nov 8, 2026
13 checks passed
@kixelated
kixelated deleted the quest/m1/explicit-state/transport-close branch November 8, 2026 05:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant