Skip to content

feat(net)!: lite-07 routes through the route layer - #8023

Merged
kixelated merged 55 commits into
mainfrom
quest/m2/cluster-routing/routes/session
Nov 7, 2026
Merged

kixelated merged 55 commits into
mainfrom
quest/m2/cluster-routing/routes/session

Conversation

@kixelated

@kixelated kixelated commented Nov 6, 2026 •

Copy link
Copy Markdown
Collaborator

Claims quest/m2/cluster-routing/routes/session.md.

Summary

moq-lite-07-wip (opt-in in both Rust and JS) now runs on the route layer instead of hop chains.

  • One router per origin. The first lite-07 session on an origin creates its route::Node with a random routing identity. The node is fed the origin's own claims: everything in the table the route layer did not select, meaning local publishers and older-version peers. Every lite-07 announce stream on that origin is a neighbor key on the node, so an interest change is just another stream, and the old one keeps its routes until the new one answers.
  • Wire. ANNOUNCE_START and ANNOUNCE_TAKEOVER name a Route ID, ANNOUNCE_UPDATE carries only the Announce Cost, and ANNOUNCE_RESTART carries no hops. ROUTE_START/UPDATE/HOLD/END, ANNOUNCE_FLOOR, ROUTE_REQUEST and the ROUTE_END echo are live. ANNOUNCE_OK has no body, SETUP has no Hop parameter, a Cost of 0 is refused, and SUBSCRIBE, FETCH, TRACK, TRACKS and DATAGRAMS carry the Node ID.
  • Takeovers ride the route layer. A route-layer claim records the claim of its identity it took over and the instance that one served (Claim::continues). The origin's own takeovers feed it, and a relay keeps it from the ANNOUNCE_TAKEOVER it received. A stream sends the claim as a takeover only when the peer held the predecessor, under the same route, before the batch, and every claim the peer holds at that path serves the same instance; otherwise it is a start or a restart, as the draft says. The receiver forwards the claim with its predecessor's route, and in place keeps the predecessor's sources.
  • Floors are whole-interest markers (decided 2026-11-05). The router presents each route's view at the stream's interest and certifies only what it knows across that interest at one revision.
  • Origin selection. The first hop keeps route_order's shape over the origin nodes announcing the path and names the winner on the request. Every later hop follows its own route to that node. Pool ties hash the path with the node's identity.
  • Scoped peers. A scoped peer's identities are qualified under its route key: the key it proved in SETUP, or a per-session key when it proved none. A permit change that crosses the scope boundary re-files the peer's routes under the new key before the old key lets go (decided 2026-10-15). The node holds only the peer's claims the session's limit admits, not every claim within the limit's regions, and a permit change that admits a different set files the stream again.
  • Bounded per-stream state. Between floors a received route keeps only its net change to the committed claims. A stream whose reader stalls holds one entry per route it holds: route changes and batches coalesce per identity, and a route or claim the reader was never sent leaves nothing when it ends. ROUTE_REQUESTs coalesce per identity too, the highest seqno winning, and a stream's replies reach its writer only once the last ones flushed, so a peer leaving them unread holds one request per identity it advertised.
  • Mounts. A lite-07 answer reads each of the handle's mounts beneath its prefix through a node stream of its own, at the target, and names what it presents under the mount point, as in process and on lite-06.
  • lite-06 translation at the relay. Older-version routes enter the route layer as the relay's own claims. lite-07 routes reach a lite-06 peer as a hop list that starts with their origin node's hop. A lite-06 chain carrying this node's own routing identity is dropped as a reflection. Without that check, a claim that left over lite-07 and came back over lite-06 kept itself alive after its publisher left (mixed_version_ring_withdraw_retracts).
  • Proxying is opt-in. On lite-07 a session sends its peer only the routes to its origin's own claims unless the handle is marked with_proxy(true). moq-relay marks every session.
  • cluster.id / --cluster-id removed (decided 2026-10-13).
  • Native JS lite-07 lanes are skipped until routes/js.md ports js/net, since it still speaks lite-07 from before the route layer. Each skip names that quest, which restores them.

Two pre-existing races surfaced and are fixed at the root:

  • A track that already ended now keeps its copy when its front ends, so a fetch still reads what it cached. The routed restart arrives later than the old one did, which exposed this in moq-hls finished_first; the fix comes with a deterministic finished_settled test.
  • Only a grant this side issued scopes a lite-07 peer. The peer's grant to us no longer qualifies its identities, which broke socket_auth.

Decisions

Settled by the maintainer on 2026-11-06 in /quest-complete:

  1. Interop: the in-tree lanes running JS on lite-07-wip speak the old wire and fail against this until routes/js.md lands. How should they be handled?
    • ✅ Skip the JS lite-07 lanes until js.md lands, so the nightly report stays meaningful, with each skip naming js.md so it is removed there.
    • Leave them red until then.
    • Hold this PR until js.md is ready.
  2. Takeover: announce_takeover.rs's two lite-07 expectations had become a restart at the claim's own path and a start beneath it. Port ANNOUNCE_TAKEOVER onto Route IDs here, or accept the changed expectations?
    • ✅ Port ANNOUNCE_TAKEOVER onto Route IDs in this PR, so both stay takeovers. This follows the routes epic's decision 8 (2026-11-05: it keeps its shape, a Route ID in place of hops, the predecessor under the same route). announce-takeover still deletes the message later.
    • Accept the changed expectations until announce-takeover removes the message.
  3. Contract: amend the handoff contract's "a sender that certifies anything for a route answers every stream" (routes/README.md), or keep it?
    • ✅ Amend it: send a route on a stream only while it meets the stream's interest, with a floor certifying the last withdrawal before the route ends.
    • Keep "a sender that certifies anything for a route answers every stream".

Settled by the maintainer on 2026-11-07 in /quest-complete, after the merge queue dropped this PR because a default moq-net build failed (route/node.rs called the test-only Hop::new):

  1. Fix here, or land moq-dev/moq#8063's fix first?
    • ✅ Port 8063's fix (8cc983e949) onto this PR and requeue it. This PR still lands first, then 8063 (decided 2026-11-06).

The earlier body cited a "2026-11-06 decision" to carry ANNOUNCE_TAKEOVER only as far as compiling. No such decision is recorded on main or this branch, so that claim is dropped; decision 2 replaces it. Decisions 2 and 3 are recorded in the routes README's Plan (items 8 and 15), decision 1 in routes/js.md's Plan, and session.md now plans the scoping work decision 3 asks for.

Settled by the maintainer on 2026-11-07 in /quest-complete, after moq-dev/moq#8063's review found that a downstream peer raising its ROUTE_REQUEST seqno through a stream whose peer leaves replies unread grew relay memory without bound (recorded in this comment):

  1. Where to fix the ROUTE_REQUEST backlog?

Choices made without asking

  • Requests coalesce in the node, apart from the ordered outputs. Each stream's outbox keeps one ROUTE_REQUEST per identity (Stream::poll_requests), and the session takes them only once its writer has flushed. Its serves keep flowing meanwhile, so a peer that stops reading replies stalls only its replies. The identities are those the peer advertised on the stream, which the announce cap bounds. node::Output::Request is gone.

  • A lite-07 answer takes one reply and one output per pass, on a cooperative budget. Round 1 of this iteration's review found a peer flooding ROUTE_REQUEST starved the answer's route outputs and held the task.

  • Serve a claim through every feasible route that can carry it, not only the selected one. Losing the next hop to the same origin node then continues with nothing announced (next_hop_lost_lite07). Epochless wires still restart onto the minted instance (next_hop_lost_restarts_older_lite, next_hop_lost_restarts_moq_transport), as the 2026-10-27 decision asked. A route carries a claim when the claim lies within what this node asked that neighbor for and within its scope, since a route reaches an origin node, not each of its claims.

  • Proxy preference lives on the origin handle (origin::Producer::with_proxy, origin::Consumer::with_proxy), next to with_preference and with_link. Each connection already gets its own handle, so this is the per-connection preference the plan asked for. The library defaults to off and moq-relay turns it on.

  • No hop chain on lite-07. RouteInfo::hops() on a lite-07 route names only its origin node, and max_hops does not bound lite-07. Feasibility keeps it loop-free.

  • The route key waits for SETUP only when the peer is scoped. Unscoped identities file as they arrive.

  • Regions come from the scope this side issued, not the union of both grants.

  • A limit narrower than its regions certifies nothing. The node holds only the claims the limit admits, so a floor would certify the ones it left out absent. The stream then forwards no floor revision, as the publisher side already does for a grant that leaves claims out, and the admitted claims stay only while the peer carries them.

  • What survives of the old origin model: the route trie, fronts and route_order survive. route_order gains the node key, and Preference still ranks feasible transit routes.

  • A compression mismatch is refused one hop further downstream. D holds a route to each origin node, so on failover D refuses the replica that frames the track differently with Unsupported. Before, R refused it and the error reached D as INTERNAL.

  • The lite-06 guard checks only this node's own identity. Dropping every identity the router knows would strand a route: a lite-06 subscriber never reconsiders one it declined, even once the lite-07 path fails.

  • A takeover's lineage is claim state, kept while the instance stays. A re-price or a newer epoch keeps it, another instance drops it. Views of one identity can disagree only on whether a hop received the takeover; that changes no wire message, since a takeover needs the peer to hold the predecessor at the instance it continued.

  • A snapshot replays no takeover: a claim takes over only a predecessor the peer held before the batch, as a late announce cursor did before. A stalled reader's coalesced batch is a snapshot in this sense.

  • The node acks a route end its reader never read. When a route starts, or restarts, and ends before its stream's reader reads it, the outbox drops it and echoes the ROUTE_END to the router itself, since no peer holds the route.

  • Mounts merge onto the same routes. Every node stream of one answer shares a Route ID per origin node. The first stream's reach is the peer's, the route ends on the wire once no stream carries it, and a stream that stops early ends the claims it presented and is acked at once. A takeover's lineage crosses a mount only when the predecessor has a name under the mount point; otherwise the claim is a start.

  • The legacy publisher and subscriber takeover paths are gone. lite-07 always routes, so they were unreachable; the in-place test (a_takeover_in_place_keeps_the_predecessors_sources) moved to the routed subscriber.

  • How the JS lanes skip. Native JS dials its defaults in the matrix rather than skipping its media and finish cells, so only its grant check is lost. Claims, track replacement, sockets, and the lite-07 compress and drop legs record a skip. The compressed catalog runs without its lite-07 reader. --versions moq-lite-07-wip with JS clients is refused at setup, like --sockets with --versions. lite_tracks_interop leaves just test interop, and max_age_relay_javascript and relay_fill_javascript drop their lite-07 legs. bare_fin_interop and owed_tail_death_interop pass on lite-07 here, so bare-fin.ts is not skipped.

  • A limit change always files the stream again. Comparing only regions and admitted claims missed changes in what the floors certify, and in which claims another session of the same peer may serve here. Permits that change only the grant leave the stream alone.

  • Echoes are bounded by the announce cap, not by read backpressure. An ended route keeps its slot until its ROUTE_END echo is flushed. Stopping reads until replies flush could deadlock, since the answering side stops reading replies while its own writes are blocked; only the writes of replies wait for a flush.

  • A lite-06-learned own claim goes only to a peer this node proxies for. A route published here has an empty hop list, and every route learned over an older version has a hop, so the claim's route marks it. An answer without proxying leaves it out and certifies no floor for it, so an application is never transit between versions.

  • An upstream-learned own claim is one whose instance no route outside an upstream link serves at that path. Such a claim is left out of an upstream answer, and the floors of that route certify nothing there, as the grant filter does.

  • Scoping stays in session.md. Decision 3 settles the contract; this PR keeps the router answering every stream, and session.md implements item 15 with incremental presentation, which reshapes the same per-stream code.

Impact

Breaking, all behind the opt-in lite-07-wip ALPN unless noted:

  • The lite-07-wip wire changes as listed above. ANNOUNCE_TAKEOVER (0x5) is sent on lite-07, with a Route ID and a predecessor under the same route, as the draft specifies.
  • A library relay must call with_proxy(true), or its lite-07 peers learn only its own claims.
  • Client::with_cost(0) fails the session.
  • cluster.id and --cluster-id are removed from moq-relay and moq, and a zero-cost cluster link is refused. This applies to every version.

This round's fixes change no public API and no wire message.

Added: origin::Producer::with_proxy and origin::Consumer::with_proxy. doc/bin/relay/cluster.md and doc/lib/rs/moq-net.md cover this.

Migration notes for the release cut (main now leaves MIGRATE.md to the cut, so this PR no longer edits it):

moq-net: moq-lite-07 routes through the route layer. moq-lite-07-wip (opt-in) replaces the hop list with routes: each origin node declares a random routing identity, every session on an origin feeds one router, and a request names the origin node the first hop picked. Older versions are unchanged. On moq-lite-07:

  • A relay built on the library must opt into proxying, or its peers are sent only the routes to its origin's own claims: publish origin.clone().with_proxy(true), or mark one consumer with consumer.with_proxy(true). moq-relay does this for every session; an application publishing only its own broadcasts leaves it off.
  • A link cost of 0 is refused at SETUP: Client::with_cost(0) fails the session. Use 1, the price of an unpriced link.
  • A route carries no hop chain. RouteInfo::hops() on a route learned over moq-lite-07 names only its origin node, and origin::Config::max_hops does not bound it: routes are loop-free by feasibility instead.
  • A pool's paths spread by a hash of the path and each origin node's routing identity, which every relay computes alike, rather than by publisher instance, so every relay picks the same origin for a path.

moq-relay: cluster.id is removed. Remove --cluster-id, MOQ_CLUSTER_ID, and [cluster] id; in Rust, drop cluster::Config::id. A relay that still sets one stops at startup with a message naming the removal, for the flag, the variable, and the key alike. Each relay picks a random Hop ID on every start and nothing pins it, so a restarted relay is a new node to the mesh. Cluster::id() still reports the one in use.

moq-relay: zero-cost cluster links are refused. A link costs at least 1, the price of an unpriced link, since routing needs every link to cost something. A peer priced at 0 (?cost=0 on its URL, or an object cost or egress of 0) stops startup in cluster.connect and rejects a cluster.api.url update, which keeps the last good list. With LAN discovery on, a cluster.node whose ?cost= is 0 or malformed stops startup, and a LAN peer advertising one is skipped. Price a same-datacenter sibling at 1, and raise the other links' costs if it must stay relatively cheaper.

moq (CLI): --cluster-id is removed. Remove --cluster-id and MOQ_CLUSTER_ID: moq stops at startup naming the removal. The Hop ID is random on every run and nothing pins it.

moq (CLI): zero-cost cluster links are refused. A --cluster-connect or --cluster-api-url peer priced at 0 (?cost=0, or an object cost or egress of 0) is refused, as is a --cluster-node carrying ?cost=0 under --cluster-lan. A link costs at least 1, the price of an unpriced link: use ?cost=1 for a same-datacenter sibling.

Testing

On 2a7e718e03 (merge of main at 58fedef960, the ROUTE_REQUEST bound, and the answer's fairness fix; it merges cleanly with main at 10fc09e5cc):

  • quest gate just ci origin/main: every module passed, Rust 11160 tests (145 skipped).
  • cargo clippy -p moq-net --lib --no-default-features -- -D warnings: clean.
  • a_stalled_writer_holds_one_request_per_identity (routed subscriber): with the writer stalled, 1000 raised requests for one identity go out as two, the first and the highest. Before the fix the writer sent all 1000.
  • a_lite07_reply_flood_yields_and_announces (publisher): an endless ROUTE_REQUEST stream yields the task and still announces a claim made mid-flood. Before the fix it hangs until nextest's timeout.
  • The merge of main adds the node field to main's new lite-07 TrackServe test helper from moq-dev/moq#7670. Check's repository_configs_load failure was already fixed by moq-dev/moq#8144, merged in on b327b8eaa4; the one failing check, the optional pullfrog, fails because its OpenRouter key was rejected.

On b327b8eaa4 (merge of main at 812e38fc90, plus the route key skip):

  • quest gate just ci origin/main: every module passed, Rust 11094 tests (145 skipped).
  • quest gate just test interop --all on the merge a6ded820d5: 191 passed, 22 skipped, 2 failed. Both failures were main's new browser route key lane (moq-dev/moq#8116), which dials the relay from a tab on moq-lite-07-wip in js/net's pre-route wire: the relay ended Firefox's session with a protocol violation after its old-wire announce, and Chromium's page crashed at the same point. It now records a skip naming routes/js.md, like the other JS lite-07 lanes (decision 1).
  • quest gate just test interop --publishers rust --subscribers js,js-native-node on b327b8eaa4: all checks passed, both route key lanes skipped.
  • The rust -> js-native-node audio timestamp gap the land fixer saw on b02bdff9f8 did not reproduce in either run here, nor in the fixer's targeted rerun. That cell runs JS on its default version, not lite-07, so it is recorded as a load flake rather than this PR's.

On 70771b3cd6 (the final head adds only a conflict-free merge of main, after which cargo nextest run -p moq-net --features async,fuzz passed: 3624 tests):

  • quest gate just ci: every module passed, Rust 11066 tests (145 skipped).

On f51f5292be, before the cluster ID's deprecation message:

  • quest gate just test interop --all: 182 passed, 22 skipped (the JS lite-07 lanes), 0 failed.
  • cargo nextest run -p moq-net -p moq-tokio -p moq-relay -p moq-net-tests: 4664 passed.

On d5ed89fd60, before the takeover-ordering fix, which touches only the node's outbox:

  • quest gate just ci: every module passed, Rust 11060 tests (145 skipped).
  • cargo check -p moq-net --lib, the same with --features async, and cargo clippy -p moq-net --lib --no-default-features -- -D warnings: clean. This was the merge queue's failure.
  • test/drill/sensitivity.sh relay-withdraws-lost-publisher: the drill fails with "the interrupted publisher: no announcement change" once mutated, so it is sensitive again.

On fb3cc76aee, before the review fixes, which touch only the Rust lite-07 route layer:

  • just test interop --tail: 46 passed, 22 skipped (the JS lite-07 lanes), 0 failed.
  • interop.sh --publishers rust --subscribers rust,js-native-node,js-native-bun: all checks passed.
  • interop.sh --subscribers rust,js-native-node --versions moq-lite-07-wip: refused at setup, exit 2: "JS speaks pre-route lite-07 until quest/m2/cluster-routing/routes/js.md; pin moq-lite-07-wip without JS clients".
  • max_age_relay_javascript and relay_fill_javascript (moq-tokio): pass without their lite-07 legs.

New tests:

  • mixed_version_ring_withdraw_retracts: hangs without the reflection guard.
  • next_hop_lost_*: covers both the lite-07 continuation and the epochless restart.
  • a_lite07_claim_past_the_cap_closes_the_session: shows a peer's routes and claims share the announce cap, which bounds what a peer makes the router hold.
  • announce_takeover.rs is back to main's expectations: lite-07 announces a job answering a claim as a takeover, at the relay and the edge, in place and beneath, and as a start to a lite-06 edge or with another worker's claim visible.
  • a_takeover_in_place_keeps_the_predecessors_sources (routed subscriber): fails if the in-place takeover drops the predecessor's sources.
  • the_limit_admits_each_claim (routed subscriber): a limit of live/*/public/** shares its regions with live/**, and still leaves the private claim out, brings it in when widened, and out again when narrowed.
  • pending_changes_net_out (routed subscriber): 999 updates, an update back to the committed claim, and 100 start/end pairs without a floor leave at most one pending change.
  • a_claim_is_served_only_through_streams_that_carry_it (node): a cheap stream asking for live/a and a costly one asking for live/b each serve only their own claim.
  • a_stalled_reader_holds_the_net_change and routes_ended_unread_leave_nothing (node): 1000 repricings and start/end churns read as one route and one batch, 100 identities churned through a stalled reader leave nothing and no unacked end, and an end, restart, and end of a held route reads as one end with one echo owed.
  • a_new_scope_certifies_against_its_own_regions (routed subscriber): unscoped, then live/*/public/**, then live/**, and the private claim comes back.
  • a_claim_is_served_only_where_the_limit_admits_it (routed subscriber): two sessions proving one route key, live/** and live/*/public/**, and only the wide one serves the private claim.
  • an_ended_route_holds_its_slot_until_its_echo_is_flushed (routed subscriber): with the cap at 2 and the writer stalled, two ended routes still refuse a third.
  • a_stalled_stream_holds_one_serve_per_claim, a_stalled_takeover_reads_its_successor_first, and a_claim_without_a_route_keeps_the_peers_name (node).
  • an_upstream_learned_claim_never_reaches_another_upstream (integration): a relay between a lite-06 and a lite-07 upstream forwards its own broadcast up, never the lite-06 one.
  • a_lite07_prefix_past_the_root_announces_nothing (publisher).
  • Each of these fails with its fix reverted.
  • mount.rs (integration), each on lite-06 and lite-07: a tenant with another tenant's public mounted at shared discovers and reads shared/cam, never what the origin holds at the mount point, and sees it end with the target; a session rooted at the mount point reads the target; and a relay learning the target from upstream presents it and lets it go with the upstream. The lite-07 cases fail before the mount port.

Review

Reviewer: Codex (gpt-6.1-sol, medium) via quest review --base origin/main, session 01a1b783 (a new session for this iteration).

  • Round 2 reviewed head 2a7e718e03 and reported no findings.
  • Round 1 reviewed 0b70a604ff (the merge and the ROUTE_REQUEST bound) and found one, fixed in 2a7e718e03: a lite-07 peer sending ROUTE_REQUEST without pause kept Answer::poll (rs/moq-net/src/lite/publisher/routed.rs) reading replies, so it never served route outputs nor yielded the task. Each pass now takes one reply and one output, on a budget of 32 passes (a_lite07_reply_flood_yields_and_announces).

The earlier record, before this iteration: session 01a1b1fe round 8 reviewed 70771b3cd6 with no findings, and b327b8eaa4 added only a merge of main that left the diff identical, plus the browser route key skip. Its findings across sessions 01a1b1da and 01a1b1fe, and the online reviews in between, were all fixed or refuted as recorded in this body's history: the hidden-floor refutation (drafts/draft-lcurley-moq-lite.md:450) and the mount takeover the maintainer moved to mount-takeover.md stand.

Suggested follow-ups

  • Fix the no-async errors on main. moq-net's no-async doc links and the track.rs lib test (read_frame on group::Consumer) fail to build on main.

The decision on when Hops, RouteInfo::hops and the saturating Cost leave the public API waits in session.md, which keeps them; this PR does not depend on it.

🤖 Generated with Claude Code

(Written by Claude Opus 5.5)

kixelated and others added 19 commits November 5, 2026 22:33
A floor is now one marker per route across the stream's interest. The
router presents its per-region view onto it: claims inside keep their
path, the most specific covering claim moves onto the interest (moved,
not ended, when withdrawn), and the floor certifies only when the view
knows the whole interest at one revision. Claims a stream carries
uncertified stay in its heard view and show where no certified floor
proves them absent; they go offline, never END, when their stream drops
them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The origin keeps a route layer node its lite-07 sessions will share,
created on first use with a driver task that announces every route the
origin serves itself. A route the route layer selected names its origin
node, which a reader can filter on and which ranks routes by the
rendezvous hash of the path and that node.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every relay now takes a random Hop ID on each start: `--cluster-id`,
`MOQ_CLUSTER_ID`, and `[cluster] id` are gone and refused as unknown
settings, matching the route layer's per-process node identity.

A cluster link costs at least 1, the price of an unpriced link. A peer
priced at 0 (`?cost=0`, or an object `cost` or `egress` of 0) is a
configuration error, as is a LAN-advertised `cluster.node` whose cost is
0 or malformed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ANNOUNCE_START and ANNOUNCE_TAKEOVER carry the Route ID, ANNOUNCE_UPDATE
only the Announce Cost, and ANNOUNCE_RESTART no hops; hop bases are
gone. ANNOUNCE_OK has no body and SETUP no Hop parameter on lite-07, a
Cost of 0 is refused, and SUBSCRIBE, FETCH, TRACK, TRACKS, and DATAGRAMS
carry the Node ID.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each lite-07 announce stream registers with the origin's route node: the
subscriber feeds it the peer's routes and claim batches (closed by
ANNOUNCE_FLOOR) and serves the claims the node hands it, minting sources
whose requests name the claim's Node ID; the publisher answers from the
node, filtering hidden, out-of-scope, and ungranted claims (a floor then
certifies nothing). Requests resolve through the Node ID they name.

A scoped peer's identities are qualified under its proven route key. A
per-connection proxy preference decides whether a peer is sent routes
learned from other peers: relays on, the library default off.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A claim is served through every stream with a usable feasible route to
its origin node, so the origin holds the standby and fails over at once
when a session drops, as it did with hop chains. Own claims reach the
node before the first stream answers, and feeding them asks no peer.

A re-filed stream withdraws what it no longer serves, a covering claim
is answered like a covering reader sees it, and a route that entered
here for exactly the requested path ranks newest-first. The mesh
counterexamples now retract once everywhere.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A front ending released the copy of every track nobody read, including one
that had already finished. A replaced run whose track finished before the
restart reached the subscriber therefore lost its cached groups, and a
recorder fetching them failed with the segment unavailable. Main hits the
same failure once the old run's end lands first; lite-07's routed restart
just arrives a little later. An ended copy subscribes nothing, so keeping it
costs only what its readers already hold.

moq-hls's restart suite gains a case that lets the old run's end settle
before the new instance announces, so the ordering no longer decides it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A client took the grant its relay answered with as a scope on the relay, so
it qualified the relay's identities and refused every one the relay had
already qualified: two clients behind a relay never saw each other's
endpoints. A connection's identities are qualified when its publish grant is
scoped, and that grant is the one this side issued, the permit's limit. What
the peer granted this side limits only what this side does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A relay built from the library now opts into proxying, as the relay binary
does, or a lite-07 peer is sent only its own claims: moq-mux's links,
moq-stats' and moq-tokio's relays, and the goaway suite's fake upstream.

Lite-07 carries no hop chain, so the hop-limit tests run on the older wires
only, and tiered_regions checks reachability and core failover rather than
which core a path's chain names. The first hop picks a pool's origin by a
hash every node agrees on, so prefix_pool checks that the downstream relay
agrees with the upstream one and that a joining worker takes only paths it
wins. With lite-07's takeover gone until quest/m2/announce-takeover.md,
claim_restart wakes the worker before subscribing, as the older wires do.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The lite-07 session drives the route layer now, so its blanket dead_code
allowance goes: test-only queries are gated on test, and builds without
async allow only what the async session alone drives. ROUTE_REQUEST and the
announce replies lose their allowances too. The routed streams log each
announce and restart as the older wire does, and `Client::with_cost`
documents that lite-07 refuses zero.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A relay translating a moq-lite-07 claim for a moq-lite-06 peer starts the chain
with the claiming node's route identity hop. When that chain came back to the
claiming node over lite-06, nothing in it named the node's own origin hop, so it
was filed as one of the node's own claims again and kept the broadcast circling
after its publisher left.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…arries on

One publisher reaches a relay through two neighbors and the preferred one goes
away. Wires that mint an instance per announcement restart the route onto the
standby's; lite-07 carries the publisher's instance and already serves the claim
through both, so it continues with nothing announced.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated kixelated changed the title Route session integration feat(net)!: lite-07 routes through the route layer Nov 6, 2026
kixelated and others added 9 commits November 6, 2026 08:34
…ontract

# Conflicts:
#	rs/moq-net/MIGRATE.md
#	rs/moq-net/src/route/mod.rs
A route-layer claim records the claim it took over and the instance that
one served. The origin's own takeovers feed it, a relay keeps it from the
ANNOUNCE_TAKEOVER it received, and an answering stream sends the claim as
a takeover of its predecessor when the peer held that predecessor, under
the same route, before the batch, and every claim it holds at that path
serves the same instance. The receiver forwards the claim with its
predecessor's route, and in place keeps the predecessor's sources.

The legacy publisher and subscriber takeover paths were unreachable once
lite-07 routed, so they go.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
js/net still speaks lite-07 from before the route layer, so the in-tree
lanes running it on lite-07-wip fail against this checkout's relay. Native
JS dials its defaults in the interop matrix, and every lane that would run
it on lite-07 records a skip naming quest/m2/cluster-routing/routes/js.md,
which restores them. The same goes for lite_tracks_interop and the lite-07
legs of the max-age and relay-fill JS suites.

The routes epic records the maintainer's 2026-11-06 picks: ANNOUNCE_TAKEOVER
ported onto Route IDs, and a route sent on a stream only while it meets the
stream's interest. session.md keeps the scoping work it implements.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ontract

# Conflicts:
#	rs/moq-net/MIGRATE.md
…ing, serves per claim

- A lite-07 stream hands the node only the peer's claims the session's limit admits,
  not every claim within its regions. A limit narrower than its regions certifies
  nothing, so a floor never certifies a claim it left out absent. A permit change
  that admits a different set files the stream again, as a scope change does.
- Claim changes between floors are kept as their net change to the committed claims,
  so what a stream holds is bounded by its claims, not by the messages it read.
- The node serves a claim only through the streams that can carry it: within what
  this node asked the neighbor for and within its scope.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A stream whose reader stalls kept every route change and batch the node queued.
Its outbox now holds one entry per identity: an end first, then the last start and
hold, and one batch with each claim's last change and the last floor, starts before
ends as the router orders them. A claim the reader was never sent leaves nothing
when it ends, so what a stalled stream holds is bounded by its claims.

A route started and ended unread reaches the reader as an end alone, which it acks
at once since the peer never held the route.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Main now has the release cut write migration notes; this PR carries them in its
Impact section instead.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@pullfrog

pullfrog Bot commented Nov 7, 2026

Copy link
Copy Markdown
Contributor

Your OpenRouter API key was rejected. Issue a new key with the provider, then update the copy Pullfrog uses, in Pullfrog or in this repo's GitHub Actions secrets.

The provider's response
provider error: User not found.

Pullfrog provider settings → · GitHub Actions secrets →

Pullfrog  | Rerun failed job ➔ | View workflow run | via Pullfrog | Using deepseek-v4.1-flash-1017 (free via Pullfrog for OSS)

@kixelated kixelated left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated review by review (OpenAI)

Incremental review of commit b327b8e against base 812e38f, following reviewed head 8a56b87 and accounting for inherited base changes.

No new actionable findings. The substantive new change extends the existing pre-route JavaScript lite-07 exclusion to the browser route-key lane. It records an explicit skip for each engine with the same restoration quest, and both documentation locations now list that lane. The additional node: 0 initializers adapt inherited subscriber test fixtures. The previously reviewed production routing changes, including the mixed-version learned-claim filtering fix, are unchanged in the PR-owned delta.

Direction: retain the explicit skip reporting and restore this lane with routes/js.md. While this exclusion is present, browser route-key proof is not exercised, including under CI_STRICT; a passing interop run must not be interpreted as validating that proof.

Verification limits: GitHub-only static, base-aware patch comparison and inspection of the surrounding browser-lane dispatch. I did not run builds, interop sessions, or tests, and have not independently reproduced the author-reported passing reruns. This is a COMMENT, not an approval or a fresh exhaustive review of inherited changes.

(Written by review (OpenAI))

@kixelated
kixelated added this pull request to the merge queue Nov 7, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to a conflict with the base branch Nov 7, 2026
@moq-bot

moq-bot Bot commented Nov 7, 2026

Copy link
Copy Markdown
Contributor

quest land watch: the merge queue removed this pull request at b327b8e because it conflicts with main (merge_conflict).

The land fixer works on it in the background, and the watcher re-enqueues it at the back of the queue once it is fixed or judged flaky, or hands it back here with what it needs.

(written by quest land watch)

@moq-bot

moq-bot Bot commented Nov 7, 2026

Copy link
Copy Markdown
Contributor

Codex land fixer: fixer crashed (exit 1); run /quest-iterate. Recommendation: run /quest-iterate before picking it again.

exec
/bin/bash -lc 'tail -20 /tmp/quest-8023-interop-final.log && tail -8 /tmp/tmp.hzhLGSiV0l/bun.log && tail -6 /tmp/quest-8023-ci-merged.log' in /work
 succeeded in 0ms:
   Compiling moq-mux v0.10.11 (/work/rs/moq-mux)
   Compiling qmux v0.7.0 (/work/rs/qmux)
   Compiling moq-tokio v0.19.23 (/work/rs/moq-tokio)
   Compiling moq-hls v0.5.11 (/work/rs/moq-hls)
   Compiling moq-srt v0.3.11 (/work/rs/moq-srt)
   Compiling moq-audio v0.1.10 (/work/rs/moq-audio)
   Compiling moq-rtmp v0.3.11 (/work/rs/moq-rtmp)
   Compiling moq-rtsp v0.1.0 (/work/rs/moq-rtsp)
   Compiling moq-relay v0.17.2 (/work/rs/moq-relay)
   Compiling moq-webrtc v0.3.12 (/work/rs/moq-webrtc)
   Compiling moq-cli v0.14.2 (/work/rs/moq-cli)
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 2m 03s
relay:   /tmp/moq-test-1000/interop-SlbzVZow/bin/moq-relay
moq-cli: /tmp/moq-test-1000/interop-SlbzVZow/bin/moq
building python client (workspace moq wheels)...
building go client (workspace moq-go via uniffi-bindgen-go)...
building c++ client (workspace cpp/moq via uniffi-bindgen-cpp + cmake)...
installing js clients (workspace @moq/* via bun)...
building c client (workspace moq-c + cc)...
building gstreamer client (workspace moq-gst plugin)...
python3 sh/rs/test-lints.test.py
..
----------------------------------------------------------------------
Ran 2 tests in 398.572s

OK
python3 sh/gh/nightly.test.py
..............        PASS [   0.829s] (11153/11156) qmux session::tests::reading_returns_flow_control_credit
        PASS [   0.517s] (11154/11156) qmux::interop ours_as_client
        PASS [   0.684s] (11155/11156) qmux::interop ours_as_server
        PASS [   2.370s] (11156/11156) qmux session::tests::websocket_keep_alive_runs_during_continuous_output
────────────
     Summary [ 247.374s] 11156 tests run: 11156 passed, 145 skipped

codex
The local test-runner fixture tests passed. Interop has moved on to building the GStreamer client.

 succeeded in 3405894ms:
[pr-8023 bee2ed4fb] merge: resolve moq-net documentation conflict with base

exec
/bin/bash -lc 'tail -14 /tmp/quest-8023-interop-final.log && tail -7 /tmp/tmp.hzhLGSiV0l/bun.log && tail -8 /tmp/quest-8023-ci-merged.log' in /work
 exited 1 in 0ms:
building c client (workspace moq-c + cc)...
building gstreamer client (workspace moq-gst plugin)...
starting relay on 127.0.0.1:4500...
endpoint: relay http://127.0.0.1:4500
=== publisher: rust  broadcast: interop-rust-123319-22689.hang ===
  PASS  rust -> rust (1s)
  PASS  rust -> python (1s)
  PASS  rust -> go (0s)
  PASS  rust -> cpp (1s)
  PASS  rust -> js (12s)
  PASS  rust -> js-native-node (2s)
  PASS  rust -> js-native-bun (1s)
  PASS  rust -> c (0s)
  PASS  rust -> gst (1s)
tail: cannot open '/tmp/tmp.hzhLGSiV0l/bun.log' for reading: No such file or directory

exec
/bin/bash -lc 'tail -30 /tmp/quest-8023-ci-merged.log && git status --short' in /work
 succeeded in 0ms:
publish-wrapper test: no rsync on PATH, skipping
bash .claude/hooks/direnv.test.sh
direnv hook: could not load the dev shell, see the direnv output above.
direnv hook: regression tests passed
bash .codex/hooks/direnv.test.sh
Codex snapshot capture: regression tests passed
python3 sh/rs/uring.test.py
.
----------------------------------------------------------------------
Ran 1 test in 1.008s

OK
ci: seconds per module
  1767  check rs
  1231  test rs
    13  check biome
   710  check js
  1787  test js
   133  check markdown
   121  check gh
   642  test gh
     1  check drill
     1  check quest
    45  check shell
     1  check large_files
     1  check text
     0  check toml
     1  check nix
     1  check justfile
  3406  wall time

codex
`just ci base` passed, including all 11,156 Rust tests and the JavaScript and tooling suites. The full interop matrix is running; the Rust publisher passed against every subscriber.

quest land check: truncated check protocol

(written by Codex)

@kixelated

Copy link
Copy Markdown
Collaborator Author

Maintainer decision (2026-11-07, /quest-complete) on the ROUTE_REQUEST backlog that moq-dev/moq#8063's review found in this PR's code (rs/moq-net/src/lite/subscriber/routed.rs, Routed::poll):

Working on it in /quest-iterate now, alongside merging main and the Check fix.

(Written by Claude Opus 5.5)

kixelated and others added 3 commits November 7, 2026 11:29
# Conflicts:
#	doc/lib/rs/moq-net.md
A downstream peer raising its ROUTE_REQUEST seqno for a route through a
stream whose peer leaves its replies unread queued one request per raise,
and the session buffered every reply into the writer while its flush was
pending, so relay memory grew without bound.

The node now keeps each stream's requests per identity, the highest seqno
subsuming lower ones, and the session takes them, and its ROUTE_END
echoes, only once the writer has flushed. A stalled writer holds one
request per identity the peer advertised, which the announce cap bounds.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A peer sending ROUTE_REQUEST without pause kept the answer loop reading
replies, so it never served the node's outputs and never yielded the
task. Each pass now takes one reply and one output, and the loop yields
on a cooperative budget like the publisher's other loops.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@kixelated kixelated left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated review by review (OpenAI)

Reviewed head 2a7e718 against base 58fedef, incrementally since b327b8e and excluding inherited main changes.

No new actionable findings. The reported ROUTE_REQUEST backlog is addressed structurally: each identity retains only its highest pending seqno, replies enter the writer only after its previous buffer flushes, and ended-route slots stay charged until their echoes flush. Normal serve outputs continue independently. The answer's one-reply/one-output passes and self-waking cooperative budget also address the local-review starvation finding without starving route announcements.

Direction: keep coalescing and flush-before-buffer backpressure rather than a fixed queue cap. The added regressions cover 1-through-1000 request coalescing with a stalled writer and announcement progress under an endless reply stream.

Verification limits: GitHub-only static comparison of all 72 PR-owned patches against their respective bases, plus inspection of the five differing files, relevant routing/slot paths and cooperative budget. No builds, tests or interop run. This incremental review is not a new exhaustive audit of the full routing change.

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ No new issues found.

Reviewed changes

Reviewed the three PR-side commits since the prior pullfrog review at b02bdff9f: the ROUTE_REQUEST backlog bound, the answer's fairness fix, and a merge of main.

  • Bounded the ROUTE_REQUEST backlog per identity. node::Output::Request is gone: Outbox.requests keeps one entry per identity with the highest seqno winning, Stream::poll_requests hands the map over only when it is non-empty, and requests stay out of is_empty, so poll_output/try_output never see them (rs/moq-net/src/route/node.rs).
  • Stopped buffering replies until the writer flushes. Routed::poll writes the queued ends (one u64 per ROUTE_END echo, each backed by a slot in echoing) and the taken requests only after poll_flush returns Ready, truncating echoing to ends.len() so a stalled writer holds exactly one slot per unwritten echo (rs/moq-net/src/lite/subscriber/routed.rs). a_stalled_writer_holds_one_request_per_identity pins the coalescing: 1000 raised seqnos stall into two writes, the first and the highest.
  • Gave the answer a cooperative budget. Answer::poll spends a kio::coop::Budget(32) pass per loop and takes at most one reply and one output per pass, so a peer flooding ROUTE_REQUEST can no longer starve route outputs or hold the task (rs/moq-net/src/lite/publisher/routed.rs, covered by a_lite07_reply_flood_yields_and_announces).
  • Skipped the browser route-key lane with the other pre-route JS lite-07 lanes. The lane records a skip naming routes/js.md before is_broken js and before CI_STRICT can force it, and test/interop/README.md plus routes/js.md list it.
  • Merged main at 58fedef96. The range diff over the PR-owned files shows only the incoming moq-webrtc rename and socket endpoint .client/.server wording in the guide, main's new route-key paragraph, and one more node: 0 test fixture; no PR hunk moved.

Verified cargo check -p moq-net --lib --tests --features async finishes clean at this head. Also traced the two new invariants that carry the fix: echoing.len() == ends.len() holds after every receive (a RouteEnd appends exactly one held slot and one end, and slots are fungible, so the count-based truncate is sufficient), and a queued ROUTE_REQUEST always wakes the stream task through the poll_output registration on the same outbox, so the flush gate cannot leave a request stranded.

Pullfrog  | View workflow run | Using deepseek-v4.1-flash-1017 (free via Pullfrog for OSS) | 𝕏

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant