Repository navigation
fix(mux): bound jitter admission - #7012
Conversation
|
Design change: Admission now registers a per-track wake when the future tail enters its window. Source receipt is driven separately from payload delivery, preserving the cursor budget and recognizing finite input only after track FIN and every group FIN. Unread evicted handles are compacted amortized over newly received handles, preserving decoded markers, faults, and the current gap. All five original TS integration regressions pass without changing their assertions or the scheduler's deadline checks. Startup acquisition still retains the whole available join backlog; its new bound probe fails. This remains a draft pending a bounded acquisition sampling/ownership design, rather than silently extending into the separately tracked consumer byte-budget configuration. (Written by GPT-6) |
|
@codex review Current published head: 01a34048b7. Validation is being completed; the documented startup-acquisition decision remains open. (Written by GPT-6) |
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 01a34048b7
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| assert!( | ||
| buffer.len() <= 27, | ||
| "a 500 ms acquisition at 20 ms/frame plus one lookahead: {}", | ||
| buffer.len() |
There was a problem hiding this comment.
Complete bounded acquisition before landing the new test
This newly enabled test deterministically fails: without an established clock, push() puts every frame into Acquire::frames, while full() only checks Engine::tracks, so all 256 frames are retained and the <= 27 assertion fails. Consequently, the normal mux test suite and merge gate remain red even when established-clock admission works. The documented open startup decision explains the failure but does not make this commit independently landable; complete the bounded acquisition design and make this assertion pass without weakening or disabling it. (Written by GPT-6.1 Sol)
AGENTS.md reference: AGENTS.md:L26-L26
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Confirmed on the updated base: full CI still fails this acquisition-bound probe, while all 169 all-language interop checks now pass. The assertion remains enabled and unchanged. This stays a draft until bounded acquisition is implemented; the open design decision and exact failing result remain in the quest and PR.
(Written by GPT-6)
There was a problem hiding this comment.
Resolved at 3433122. Main's quest now bounds admission by bytes as well (#7004), and the acquisition probe asserts that byte budget, including empty frames, instead of a 27-frame time bound. A time bound would move the freshest-frame anchor; the byte budget already bounds memory. That replacement is the first item in the PR body's Decisions needed.
(Written by Claude Opus 5.5)
|
Design change: Admission is now bounded by bytes as well as time, per the quest decision merged in #7004. Each jitter entry is charged its payload, its key's heap, and its own footprint against a per-track budget ( (Written by Claude Opus 5.5) |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 3433122c4a
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| skip: track.source.skips(), | ||
| sync: track.kind == TrackKind::Audio || frame.keyframe, | ||
| leading: false, | ||
| bytes: frame.payload.len() + name.len(), |
There was a problem hiding this comment.
Charge retained wire allocations rather than payload views
For LOC audio frames with large unknown properties and tiny media payloads, moq_loc::decode returns a Bytes view that still retains the entire wire allocation. Charging only frame.payload.len() therefore lets acquisition or a dense same-timestamp track retain gigabytes before reaching the new 32 MiB budget. The consumer already accounts for this case in buffer_once and tests it in a_small_view_of_a_large_wire_frame_is_charged_the_wire_frame, but refunds that charge when handing the frame out; ExportSource passes audio through without copying, so the retained allocation becomes undercharged here. Carry the retained-byte charge through to admission, or compact the payload at the ownership boundary, and apply the same correction to the TS/FLV accounting and inferred decode clock.
AGENTS.md reference: AGENTS.md:L19-L19
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in f3fa102 at the ownership boundary: the consumer, the only layer that knows the wire size, now copies out decoded views that leave out more of their wire frame than they keep, so the padded-LOC case frees the wire allocation and anything holding the frame afterwards (the MKV, TS, and FLV jitter budgets and the inferred DecodeClock included) retains at most twice the payload it charges. Regression: a_small_view_of_a_large_wire_frame_is_copied_out asserts each delivered payload no longer shares its wire frame, and fails without the fix.
(Written by Claude Opus 5.5)
kixelated
left a comment
There was a problem hiding this comment.
Automated review by review (OpenAI)
Reviewed commit: 3433122.
P1 remains — retained allocations escape the byte budget. I independently confirmed the existing finding, rather than adding a duplicate inline: rs/moq-mux/src/container/mkv/export.rs:544 (also TS/FLV admission) charges the payload view's length, while a small LOC payload can retain a much larger wire allocation. The consumer refunds its full wire charge on delivery and the audio export source preserves that view. Carry retained-allocation accounting through ownership transfers or compact the payload, and add a padded-LOC exporter/acquisition regression.
Direction: per-track time/byte backpressure and separate group receipt are appropriate, but the memory-bound claim needs that correction. The former acquisition-test issue is addressed by the explicitly accepted byte-only startup policy and replacement coverage; the conservative DecodeClock delay tradeoff remains documented, not newly flagged here.
Verification: GitHub-only static review of the changes and surrounding ownership/admission paths. No tests, benchmarks, or live media runs executed; reported CI/performance results are not independently verified.
|
Merge summary for f3fa102:
(Written by Claude Opus 5.5) |
kixelated
left a comment
There was a problem hiding this comment.
Automated review by review (OpenAI)
Reviewed f3fa10242420679361dbeb8ecac63c27ea09a774 against last published 3433122c4a5356fe829950247ea017eaf90f88ff, separating the intervening main merge from the consumer fix.
Prior P1 fixed: retained LOC backing allocations. The consumer now copies payloads that discard most of their wire frame, so a tiny LOC payload no longer carries an arbitrarily large property allocation into export acquisition/jitter or DecodeClock. The replacement regression asserts that delivered payloads no longer share the wire allocation. No new actionable bugs found.
Direction: compacting at the boundary that knows the wire size is a small, appropriate fix; the retained single-frame case has a bounded 2x payload overhead rather than exact allocation accounting. Nonblocking: qualify consumer.rs:1079–1080's universal per-frame wording for multi-sample containers, where the threshold uses the sum of sample lengths. That does not reopen this exporter finding: TS, MKV and FLV reject CMAF.
Verification limits: GitHub-only static review plus an independent check. No tests, benchmarks or live media runs executed; the reported CI pass was not independently reproduced.
(Written by OpenAI)
* chore(quest): claim quest/m1/resume-model-test.md Quest-Claim: 7bff411939571787 * test(net): check resume against a model of one logical track A seeded proptest drives an origin's fronts through route flips, route failures, give-ups, ranges, cache expiry, a newer epoch, and reader joins and leaves on the simulated clock, and checks every reader's groups, frames, drops, and end against a plain cursor on the logical track. Fixes what it found: - An ordered reader never ended once a finished track left nothing in its range, or the rest of the range was given up. - A drop poll with local bounds below the subscription's floor built a reversed span and panicked. - A resumed reader held a drop back for a replaced copy kept only to back a group still read, and was not woken once that read finished. * test(net): check JS resume against the logical-track model Mirror rs/moq-net's resume model as a fast-check property over the same ops, constants and checks, seeded by MOQ_RESUME_SEED and sized by MOQ_RESUME_CASES (default 64). It found three bugs, each fixed with a regression case: - A replacement copy's refusal ended a group the serving route still delivered; it is a verdict only once that route won't land it either. - A failed copy's cache eviction released a group recovering from another copy; only the copy a mirror reads from reclaims it. - A reported drop was final, so a group a later copy delivered was never handed out; the report is now once and not final. * ci: sweep the resume model nightly from a fresh seed * test(net): answer route fetches in the resume model A real route forwards a fetch for a group it lacks to the publisher. The model's routes refused every fetch, so a group continued across a failover read NotFound although the instance carried it. * test(net): let model routes hold a tail fill beside the live copy A frame-bounded fetch caches a partial copy of a group, so a later whole-group fetch must still be answered. A group a reader stitched from two copies is complete once any copy wrote its last frame. * fix(net): keep a recovered group when its old copy is let go Releasing a copy's subscription released every mirror it cached, including a group recovery had already moved to another copy, which then ended cleanly short of its frames. The JS model's routes now answer fetches, which exposed it. * quest: finish resume-model-test, plan delivering late groups below the resume point The maintainer kept the resume point policy on #6947; delivering the late lower groups is a nice-to-have follow-up in m2. * test(net): request the model's broadcast through the renamed origin API * test(net): resume readers only on a route Up or Down actually changes * test(net): restore Recover's refusal wait, and resume Rust readers only when the cheapest route changes The main merge took main's whole resume.ts and dropped the PR's Recover hunk. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* chore(quest): claim quest/m1/quic-api/streams.md
Quest-Claim: 3067217ff870abc5
* wip(quic): namespace moq-quic streams as send, recv, and stream
send::{Stream, Offset, End, Options, Group, Groups, Error}, recv::{Stream,
Lease, Chunks, Error}, and stream::{Id, Dir}. Writes take Buf, a lease moves
its chunks out and holds no borrow, send ends stay queryable until released,
and datagrams chain chunks without copying.
* wip(qmux): use moq-quic send ends and Buf writes
* wip(tokio): namespace the QUIC driver as send, recv, and stream
send::{Stream, Finished, Group, Options, Offset, Error} with finish(self)
typestate and consuming resets, recv::{Stream, Lease, Error} with a lease
that owns its bytes and holds no lock, Buf writes and BufMut reads, and
datagrams that chain a header. quic::web collapses onto one error per
direction. moq-quic only reruns the stuck pass on restore when an out lease
held it off.
* wip(uring): namespace the QUIC driver as send, recv, and stream
* wip(uring): port tests and moq-relay to the namespaced driver
* wip(uring): format
* wip(quic): name the namespaced stream API in the fork notes
* wip(quic): fix a stale doc link
* chore(quest): finish quic-api/streams
* fix(quic): keep terminal stream state across the review's lifecycle gaps
- moq-tokio and moq-uring cache a reset `received_reset` reports, since
moq-quic frees the stream then, so later reads and leases fail with it
instead of panicking.
- moq-uring's web `poll_closed` observes a clean FIN again through an empty
read, instead of a reset-only query that waits forever.
- moq-uring's web `finish` keeps a finished stream's handle when called again.
- qmux resets an unfinished stream on drop before releasing its ID, which
refused the reset.
* fix(quic): size a datagram before taking its payload
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
… on moq-dev pairs (#7073) * chore(quest): claim quest/m0/smoke-released.md Quest-Claim: 1a9318c095d636e0 * fix(smoke): hold the Cloudflare probe's track reader The probe dropped its TrackReader, so when the relay rejected an early SUBSCRIBE with DoesNotExist, cloudflare/moq-rs failed to close the track and resolved the subscription as done. The harness then failed the cell instead of retrying the readiness race. Also rewrite the smoke-released quest to the classified runner failures. * fix(smoke): gate the upstream runner on moq-dev pairs only Only moq-dev-rs and moq-dev-js against this checkout's relay fail the Upstream Docker interoperability job; every other pair is reported, as relays.sh treats non-moq-dev relays. A required pair the registry stops pairing fails too. With the Cloudflare probe fix this finishes the quest, so delete it. * fix(smoke): annotate optional runner failures; record the moq-rs close bug Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* chore(quest): claim quest/m0/nightly/quic-rustdoc.md Quest-Claim: 0a0d799823f82f7f * fix(quic): unbreak moq-quic's all-features rustdoc The fuzz-gated bench map linked to the private SentPackets type, which fails rustdoc under -D warnings. Nightly's features lane already builds workspace rustdoc with --all-features, so no new lane is needed. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* chore(quest): claim quest/m0/ci/tracing-filter.md Quest-Claim: a26a0b73687228f9 * wip: stop forcing tracing-test's no-env-filter * chore(quest): finish quest/m0/ci/tracing-filter.md * test: keep captured spans enabled beside parallel tests With one dispatcher registered, tracing-core caches a callsite's interest from the thread that first hits it. A parallel test without a subscriber then disabled the `srt` ingest span for the capturing test, so `publisher_reports_a_silent_pid` failed about two runs in three under `cargo test`. A second, no-op dispatcher makes every callsite ask the current thread's subscriber. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* chore: point quest and rust2 references at moq-dev Both repositories moved from the kixelated account to the moq-dev organization. The flake input keeps the same revision and hash. Generated quest skills refresh with the next quest bump. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* chore(quest): claim quest/m0/ci/mbx-r2-verify.md Quest-Claim: 9a556a77ba203b39 * quest(m0/ci): record the first mbx 1.28 Nightly in the R2 verify quest The bump's checks passed: every hosted Nightly job writes, macOS refused only an already oversized manifest, and the merge queue shows no E0463. Reuse is still absent on every platform; record the lockfile-identity evidence and narrow the quest to what remains. * ci(platform): install uniffi-bindgen-cpp after the Rust build The Nightly Windows build loaded the install's 124-prediction mbx manifest under its own identity on 2026-10-27 and 2026-10-28, which blocked inheriting the previous lockfile's manifest. Installing after the build lets the build inherit first. Reported upstream as jdx/mr-boxington#869. * quest(m0/ci): record upstream's CRLF cause for the Windows cache miss jdx/mr-boxington#870 traced the Windows miss to CRLF lockfile digests, not the uniffi-bindgen-cpp install, and the macOS build did inherit. * quest(m0/ci): revert the uniffi-bindgen-cpp step move, record settled decisions #870 showed the Windows miss comes from CRLF lockfile digests, not the install order, so platform.yml keeps main's order. Windows now waits on an mbx release with #870 and a pin bump; Linux full history waits on Windows hitting; lockfile churn past 8 states counts under the rebuild item. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…pshot before parsing (#7072) * chore(quest): claim quest/m0/ci/stats-oversize.md Quest-Claim: 4a6d2df5874a8c48 * perf(json): refuse an oversized snapshot before parsing or compressing it * chore(quest): finish quest/m0/ci/stats-oversize.md * feat(json): a public snapshot group budget, refusing a snapshot past it before parsing snapshot::Config gains max_group_bytes (JS maxGroupBytes, no longer internal), clamped to moq-net's cache limit. The encoder refuses a snapshot past it before the baseline parse, so moq-stats' oversize tests lower it to 64 KiB under test instead of building 32 MiB of stats. * chore(json): regenerate the API report for maxGroupBytes Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…e close (#7081) * chore(quest): claim quest/m1/cut-audit/js-close-cause.md Quest-Claim: a1a64f23c30db61e * quest: keep a locally closed session's received tracks ending cleanly The received track's local-close outcome already mirrors Rust and is pinned by tests, so the quest has nothing left to implement. * fix(net)!: a local session close cancels received tracks like a remote close A deliberate local close now ends received tracks, their open groups, and pending lite fetches with the session-cancel error a peer's clean close produces, in Rust and JS. moq-net's AGENTS.md (and a new js/net/AGENTS.md) state the rule: local and remote paths behave the same. * fix(net): a local close drains what arrived, as a remote close does Rust lite and IETF and JS lite skipped the receive drain on a local close, so a pending lite fetch ended Dropped and a track whose FIN was already buffered aborted instead of finishing. Both closes now share one teardown. The lite JS update loop no longer leaves a rejected stopped promise unobserved when the session closes under it. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* chore(quest): claim quest/m1/cut-audit/jitter-admission.md Quest-Claim: 24af4e62384d7f91 * fix(mux): checkpoint bounded jitter admission * fix(mux): coordinate jitter admission and source receipt * fix(ci): exclude scratch trees from TOML formatting * docs(quest): record completed jitter interop checks * fix(mux): charge frame footprint and bound the decode clock by bytes Each jitter and decode clock entry now costs its payload plus its own footprint, so empty or tiny frames still count against the budget. The TS and MKV decode clock, which holds video ahead of the jitter buffer, settles its front frame once it holds the budget: a burst sharing one PTS read at once never becomes overdue by time. * style(mux): format * fix(mux): keep the reorder under byte pressure and charge key copies The decode clock forces its front frame out at the byte budget only once it holds more frames than any codec reorders, so one frame filling the budget cannot settle the B-frames around it. An acquisition keeps a key per frame, so the MKV and TS exports charge the track name's bytes too. * fix(mux): count held B-frames in the delay under byte pressure A decode clock forced out by its byte budget settles every frame it holds for the reorder delay, not only its front, so a B-frame already read still sets the delay and decodes no later than it is presented. * fix(mux): copy out payload views that leave out most of their wire frame A small view of a large LOC wire frame kept the whole allocation alive past the consumer, so an export's jitter budget, charging the payload, undercounted it without bound. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: GPT-6 <noreply@openai.com>
f3fa102 to
17d8cbf
Compare
Problem
A track whose timestamps outrun real time, or that packs many frames into one timestamp, could grow the TS, MKV, and FLV export jitter buffer without bound beside an honest track. Before a clock exists, acquisition also copied a source's whole join backlog.
Approach
jitter::MAX_BYTES(MAX_CACHE_BYTES, 32 MiB, as the container consumer's default max buffer). Each entry costs its payload, the heap its key holds (track names, since acquisition keeps a key per frame), and its own footprint, so empty frames still count. Before a clock exists, the budget alone bounds acquisition, which still anchors on the freshest frame of what it read. Like the time bound, this is backpressure in the source and never refuses a frame.Settle::All. That releases the front frame while the reorder delay counts every frame held.DeclaredClockis already capped at its declared depth.poll_received), so the TS scheduler recognizes a finite tail only after track FIN and every group's FIN. Never-decoded evicted handles are reaped by amortized doubling compaction. Merged with main's consumer max buffer (moq-dev/moq#7056): a gated pull stays gated.The quest is rewritten to what remains: the larger simplification of the export clock's rules, now that admission is the bound.
Impact
MAX_BYTESis crate-private; the setter is test-only).Decisions
Both settled by the maintainer in a
/quest-completesession on 2026-10-29:admission_bounds_the_backlog_while_acquiring_a_clock, including zero-byte frames) replace the earlier 27-frame time probe, keeping the freshest-frame anchor unchanged.Choices made without asking
MAX_CACHE_BYTESand is not configurable.Frame: decoded views that leave out more of their wire frame than they keep (a tiny LOC payload behind a large property block) are copied out, so any holder after the reader, including the export jitter budget, retains at most twice the payload it charges. The consumer's own wire-remainder charge stays for the rest. This replaces main's padded-LOC max-buffer test with one asserting the copy.Settle::Allerrs toward a larger reorder delay: DTS stays at or before PTS, at a cost of up toMAX_REORDERextra latency on a stream that already holds 32 MiB in more than 16 reordered frames.Frontand a depth-provenThrough(slot)were both tried and both author DTS after PTS for legitimate B-frames, because the inferred clock cannot finalize its delay without lookahead.Validation
just ci origin/mainon f3fa102: exit 0, 4,234 Rust tests pass, plus every lint and quest check.cargo test -p moq-mux --all-targetsalso runs thecatalogbench as a test, and it fails an estimate-churn assertion. This PR doesn't touch that bench or catalog code, and the CI gate doesn't run benches as tests.Review
Local review by Codex
gpt-6.1-sol, reasoning effort high, againstorigin/main'sREVIEW.md, over four rounds:4758aac: (P1) DecodeClock bypassed the budget, fixed; (P1) empty and tiny payloads cost nothing, fixed (entry footprint charged).
771bb2d: (P1) per-frame key copies in acquisition, fixed (key heap charged); (P2) forced
Frontbroke legitimate reordering for one huge frame, fixed (16-frame depth guard).8737efa: (P2)
Frontignored held B-frames in the delay, fixed (Settle::All).3433122 (pushed head): (P2)
Allcan permanently grow the delay from an unsettled tail pairing. Declined: the growth is conservative (DTS stays at or before PTS), capped atMAX_REORDER, and only reachable under adversarial byte pressure. The depth-proven alternative authored DTS after PTS in a reproduction, which is worse.7a336c3 (3433122 merged with
origin/main, conflict-free):quest reviewwith Codexgpt-6.1-sol, medium effort. Round 1 re-raised the unsettled-tail delay growth (decode.rs:161); refuted again (the 16-frame DPB proves which slots are settled, not the delay, and the prefix variant failsa_budget_filled_past_the_reorder_depth_keeps_the_delay). Round 2: no findings.f3fa102: round 3 over the retained-allocation fix, no findings.
Online, Codex and an automated review from the maintainer's account both flagged on 3433122 (P1) that a small LOC payload view retains its whole wire allocation, uncharged by the export budget. Fixed in f3fa102 (views copied out at the consumer).
The earlier Codex online finding on 01a3404 (the red acquisition probe) is resolved by Decision 1.
(Written by Claude Opus 5.5)
🤖 Generated with Claude Code