Skip to content

fix(congestion): preserve low-rate pacing debt and retune BBR3 - #7606

Merged
moq-bot[bot] merged 8 commits into
mainfrom
quest/m1/pacer-low-rate
Nov 2, 2026
Merged

moq-bot[bot] merged 8 commits into
mainfrom
quest/m1/pacer-low-rate

Conversation

@kixelated

@kixelated kixelated commented Nov 2, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

Below 2 Mbit/s, moq-quic can release two datagrams on one packet of pacing credit. Saturating subtraction discards the second packet's cost, allowing up to twice a controller's rate. This finishes quest/m1/pacer-low-rate.md and deletes it.

Approach

  • Carry overshoot as debt and repay it before earning burst credit, for both pacing paths. Keep unpaced loss-probe repayment bounded to a burst, including when the burst budget shrinks, following the recorded debt approach.
  • Retune BBR3's minimum send quantum from two datagrams to one. The initial flight also starts at one datagram until an ACK measures the RTT. The 1 ms target and 64 KiB ceiling stay in place. This avoids forcing packet pairs at low rates; pacing gains, native probe limits, and acceptance budgets stay unchanged. Smaller feasible quanta are permitted by BBR draft §5.6.3.
  • Sweep deterministic pacing throughput over 10 kbit/s through 100 Mbit/s with one- and two-datagram sends. Replace the former 5% throughput slack with the actual initial-credit plus final-batch bound. Extend first-RTT calibration through low-rate, long-RTT cases.

Concrete source overlap checked before edits: moq-dev/moq#7602 owns build costs, currently fMP4 decoding and audit evidence. moq-dev/moq#7605 owns bench/ relay measurement support. Neither currently overlaps rs/moq-congestion/src/{pacer.rs,bbr3/mod.rs}. No status/FIN work duplicates moq-dev/moq#7439.

Impact

  • Public API: none.
  • Wire: none.
  • Rate-paced senders repay overshoot instead of sending it free. BBR3 emits smaller aggregates where the 1 ms quantum is below two datagrams. This can increase transmit calls in exchange for smaller bursts.
  • BBR3's offload_budget (send_quantum plus the delayed-ACK term) also drops by one datagram at low rates. It only binds when a connection negotiates ack_eliciting_threshold >= 4, so it has no effect under the default threshold (noted by the Pullfrog review).

Measurements

Before is base 21d6da9fc108b48373a961f34e43c3fcd38b0146; after is this pacer correction plus the BBR3 quantum change. Every measured release run below enables CARGO_PROFILE_RELEASE_DEBUG_ASSERTIONS=true and CARGO_PROFILE_RELEASE_OVERFLOW_CHECKS=true. All expensive commands use quest gate.

All 20 probe tests pass before and after. The table includes every paired comparison in that suite: the nightly scenarios use their complete seeds 1-8; short merge-gate comparisons use their complete seed set, seed 1. Values are means of per-seed outcomes, including per-seed p99. Probe overhead is redundant / unique media bytes.

All probe comparison rows
Scenario Seeds Late / baseline, before → after p99 / baseline ms, before → after Probe overhead %, before → after
bbr 8 2.500 / 2.625 → 2.750 / 2.125 235.33 / 243.98 → 232.34 / 240.74 1.055 → 0.949
cubic 8 378.125 / 381.125 → 429.625 / 431.875 1433.75 / 1433.75 → 1468.48 / 1468.48 2.047 → 2.088
acks 8 1.000 / 1.000 → 1.000 / 1.000 159.36 / 159.36 → 159.36 / 159.36 0.649 → 0.649
congested buffer 187500 1 389.000 / 389.000 → 381.000 / 381.000 685.35 / 685.35 → 678.88 / 678.88 0.000 → 0.000
congested buffer 7500 1 375.000 / 375.000 → 386.000 / 386.000 737.22 / 737.22 → 759.58 / 759.58 0.000 → 0.000
credit 8 1.000 / 1.000 → 1.000 / 1.000 158.59 / 158.79 → 158.59 / 158.79 1.289 → 1.289
idle 8 1.000 / 1.000 → 1.000 / 1.000 50.58 / 42.78 → 50.58 / 42.78 3.049 → 3.049
buffer 15000 8 1.000 / 1.125 → 1.000 / 1.125 204.56 / 204.83 → 204.56 / 204.83 1.026 → 1.026
buffer 60000 8 1.000 / 1.000 → 1.000 / 1.000 159.36 / 159.36 → 159.36 / 159.36 1.168 → 1.168
buffer 240000 8 1.000 / 1.000 → 1.000 / 1.000 159.36 / 159.36 → 159.36 / 159.36 1.168 → 1.168
buffer 960000 8 1.000 / 1.000 → 1.000 / 1.000 159.36 / 159.36 → 159.36 / 159.36 1.168 → 1.168
rises 1 1.000 / 1.000 → 1.000 / 1.000 96.97 / 92.58 → 96.97 / 92.58 7.899 → 7.899
rtt 100ms keyframe 0 8 2.000 / 2.000 → 2.000 / 2.000 94.35 / 72.78 → 94.35 / 72.78 6.205 → 6.205
rtt 100ms keyframe 50000 8 2.000 / 2.000 → 2.000 / 2.000 189.36 / 186.42 → 189.36 / 186.42 2.668 → 2.668
rtt 10ms keyframe 0 8 0.000 / 0.000 → 0.000 / 0.000 28.36 / 27.78 → 28.36 / 27.78 0.618 → 0.618
rtt 10ms keyframe 50000 8 0.000 / 0.000 → 0.000 / 0.000 144.36 / 144.36 → 144.36 / 144.36 0.351 → 0.351
rtt 200ms keyframe 0 8 5.000 / 3.000 → 5.000 / 3.000 250.63 / 250.63 → 262.56 / 262.56 9.088 → 9.082
rtt 200ms keyframe 50000 8 17.000 / 17.000 → 17.750 / 17.750 352.14 / 352.14 → 364.06 / 364.06 7.452 → 7.590
stale 375000 8 1.000 / 1.000 → 1.000 / 1.000 159.36 / 159.36 → 159.36 / 159.36 1.444 → 1.444
stale 187500 8 416.000 / 416.000 → 400.125 / 400.125 1281.08 / 1281.08 → 1281.08 / 1281.08 0.000 → 0.000
steps constant 8 1.000 / 1.000 → 1.000 / 1.000 50.40 / 42.78 → 50.40 / 42.78 2.859 → 2.859
steps keyframes 8 1.000 / 1.000 → 1.000 / 1.000 159.36 / 159.36 → 159.36 / 159.36 1.168 → 1.168
unreachable 8 1.000 / 1.000 → 1.000 / 1.000 159.36 / 159.36 → 159.36 / 159.36 1.871 → 1.871

Larger competition sample: seeds 1-256, temporarily substituted only in competing(); no budget or assertion changed, and the measurement override is absent from the final source. The command was quest gate env CARGO_PROFILE_RELEASE_DEBUG_ASSERTIONS=true CARGO_PROFILE_RELEASE_OVERFLOW_CHECKS=true MOQ_PACER_SEEDS=256 just rs test --release -p moq-quic --lib --run-ignored only -E 'test(/^tests::probe::competing_/)' --no-capture, with that temporary seed override. Each 256-seed test passed before and after.

Scenario Seeds Late / baseline, before → after p99 / baseline ms, before → after Probe overhead %, before → after
bbr 256 2.488 / 2.633 → 2.547 / 2.676 233.57 / 241.59 → 232.91 / 241.48 0.977 → 0.986
cubic 256 408.250 / 405.527 → 419.684 / 417.891 1444.06 / 1444.00 → 1455.49 / 1455.48 2.246 → 2.505

Paired probing minus baseline, mean ± standard error over those 256 seeds:

Controller sharing path Late frames, before → debt only → final p99 ms, before → debt only → final
CUBIC +2.72 ± 0.65 → +1.66 ± 0.94 → +1.79 ± 0.58 +0.051 ± 0.051 → +0.170 ± 0.102 → +0.009 ± 0.026
BBR3 -0.145 ± 0.057 → -0.070 ± 0.052 → -0.129 ± 0.057 -8.02 ± 1.09 → -8.59 ± 1.12 → -8.56 ± 1.15

The CUBIC late allowance is +4.5 frames; p99 allowance is +40 ms. Correcting pacing also changes the non-probing baseline, so the absolute late/p99 shifts must not be attributed entirely to probing. The quantum retune reduces the debt-only CUBIC paired spread rather than relying on a favorable eight-seed result.

Rerun after merging main through 30d6f63571

moq-dev/moq#7458 (Quinn and BBR3 correctness sync) and GCC (moq-dev/moq#7546) landed after the measurements above. So the release probe suite, nightly cases included, was rerun on origin/main (30d6f63571) and on this branch merged with it. The command was quest gate env CARGO_PROFILE_RELEASE_DEBUG_ASSERTIONS=true CARGO_PROFILE_RELEASE_OVERFLOW_CHECKS=true RUST_LOG=info just rs test --release -p moq-quic --lib --run-ignored all -E 'test(/^tests::probe::/)' --no-capture --test-threads 1. All 22 tests pass on both. Same columns as above; main is the before.

Every probe comparison row, main vs this branch, after #7458 and GCC
Scenario Seeds Late / baseline, main → PR p99 / baseline ms, main → PR Probe overhead %, main → PR
bbr 8 2.375 / 2.500 → 2.375 / 2.625 238.15 / 239.76 → 237.80 / 247.58 0.917 → 0.927
cubic 8 438.625 / 436.750 → 432.500 / 435.375 1454.69 / 1454.69 → 1427.98 / 1427.98 1.440 → 2.330
acks 8 1.000 / 1.000 → 1.000 / 1.000 159.36 / 159.36 → 159.36 / 159.36 0.645 → 0.646
congested buffer 187500 1 389.000 / 389.000 → 381.000 / 381.000 685.35 / 685.35 → 678.88 / 678.88 0.000 → 0.000
congested buffer 7500 1 375.000 / 375.000 → 386.000 / 386.000 737.22 / 737.22 → 759.58 / 759.58 0.000 → 0.000
credit 8 1.000 / 1.000 → 1.000 / 1.000 158.79 / 158.79 → 158.79 / 158.79 1.303 → 1.303
idle 8 1.000 / 1.000 → 1.000 / 1.000 50.58 / 42.78 → 50.58 / 42.78 3.049 → 3.049
buffer 15000 8 1.000 / 1.125 → 1.000 / 1.125 199.88 / 202.57 → 199.88 / 202.57 1.013 → 1.013
buffer 60000 8 1.000 / 1.000 → 1.000 / 1.000 159.36 / 159.36 → 159.36 / 159.36 1.206 → 1.206
buffer 240000 8 1.000 / 1.000 → 1.000 / 1.000 159.36 / 159.36 → 159.36 / 159.36 1.206 → 1.206
buffer 960000 8 1.000 / 1.000 → 1.000 / 1.000 159.36 / 159.36 → 159.36 / 159.36 1.206 → 1.206
rises 1 1.000 / 1.000 → 1.000 / 1.000 96.97 / 92.58 → 96.97 / 92.58 7.899 → 7.899
rises gcc 500000 8 1.000 / 1.000 → 1.000 / 1.000 73.76 / 72.76 → 73.76 / 72.76 0.378 → 0.378
rises gcc 1000000 8 1.000 / 1.000 → 1.000 / 1.000 73.76 / 72.76 → 73.76 / 72.76 0.868 → 0.864
rtt 100ms keyframe 0 8 2.000 / 2.000 → 2.000 / 2.000 94.35 / 72.78 → 94.35 / 72.78 6.205 → 6.205
rtt 100ms keyframe 50000 8 2.000 / 2.000 → 2.000 / 2.000 189.36 / 186.42 → 189.36 / 186.42 2.665 → 2.665
rtt 10ms keyframe 0 8 0.000 / 0.000 → 0.000 / 0.000 28.36 / 27.78 → 28.36 / 27.78 0.618 → 0.618
rtt 10ms keyframe 50000 8 0.000 / 0.000 → 0.000 / 0.000 144.36 / 144.36 → 144.36 / 144.36 0.351 → 0.351
rtt 200ms keyframe 0 8 5.000 / 3.000 → 5.000 / 3.000 249.67 / 249.67 → 259.86 / 259.86 9.086 → 9.084
rtt 200ms keyframe 50000 8 17.000 / 17.000 → 17.000 / 17.000 351.17 / 351.17 → 361.37 / 361.37 7.432 → 7.593
stale 375000 8 1.000 / 1.000 → 1.000 / 1.000 159.36 / 159.36 → 159.36 / 159.36 1.444 → 1.444
stale 187500 8 415.875 / 415.875 → 400.500 / 400.500 1277.55 / 1277.55 → 1277.55 / 1277.55 0.000 → 0.000
steps constant 8 1.000 / 1.000 → 1.000 / 1.000 50.40 / 42.78 → 50.40 / 42.78 2.859 → 2.859
steps keyframes 8 1.000 / 1.000 → 1.000 / 1.000 159.36 / 159.36 → 159.36 / 159.36 1.206 → 1.206
unreachable 8 1.000 / 1.000 → 1.000 / 1.000 159.36 / 159.36 → 159.36 / 159.36 1.892 → 1.892

GCC's in-tree low-rate row is rises_to_target_under_gcc: a 1 Mbit/s start on a 3 Mbit/s path that opens to 12 Mbit/s at 5 s. The correction leaves it unchanged. Late frames stay at 1 probing and baseline, and p99 at 73.76 / 72.76 ms. On every seed the target is reached 5 s (1x target) and 9 s (2x target) after the path opens. GCC's tests::realtime rows in #7546 came from an out-of-tree harness and can't be rerun here. BBR3 delay bound (moq-dev/moq#7544) and SCReAMv2 (moq-dev/moq#7547) are still unmerged. The realtime epic now records the GCC rerun and the need to use the corrected pacer for its bake-off.

Validation

  • Reproduced before the fix: at 1,250 B/s with two-packet batches, 27,000 bytes in 10 s against 12,500 bytes of rate credit, exceeding the finite burst allowance.
  • quest gate just rs test -p moq-congestion -p moq-quic --lib: 810 passed, 16 skipped, including Clippy and the final extended low-rate calibration test.
  • Final release probe suite: 20 passed; all 20 also pass before the fix. The two larger 256-seed competition tests pass both ways. Measurements were rerun after fixing every first-round review finding.
  • quest check --base origin/main: 626 documents pass. Formatting and git diff --check pass.
  • Earlier full gate on head dc824374944af4cf2f557dbadf7c236f96d0514f (this branch merged with main as of then): quest gate nix develop --command just ci passed, exit 0, after deduplicating PATH in the invoking shell with the same line moq-dev/moq#7594 adds to the dev shell's shellHook. That's an environment fix, not part of this PR. No mbx E2BIG was hit. just ci selects modules from the diff against origin/main: check rs, test rs (6526 passed, 110 skipped), markdown, gh, test gh, drill, quest (613 documents), shell, large_files, toml, nix, justfile. The flake check and JS suites weren't selected, so the host's known rootless-namespace check flake limit and the flaky @moq/net overtaken-group test weren't reached.
  • Earlier attempts (before the PATH fix) failed on mbx Argument list too long (os error 7) in Rustdoc and the Clippy-warning recipe fixture. One earlier attempt also hit a bind collision in bench/relay.test.py::PortOwnership.test_a_stream_transport_needs_its_own_listener that passed on rerun.
  • Merging main twice, through 96244dd854 (moq-dev/moq#7458, which touches rs/moq-congestion/src/bbr3/mod.rs) and then 30d6f63571, needed no conflict resolution. Afterwards git diff origin/main...HEAD matches the reviewed diff except for hunk offsets and a quest-list context line. fix(quic): sync Quinn and BBR3 correctness fixes #7458 has no send_quantum, smss floor, or burst references, so it doesn't interact with the quantum retune.
  • Full gate on merged head 8b99e034b1: quest gate nix develop --command just ci passed, exit 0 (PATH deduplicated as above). Test rs: 6817 passed, 113 skipped. The first attempt, on 5a9aeeb4da, failed only moq-tokio::integration backend::quic_client_close_drains_finished_track with subscribe failed: Unroutable at rs/moq-tokio/tests/integration/backend.rs:902. That's a pre-existing flake: plain origin/main (103e98e362) fails it 3 of 60 in --stress-count 60 with the same panic. Branch heads failed it at similar rates: 5/30 merged, 3/30 pre-merge, 3/30 with only the pacer change, 3/30 with only the BBR3 change. One main run of 0/30 was luck.
  • Probe suite after the merges: 22 passed, on main and on this branch (table above).

Alternatives

Keep saturating subtraction or compensate with larger pacing gains: both preserve incorrect rate accounting. Broad gain changes were unnecessary; the measured retune uses smaller feasible aggregates. No probe budgets or seed-count changes are included.

Choices made without asking

Used the existing BBR3 send-quantum mechanism to retune burst size rather than adding a knob or changing pacing gains. Used 256 paired seeds for the competing cases to separate their effect from the documented eight-seed noise. Kept all permanent seed counts unchanged.

Review

Head 4d1974a5ab74afe3457547fe12adf63f383bd5be: Codex, gpt-6.1-sol, medium effort, via quest review --base origin/main (base 24b04633a5), round 1: no findings. Since the previous reviewed head dc824374944af4cf2f557dbadf7c236f96d0514f, this head adds two conflict-free merges of origin/main and one quest-prose commit: the realtime epic records GCC's rerun row under the corrected pacer. Source is unchanged.

Automated reviews on dc824374944af4cf2f557dbadf7c236f96d0514f: the OpenAI review and Pullfrog reported no actionable findings. Pullfrog's note that offload_budget also drops by one datagram is now in Impact. CodeRabbit was rate-limited and didn't review.

Earlier rounds on the source change:

Code review: Claude Code, claude-opus-5-5, medium effort, via quest review --base origin/main --harness claude.

  • Round 2 reviewed head e8649b81a2347ac3a5dc1dca75d03af42f3c46f3: no findings.
  • Round 1 on 6b4dd0bfccb63a55c10bdb784f853852b05226c2, all four findings fixed:
    • Capacity-change debt bound: fixed, with a reproducing low-rate sweep across both pacing paths.
    • Initial send quantum: fixed and covered by first-RTT calibration.
    • Sub-packet-BDP floor documentation: fixed.
    • Poll-bound documentation: fixed.

Follow-ups

Suggested: plan the pre-existing moq-tokio::integration backend::quic_client_close_drains_finished_track flake (subscribe failed: Unroutable, 3 of 60 on origin/main) as a quest. Plan the observed bench/relay.test.py::PortOwnership.test_a_stream_transport_needs_its_own_listener bind collision as a flake quest if it recurs. Benchmark A/A calibration still requires maintainer dispatch and was not claimed here.

(Written by GPT-6.1 Sol; gate, quest completion, readiness, and the merge-and-rerun iteration by Claude Opus 5.5)

Quest-Claim: 403dc0057011512e
@kixelated kixelated changed the title The shared pacer holds its rate under 2 Mbit/s fix(congestion): preserve low-rate pacing debt and retune BBR3 Nov 2, 2026
kixelated and others added 2 commits November 1, 2026 19:22
Co-Authored-By: GPT-6.1 Sol <noreply@openai.com>
Co-Authored-By: GPT-6.1 Sol <noreply@openai.com>
kixelated and others added 2 commits November 1, 2026 22:05
The full local gate now passes, so the pacer correction and BBR3 retune
need no further work under quest/m1/pacer-low-rate.md.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated
kixelated marked this pull request as ready for review November 2, 2026 06:13
@coderabbitai

coderabbitai Bot commented Nov 2, 2026 •

Copy link
Copy Markdown
Contributor

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 9 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used all 4 included reviews currently available.

Learn how review limits work.

Review configuration:

Run configuration
  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 90dd209e-1631-477e-83e1-4e89cf26580d
Commits

Reviewing files that changed from the base of the PR and between 30d6f63 and 4d1974a.

Files selected for processing: 5
  • quest/m1/README.md
  • quest/m1/pacer-low-rate.md
  • quest/m2/quic-realtime/README.md
  • rs/moq-congestion/src/bbr3/mod.rs
  • rs/moq-congestion/src/pacer.rs
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@kixelated kixelated left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated review by review (OpenAI)

Reviewed commit: dc82437

No actionable bugs found in the changed areas. Carrying overshoot into the shared pacer's next refill addresses the lost-credit root cause in both pacing paths; the capacity-change clamp preserves bounded repayment for unpaced probes. The existing BBR3 quantum mechanism is a proportionate place to reduce low-rate aggregation, without adding API/wire surface or compensating with higher gains. The deterministic one-/two-datagram sweep and first-RTT coverage target the relevant boundaries.

Verification: static review of the diff, pacer accounting/tests, BBR3 quantum/offload integration, and moq-quic's pacing/batch caller. I did not run tests or reproduce the PR's reported benchmark/CI results; the competition measurements remain author-reported evidence.

(Written by OpenAI)

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ No new issues found.

Reviewed changes

  • Pacer debt accounting (rs/moq-congestion/src/pacer.rs) — on_transmit now records the bytes sent past available credit in a debt field, bounded to one burst, and refill repays that debt before adding earned credit. Both delay and delay_at_rate require debt == 0 and fold debt into the wait. This charges the second datagram the transport releases on one packet of credit at low rates, instead of discarding its cost through saturating subtraction.
  • BBR3 quantum retune (rs/moq-congestion/src/bbr3/mod.rs) — the initial send_quantum and the set_send_quantum floor drop from 2 * smss to smss, so burst() yields one-datagram aggregates below roughly 19 Mbit/s. Draft §5.6.3's default floor is max(..., 2 * C.SMSS) but the section explicitly permits alternatives and directs implementations to use the smallest feasible quanta. One coupled effect the description doesn't mention: offload_budget = send_quantum + delayed_ack_term also drops by smss, which only binds when a connection negotiates ack_eliciting_threshold >= 4 (inert under the default threshold).
  • Tests — new repays_a_packet_sent_past_the_burst_budget and capacity_changes_bound_unpaced_debt; the throughput sweep now spans 1_250..12_500_000 B/s over 10 s with one- and two-datagram batches against the initial-credit-plus-final-batch bound; first-RTT calibration extends through long-RTT low-rate cases.
  • Quests — deletes the finished quest/m1/pacer-low-rate.md and updates the m1 and quic-realtime READMEs.

Verified the debt invariants (tokens <= capacity; debt > 0 ⇒ tokens == 0) across both pacing paths and their capacity-change clamps: no stall, no underflow/overflow, and the paced path is bounded to one datagram of overshoot. Unpaced loss probes can leave more than one burst unrepaid, but that is the documented hostile-peer bound, strictly more conservative than the pre-fix code, and externally bounded by the window and send-rate cap. Two independent specialist passes on the pacer accounting and the BBR3 quantum integration found no violation.

Pullfrog  | View workflow run | Using deepseek-v4.1-flash (free via Pullfrog for OSS) | 𝕏

@kixelated
kixelated added this pull request to the merge queue Nov 2, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Nov 2, 2026
@moq-bot

moq-bot Bot commented Nov 2, 2026

Copy link
Copy Markdown
Contributor

quest land watch: the merge queue removed this pull request at 4d1974a because its merge_group build failed (failed_checks).

The land fixer works on it in the background, and the watcher re-enqueues it at the back of the queue once it is fixed or judged flaky, or hands it back here with what it needs.

Queue / Queue (job) failure at step Check:

…0.032s] (1774/6817) moq-gst source::imp::session_tests::malformed_input_is_not_a_lost_source
        PASS [   0.289s] (1775/6817) moq-gst source::imp::session_tests::a_relisted_rendition_keeps_its_pad
        PASS [   0.067s] (1776/6817) moq-gst source::imp::session_tests::keylog_reaches_the_client
        PASS [   0.052s] (1777/6817) moq-gst source::imp::session_tests::rendition_subscribes_on_its_referenced_broadcast
        PASS [   0.200s] (1778/6817) moq-gst source::imp::session_tests::a_sibling_rendition_relisted_while_draining_does_not_replay_its_finish
        PASS [   0.177s] (1779/6817) moq-gst source::imp::session_tests::an_unserved_rendition_does_not_claim_the_first_pad_name
        PASS [   0.198s] (1780/6817) moq-gst source::imp::session_tests::an_ended_broadcast_resumes_on_the_same_pad
        PASS [   0.123s] (1781/6817) moq-gst source::imp::session_tests::paused_to_ready_returns_after_the_connection_closes
        FAIL [   0.108s] (1782/6817) moq-gst source::imp::session_tests::restarts_under_reconnects_interleave_on_one_pad
  stdout ───

    running 1 test
    test source::imp::session_tests::restarts_under_reconnects_interleave_on_one_pad ... FAILED

    failures:

    failures:
        source::imp::session_tests::restarts_under_reconnects_interleave_on_one_pad

    test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 170 filtered out; finished in 0.07s

  stderr ───

    thread 'source::imp::session_tests::restarts_under_reconnects_interleave_on_one_pad' (1245554) panicked at rs/moq-gst/src/source/imp.rs:3788:13:
    assertion `left == right` failed: [Pad("video_0"), Segment("video_0", 79000), Buffer("video_0", 1), Start("video_0"), Caps("video_0", "video/x-h264"), Segment("video_0", 94000), Buffer("video_0", 2), Buffer("video_0", 3), Buffer("video_0", 4), Start("video_0"), Caps("video_0", "video/x-h264"), Segment("video_0", 94000), Buffer("video_0", 2), Buffer("video_0", 3), Start("video_0"), Caps("video_0", "video/x-h264"), Segment("video_0", 62000), Buffer("video_0", 5), Buffer("video_0", 6), Buffer("video_0", 7), Start("video_0"), Caps("video_0", "video/x-h264"), Segment("video_0", 62000), Buffer("video_0", 5), Buffer("video_0", 6), Buffer("video_0", 7), Start("video_0"), Caps("video_0", "video/x-h264"), Segment("video_0", 73000), Buffer("video_0", 8), Buffer("video_0", 9), Buffer("video_0", 10), Start("video_0"), Caps("video_0", "video/x-h264"), Start("video_0"), Caps("video_0", "video/x-h264"), Segment("video_0", 2809000), Buffer("video_0", 11), Buffer("video_0", 12), Buffer("video_0", 13), Eos("video_0")]
      left: 1
     right: 0
    note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

  Cancelling due to test failure: 23 tests still running
        PASS [   0.051s] (1783/6817) moq-gst source::imp::session_tests::the_source_retries_without_a_budget
        PASS [   0.049s] (1784/6817) moq-gst source::imp::tests::aac_caps_follow_the_description_framing
        PASS [   0.136s] (1785/6817) moq-gst source::imp::session_tests::stop_from_a_runtime_worker_does_not_deadlock
        PASS [   0.137s] (1786/6817) moq-gst source::imp::session_tests::stop_from_its_own_reader_does_not_deadlock
        PASS [   0.042s] (1787/6817) moq-gst source::imp::tests::an_unrepresentable_timestamp_is_refused
        PASS [   0.202s] (1788/6817) moq-gst source::imp::session_tests::audio_and_video_share_a_run_timeline_across_scales_and_restarts
        PASS [   0.135s] (1789/6817) moq-gst source::imp::session_tests::stop_inside_another_executor
        PASS [   0.121s] (1790/6817) moq-gst source::imp::session_tests::stop_returns_after_the_connection_closes
        PASS [   0.125s] (1791/6817) moq-gst source::imp::session_tests::stop_releases_a_blocked_push
        PASS [   0.643s] (1792/6817) moq-gst source::imp::session_tests::a_delisted_sibling_rendition_follows_no_later_instance
        PASS [   5.248s] (1793/6817) moq-cli::integration import::a_short_import_drains_its_media_to_a_waiting_subscriber
        PASS [   0.379s] (1794/6817) moq-gst source::imp::session_tests::an_update_keeps_the_run
        PASS [   5.430s] (1795/6817) moq-cli::integration import::import_delivers_the_catalog_finish_at_eof
        PASS [   0.839s] (1796/6817) moq-gst source::imp::session_tests::a_retired_rendition_drains_to_eos
        PASS [   1.196s] (1797/6817) moq-gst source::imp::session_tests::a_protocol_violation_posts_a_bus_error_after_playback
        PASS [   7.066s] (1798/6817) moq-audio encode::producer::tests::hls_stops_the_segment_where_retained_history_expired
        PASS [   2.367s] (1799/6817) moq-ffi test::shutdown_times_out_on_an_unfinished_track
        PASS [   2.157s] (1800/6817) moq-gst source::imp::session_tests::a_refused_redial_posts_a_bus_error_after_playback
        PASS [   2.287s] (1801/6817) moq-gst source::imp::session_tests::a_closed_publisher_session_resumes_on_the_next_start
        PASS [   7.061s] (1802/6817) moq-cli::integration report::import_reports_count_what_each_rendition_wrote
        PASS [   3.247s] (1803/6817) moq-gst source::imp::session_tests::a_restart_renders_through_a_synced_sink
        PASS [   5.871s] (1804/6817) moq-gst source::imp::session_tests::relay_loss_reconnects_after_refused_redials
        PASS [  10.277s] (1805/6817) moq-ffi test::session_state_names_a_failing_first_dial
────────────
     Summary [  15.550s] 1805/6817 tests run: 1804 passed, 1 failed, 114 skipped
        FAIL [   0.108s] (1782/6817) moq-gst source::imp::session_tests::restarts_under_reconnects_interleave_on_one_pad
  stdout ───

    running 1 test
    test source::imp::session_tests::restarts_under_reconnects_interleave_on_one_pad ... FAILED

    failures:

    failures:
        source::imp::session_tests::restarts_under_reconnects_interleave_on_one_pad

    test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 170 filtered out; finished in 0.07s

  stderr ───

    thread 'source::imp::session_tests::restarts_under_reconnects_interleave_on_one_pad' (1245554) panicked at rs/moq-gst/src/source/imp.rs:3788:13:
    assertion `left == right` failed: [Pad("video_0"), Segment("video_0", 79000), Buffer("video_0", 1), Start("video_0"), Caps("video_0", "video/x-h264"), Segment("video_0", 94000), Buffer("video_0", 2), Buffer("video_0", 3), Buffer("video_0", 4), Start("video_0"), Caps("video_0", "video/x-h264"), Segment("video_0", 94000), Buffer("video_0", 2), Buffer("video_0", 3), Start("video_0"), Caps("video_0", "video/x-h264"), Segment("video_0", 62000), Buffer("video_0", 5), Buffer("video_0", 6), Buffer("video_0", 7), Start("video_0"), Caps("video_0", "video/x-h264"), Segment("video_0", 62000), Buffer("video_0", 5), Buffer("video_0", 6), Buffer("video_0", 7), Start("video_0"), Caps("video_0", "video/x-h264"), Segment("video_0", 73000), Buffer("video_0", 8), Buffer("video_0", 9), Buffer("video_0", 10), Start("video_0"), Caps("video_0", "video/x-h264"), Start("video_0"), Caps("video_0", "video/x-h264"), Segment("video_0", 2809000), Buffer("video_0", 11), Buffer("video_0", 12), Buffer("video_0", 13), Eos("video_0")]
      left: 1
     right: 0
    note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

warning: 5012/6817 tests were not run due to test failure (run with --no-fail-fast to run all tests, or run with --max-fail)
error: test run failed
mbx[cache]: object cache: 777 hits, 9 misses, 50 not looked up, 3 bypassed; 0 B downloaded, 0 B uploaded, 432.8 MiB stored locally
mbx[cache]: 1m 55s session; 27m 21s estimated compiler time avoided (summed across compilations)
mbx[cache]: 50 compilations had no usable prior inputs or matching prediction for a lookup
mbx[cache]: bypass reasons: 3 unsupported-crate-type
mbx[cache]: Cargo artifact reuse and CI cache archive transfers are not included above
mbx[savings]: 816567 compilations on file. the box remembers.
error: recipe `test` failed on line 294 with exit code 100
error: recipe `test-changed` failed on line 86 with exit code 100
##[endgroup]
##[error]ci: test rs failed
##[error]Process completed with exit code 100.
##[group]Run actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9
with:
  name: cargo-timings
  path: target/cargo-timings/cargo-timing-*.html
  if-no-files-found: ignore
  retention-days: 14
  compression-level: 6
  overwrite: false
  include-hidden-files: false
  archive: true
env:
  BASE: 20955c09e94ae2dcc579060365f6dcee2855a328
  CHECK_FAIL_FAST: 1
  NEXTEST_TEST_THREADS: 24
  MBX_CACHE_LINKS: 1
  MBX_REMOTE_URL: 
##[endgroup]
With the provided path, there will be 1 file uploaded
Artifact name is valid!
Root directory input is valid!
Uploading artifact: cargo-timings.zip
Beginning upload of artifact content to blob storage
Uploaded bytes 69996
Finished uploading artifact content to blob storage!
SHA256 digest of uploaded artifact is 7643f9635dbbd9f46e472050ecbccb000395e73b23a889e31d30374fc2ee0a16
Finalizing artifact upload
Artifact cargo-timings successfully finalized. Artifact ID 13174579057
Artifact cargo-timings has been successfully uploaded! Final size is 69996 bytes. Artifact ID is 13174579057
Artifact download URL: https://github.com/moq-dev/moq/actions/runs/41516535364/artifacts/13174579057
Post job cleanup.
##[start-action display=Run jdx/mr-boxington-action@47ce49b32025d93a7a2f4570815afaa2b7092240;id=__75c9f721-c833-4be0-b099-3bf6c7b94343.mbx]
##[end-action id=__75c9f721-c833-4be0-b099-3bf6c7b94343.mbx;outcome=skipped;conclusion=skipped;duration_ms=0]
Post job cleanup.
[command]/nix/store/lg46w7hrjx7kylsh6645c9l975i8h43b-git-2.55.0/bin/git version
git version 2.55.0
Temporarily overriding HOME='/var/cache/github-runner-work/_temp/fcd029b8-caeb-4e74-b5c6-86a8307b0fa1' before making global git config changes
Adding repository directory to the temporary git global config as a safe directory
[command]/nix/store/lg46w7hrjx7kylsh6645c9l975i8h43b-git-2.55.0/bin/git config --global --add safe.directory /var/cache/github-runner-work/moq/moq
Removing SSH command configuration
[command]/nix/store/lg46w7hrjx7kylsh6645c9l975i8h43b-git-2.55.0/bin/git config --local --name-only --get-regexp core\.sshCommand
[command]/nix/store/lg46w7hrjx7kylsh6645c9l975i8h43b-git-2.55.0/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :"
Removing HTTP extra header
[command]/nix/store/lg46w7hrjx7kylsh6645c9l975i8h43b-git-2.55.0/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader
[command]/nix/store/lg46w7hrjx7kylsh6645c9l975i8h43b-git-2.55.0/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :"
Removing includeIf entries pointing to credentials config files
[command]/nix/store/lg46w7hrjx7kylsh6645c9l975i8h43b-git-2.55.0/bin/git config --local --name-only --get-regexp ^includeIf\.gitdir:
[command]/nix/store/lg46w7hrjx7kylsh6645c9l975i8h43b-git-2.55.0/bin/git submodule foreach --recursive git config --local --show-origin --name-only --get-regexp remote.origin.url
A job completed hook has been configured by the self-hosted runner administrator
##[group]Run '/nix/store/gx2v6q84b0w43bafm0whkdx93qyxbpps-ci-cache-gc.sh/bin/ci-cache-gc.sh'
shell: /nix/store/1mv3qz005gkbalxfghk4y3s8ayv4d2dl-bash-interactive-5.3p15/bin/bash --noprofile --norc -e -o pipefail {0}
##[endgroup]
evicted 263 objects and 49 action results (794.4 MiB logical); 49.9 GiB logical remain
dropped 3 stale checkout records
dropped 8 session event streams
removed 0 learned incremental units and 2 checkout directories (3.7 KiB logical); 51 units (4.7 GiB logical) remain
evicted 35 objects and 5 action results (508.3 MiB logical); 49.8 GiB logical remain
removed 2 target directories (13.7 MiB logical, 2 abandoned and 0 live); 12.7 GiB logical remain
Cleaning up orphan processes

(written by quest land watch)

@moq-bot
moq-bot Bot added this pull request to the merge queue Nov 2, 2026
@moq-bot

moq-bot Bot commented Nov 2, 2026

Copy link
Copy Markdown
Contributor

codex / Codex land fixer note:

The queue failed in moq-gst source::imp::session_tests::restarts_under_reconnects_interleave_on_one_pad, asserting that no EOS was recorded (left: 1, right: 0). This test follows an in-process origin and does not exercise the congestion pacer changed here. With base merged into a detached checkout, both packages built and all 578 package tests passed, including the failing test; 2 tests were skipped. This appears to be an unrelated flaky test. The PR branch is unchanged.

(written by codex / Codex)

@moq-bot

moq-bot Bot commented Nov 2, 2026

Copy link
Copy Markdown
Contributor

quest land watch: re-enqueued this pull request at 4d1974a, at the back of the merge queue, unchanged, since the land fixer judged the failure flaky: cargo nextest run --locked --tests: moq-gst source::imp::session_tests::restarts_under_reconnects_interleave_on_one_pad.

(written by quest land watch)

Merged via the queue into main with commit 7059a71 Nov 2, 2026
9 of 10 checks passed
@moq-bot
moq-bot Bot deleted the quest/m1/pacer-low-rate branch November 2, 2026 21:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant