Repository navigation
quest: plan the passive-reader spin, retry intervals, and a real-clock test guard #8043
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,45 @@ | ||
| # [M] Tests can't wait on the real clock | ||
|
|
||
| ## Goal | ||
|
|
||
| A new test that waits on the real clock fails every run instead of flaking | ||
| 1 in 20 under load. Today's lints ban how a timer is spelled, not the clock | ||
| it runs on: `clippy.toml` and biome send every timer to `kio::clock` or | ||
| `@moq/clock`, whose `sleep`/`timeout` still wait in real time unless the | ||
| test installs a paused, simulated, or `Manual` clock. So a test passes every | ||
| lint and still proves "nothing happened" by waiting 200 ms of wall time, | ||
| which is how the `copy_fill` `idle()` flake landed | ||
| ([moq-dev/moq#7845](https://github.com/moq-dev/moq/pull/7845)) three days | ||
| after the lint. | ||
|
|
||
| The tests already waiting in real time are burned down in m1's | ||
| [Tests run on a mocked clock](/quest/m1/mock-clock-tests/README.md). | ||
|
|
||
| ## Plan | ||
|
|
||
| Decided 2026-11-06 (maintainer, /quest-complete, from an audit of every | ||
| test's clock): | ||
|
|
||
| - ✅ **JS fails a test when a real-clock timer fires**, with a scoped | ||
| opt-out, plus a `REVIEW.md` rule and a biome deny on the `Bun` global. Not | ||
| chosen: a grep check over changed lines (misses shared helpers); a review | ||
| rule alone (reviewers flagged #4039, #3641, #1717, and #7845, and the | ||
| waits landed anyway). | ||
| - ✅ **Rust spikes the same guard**, falling back to a grep check if the | ||
| spike fails. Not chosen: a grep check only; JS only for now. | ||
| - ✅ m0 for the guards, since each real-time wait is a future merge-queue | ||
| flake; the burn-down is m1. | ||
|
|
||
| This README's own work: restore the rule to `REVIEW.md`, for every language. | ||
| "Unit tests mock time instead of depending on wall-clock timing or sleeps" | ||
| sat in root `AGENTS.md` from #4286 until | ||
| [moq-dev/moq#7761](https://github.com/moq-dev/moq/pull/7761) moved it into | ||
| `quest context`, where online reviewers no longer see it; today only | ||
| `rs/AGENTS.md` and `rs/moq-net/AGENTS.md` say anything. The rule should say | ||
| what a fix looks like: a mocked clock or an awaited event, never a shorter | ||
| or longer real wait, and the same fix in every language's sibling test. | ||
|
|
||
| ## Required | ||
|
|
||
| - [JS tests fail when a real timer fires](/quest/m0/clock-guard/js.md) - `@moq/clock` fails a test whose real-clock timer fires, with today's offenders opted out | ||
| - [Spike a Rust real-timer guard](/quest/m0/clock-guard/rust.md) - try the same guard in `kio`, or fall back to a grep check for new real-clock tests |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,37 @@ | ||
| # [M] JS tests fail when a real timer fires | ||
|
|
||
| ## Goal | ||
|
|
||
| Under `bun test`, a test fails when a timer armed on the host clock fires, | ||
| unless the test opted out. That catches both ways a test waits on real | ||
| time, and leaves progress watchdogs alone: | ||
|
|
||
| - A negative proof by waiting (`Promise.race([x, sleep(50)])`, a helper | ||
| that returns `undefined` when its timeout runs out) fires its timer. | ||
| - A settle sleep fires its timer. | ||
| - A `timeout()` that settles first cancels its timer | ||
| (`js/clock/src/timer.ts`), so a watchdog on progress never fires on | ||
| success. | ||
|
|
||
| ## Plan | ||
|
|
||
| Decided 2026-11-06 (maintainer, /quest-complete); see the | ||
| [epic](/quest/m0/clock-guard/README.md). | ||
|
|
||
| - The check lives where host timers are armed (`js/clock/src/host.ts`) and | ||
| is switched on by a bun test preload, so production code is unaffected. | ||
| - A scoped opt-out (for example `hostTime()`) covers tests that need real | ||
| time: real sockets, devices, or processes. | ||
| - Land the guard first with today's offenders opted out. The audit counted | ||
| about 170 JS tests that prove nothing happened by waiting, sleep to | ||
| settle, or assert on elapsed time, mostly behind shared helpers | ||
| (`nextUni`, `nextStream`, `closeCode`, `settlesWithin`, `idle`, | ||
| `settle(20)`, fanout `take`/`flush`). The opt-out list is the burn-down | ||
| list for m1's [Tests run on a mocked clock](/quest/m1/mock-clock-tests/README.md). | ||
| - Add `Bun` to biome's `noRestrictedGlobals` and extend that override to | ||
| `test/**`, so `Bun.sleep` and raw host timers can't bypass the guard. | ||
| - Prove it: a test that waits 50 ms on the real clock fails, and one that | ||
| races `timeout()` against a promise that settles first passes. | ||
|
|
||
| Public API: none expected; `hostTime()` may be a test-only export of | ||
| `@moq/clock`. Wire: none. |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,31 @@ | ||
| # [M] Spike a Rust real-timer guard | ||
|
|
||
| ## Goal | ||
|
|
||
| Find out whether `kio` can fail a Rust test whose timer fires on the real | ||
| clock, as the [JS guard](/quest/m0/clock-guard/js.md) does, and land it if | ||
| so. Otherwise land a grep check that flags a new `#[tokio::test]` without | ||
| `start_paused` and without a stated real-I/O reason. | ||
|
|
||
| `clippy.toml` already bans `tokio::time::*` and `std::thread::sleep`, but | ||
| `kio::clock::sleep` is the approved replacement and waits in real time on an | ||
| unpaused runtime. Clippy can't scope a ban on it to tests, since production | ||
| code uses it. | ||
|
|
||
| ## Plan | ||
|
|
||
| Decided 2026-11-06 (maintainer, /quest-complete); see the | ||
| [epic](/quest/m0/clock-guard/README.md). ✅ Spike first, with the grep | ||
| check as the fallback. Not chosen: the grep check only; leaving Rust out. | ||
|
|
||
| - `kio` already knows whether a runtime is paused | ||
| (`rs/kio/src/clock/runtime.rs`). The open questions are how a test opts | ||
| out (real sockets, the ~150 legitimate real-I/O tests), and how to keep | ||
| the switch from leaking into the moq binaries that integration tests | ||
| spawn, whose QUIC timers fire for real. | ||
| - Std channels' `recv_timeout` also waits in real time; decide whether the | ||
| guard or a clippy entry covers it. | ||
| - Record the spike's outcome here before choosing; the guard lands with | ||
| today's offenders opted out, as in JS. | ||
|
|
||
| Public API: none expected. Wire: none. |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,33 @@ | ||
| # [S] A lite subscriber parks once only passive readers remain | ||
|
|
||
| ## Goal | ||
|
|
||
| A lite datagram subscription whose last demanding reader leaves while | ||
| passive readers remain parks until demand returns or the readers go, instead | ||
| of busy-looping. Today `DatagramState::Live` and `Shared` in | ||
| `rs/moq-net/src/lite/subscriber.rs` call | ||
| `producer.poll_priority(waiter, Some(*priority))`. Once only passive | ||
| consumers are left, the aggregate priority is `None`, so that call returns | ||
| `Ready(None)` on every pass. `close_unused()` returns false because the | ||
| passive slots remain, and the loop `continue`s, burning a core. | ||
|
|
||
| Non-goals: changing what a passive reader is, or the IETF subscriber, which | ||
| already tracks `Option<i8>` (`rs/moq-net/src/ietf/subscriber/datagrams.rs`) | ||
| and does not spin. | ||
|
|
||
| ## Plan | ||
|
|
||
| Decided 2026-11-06 (maintainer, /quest-complete, a follow-up of | ||
| [moq-dev/moq#8000](https://github.com/moq-dev/moq/pull/8000), first seen in | ||
| [moq-dev/moq#7959](https://github.com/moq-dev/moq/pull/7959)): ✅ m0, since a | ||
| relay can burn a core in shipped code. Not chosen: m1; m3. | ||
|
|
||
| - Reproduce it first in a test that fails on the spin: a relayed lite-07 | ||
| datagram copy whose demanding reader leaves while a lite-05/06 stream | ||
| subscriber's passive join, or a TRACK hold, remains. Assert progress by | ||
| event or poll count on a paused clock, not by wall time. | ||
| - Track the last priority as `Option<i8>`, as the IETF subscriber does, so a | ||
| `None` priority parks on the waiter instead of re-polling. The fix lands | ||
| for both `Live` and `Shared`. | ||
|
|
||
| Public API: none. Wire: none. | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,18 @@ | ||
| # [XS] lite-05 copy fill checks idleness without wall time | ||
|
|
||
| ## Goal | ||
|
|
||
| `copy fill: lite-05 fills below the live edge only what a reader's ranges | ||
| name` (`js/net/src/lite/copy_fill.test.ts`) passes reliably alone and in the | ||
| loaded suite. It proves no FETCH went out by waiting 200 ms of real time in | ||
| `idle()`, and fails about 1 in 20 on main. | ||
|
|
||
| ## Plan | ||
|
|
||
| Decided 2026-11-06 (maintainer, /quest-complete, a follow-up of | ||
| [moq-dev/moq#7670](https://github.com/moq-dev/moq/pull/7670)): ✅ fix it as | ||
| stated, as a child of this epic under its rules. Drive the wait with the | ||
| `Manual` clock the file's other cases already install, or assert on the event | ||
| that settles the copy, rather than lengthening the wait. | ||
|
|
||
| Public API: none. Wire: none. |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,47 @@ | ||
| # [XL] Tests run on a mocked clock | ||
|
|
||
| ## Goal | ||
|
|
||
| No test proves that nothing happened by waiting, sleeps to let things | ||
| settle, or asserts on elapsed real time. Each one runs on a paused, | ||
| simulated, or `Manual` clock, or awaits the event it means. About 300 tests | ||
| do one of these today, most of them behind a dozen shared helpers. Tests | ||
| that only wait for progress under a generous timeout, and tests on real | ||
| sockets, devices, or processes, stay on the real clock. | ||
|
|
||
| ## Plan | ||
|
|
||
| Decided 2026-11-06 (maintainer, /quest-complete, from an audit of every | ||
| test's clock): | ||
|
|
||
| - ✅ **Scope: negative proofs, settle sleeps, and elapsed-time | ||
| assertions.** Not chosen: only the first two; also the ~530 tests that | ||
| only wait for progress under a timeout, which flake little and would cost | ||
| a much larger migration. | ||
| - ✅ m1, while the [guards](/quest/m0/clock-guard/README.md) that stop new | ||
| ones are m0. | ||
|
|
||
| Rules every child keeps: | ||
|
|
||
| - Fix the shared helper first, then its callers. A helper that resolves | ||
| `undefined` when a timeout runs out becomes one that advances a mocked | ||
| clock past the deadline, or awaits the event. | ||
| - Never replace one real wait with a shorter or longer one, or with a | ||
| quiet window. | ||
| - Fix a test and its siblings in other languages together. | ||
| - Each JS child removes the opt-outs the | ||
| [JS guard](/quest/m0/clock-guard/js.md) added, so the guard covers the | ||
| converted tests. | ||
|
|
||
| The counts come from a heuristic scan on 2026-11-06; re-scan each area | ||
| before starting, since tests land and move. | ||
|
|
||
| ## Required | ||
|
|
||
| - [js/net tests run on a mocked clock](/quest/m1/mock-clock-tests/js-net.md) - about 120 js/net tests stop waiting on real time, through their shared helpers | ||
| - [Other JS tests run on a mocked clock](/quest/m1/mock-clock-tests/js-rest.md) - hang's `settle(20)`, publish's fanout helpers, and the rest of JS | ||
| - [moq-relay tests run on a mocked clock](/quest/m1/mock-clock-tests/rs-relay.md) - the relay's `timeout(..).is_err()` session proofs and settle sleeps | ||
| - [Transport crate tests run on a mocked clock](/quest/m1/mock-clock-tests/rs-transport.md) - moq-tokio, moq-quic, moq-sock, and moq-uring | ||
| - [Media crate tests run on a mocked clock](/quest/m1/mock-clock-tests/rs-media.md) - moq-audio, moq-video, moq-hls, moq-mux, and the bridges | ||
| - [Binding and CLI tests run on a mocked clock](/quest/m1/mock-clock-tests/rs-bindings.md) - moq-ffi, moq-c, moq-cli, and kio | ||
| - [Go and Python tests run on a mocked clock](/quest/m1/mock-clock-tests/go-py.md) - the Go wrapper's and Python packages' real-time waits |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,22 @@ | ||
| # [S] Go and Python tests run on a mocked clock | ||
|
|
||
| ## Goal | ||
|
|
||
| The Go wrapper's and Python packages' tests stop proving nothing happened by | ||
| waiting, and stop sleeping to settle. See the | ||
| [epic](/quest/m1/mock-clock-tests/README.md) for the rules. | ||
|
|
||
| ## Plan | ||
|
|
||
| As of 2026-11-06: | ||
|
|
||
| - Go: `go/wrapper/moq_test.go:1477` (`untilWanted` must hit a 50 ms | ||
| deadline), `:1283` (a 5 ms context), the poll sleeps at `:1570` and | ||
| `reconnect_test.go:217`, and `internal/bridge/call_test.go:87`. Go has no | ||
| fake clock yet; prefer awaiting the event over adding one. | ||
| - Python: `py/moq-rs/tests/test_local.py:137`, `:1482`, and `:1516` each | ||
| sleep, then assert a task isn't done. `moq.testing.Clock` exists, but only | ||
| its own tests use it. Also `py/moq-ffi/tests/exit_inside_callback.py:49` | ||
| and `demo/carla/sim/tests/test_world.py:960`, `:969`. | ||
|
|
||
| Public API: none. Wire: none. |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,32 @@ | ||
| # [L] js/net tests run on a mocked clock | ||
|
|
||
| ## Goal | ||
|
|
||
| About 120 `js/net` tests stop proving nothing happened by waiting, sleeping | ||
| to settle, or asserting on elapsed time. See the | ||
| [epic](/quest/m1/mock-clock-tests/README.md) for the rules. | ||
|
|
||
| ## Plan | ||
|
|
||
| Start with the shared helpers, which cover most of them: | ||
|
|
||
| - `nextUni` (`src/ietf/publisher.test.ts`) and `nextStream` | ||
| (`src/ietf/subscriber.test.ts`) resolve `undefined` when their timeout | ||
| runs out. | ||
| - `closeCode` (`src/ietf/malformed.test.ts`) races `Bun.sleep(50)`. | ||
| - `settlesWithin` (`src/lite/track_stream.test.ts`). | ||
|
|
||
| Then the rest, by file, as of 2026-11-06: `ietf/publisher` 31, | ||
| `ietf/subscriber` 22, `lite/publisher` 19, `ietf/malformed` 10, | ||
| `ietf/join` 9, `lite/tail` 8, `lite/track_stream` 7, `ietf/goaway` 2, and | ||
| one each in `auth`, `connection/connect`, `connection/reload`, | ||
| `ietf/datagram`, `ietf/request_stream`, `ietf/tail`, `lite/connection`, | ||
| `tail`, and `track`. The `copy_fill` `idle()` flake belongs to | ||
| [test-flakes-2](/quest/m0/test-flakes-2/copy-fill-idle.md); this quest | ||
| converts the file's other cases. | ||
|
|
||
| Public API: none. Wire: none. | ||
|
|
||
| ## Required | ||
|
|
||
| - [JS tests fail when a real timer fires](/quest/m0/clock-guard/js.md) - `@moq/clock` fails a test whose real-clock timer fires, with today's offenders opted out |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,25 @@ | ||
| # [M] Other JS tests run on a mocked clock | ||
|
|
||
| ## Goal | ||
|
|
||
| The JS tests outside `js/net` that wait on real time move to a mocked clock | ||
| or an awaited event. See the [epic](/quest/m1/mock-clock-tests/README.md) | ||
| for the rules. | ||
|
|
||
| ## Plan | ||
|
|
||
| As of 2026-11-06: | ||
|
|
||
| - `js/hang/src/container/consumer*.test.ts`: 36 tests, most through | ||
| `settle(20)` in `consumer.test.ts`. Fix the helper first. | ||
| - `js/publish/src/fanout.test.ts`: 9 tests through its `take` and `flush` | ||
| helpers. | ||
| - `js/clock/src/index.test.ts` (2), `js/json/src/snapshot/snapshot.test.ts`, | ||
| `js/signals/src/index.test.ts`, and | ||
| `test/audio-quality/clients/js/src/capture.test.ts` (2). | ||
|
|
||
| Public API: none. Wire: none. | ||
|
|
||
| ## Required | ||
|
|
||
| - [JS tests fail when a real timer fires](/quest/m0/clock-guard/js.md) - `@moq/clock` fails a test whose real-clock timer fires, with today's offenders opted out |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,24 @@ | ||
| # [M] Binding and CLI tests run on a mocked clock | ||
|
|
||
| ## Goal | ||
|
|
||
| The FFI, C, CLI, and `kio` tests that wait on real time move to a paused or | ||
| simulated clock, or an awaited event. See the | ||
| [epic](/quest/m1/mock-clock-tests/README.md) for the rules. | ||
|
|
||
| ## Plan | ||
|
|
||
| As of 2026-11-06: | ||
|
|
||
| - moq-ffi, 10: `src/test.rs` 8, `src/test/rung.rs`, and | ||
| `src/bandwidth.rs`. | ||
| - moq-c, at least 6 in `src/test.rs`, including the 50 ms proof in | ||
| `States::recv_terminal`, which the scan undercounted. | ||
| - moq-cli, 9: `tests/integration/export.rs` 3, `auth_serve.rs` 2, | ||
| `listen.rs`, `src/archive.rs` 2, and `src/main.rs`. | ||
| - kio, 3: `src/clock/sim.rs` 2 and `src/channel.rs`. | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. These three kio entries already meet the epic's goal: Technical details# Inventories include tests the epic's scope excludes
## Affected sites
- `quest/m1/mock-clock-tests/rs-bindings.md:19` — kio: `src/clock/sim.rs` 2 are simulator tests (`rs/kio/src/clock/sim.rs:831`, `:848` run inside `run(async { .. })`), and `src/channel.rs` is the real-thread `until` hang guard (`rs/kio/src/channel.rs:512-516`).
- `quest/m1/mock-clock-tests/rs-transport.md:18-19` — moq-quic: `src/connection/mtud.rs` 4 and `src/tests/characterize/endpoint.rs` 2 are plain `#[test]`s passing explicit `Instant` arguments (e.g. `mtud.rs:544`, `:578`); they take `Instant::now()` as an arbitrary base and never wait.
- `quest/m1/mock-clock-tests/rs-media.md:16` — `src/capture/wgc.rs` and `src/frame/cuda_test.rs` drive a real device or encoder, which the same file's closing paragraph excludes.
## Required outcome
- Each area list names only real-clock negative proofs, settle sleeps, and elapsed-time assertions.
- Tests already on a simulated clock, explicit instants, real threads, or real devices leave the lists (or are marked as left alone).
## Suggested approach
- Re-derive the lists with the classifier restricted to the three agreed categories, then drop the entries above. |
||
|
|
||
| Fix each FFI test together with its Python, Go, Swift, and Kotlin siblings | ||
| where they exist. | ||
|
|
||
| Public API: none. Wire: none. | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,26 @@ | ||
| # [M] Media crate tests run on a mocked clock | ||
|
|
||
| ## Goal | ||
|
|
||
| The media and bridge crates' tests that wait on real time move to a paused | ||
| or simulated clock, or an awaited event. See the | ||
| [epic](/quest/m1/mock-clock-tests/README.md) for the rules. | ||
|
|
||
| ## Plan | ||
|
|
||
| As of 2026-11-06: | ||
|
|
||
| - moq-audio, 9: `tests/integration/playback.rs` 5, `src/aec.rs` 2, and | ||
| `src/playback/driver.rs` 2. | ||
| - moq-video, 8: `src/decode/consumer.rs` 3, `src/encode/producer.rs` 3, | ||
| `src/capture/wgc.rs`, and `src/frame/cuda_test.rs`. | ||
| - moq-hls, 8: `src/export/mod.rs` 7 and `src/import.rs`. | ||
| - moq-mux, 5: `src/rate.rs`. | ||
| - moq-rtc 3 (`src/lib.rs`, `src/server/whep.rs`), moq-rtsp 3 | ||
| (`src/client.rs`), moq-srt 1 (`src/server.rs`), and moq-transcode 1 | ||
| (`src/lib.rs`). | ||
|
|
||
| Tests that drive real devices or encoders keep real time where the device | ||
| needs it, and say so. | ||
|
|
||
| Public API: none. Wire: none. |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,21 @@ | ||
| # [M] moq-relay tests run on a mocked clock | ||
|
|
||
| ## Goal | ||
|
|
||
| moq-relay's tests stop proving a session stayed open with | ||
| `timeout(200ms, session.closed()).is_err()` on the real clock, and stop | ||
| sleeping to settle. See the [epic](/quest/m1/mock-clock-tests/README.md) | ||
| for the rules. | ||
|
|
||
| ## Plan | ||
|
|
||
| As of 2026-11-06, 28 tests: `tests/integration/auth_lifetime.rs` 9, | ||
| `session_revalidate.rs` 4, `public_rules.rs` 3, `goaway_cluster.rs` 2, | ||
| `runtime_uring.rs` 2, `tenant_matrix.rs` 2, `web.rs` 2, and one each in | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
|
||
| `cluster_unknown.rs`, `socket_auth.rs`, `src/auth.rs`, and `src/session.rs`. | ||
| Many run over real sockets: assert on the event the relay emits (a | ||
| revalidation, a close reason), or drive the deadline with the | ||
| [simulator or a `Manual` clock](/quest/m0/test-flakes-2/README.md), rather | ||
| than waiting out a window. | ||
|
|
||
| Public API: none. Wire: none. | ||
Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This mechanism drops the
close_unused()path. With only passive readers left the aggregate priority isNone, sopoll_priority(waiter, None)staysPendingwhen the last reader leaves and nothing re-checks whether the subscription is unused, which strands the upstream subscription rather than cancelling it — the goal's "or the readers go" half.Technical details