Repository navigation
quest: stop per-PR MIGRATE.md entries; four flakes join test-flakes-2 #8024
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
8 commits
Select commit
Hold shift + click to select a range
5f7e951
quest: stop per-PR MIGRATE.md entries; three flakes join test-flakes-2
kixelated 9b8bd9b
quest: drop the pool tick lead from pool-debt-flight
kixelated a0bfe4b
test(ffi): quarantine concurrent_ready_calls_all_wake under its own f…
kixelated 92e8e9c
test(py): stop pinning one cause for an endpoint close
pullfrog[bot] 64ace62
Merge remote-tracking branch 'origin/main' into chore/quest-complete-…
kixelated b52de16
test(py): quarantine the audio playout-delay test under its flake quest
kixelated 11478a9
Merge origin/main
kixelated d9ad5ca
AGENTS.md: the release cut is the one PR that edits MIGRATE.md
kixelated File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,53 @@ | ||
| # [S] The Python audio playout-delay test reads its frames | ||
|
|
||
| ## Goal | ||
|
|
||
| `test_audio_consumer_reports_its_playout_delay` in | ||
| `py/moq-rs/tests/test_local.py` reads both written frames from each of its | ||
| consumers in loaded runs and in the merge queue, fixed at its cause under | ||
| the [epic](/quest/m0/test-flakes-2/README.md)'s rules, never by raising its | ||
| 5-second wait. Done when the PR explains the stalled read and shows the run | ||
| that reproduced it now passing. | ||
|
|
||
| ## Plan | ||
|
|
||
| Added 2026-11-05 (maintainer, /quest-complete follow-ups), from the merge | ||
| queue entry for [moq-dev/moq#7951](https://github.com/moq-dev/moq/pull/7951) | ||
| at `04a53405db`, | ||
| [run 42145929336](https://github.com/moq-dev/moq/actions/runs/42145929336/job/127657266646) | ||
| (2026-11-06 01:51Z): `1 failed, 124 passed`. The first read timed out: | ||
|
|
||
| ```text | ||
| > assert isinstance(await asyncio.wait_for(anext(fixed), timeout=5.0), moq.audio.Frame) | ||
| moq-rs/tests/test_local.py:1630: | ||
| E TimeoutError | ||
| ``` | ||
|
|
||
| The consumer with the fixed 80 ms delay (`estimate=False`) never returned | ||
| its first frame within 5 s. The land fixer | ||
| ([note](https://github.com/moq-dev/moq/pull/7951#issuecomment-6674925814)) | ||
| found the test and its implementation unchanged by the PR, and the full | ||
| Python suite passed locally (125 tests). The PR landed on requeue. No other | ||
| failed run from 2026-11-03 to 2026-11-06 shows this test, and no other PR | ||
| comment names it. | ||
|
|
||
| It failed again on 2026-11-06 in the merge queue entry for | ||
| [moq-dev/moq#7904](https://github.com/moq-dev/moq/pull/7904) at `565d07a3f6` | ||
| (same TimeoutError). The land fixer found the PR unrelated and the full | ||
| Python suite passing locally, then stopped on the repeated failure. The test | ||
| is now skipped with `@pytest.mark.skip` pointing here; remove the skip with | ||
| the fix. | ||
|
|
||
| Lead, unverified: the test writes only two 20 ms Opus frames (40 ms) and | ||
| doesn't finish the track until after both reads. Any path that needs more | ||
| input before the first frame goes out can stall the read for good. Candidates | ||
| are the encoder's lookahead, the decoder's warmup trim, the fixed delay | ||
| waiting for 80 ms of buffered audio, or a subscription that isn't live when | ||
| the frames are written. Which one applies depends on timing, which fits a | ||
| failure that appears only under load. | ||
|
|
||
| Reproduce first: run the test repeatedly while `just check` loads the host, | ||
| and record whether the encoder published the frames and where the decoder | ||
| held them. Then fix whichever of the test or the decoder is wrong. | ||
|
|
||
| Public API: none. Wire: none. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,33 @@ | ||
| # [S] The FFI concurrent-ready test wakes both calls under load | ||
|
|
||
| ## Goal | ||
|
|
||
| `concurrent_ready_calls_all_wake` in `rs/moq-ffi/src/test.rs` passes | ||
| reliably in loaded runs, fixed at its cause under the | ||
| [epic](/quest/m0/test-flakes-2/README.md)'s rules, and its `#[ignore]` and | ||
| `sh/gh/test-inventory.allow` entry are removed. Done when the PR explains | ||
| the stall and shows the loaded run that reproduced it now passing. | ||
|
|
||
| ## Plan | ||
|
|
||
| Added 2026-11-06 (maintainer's standing rule that flakes join this epic, | ||
| `/quest-complete`). The `/quest-iterate` agent on | ||
| [moq-dev/moq#7981](https://github.com/moq-dev/moq/pull/7981) saw it time out | ||
| in a local `quest gate just ci` while draining the reader during the two | ||
| concurrent `ready` calls; it passed on unchanged `main`, and #7981 doesn't | ||
| touch moq-ffi. Quarantined in | ||
| [moq-dev/moq#8024](https://github.com/moq-dev/moq/pull/8024) with every | ||
| assertion kept. | ||
|
|
||
| Lead, unverified: the drain loop reads until both parked `ready` calls have | ||
| finished, but it checks them only between reads. If the reader takes the | ||
| last of the 25 frames before both calls finish (their wakes run on the FFI | ||
| runtime, which lags under load), the next `read_frame` waits on a group that | ||
| gets no more frames and never finishes, so it times out as "the read | ||
| stalled". Check whether that is the failure before choosing between waiting | ||
| on the calls once the window has room and fixing a lost wake in the group. | ||
|
|
||
| Reproduce first: run the test repeatedly with `--run-ignored all` beside | ||
| `just check --all` load under `quest gate`, and keep the failing output. | ||
|
|
||
| Public API: none. Wire: none. | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,67 @@ | ||
| # [S] moq-net's pool-debt flight test holds its bound in parallel runs | ||
|
|
||
| ## Goal | ||
|
|
||
| `pool_debt_sheds_a_flight_without_its_cached_group` in | ||
| `rs/moq-net/src/model/track.rs` keeps the pool under its `2 * POOL` bound | ||
| when it shares a process with the rest of moq-net's tests, fixed at its | ||
| cause under the [epic](/quest/m0/test-flakes-2/README.md)'s rules, without | ||
| loosening the bound. Its sibling `pool_debt_sheds_a_flight` failed beside it | ||
| in two of the runs below, so check it against the same cause. Done when the | ||
| PR explains the overshoot and shows the parallel run that reproduced it now | ||
| passing. | ||
|
|
||
| ## Plan | ||
|
|
||
| Added 2026-11-05 (maintainer, /quest-complete follow-ups). The test landed | ||
| in [moq-dev/moq#7944](https://github.com/moq-dev/moq/pull/7944) (merged | ||
| 2026-11-05 18:36Z). No CI or merge-queue run has failed it: CI runs nextest, | ||
| one process per test, and it passed there in every run checked (for example | ||
| [run 42104077314](https://github.com/moq-dev/moq/actions/runs/42104077314)). | ||
| Every failure came from land fixers running plain `cargo test`, where tests | ||
| share one process on parallel threads. Each passed alone, with | ||
| `--test-threads=1`, and under nextest: | ||
|
|
||
| - [moq-dev/moq#7937](https://github.com/moq-dev/moq/pull/7937) | ||
| ([note](https://github.com/moq-dev/moq/pull/7937#issuecomment-6667877930), | ||
| 2026-11-05 19:58Z): `frame 763: 8402340 bytes held`. | ||
| - [moq-dev/moq#7670](https://github.com/moq-dev/moq/pull/7670) | ||
| ([note](https://github.com/moq-dev/moq/pull/7670#issuecomment-6668545954), | ||
| 20:30Z): both pool-debt tests failed in parallel. Then | ||
| ([note](https://github.com/moq-dev/moq/pull/7670#issuecomment-6670516495), | ||
| 21:52Z) `cargo test --locked -p moq-net -p moq-archive --all-features` | ||
| failed twice: `frame 500: 8402340 bytes held`, then | ||
| `frame 523: 8402372 bytes held`. | ||
| - [moq-dev/moq#7951](https://github.com/moq-dev/moq/pull/7951) | ||
| ([note](https://github.com/moq-dev/moq/pull/7951#issuecomment-6670771027), | ||
| 22:06Z): `frame 500: 8402340 bytes held`. It failed again in that PR's | ||
| later validation | ||
| ([note](https://github.com/moq-dev/moq/pull/7951#issuecomment-6674925814)). | ||
| - [moq-dev/moq#7988](https://github.com/moq-dev/moq/pull/7988) | ||
| ([note](https://github.com/moq-dev/moq/pull/7988#issuecomment-6671969800), | ||
| 23:12Z, and again at | ||
| [00:05Z](https://github.com/moq-dev/moq/pull/7988#issuecomment-6672860745) | ||
| and [01:07Z](https://github.com/moq-dev/moq/pull/7988#issuecomment-6673858792)): | ||
| both pool-debt tests failed in parallel, holding about 8.4 MB. | ||
|
|
||
| The overshoot is always about 13.7 KB past the 8,388,608-byte bound, less | ||
| than one of the test's frames (8 MiB is about 500 of them). | ||
|
|
||
| Ruled out (pullfrog on | ||
| [moq-dev/moq#8024](https://github.com/moq-dev/moq/pull/8024#discussion_r4489748136)): | ||
| the pool's 100 ms access tick. `Inner::tick` moves only in `Pool::advance` | ||
| (`rs/moq-net/src/model/cache.rs`), reached from `Pool::poll`, and the test | ||
| builds a bare `Pool::new` that it never polls, so every access stamp is the | ||
| same tick plus its boost however the threads are scheduled. | ||
|
|
||
| Lead, unverified: the overshoot is a near-constant 13.7 KB, not a drifting | ||
| amount, so look at the per-write charge and shed path, at what a write | ||
| charges before the flight's debt is paid. Since the failures need parallel | ||
| threads, also rule out process-wide state shared between tests in one | ||
| binary, and any other time source the pool reads (its wall-clock epoch). | ||
|
|
||
| Reproduce first: run plain `cargo test -p moq-net --all-features` (not | ||
| nextest) on parallel threads until it fails, and record the frame and bytes | ||
| held. Then fix the cause. | ||
|
|
||
| Public API: none. Wire: none. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,80 @@ | ||
| # [S] io_uring tests fit the shared locked-memory budget | ||
|
|
||
| ## Goal | ||
|
|
||
| moq-relay's and moq-uring's io_uring tests stop failing ring setup with | ||
| `ENOMEM` on the user's shared `RLIMIT_MEMLOCK` when several test runs share | ||
| one host, as they do under parallel quest sessions. The fix works at the | ||
| cause, by bounding how much locked memory the tests take at once across | ||
| invocations or by sizing their rings to fit, not by raising the limit | ||
| blindly. Done when the PR shows concurrent runs that failed before now | ||
| passing on a host with the default 8 MiB limit. | ||
|
|
||
| ## Plan | ||
|
|
||
| Added 2026-11-05 (maintainer, /quest-complete follow-ups). | ||
|
|
||
| What [moq-dev/moq#7593](https://github.com/moq-dev/moq/pull/7593) (merged | ||
| 2026-11-02 17:04Z) did: it widened the `io-uring` nextest group in | ||
| `.config/nextest.toml` to every moq-relay test naming `uring` as a whole | ||
| segment, with `package(moq-uring)` already in it. It also made a refused | ||
| relay worker report its `RLIMIT_MEMLOCK` error through `Ready::wait` | ||
| instead of a timeout. It kept the group at `max-threads = 4`. Its quest | ||
| scoped out the rest on purpose: "The group caps rings only within one | ||
| nextest invocation; separate invocations still share the budget, as | ||
| `rs/moq-uring/README.md` documents, and this quest does not change that." | ||
| It gives no reason beyond that scope. A nextest group also does nothing | ||
| under plain `cargo test`, which the land fixers run. | ||
|
|
||
| So the budget is still shared by: | ||
|
|
||
| - separate nextest invocations from the same user (several checkouts or | ||
| sessions running `just check` at once), each allowed four ring tests; | ||
| - plain `cargo test`, which runs moq-uring's and moq-relay's ring tests in | ||
| parallel with no group at all. | ||
|
|
||
| `sh/rs/nextest.sh` raises hosted CI to 64 MiB, and no CI or merge-queue run | ||
| from 2026-11-03 to 2026-11-06 failed on it. The one io_uring failure in that | ||
| window, | ||
| [run 41736459964](https://github.com/moq-dev/moq/actions/runs/41736459964)'s | ||
| `embed::uring_custom_route_and_quic`, was the socket-ownership check | ||
| [moq-dev/moq#7823](https://github.com/moq-dev/moq/pull/7823) fixed. The | ||
| failures below are local runs, on hosts with an 8 MiB limit where the | ||
| report names it, and all but the last are on bases that include #7593: | ||
|
|
||
| - [moq-dev/moq#7951](https://github.com/moq-dev/moq/pull/7951) (PR body, | ||
| 2026-11-05): two `quest gate just ci` runs on `eb0c56312b` "failed only in | ||
| io_uring tests in moq-relay and moq-uring: `io_uring setup failed: Cannot | ||
| allocate memory ... RLIMIT_MEMLOCK (8192 KiB), shared by every process of | ||
| this user`". The third run passed unchanged. | ||
| - [moq-dev/moq#7375](https://github.com/moq-dev/moq/pull/7375) (PR body, | ||
| merged 2026-11-03): `quest gate just check` failed once each in | ||
| `bitrate_caps::uring_session_is_held_to_its_caps` and | ||
| `uring::tests::a_group_stopped_before_serving_joins`, both `ENOMEM` on the | ||
| 8 MiB limit. | ||
| - [moq-dev/moq#7313](https://github.com/moq-dev/moq/pull/7313) (PR body, | ||
| merged 2026-11-03): a moq-relay uring test in `just check`. Then 11 | ||
| moq-uring integration tests failed in a single | ||
| `just rs test --workspace --no-fail-fast`. One invocation stays within its | ||
| four-test group, so other processes of the same user must have held part | ||
| of the budget. | ||
| - [moq-dev/moq#7508](https://github.com/moq-dev/moq/pull/7508) (PR body, | ||
| merged 2026-11-03): `moq-uring::integration mtu::discovery_finds_a_larger_mtu` | ||
| with `Cannot allocate memory`. The PR doesn't say whether that run | ||
| included #7593. | ||
|
|
||
| Reproduce first: on a host with an 8 MiB limit, run two or three | ||
| `just rs test -p moq-uring` or relay `--features io-uring` invocations at | ||
| once (and one plain `cargo test`) until a ring setup fails. Then measure what | ||
| one ring test locks. `rs/moq-uring/src/worker.rs` counts 16 bytes per CQ | ||
| entry, but the registered buffer rings (`register_buf_ring_with_flags` in | ||
| `udp.rs`) may be most of it. Integration tests keep the production | ||
| 256/2048 geometry, while unit tests use 128/512. | ||
|
|
||
| Choose the fix from that measurement and prefer the simplest. Options | ||
| include serializing ring tests across invocations through a user-wide lock | ||
| that both nextest and `cargo test` respect, or sizing the test rings and | ||
| buffer pools so the worst-case concurrency fits. Raise the limit only with | ||
| a measured reason, not as the fix. | ||
|
|
||
| Public API: none. Wire: none. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.