Skip to content

docs(moq-uring): the shared locked-memory budget is mostly other programs' rings - #8051

Merged
kixelated merged 4 commits into
mainfrom
quest/m0/test-flakes-2/uring-memlock
Nov 7, 2026
Merged

kixelated merged 4 commits into
mainfrom
quest/m0/test-flakes-2/uring-memlock

Conversation

@kixelated

@kixelated kixelated commented Nov 6, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

The quest asked for a test-side fix to io_uring ring setup failing with ENOMEM on the user's shared 8 MiB RLIMIT_MEMLOCK when parallel sessions run tests. Measurement shows the tests are a small share of that budget. The rest is held by other processes, and it grows with the number of agent sessions.

Measured 2026-11-06 on the maintainer's host (kernel 7.0, ulimit -Hl = 8192). Each ring's cost came from calling io_uring_setup until ENOMEM; holdings came from /proc/*/fdinfo SQ/CQ masks.

Measure Result
Ring cost SQE pages + (CQ ring + SQ array) pages. Production worker 256/2048 = 13 pages (52 KiB), unit-test worker 128/512 = 5 pages (20 KiB), libuv 256/512 = 7 pages (28 KiB)
Freed ring Its charge comes back asynchronously, about 10 to 20 ms after close on an idle host
Non-test holders 4.1 to 4.9 MiB of the 8 MiB: 118 Node.js rings (3.3 MiB), plus Electron apps. Each agent session's chrome-devtools-mcp holds 9 rings (about 250 KiB)
libuv Every event loop gets a 256-entry epoll-batching ring. UV_USE_IO_URING does not disable it (linux.c uv__use_io_uring: non-SQPOLL rings return 1)
One just rs test -p moq-uring -p moq-relay --features moq-relay/io-uring Peaks at about 0.3 to 0.5 MiB of live rings. With only my process's limit lowered to (held + headroom): passes with 1024 KiB of headroom, fails 66 to 113 tests with 256 or 512 KiB
Concurrency 3, 6, and 12 such runs started at once all passed at today's baseline. A full workspace run's test binaries peaked at 312 KiB
Reports moq-dev/moq#7313's single run, with 11 moq-uring failures, means the baseline was above about 7 MiB at that point. No test-side cap brings that back under 8 MiB

Approach

No code change. The PR records the measurement and rewrites the quest into what remains: a host-limit change only the maintainer can make (see Waiting).

  • rs/moq-uring/README.md: says that other programs (libuv rings) share the budget, gives the measured costs and a way to count rings, and names the user@.service LimitMEMLOCK drop-in.
  • quest/m0/test-flakes-2/uring-memlock.md: rewritten to [XS] The dev host's locked-memory limit fits parallel sessions, with the measurement, the options set aside, and a Waiting item for the maintainer.

Alternatives

Each one was measured and set aside:

  • Cap the io-uring nextest group at 1 thread (one-line change). A single run then passed with 512 KiB of headroom, where it had failed 113 tests. But just rs test --workspace --features moq-relay/io-uring went from 101 / 108 s to 153 / 149 s, because the group became the tail. max-threads = 2 still failed with 512 KiB, in one of two tries. Most of the group's ~45 s of serial time is three endpoint.rs tests that each wait out moq-quic's real 10 s handshake timeout.
  • Share one lock across invocations (flock slots). A test waiting for a slot counts against nextest's 60 s hang timeout, so this would add a new flake. It needs either nextest's experimental wrapper-scripts or a hook in every ring test, and plain cargo test would still bypass it.
  • Give integration and relay tests the small unit-test rings. This needs a public ring-geometry knob on moq_uring::Config plus relay plumbing, and the tests would stop exercising production geometry.
  • Plain cargo test is not covered. rs/AGENTS.md says to run tests through nextest, and none of this repo's recipes runs ring tests that way.

Impact

  • Public API: none. Wire: none.

Follow-ups

  • The three 10 s moq-uring endpoint tests (an_unanswered_dial_times_out, the_backlog_bounds_pending_handshakes, the_backlog_bounds_queued_connections) could set a short handshake_timeout in their test config. That would make them fast, and would make a smaller io-uring group cheap if one is still wanted after the host change.

Decision

Decided 2026-11-06 (maintainer, /quest-complete): ✅ raise the host limit and keep the quest as a host action (64 MiB hard RLIMIT_MEMLOCK via a user@.service.d LimitMEMLOCK=64M drop-in plus limits.d). Not chosen: cap the io-uring group at 1 thread (about +45 s per workspace check), or first make the 10 s endpoint tests fast; add a public ring-geometry knob. Recorded in the quest file. The host change needs root, so it stays there as a Maintainer: Waiting item, checked with ulimit -Hl in a fresh session.

Validation

  • quest check: 606 documents ok, after merging origin/main.
  • quest gate just ci origin/main on the merged head: every module passed except test rs, where io_uring tests in moq-relay failed ring setup with ENOMEM against this host's 8 MiB RLIMIT_MEMLOCK. That is the host problem this quest tracks. The PR changes only docs.

Review

  • quest review round 1 (Codex gpt-6.1-sol, medium) on f0b7075: no findings.

(Written by Claude Opus 5.5)

🤖 Generated with Claude Code

kixelated and others added 2 commits November 6, 2026 11:49
… on the host limit

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated kixelated changed the title io_uring tests fit the shared locked-memory budget docs(moq-uring): the shared locked-memory budget is mostly other programs' rings Nov 6, 2026
@kixelated
kixelated marked this pull request as ready for review November 6, 2026 21:53
@coderabbitai

coderabbitai Bot commented Nov 6, 2026

Copy link
Copy Markdown
Contributor

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 1 minute.

Check out review usage here.

View limit details

Limit details: You’ve used all 4 included reviews currently available.

Learn how review limits work.

Review configuration:

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 43d750d4-5328-464c-9bd9-c03b76147eb3
Commits

Reviewing files that changed from the base of the PR and between fe5a6fa and f0b7075.

Files selected for processing: 3

  • quest/m0/test-flakes-2/README.md
  • quest/m0/test-flakes-2/uring-memlock.md
  • rs/moq-uring/README.md
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@kixelated kixelated left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated review by review (OpenAI)

Reviewed head f0b7075980a63b4f530550e5ff5407b3cd1e46cd against base fe5a6fa7aecb437328a36360da272574f4a68d5c.

No actionable findings in the three changed documentation files. The documented ring geometry and four-test group match rs/moq-uring/src/worker.rs and .config/nextest.toml; sh/rs/nextest.sh already raises the local soft limit to the inherited hard limit and uses 64 MiB on hosted CI. The linked libuv source also confirms that its non-SQPOLL path does not consult UV_USE_IO_URING.

Direction: keeping the measured host-limit change as an explicit maintainer-owned Waiting item is consistent with this docs-only scope. This PR does not itself fix the host or prove that parallel tests pass after the limit change.

Verification limits: GitHub-only static review. I did not run quest check, just ci, host measurements, or a concurrent-test reproduction, and did not change host settings. At inspection, the Quest check passed and the Check job was still running. The timings and memory measurements remain author-reported; this is a COMMENT review, not a merge approval.

@kixelated
kixelated added this pull request to the merge queue Nov 6, 2026

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ℹ️ No critical issues — one factual precision fix on the libuv claim, inline.

Reviewed changes

Read the full three-file diff at f0b7075 and checked its claims against the tree and upstream sources.

  • rs/moq-uring/README.md: the "Parallel test runs" section now says other programs share the user's locked-memory budget, giving per-ring costs, the libuv ring behavior, a ring-count command, and a user@.service LimitMEMLOCK=64M drop-in.
  • quest/m0/test-flakes-2/uring-memlock.md: rewritten from "fix the tests" into [XS] The dev host's locked-memory limit fits parallel sessions, recording the 2026-11-06 measurement, the options measured and set aside, and a Maintainer: Waiting item.
  • quest/m0/test-flakes-2/README.md: the child line follows the new title.

The rest holds up: .config/nextest.toml still puts the group at max-threads = 4; all three named endpoint tests exist and wait out the 10 s handshake_timeout (rs/moq-quic/src/config/transport/mod.rs:296) while the worker's ring is live; sh/rs/nextest.sh raises to 64 MiB with sudo prlimit on hosted CI only and otherwise lifts the soft limit to the hard limit, so ulimit -Hl printing 65536 is the right Done check; and #7313's body does record the eleven ENOMEM moq-uring failures in one --no-fail-fast workspace run, so the "baseline above 7 MiB" inference is grounded.

Pullfrog  | Fix all ➔ | Fix 👍s ➔ | View workflow run | Using deepseek-v4.1-flash-1017 (free via Pullfrog for OSS) | 𝕏

Comment thread rs/moq-uring/README.md
Comment on lines +157 to +159
256-entry ring of 28 KiB, and it creates these without checking
`UV_USE_IO_URING`. A desktop running nine agent sessions held 4 to 5 MiB of an
8 MiB limit before any test started, mostly in Node.js tool servers. A

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

UV_USE_IO_URING=0 does drop this ring on released libuv below 1.50: uv__use_io_uring() consults the env var for every ring and uv__iou_init returns before io_uring_setup, so Node 20 (libuv 1.46) creates none of these rings when it is set. The unconditional return 1 for non-SQPOLL rings is only on v1.50.0 and later (the body links v1.x), so as written the sentence is false for anyone on that boundary and hides a remedy that still works there. The same sentence appears in quest/m0/test-flakes-2/uring-memlock.md:22-24.

Suggested change
256-entry ring of 28 KiB, and it creates these without checking
`UV_USE_IO_URING`. A desktop running nine agent sessions held 4 to 5 MiB of an
8 MiB limit before any test started, mostly in Node.js tool servers. A
256-entry ring of 28 KiB, and libuv 1.50 and later creates these without checking
`UV_USE_IO_URING`; earlier versions let `UV_USE_IO_URING=0` turn it off. A desktop
running nine agent sessions held 4 to 5 MiB of an

Merged via the queue into main with commit 1a1940c Nov 7, 2026
8 checks passed
@kixelated
kixelated deleted the quest/m0/test-flakes-2/uring-memlock branch November 7, 2026 00:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant