[kixelated]

Problem

Claims the cluster spike: measure a relay per worker thread, each with its own Cluster and origin and nothing shared, clustered over the in-memory transport, against today's shared origin. Its numbers and a recommendation go in the kio channel contention epic.

Approach

The prototype is moq-dev/moq@ddd8b04. The next commit removes it, so this PR merges only the epic README update and the quest deletion, as moq-dev/moq#7057 did for its spike.

• SPIKE_CLUSTER=1 with runtime.workers: each tokio worker builds its own Cluster (origin, cache pool, origin driver) on its thread. The workers link in a full mesh of moq-lite-06 memory sessions that publish and subscribe like a dialed cluster peer (Cluster::run_remote_session), so announcements, routes, and groups all cross workers through moq-lite.
• just bench-profile lock for fanout and chat at 1, 4, and 16 workers, both arms on the same build in the same session. Two pinned passes, a third for pinned chat base, and one unpinned pass. The bench gained SPIKE_PERF_STAT (perf stat instructions and cycles, user and kernel, both core types, over the same 5 s steady window as the lock capture), BENCH_PIN, and SPIKE_PERF_BUFFER.
• rs/moq-relay/examples/spike_announce.rs meshes W origin threads the same way to measure what the generator cannot: announce propagation delay, announce table memory (counting allocator), burst convergence, and the frame delay one hop adds.

The host stayed loaded (load average 15 to 25, other agents benchmarking). So instructions per delivered frame rank the arms, and wall-clock numbers only corroborate them.

Findings

Full tables are in the README diff. Thousand instructions per delivered frame, base to cluster (medians of passes):

shape	4 workers	16 workers	16 workers, unpinned
chat	219 to 185 (-15%)	236 to 190 (-20%)	309 to 191 (-38%)
fanout	204 to 200 (-2%)	255 to 244 (-4%)	221 to 244 (+10%)

• Lock wait: zero. Every cluster run measured 0 to 2 ms, against base's 2.1 to 3.2 s for chat and 0.67 to 0.77 s for fanout at 16 workers.
• Where the saving comes from: kernel (futex) instructions and off-CPU wait.
• Per-hop cost: about 140 thousand user instructions per one-frame group. Fanout spreads its 64 subscribers over 15 hops, four readers each, so user instructions rise 19% there. Chat's 2,000 subscribers amortize the hop to nothing.
• CPU and memory: chat relay CPU, pinned, fell from 1.8 to 2.4 cores to 1.2 to 1.3 at 16 workers. Resident memory rose 14 to 20 MiB at 16 workers.
• Hop latency (in process): p50 75 to 175 us and p99 0.26 to 1.6 ms, against 14 to 44 us on the writing worker. The generator's glass-to-glass p50 didn't move at its 1 ms resolution.
• Control plane: announce state grows with W squared. Per broadcast, heap goes from 4.9 KB with one origin to 45 KB at 4 workers and 886 KB at 16, and allocations from 64 to 14,600. Announce delay p99 reaches about 1.2 ms at 16 workers, and 1,000 broadcasts take 0.5 to 0.9 s to converge. At 16 workers, 10,000 broadcasts would hold about 9 GB of announce state, where a shared origin holds 49 MB.

Findings: the data plane wins where a track fans out widely per worker (chat), and breaks even or loses where it doesn't (fanout). Sharing nothing through today's full mesh then costs announce state and propagation that grow with W squared.

Decisions

Decided (maintainer, 2026-11-01). Does the cluster spike count as a win for /quest-plan?

• (a) No-win as a design. Let the replica spike (claimed in moq-dev/moq#7533) decide sharding, using these data-plane numbers as its upper bound.
• ✅ (b) A conditional win. Plan relay-per-thread with a mesh that never re-announces peer routes to peers, which cuts announce state to about one route per worker.
• (c) No-win, and restore the no-sharding rule now.

The epic README records the verdict and its condition, and notes:

• It changes cluster routing semantics: today a relay re-announces every route it learned to every other peer, with hop chains preventing loops.
• The cost is unmeasured: the spike measured only the full mesh.
• The replica spike (moq-dev/moq#7533) has since landed as its own conditional win. It chases the same data-plane win, which these numbers bound, at a different control-plane cost; weighing the two is for the relay-per-thread planning.

The relay-per-thread quest is not written here; it is planned with the maintainer through /quest-plan (see Suggested follow-ups).

Merge with main (2026-11-02)

Merged origin/main (945aec5a), which had landed the version counter (moq-dev/moq#7532) and replica spike (moq-dev/moq#7533) reports in the same epic, and moq-dev/moq#7648's ## Waiting and P2P edits elsewhere. The report still applies: the cluster spike quest was still open on main, the replica report says to weigh it against this one, and the cluster code still re-announces every learned route with hop chains.

• The cluster spike section follows the version counter and replica spike sections, in date order.
• The replica report's link to the deleted spike-cluster.md now points to the cluster spike section below it. The verdict's "replica spike still runs" bullet now says it has reported.
• The verdict's open question (how outside routes reach every worker without re-announce) now cross-references Routes and announces' planned per-connection proxy preference. A worker link with it off is the condition. Nothing about it is decided here.
• The route-memory paragraph links Cluster route memory, already planned from this PR's follow-ups in moq-dev/moq#7568.
• Log adoption is now unblocked. main had already dropped its replica spike link, so removing the cluster spike link empties its Required. That follows the 2026-10-14 decision ("Log adoption waits on the spikes"). Its Plan now records that neither conditional win settles whether the shared model survives, and that adoption is likely abandoned if relay-per-thread lands.

Choices made without asking

• Topology: a full mesh, like a relay's static peer list. The lower-indexed worker dials and links speak moq-lite-06 with the memory transport's default 1 MiB stream buffer.
• Per-worker resources: each worker gets its own cache pool built from the same [cache] config, unbounded in the bench. Per-worker clusters keep stats disabled; the bench config enables none.
• Pinning: main passes pin workers (runtime.pin) for the instruction counts, plus one unpinned pass for the relay's default.
• perf buffer: lock captures ran with a 32 MiB perf buffer, because the default 2 MiB dropped events on pinned base chat (three refused runs).
• Unpinned base chat: its lock capture failed three times (perf "nonalternating switch records"). Its CPU and instructions come from a cpu-mode pass, so it has no lock-wait number.
• Env var name: the spike reads SPIKE_CLUSTER, not MOQ_SPIKE_CLUSTER, because the relay refuses unknown MOQ_ variables.

Suggested follow-ups

• Relay-per-thread with a no-re-announce worker mesh, planned through /quest-plan per the maintainer's 2026-11-01 decision above. It is not planned yet; the planning should weigh the replica spike's per-track replicas against it, and decide whether Log adoption waits on it.
• moq-bench latency: it reports latency over the whole lifetime (p99 about 1.4 s from startup in every run) at 1 ms resolution. A steady-window, sub-millisecond latency would let the bench see a hop. (uring-bench-followups notes the startup artifact.)
• Already planned in Cluster route memory: announce state per route in a full mesh, the in-memory hop's per-group cost, and bench-profile lock robustness.

Review

Codex (quest review), gpt-6.1-sol, medium effort, round 1 on head 8a1bd49c against origin/main 945aec5a: no findings. The online reviews on 8ccf8e86 (Pullfrog, OpenAI) reported none either. On 8a1bd49c, quest check --base origin/main (dev shell) and just check origin/main both passed.

🤖 Generated with Claude Code

(Written by Claude Opus 5.5)

[coderabbitai]

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 4 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used all 4 included reviews currently available.

Learn how review limits work.

Review configuration:

Run configuration

• Source mode: Auto
• Repository: Unavailable
• Branch: Unavailable
• File: No configuration file from this repository applied
• Commit: Unavailable
• Layers: Organization UI
• Review profile: CHILL
• Plan: Advanced
• Run ID: 98b46df3-5322-431c-af45-b52f98b87610

Commits

Reviewing files that changed from the base of the PR and between 945aec5 and 8a1bd49.

Files selected for processing: 3

• quest/m2/perf/kio-channel-contention/README.md
• quest/m2/perf/kio-channel-contention/log-adoption.md
• quest/m2/perf/kio-channel-contention/spike-cluster.md

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

• X
• Mastodon
• Reddit
• LinkedIn

CodeRabbit Verify

• Run CodeRabbit Verify to test the changes in this pull request.

[pullfrog]

✅ No new issues found.

Reviewed changes

• Epic README records the cluster spike. Adds the "Cluster spike (2026-11-01)" section: data-plane table (chat/fanout at 1/4/16 workers, pinned and unpinned), lock wait dropping to zero, CPU and resident-memory deltas, per-hop instruction cost, and the spike_announce control-plane table showing W-squared announce state and convergence.
• Verdict and conditions. Records the maintainer's conditional win: relay-per-thread proceeds only with a worker mesh that never re-announces peer routes, which shrinks announce state to about one route per worker, and notes that this changes cluster routing semantics and is unmeasured.
• Quest deleted and links dropped. Removes spike-cluster.md (spike reported; prototype at ddd8b04 never merged) and its bullets from both the epic Required list and log-adoption.md.

I checked the tree for dangling references and cross-checked the figures against the PR body and the follow-up quest quest/m2/perf/cluster-route-memory.md; they agree (886 KB at 16 workers = 240 routes x ~3.7 KB; per-worker per-broadcast 5.4/11.3/25.3/55.4 KB at 2/4/8/16). Documentation-only, no behavioral surface.

  ^{｜ View workflow run ｜ Using deepseek-v4.1-flash (free via Pullfrog for OSS) ｜ 𝕏}

[kixelated]

Automated review by review (OpenAI)

Reviewed commit: 8ccf8e86052595771dc9c1bcf0d592bafec79ffe.

No actionable bugs found in the three-file quest-only diff. Keeping the prototype out of the merged tree is the simpler, appropriate scope. The report separates the measured full mesh from the unmeasured no-re-announce design and preserves the shared-control-plane replica comparison; external-route propagation remains explicitly a planning question, not an established property. No runtime, public API, or wire change lands here.

Verification: static inspection of the changed quests, related replica plan, cluster session setup, and linked prototype. I did not rerun the benchmarks or tests, independently verify the reported measurements, or validate the proposed replacement mesh.

(Written by OpenAI)

[kixelated]

Automated review by review (OpenAI)

Reviewed commit: 8a1bd49

No new actionable issues found. Accounting for the merged replica and version-counter reports, the reconciliation keeps both conditional wins and the unmeasured routing trade-off explicit, and removes the completed spike dependency.

Static GitHub review only; I did not rerun measurements or tests.

[pullfrog]

✅ No new issues found.

Reviewed changes

• Merged origin/main and reconciled the epic README with the reports that landed there: the landed version-counter and replica-spike sections are absorbed, and the ## Required list drops to Log adoption.
• Cross-referenced the no-re-announce condition to the planned per-connection proxy preference in /quest/m1/cluster-routing/routes.md, and linked the already-planned cluster-route-memory.md follow-up.
• Rewrote the spike references in the epic README and log-adoption.md to past tense, recording that neither conditional win settles adoption.

The reconciliation is clean: no dangling spike-cluster/spike-replica/version.md references, every linked quest path resolves, no conflict markers, and both cited PRs (#7532, #7533) are merged. Documentation-only, no behavioral surface.

  ^{｜ View workflow run ｜ Using deepseek-v4.1-flash (free via Pullfrog for OSS) ｜ 𝕏}