Conversation
… caveat Ran the expert cache on Qwen3.8-Flash-Next (qwen4exp, 177B, 512 experts/layer) on a single RTX 3090: +34.5% decode for 9.2 GB, where moving whole layers with --n-cpu-moe instead spent 12.5 GB to buy 10%. Three things a downstream user would want in the guide: 1. After syncing past ggml-org ebb546b (ggml-org#26802), the pack chains trip GGML_ASSERT(ggml_cuda_mul_mat_id_needs_sync) on the first decode. op_params[0]=1 correctly disqualifies both fast paths, but the new predicate predates the flag and answers 'no sync needed' for a small quantized batch. Two-line fix included; not a code PR because this branch has no needs_sync to patch. 2. On qwen4exp the output is NOT bit-identical to baseline -- it diverges at temperature 0, consistent with float non-associativity between one fused chain and two summed pack chains. Quality is unaffected as far as a proper test can see: paired over 64 chunks, dPPL +0.047% (t=+0.88) against that test's own 0.107% detection floor. 3. --moe-cache-slots without --moe-cache-profile is SILENT. Added to the troubleshooting table: tok/s reads as a small regression and VRAM is the only tell.
github-actions
Bot
added
the
documentation
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Comment |
The first draft cited only Pride and Prejudice, and that corpus is a poor discriminator: baseline PPL 1.0594 means the model is near-certain of every token, so there is little room for a defect to show. Re-ran the whole comparison on technical prose (NASA SP-4205 OCR, baseline PPL 8.2438). NASA cache-64 vs baseline -0.047% t -0.85 floor 0.112% Austen cache-64 vs baseline +0.113% t +1.68 floor 0.107% Neither detectable, and the SIGN FLIPS between corpora -- a real quality cost would keep its sign. Both rows shown rather than only the favourable one.
|
Updated the quality evidence — the first version cited only Pride and Prejudice, and that corpus turned out to be a poor discriminator: baseline PPL 1.0594 means the model is near-certain of every token, which we read as memorisation rather than a usable test. Re-ran the whole comparison on technical prose the model has not swallowed (NASA SP-4205 OCR, baseline PPL 8.2438):
64 paired chunks each, cached arm's peak VRAM 15,544 MiB against 6,286 for cache-off so it provably engaged. Neither delta is detectable — and the sign flips between corpora, which a real quality cost would not do. Both rows are in the diff rather than only the favourable one. So the claim I would stand behind for One near-miss worth passing on, since anyone comparing perplexity runs can hit it: mid-run I read the cached arm at chunk 38 (cumulative 9.29) against the other arms' final values (8.24) and it looked 12.7% worse. Cumulative perplexity is not comparable across different chunk counts. At matched indices the cached arm was the lowest of the three at every checkpoint. |
Hey, what fork did you use, GenerelSchwerz or thecodacus? I’ve been digging into your Qwen3.8-Flash-Next MoE-cache results from PR #10 and trying to reproduce the behavior on my side. Your 3090 result was: RTX 3090 24 GB My setup is very similar on the CPU/RAM side, but I’m using an RTX 3080 Ti 12 GB: Ryzen 5600X I first had trouble figuring out whether the cache was actually working. I instrumented init_moe_expert_cache() and confirmed that it is definitely active: 48/48 layers eligible But my performance goes in the opposite direction: Cache OFF: ~9.88 tok/s I also tested the two CUDA options independently: GGML_CUDA_REGISTER_HOST=1 and neither made a meaningful difference. At 32 slots I got roughly 8.0–8.2 tok/s regardless of those settings. In particular: Kind Regards |
|
Thank you for this — instrumenting First: our host, which the PR never namedYou wrote our result as Ryzen 5600X / 64 GB RAM. That's your box. Ours:
The PR says only "on a single RTX 3090" and never named the host, so you had no way to know. That omission is on us, and it's the reason the comparison looked like apples to apples. Second: the thing I think actually decides itA quarter of this model isn't experts at all. From our file: That's the PLE n-gram table — 51.2B parameters (160 × 320,001,536), 16 head vocabs of ~20M each, addressed by a hash of the last three tokens. It lives inside the GGUF shards, and it's 25.9% of our 103.7 GiB. It behaves nothing like the expert weights:
So the cache you're tuning cannot touch the n-gram path at all. Worse, uploading ~3.3 GB of experts at init pulls pages through the page cache and can evict exactly the n-gram rows that were warm — which fits the shape you measured, where the penalty appears immediately at 8 slots (9.88 → 7.78) rather than scaling with cache size the way pure VRAM pressure would. And on a 64 GB box the table cannot be resident alongside everything else. Worth stating plainly: no quant Unsloth publishes for this model fits in 64 GB — the smallest, UD-IQ1_S, is 67.6 GiB. On our box the whole 103.7 GiB sits in page cache with ~49 GiB to spare. Those are not the same experiment, and our +34.5% is partly a measurement of our RAM. What I'd actually try on your hardwareYou are roughly the twin of the box in this discussion — RTX 3080 Ti + 64 GB + NVMe. The approach there deliberately puts the n-gram table on the SSD and keeps prefetch off, because 2.7 KB per token is nothing for an NVMe once the kernel stops reading ahead around it: Three tiers: VRAM for active experts and KV, RAM for inactive expert layers, NVMe for the n-gram table. Reported there as +344% prefill and +167% generation over plain mmap, at 8–10 tok/s generation — which is already around where your cache-off number sits, so I'd expect the headroom on your box to be in prefill.
A one-minute check before any of that: run your 32-slot arm and watch Your four questions, straight
And yes, the quant differs, as you spotted — our falsifier says one card, one model, one quant ( (Aside, if you ever have the VRAM for it: SGLang has day-0 support for this model and handles the n-gram table properly — pinned host memory with async prefetch overlapped with compute, gathered via a Triton UVA kernel, rather than leaning on the page cache. vLLM has |
|
Thank you very much for the deep analysis and for taking the time to look into this so thoroughly. It’s been really helpful. In your opinion, do you think the performance results reported by Codacus are realistically within reach on the same hardware (RTX 3080ti 12GB), or do you think there is a fundamental hardware limitation that would make those numbers unrealistic? At the moment I’m getting around 11–12 tokens/sec, depending on the task. I’m mainly asking because I’d like to know whether it’s worth investing more time into trying to squeeze significantly more performance out of this hardware, or whether I’m already reasonably close to its practical limits. Thanks again! |
|
Short answer: I don't think 21.5 tok/s is reachable on a 12 GB card, and I think you're closer to its practical limit than not. But the reason is one specific number, and there's one cheap thing worth checking before you stop. First, the caveat that should colour everything below: we have no hardware resembling yours. No 12 GB card, no 64 GB host. Everything I say about your configuration is extrapolation from a machine with twice the VRAM and 2.4× the RAM, and I've been wrong twice already in this thread by reasoning past what I'd actually measured. What I measured today, on our boxI re-ran the PR's own pair on a current build, plus the SSD-offload approach I pointed you at last time, all with one client and 128-token greedy generations:
The PR's numbers reproduce (16.00 → 16.10, 21.52 → 22.47), so that part is solid. The number that decides your questionThe winning configuration used 15,896 MiB of VRAM — more than your card has in total. The +39.6% is the expert cache, and the cache is the VRAM. At 64 slots it costs ~9.2 GB on top of the ~6.6 GB the model needs. You measured 3.34 GB at 32 slots yourself, which lines up. On 12 GB, after weights, KV and compute buffers, you simply cannot hold the cache that produced that number. So it isn't a tuning gap — the configuration that gets 21.5 doesn't fit on your card. And the other two tiers are also against youYour GPU is not the problem, which surprised me: the 3080 Ti has ~97.6% of the 3090's CUDA cores and within ~3% of its memory bandwidth. It is essentially a 3090 with half the VRAM. Per-expert GPU compute on your card is as fast as ours. What differs is everything below the GPU. Measured on our host: And the one that I think actually caps you: UD-IQ3_XXS is 76.3 GiB against ~59.6 GiB usable on a 64 GB box. The model cannot be resident, so a share of every token's expert reads comes off the bottom tier. No quant Unsloth publishes fits — the smallest, UD-IQ1_S, is 67.6 GiB. Worth knowing: our two SSD-offload runs above never actually touched the SSD — the whole 103.7 GiB sat in page cache (105 GiB cached, confirmed in the logs). So 14–15 tok/s is that technique with disk latency removed entirely. It is a ceiling you won't reach, not a target. The one cheap thing worth checking firstConfirm What I'd actually do in your positionYou've gone 9.88 → 11–12 tok/s, which is real. My honest guess is that the ceiling on that machine is mid-teens, not low-twenties, and that further cache tuning has little left to give because the cache is VRAM-bound. If you want a step change, the upgrade that matters is RAM, not the GPU. Going to 128 GB would put the whole model in page cache and take the bottom tier out of the loop entirely — which is the single biggest difference between your box and ours, and it's the cheaper half of your machine. A bigger card would help too, but you'd be replacing a GPU that is already ~97% of ours. I'd stop optimising and either accept mid-teens or add RAM. Thanks for pushing on this — the comparison forced us to measure four configurations we'd never have run otherwise, and the finding that the hybrid/SSD route is worse than plain |
|
Follow-up, because the weakest part of my last answer was that we had no card like yours — so we simulated one. We held 12,800 MiB of the 3090 with a ballast process, leaving 11,055 MiB free, and re-ran the same config on the same host. A ballast rather than just configuring fewer slots, deliberately: fewer slots shows what a small cache costs, a ballast shows what happens when the allocator genuinely cannot get the memory. That's your card rather than your config.
Everything else identical to the 22.47 run. So the VRAM variable is now isolated: About 41% of the gap is the card and 59% is everything below it — dual-channel RAM, six cores, and a 76.3 GiB model that cannot be resident in 64 GB. The practical consequence, which I think is what you're actually weighing: a 24 GB card on its own would take you to roughly 14.4 tok/s, not 22. The host stays the larger term. And the mid-teens ceiling I guessed at last time is what we measured — 17.97 — on a host with 8-channel memory, 32 cores, and the whole model in page cache. You won't exceed that on a machine that pages from SSD. Two caveats, both of which cut in your favour:
So: if you want a step change it's RAM, and if you don't want to spend anything, you're nearer the practical limit than not. Either way the cache is worth keeping on — it's the single largest effect we measured anywhere in this thread (+39.6%), it's just VRAM-bound on 12 GB. |
2 participants
Ran your expert cache on Qwen3.8-Flash-Next (
qwen4exp, 177B, 512 experts/layer, 10+1 active) on a single RTX 3090 — five cherry-picks onto ggml-org PR ggml-org#27742 and it worked. Three things a downstream reader of the guide would want, all docs-only.1. A collision waiting on your next upstream sync. ggml-org
ebb546b7e(ggml-org#26802, 2026-08-11) addsggml_cuda_mul_mat_id_needs_sync()and asserts it at the singlemul_mat_idcaller. Yourop_params[0] = 1correctly disqualifies both fast paths, so the op reaches the sync path — but the new predicate predates the flag and answers "no sync needed" for a small quantized batch. First decode, every time:Two lines fix it, on the branch linked in the README text. Not sent as a code PR because
perfdoesn't carryneeds_syncyet — there's nothing for the patch to apply to. Ready whenever you take the sync.2. A caveat on "bit-identical". On
qwen4expthe cached output is not bit-identical — it diverges from baseline at temperature 0, coherent and on-topic, consistent with float non-associativity between one fused chain and two summed pack chains. Quality is unaffected as far as a proper test can tell: paired over the same 64 chunks, ΔPPL +0.047% (t = +0.88), below that test's own 0.107% detection floor, with the cached arm's peak VRAM confirming it engaged. Suggesting statistically indistinguishable rather than bit-identical for architectures beyond the three you measured — your claim may well hold exactly on those.3. The silent no-op, added to your troubleshooting table:
--moe-cache-slotswithout--moe-cache-profiledoes nothing, reports clean, and reads as a small regression in tok/s. VRAM is the only tell — it doesn't move a megabyte. Cost us an evening.The numbers
--n-cpu-moe 48, no cache--n-cpu-moe 48+--moe-cache-slots 64+ profile--n-cpu-moe 40(whole layers on GPU instead)The comparison we found most useful: moving whole layers onto the card spent 12.5 GB to buy 10%, the cache spent 9.2 GB to buy 34.5%. Oracle hit rates from
llama-moe-trace+ yoursimulate.py: 68.2% at 64 slots/layer, 88.0% at 128 (which OOMs on 24 GB).We went in expecting the routing to be too flat for a cache to help — 512 fine-grained experts with load balancing look designed against one, and Laguna measured barely-skewed. It isn't: only 51.5% of (layer, expert) pairs are ever used.
Falsifier on all of the above: one card, one model, one quant (unsloth
UD-Q4_K_XL), routing profile traced from a single prompt. Happy to adjust or drop any of it.