From de3e4596825db7c49c7261f5e190eca528722f27 Mon Sep 17 00:00:00 2001 From: Pham Hong Vinh Date: Thu, 13 Aug 2026 00:22:54 +0700 Subject: [PATCH 1/3] docs: document fused-kernel performance in Nunchaku Lite guide --- docs/source/en/quantization/nunchaku.md | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/docs/source/en/quantization/nunchaku.md b/docs/source/en/quantization/nunchaku.md index ceb663336ef6..caf4c9b839b5 100644 --- a/docs/source/en/quantization/nunchaku.md +++ b/docs/source/en/quantization/nunchaku.md @@ -122,6 +122,21 @@ List each module you want to quantize under `svdq_w4a4` or `awq_w4a16`. A module } ``` +## Fused kernels + +The original [Nunchaku](https://github.com/nunchaku-ai/nunchaku) engine gets much of its speed from model-specific fused execution paths: the Q, K, and V projections are grouped into a single matmul with RMSNorm and RoPE applied as a fused epilogue, and the MLP runs with a fused GELU kernel. Nunchaku Lite instead runs the stock Diffusers model with generic quantized linear layers, so it does not include these fusions and cannot match the speed of the original engine. + +The table below shows what each fusion contributes in the original Nunchaku engine, measured with Flux Schnell on an RTX 5090. + +| Optimization | Latency with it OFF | Latency with it ON | Speedup | +| --- | --- | --- | --- | +| QKV+norm+RoPE fusion (combined) | 1.057 s | 0.834 s | **1.27×** | +| — grouping Q/K/V into one matmul, alone | 1.059 s | 1.034 s | 1.025× | +| — RMSNorm+RoPE epilogue fusion, on top of grouping | 1.034 s | 0.836 s | 1.236× | +| GELU-MLP fusion | 0.766 s | 0.722 s | **1.06×** | + +Without these fusions, the GPU launches each of these operations as a separate kernel, incurring some overhead. However, this overhead can be recovered using [torch.compile](#torchcompile). + ## torch.compile Nunchaku Lite kernels and quantized linear layers are compatible with [`torch.compile`](../optimization/fp16#torchcompile). From a98e5a048799ce29f47926aaa1a63b59be7ed0cb Mon Sep 17 00:00:00 2001 From: "Vinh H. Pham" Date: Thu, 13 Aug 2026 09:52:48 +0700 Subject: [PATCH 2/3] Apply suggestions from code review Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com> --- docs/source/en/quantization/nunchaku.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/source/en/quantization/nunchaku.md b/docs/source/en/quantization/nunchaku.md index caf4c9b839b5..7258959237e3 100644 --- a/docs/source/en/quantization/nunchaku.md +++ b/docs/source/en/quantization/nunchaku.md @@ -124,9 +124,9 @@ List each module you want to quantize under `svdq_w4a4` or `awq_w4a16`. A module ## Fused kernels -The original [Nunchaku](https://github.com/nunchaku-ai/nunchaku) engine gets much of its speed from model-specific fused execution paths: the Q, K, and V projections are grouped into a single matmul with RMSNorm and RoPE applied as a fused epilogue, and the MLP runs with a fused GELU kernel. Nunchaku Lite instead runs the stock Diffusers model with generic quantized linear layers, so it does not include these fusions and cannot match the speed of the original engine. +The original [Nunchaku](https://github.com/nunchaku-ai/nunchaku) engine gets much of its speed from model-specific fused execution paths. It combines the Q, K, and V projections with RMSNorm and RoPE, and uses a fused GELU kernel for the MLP. Nunchaku Lite instead uses the standard Diffusers model with generic quantized linear layers, so it does not include these fusions. -The table below shows what each fusion contributes in the original Nunchaku engine, measured with Flux Schnell on an RTX 5090. +The following measurements show the latency impact of these fusions in the original Nunchaku engine, measured with Flux Schnell on an RTX 5090. | Optimization | Latency with it OFF | Latency with it ON | Speedup | | --- | --- | --- | --- | @@ -135,7 +135,7 @@ The table below shows what each fusion contributes in the original Nunchaku engi | — RMSNorm+RoPE epilogue fusion, on top of grouping | 1.034 s | 0.836 s | 1.236× | | GELU-MLP fusion | 0.766 s | 0.722 s | **1.06×** | -Without these fusions, the GPU launches each of these operations as a separate kernel, incurring some overhead. However, this overhead can be recovered using [torch.compile](#torchcompile). +Without these fusions, the GPU launches the projection, normalization, rotary-embedding, and MLP operations as separate kernels, which adds launch overhead. `torch.compile` may reduce some of this overhead, although it does not necessarily reproduce the original engine's fused kernels. Benchmark the compiled pipeline for your model and workload. ## torch.compile From 9171f758a1b29e6ae3044d67e2fe31e0cd045fc4 Mon Sep 17 00:00:00 2001 From: "Vinh H. Pham" Date: Thu, 13 Aug 2026 09:56:47 +0700 Subject: [PATCH 3/3] Apply suggestion from @rootonchair --- docs/source/en/quantization/nunchaku.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/source/en/quantization/nunchaku.md b/docs/source/en/quantization/nunchaku.md index 7258959237e3..825eecdae810 100644 --- a/docs/source/en/quantization/nunchaku.md +++ b/docs/source/en/quantization/nunchaku.md @@ -126,7 +126,7 @@ List each module you want to quantize under `svdq_w4a4` or `awq_w4a16`. A module The original [Nunchaku](https://github.com/nunchaku-ai/nunchaku) engine gets much of its speed from model-specific fused execution paths. It combines the Q, K, and V projections with RMSNorm and RoPE, and uses a fused GELU kernel for the MLP. Nunchaku Lite instead uses the standard Diffusers model with generic quantized linear layers, so it does not include these fusions. -The following measurements show the latency impact of these fusions in the original Nunchaku engine, measured with Flux Schnell on an RTX 5090. +The following measurements show the per-step latency impact of these fusions in the original Nunchaku engine, measured with Flux Schnell on an RTX 5090. | Optimization | Latency with it OFF | Latency with it ON | Speedup | | --- | --- | --- | --- |