Repository navigation
sycl: 2-3x prompt processing speedup by dequantizing PTQ1_0 and PQ2_0 to FP16 - #278
Merged
Merged
Conversation
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DSrpF2XYi9dav1e3hSAkaS
dwymark
changed the title
dwymark
changed the title
Collaborator
|
Tested on an Intel Arc B390 (Panther Lake Xe3 iGPU), which is one generation newer than the 140V you measured on. Windows 11, oneAPI 2025.3 (Level Zero), default Correctness: Precision (
So the FP16 activation conversion is within run-to-run noise here. Speed, Ternary-Bonsai-2-27B,
This reproduces your 2.2–3.1× on Xe3, and decode is unaffected as expected. Caveat: the arm order was base then PR in both rounds, not flipped, but the gap is far outside the round-to-round spread. |
Author
|
Great, glad to hear it. Let me know if you'd like anything else on my end. Feel free to merge whenever you are happy with the contribution. |
Collaborator
|
Follow-up with the rest of the checks on the Arc B390 (same builds,
Looks good to me. |
This was referenced Sep 29, 2026
Draft
bri-prism
pushed a commit
that referenced
this pull request
Oct 2, 2026
* Adds an XMX (DPAS) path for PQ2_0 and PTQ1_0 matrix multiplies on Intel Xe2 GPUs (Battlemage, Lunar Lake). It covers every batch size: prompt processing, single-token decode, parallel decode and speculative verification. On Xe2 it replaces MMVQ and the dequantize-to-FP16 prompt path from #278 for these types. The main idea: PQ2_0 packs value+1 in 2-bit fields, lowest first, which as a little-endian dword is already the DPAS 2-bit operand layout. The kernel feeds the codes to DPAS as signed 2-bit (s2) against int8 activations, converting them in registers with a borrow-free per-field subtract. Weights are never widened to 8 or 16 bits in memory. How it works: - **Weight layout.** On first use, each PQ2_0 weight is rewritten in place into an SoA layout (same size): all 32-byte qs blocks first, then all fp16 scales. The qs row pitch is then nb*32 bytes, so one transposed 2D block load fetches 16 rows x 128 weights directly in the DPAS B-operand layout. The 34-byte AoS blocks are only 2-byte aligned and can't be loaded that way. - **PTQ1_0.** Base-3 has no DPAS form, so PTQ1_0 weights are expanded once into the same 2-bit layout. That is 34 bytes a block instead of 28. On Xe2 devices the buffer type reserves the room for it (get_alloc_size), so VRAM holds only the expanded copy: about +25% for these weights, and the file on disk is unchanged. - **Activations.** They are quantized to int8 with one scale per 128 values, one PQ2_0 block. The four DPAS of a block then accumulate in int32 before a single float rescale, instead of one rescale per 32 values. - **One kernel for all batch sizes.** Tiles are 8/16/32 tokens x 32 rows. For small batches, up to 16 threads of a work-group split K for the same tile and reduce through SLM, so a mat-vec keeps enough loads in flight. Scope and fallbacks: - On by default on BMG-G21/G31 and LNL-M. `GGML_SYCL_DISABLE_XMX=1` or `GGML_SYCL_ENABLE_OPT=0` turns it off. - Only plain MUL_MAT on 2D weights in regular (non-split, non-COMPUTE) buffers with K >= 256. Everything else (op offload, split buffers, MUL_MAT_ID, other devices) takes the existing paths unchanged. - Like the existing reorders, a tensor keeps its new layout once rewritten. The gate refuses MUL_MAT_ID expert slices and views. `get_rows` asserts if it ever sees a rewritten PQ2_0/PTQ1_0 tensor. Files: `pq2_xe2.cpp/.hpp` (new), `ggml-sycl.cpp` (gate, alloc size, extras), `common.hpp` (layout flag), `getrows.cpp` (assert). Intel Arc Pro B50 (16 GB), oneAPI 2026.1, Level Zero, Linux `xe` driver, `-ngl 99`, Ternary Bonsai models. "#278" is the same build with `GGML_SYCL_DISABLE_XMX=1`, i.e. the current prism code path. Both columns were measured in the same session. `llama-bench`, t/s: | model | test | #278 | this PR | change | |---|---|---|---|---| | 1.7B PQ2_0 | pp128 | 2574 | 6133 | 2.38x | | | pp512 | 5218 | 7645 | 1.47x | | | tg64 | 152.4 | 178.1 | +17% | | 4B PQ2_0 | pp128 | 1068 | 3068 | 2.87x | | | pp512 | 2298 | 3272 | 1.42x | | | tg64 | 77.4 | 99.8 | +29% | | 27B PQ2_0 | pp128 | 148 | 346 | 2.33x | | | pp512 | 290 | 370 | 1.28x | | | tg64 | 12.39 | 16.53 | +33% | | 27B PTQ1_0 | pp128 | 127 | 348 | 2.74x | | | pp512 | 267 | 370 | 1.39x | | | tg64 | 12.88 | 16.54 | +28% | Parallel decode (`llama-batched-bench -npp 128 -ntg 64`, aggregate t/s): | sequences | 1.7B PQ2_0 | 4B PQ2_0 | 27B PQ2_0 | 27B PTQ1_0 | |---|---|---|---|---| | 1 | 151 -> 171 | 76 -> 97 | 12.2 -> 16.1 | 13.1 -> 16.1 | | 2 | 250 -> 334 | 125 -> 186 | 18.9 -> 29.7 | 20.2 -> 29.7 | | 4 | 380 -> 613 | 185 -> 355 | 25.2 -> 48.0 | 26.3 -> 47.9 | | 8 | 478 -> 1010 | 224 -> 618 | 29.0 -> 68.6 | 31.0 -> 68.0 | | 16 | 322 -> 1446 | 138 -> 900 | 20.2 -> 85.7 | 17.2 -> 85.8 | | 32 | 588 -> 1769 | 262 -> 1184 | 32.5 -> 98.4 | 29.0 -> 98.4 | On #278, batches of 9+ tokens leave MMVQ for the dequantize-to-FP16 path, which is why throughput drops at 16 sequences there. Speculative decoding (27B PQ2_0 target, Qwen3.5-0.8B Q8_0 draft, `--spec-draft-n-max 1`, greedy, 256 tokens), t/s: | | #278 | this PR | |---|---|---| | prose, target only | 13.1 | 17.6 | | prose, with draft | 12.1 | 20.5 | | code, target only | 12.7 | 17.5 | | code, with draft | 12.8 | 21.9 | With #278, drafting does not pay. With this PR, verification is cheap enough that n_max=1 gains 17-25% over the target alone. Longer drafts lose on acceptance (15-33% at n_max 2-3 with this draft model). Where the decode gain comes from: at the 27B FFN shape (17408x5120, one token), the PQ2_0 mat-vec goes from 199 us (MMVQ, about 53% of the B50's 224 GB/s) to 88 us. End to end, 27B decode now reads about 123 GB/s. Most of the remainder is the ~2300 non-matmul kernels per token. - `test-backend-ops -b SYCL0`: MUL_MAT 1329/1329, MUL_MAT_ID 1035/1035, GET_ROWS 119/119 (all types). Full suite, run per op: 14703/14759. The 56 failures (CONV_2D, ROLL, FLASH_ATTN_EXT, LIGHTNING_INDEXER) are identical with this path disabled, so they predate this PR. SET_ROWS aborts on current prism regardless of this PR (`set_rows.cpp:549: Unsupported tensor type!`), so it was run separately. - Perplexity (`-c 512`, 5 chunks), #278 -> this PR: | model | setting | #278 | this PR | |---|---|---|---| | 1.7B PQ2_0 | b1 | 8.4089 | 8.4335 | | 1.7B PQ2_0 | b512 | 8.4233 | 8.4149 | | 4B PQ2_0 | b1 | 7.0584 | 7.0565 | | 4B PQ2_0 | b512 | 7.0545 | 7.0506 | | 1.7B PQ2_0 | -ngl 0 (op offload) | 8.4225 | 8.4225 | | 4B PQ2_0 | -ngl 20 (partial) | 7.0541 | 7.0535 | | 27B PTQ1_0 | b512 | 4.2421 | 4.2399 | - KL divergence against a CPU-only reference (1.7B, `-dev none --no-op-offload`), #278 vs this PR: - b1: 0.00122 vs 0.00125. - b512: 0.00106 vs 0.00113. - Same top token: 97.4-97.8% vs 97.7-97.9%. - The differences are within the error bars. The per-128 activation scale costs nothing measurable. - This introduces ESIMD/XMX code to the SYCL backend, a new pattern here. I'm happy to adjust structure or naming to fit how the maintainers want it organized. - Tuning (tile sizes, K-split thresholds) was done on the B50 only. BMG-G31 and LNL-M are gated in by architecture but untested. - I have read and agree with the [contributing guidelines](https://github.com/ggml-org/llama.cpp/blob/master/CONTRIBUTING.md) - AI usage disclosure: YES - the code and this description were written by Claude (Anthropic) under my direction; I tested it on my own hardware (Intel Arc Pro B50) and reviewed the changes. * sycl: gate the PQ2_0/PTQ1_0 XMX path on 16-wide DPAS instead of a device list The path was limited to BMG-G21/G31 and LNL-M by name. It now runs on any XMX device whose int8 DPAS is 16 wide, as the runtime reports it through matrix_combinations: Xe-HPC, Xe2, Xe3 and later. 8-wide XMX (Xe-HPG, Arrow Lake-H) and devices without XMX keep the existing paths. AOT builds compile the kernels for every GGML_SYCL_DEVICE_ARCH device, and they do not build for targets without 16-wide DPAS and 2D block loads (ocloc crashes on acm-g10, and fails on tgllp, dg1, mtl and pvc-vg). pq2_xmx.cpp is now built only when every AOT device is Xe-HPC, Xe2 or later; otherwise it compiles to stubs. Renames pq2_xe2 to pq2_xmx to match.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
SYCL has MMQ disabled for all types, so quantized prompt batches take the dequantize-then-GEMM path in
ggml_sycl_op_mul_mat_sycl. In the default build (GGML_SYCL_F16=OFF) that path expands the weights to FP32 and runs an FP32 GEMM, which does not use the XMX matrix engines on Intel GPUs. This PR sends PTQ1_0 and PQ2_0 down the existing FP16 branch instead: half the bytes per dequantized weight, and oneDNN (or MKL withGGML_SYCL_ENABLE_DNN=0) runs an FP16 GEMM with FP32 output.PTQ1_0 and PQ2_0 weights are exactly representable after FP16 expansion; FP16 conversion of activations and the GEMM arithmetic can still change results. This PR enables the path by default only for the two measured formats. Other quantized types retain the FP32 default and can still use the FP16 path with
GGML_SYCL_F16=ON. Decode is unchanged, since single tokens and small batches go through MMVQ.Additional information
Intel Arc 140V (Lunar Lake iGPU), Windows, oneAPI 2025.3, driver 32.0.101.9030, Release. Base
prism@ adfffbe, Ternary-Bonsai-2-27B.llama-bench -ngl 99 -fa 1 -r 1, four rounds in separate processes 30 s apart with the arm order flipped each round; median (range) in t/s:This laptop lowers its memory clock under sustained load, which accounts for the spread.
Correctness:
test-backend-ops -o MUL_MAT: PQ2_0 142/142 and PTQ1_0 165/165, with oneDNN and withGGML_SYCL_ENABLE_DNN=0. 15 cases per type have n = 9, 16 or 64 and take the changed path.llama-perplexity --kl-divergence, WikiText-2,-c 2048 -b 512, 8 chunks, against a base file from prism 0781925 (identical SYCL code). Unmodified base build: PPL ratio 1.000035, mean KLD 0.000000, same top 100%. This PR: PPL ratio 1.000035 +- 0.000036, mean KLD 0.000001, max KLD 0.000898, same top 99.963%. PTQ1_0 and PQ2_0 give identical numbers, as they pack the same trits.Only tested on Xe2. Builds on #235.
Requirements