arm: NEON vec_dot for PQ2_0

[kiljoy001]

Overview

Adds NEON vec_dot for PQ2_0, closing the same gap on ARM that #248 closed on x86.

PQ2_0 had a NEON kernel, but it targeted Q8_0 activations. PQ2_0 now dispatches through Q8_K, so that kernel was dead code for normal inference (the same issue bri-prism found and I fixed on x86 in #248).

ggml_vec_dot_pq2_0_q8_K reuses the existing Q8_0 kernel's unpack, but accumulates all four sub-block dots in int32 and applies the block scale once, matching Q8_K's one-scale-per-256 layout instead of one-scale-per-32. Uses vqtbl1q_u8 (via the ggml_vqtbl1q_u8 wrapper) for the code/trit unpack and ggml_vdotq_s32 (sdot) for the reduction.

PTQ1_0 is no longer part of this PR

This PR originally also added a NEON kernel for PTQ1_0. Two independent real-hardware reports found it slower than the plain generic loop it was meant to replace:

• bri-prism, evaluating on Apple M5 Pro
• karusrus, evaluating on a Snapdragon 7 Gen 4 phone (Motorola Edge 70), with real benchmark data: arm: NEON vec_dot for PQ2_0 #265 (comment)

Root cause: the kernel decoded trits by widening into uint16 lanes, which costs more than it saves. karusrus posted a working alternative that stays in 8-bit lanes (the same trick upstream's TQ1_0 NEON kernel uses) and adds an i8mm/SMMLA 2x2-tile path for hardware that has it, measuring ~1.9x pp / ~1.2x tg over the generic loop on their device: https://github.com/karusrus/llama.cpp/tree/arm-neon-ptq1_0

I don't have i8mm-capable ARM hardware here (my Orange Pi 5 Plus is Cortex-A76/A55, noi8mm) to independently verify that kernel's fast path, so rather than merge a replacement I can't fully test myself, this PR now drops the regressive PTQ1_0 kernel entirely and aliases it back to the generic scalar loop on ARM (matching the same fallback pattern arch-fallback.h already used for PTQ1_0 on x86 before #248 added a real SIMD kernel there). A correctly-verified PTQ1_0 NEON kernel - likely karusrus's, once I or someone else can test the i8mm path on real hardware - should land as a follow-up PR.

Results (PQ2_0 only)

Orange Pi 5 Plus (RK3588, 4x Cortex-A76 + 4x Cortex-A55, asimddp). Isolated A/B: same binary, only libggml-cpu.so swapped, alternated, two pairs.

Ternary Bonsai 1.7B PQ2_0

build	pp128	tg32
scalar	4.50 / 4.53	3.63 / 3.66
NEON	23.19 / 23.21	17.54 / 15.89

~5.1x pp, ~4.6x tg.

For reference, PTQ1_0 on this same hardware now runs the generic loop (no NEON path): 27B model, pp64 0.33 t/s / tg16 0.27 t/s at 5 threads.

Correctness

test-quantize-fns gains the same exact check as #248 (unchanged - the property has nothing architecture-specific about it): packed vec_dot vs. dequantize-then-scalar-dot, power-of-two scales, bit-exact. 0 failures for PQ2_0 (NEON) and PTQ1_0 (generic fallback) on real Cortex-A76/A55 hardware.

Also validated with a private property-based harness (theft, ~124k random inputs per format across all three x86 ISA tiers when I built it for #248) and an exhaustive edge sweep (every weight byte 0-255 x activation extremes {-128,-1,0,1,127}, including out-of-range trit bytes) - not part of either PR, kept as a private correctness check before opening either.

End-to-end verification

Greedy (temp=0) completion on the 8B PQ2_0 model is byte-identical between this NEON build and an x86 SSE build (#248), same prompt, same seed - exercising the full inference pipeline, not just the kernel in isolation.

ARMv7

Fixed in a follow-up commit (22e70a2d0): the PQ2_0 kernels used raw vqtbl1q_u8, which is AArch64-only, instead of the codebase's own ggml_vqtbl1q_u8 wrapper (which has a 32-bit fallback). I don't have ARMv7 hardware or a cross-toolchain to independently verify the fix compiles there - this is a pattern-match against bri-prism's diagnosis and against how other kernels in the same file already do it correctly (ggml_vec_dot_q2_0_q8_0). What I did verify: ggml_vqtbl1q_u8 is a plain #define to the raw intrinsic on __aarch64__, so the change is a no-op by construction on the hardware I can test, and test-quantize-fns still passes 0 failures on the Orange Pi after the change.

Claude-Session: https://claude.ai/code/session_016pBjsH3XSFLbnzoDyzkcUU

[bri-prism]

Evaluated 308f4b648 from an x86 machine (no ARM hardware here), so this covers compile checks and review only. The NEON performance and on-device correctness claims are not re-verified by me.

Cross-compile checks (clang 22.1.8, -fsyntax-only on ggml/src/ggml-cpu/arch/arm/quants.c, aarch64/armv7a-w64-mingw32):

target	prism @ 078192590	this PR
AArch64, -march=armv8.2-a+dotprod (sdot path)	✅	✅
AArch64, -march=armv8-a (ggml_vdotq_s32 fallback, no dotprod)	✅	✅
32-bit ARMv7, -mfpu=neon -mfloat-abi=hard	❌ 1 site: ggml_vec_dot_pq2_0_q8_0 L339/344	❌ 2 sites: the same plus the new ggml_vec_dot_pq2_0_q8_K L409/411

The cause is the same in both: raw vqtbl1q_u8 is AArch64-only, but the kernels are guarded only by __ARM_NEON, which ARMv7 NEON also defines. ggml-cpu-impl.h already has ggml_vqtbl1q_u8 with a 32-bit fallback. Other kernels in this file use it, e.g. arm/quants.c:266. Switching the two new call sites and the two pre-existing ones to ggml_vqtbl1q_u8 fixes all of them. Low severity: no fork release target builds 32-bit ARM. But it's a four-identifier change, and the new kernel shouldn't copy the break. The new PTQ1_0 kernel compiles on ARMv7 as is.

Review of ggml_vec_dot_pq2_0_q8_K: scale handling looks right. Each 128-weight PQ2_0 block i reads its half of Q8_K block i >> 1 (offset 128 * (i & 1)) and applies d_x * d_y once to an exact int32 dot. Codes map to {−1, 0, +1} (code 3 → +2 is unreachable given d = amax). The int32 accumulator can't overflow (≤128 products of |·| ≤ 256).

Consistency with x86: the "PQ2_0 Q8_0 NEON kernel is dead code" premise is correct. PQ2_0's vec_dot_type is Q8_K in the arch-independent CPU traits, which is the same finding as on x86 (#248).

Shared files:

• tests/test-quantize-fns.cpp is byte-identical to x86: SSE2/SSSE3 vec_dot for PTQ1_0 and PQ2_0 #248's (aed0a6345), and the two branches merge cleanly together.
• On x86 that test passes with 0 failures (my x86: SSE2/SSSE3 vec_dot for PTQ1_0 and PQ2_0 #248 run on AVX2 and SSE-only builds).
• The arch-fallback.h change only removes aliases in the ARM branch, so x86 is unaffected.

On the cross-architecture PTQ1_0 greedy divergence (x86 vs ARM after ~28 tokens, reproducing with the generic path on ARM): that matches what I measured on x86. A same-kernel rebuild with different ISA flags already moves KLD to around 5e-4 on a sensitive model (see #255), so I agree it isn't attributable to this PR.

Evaluated with Claude Code.

[bri-prism]

Thanks, this is a big improvement for PQ2_0. I tested on an Apple M5 Pro (NEON + dotprod): test-quantize-fns passes with 0 failures for both formats. PQ2_0 CPU throughput is about 6.7x for prompt processing and 5.7x for generation on a 2B model.

For PTQ1_0 I saw the opposite on this chip. On the 27B, CPU-only, the NEON path is about 9% slower than the generic C loop (pp32 5.3 vs 5.8 t/s, tg8 4.7 vs 5.1 t/s, two alternating runs). Clang seems to vectorize the generic loop well here. Could you share PTQ1_0 numbers from the RK3588? If it isn't faster there either, one option is to land the PQ2_0 kernel now and keep PTQ1_0 on the generic path until it wins.

Two smaller things: this PR and #248 both touch arch-fallback.h and tests/test-quantize-fns.cpp (including the same test function), so whichever merges second will need a rebase. And for 32-bit ARM builds, ggml_vqtbl1q_u8 rather than vqtbl1q_u8 would match the rest of the file.

[bri-prism]

For context: the review above is the on-device follow-up to the earlier x86-only comment, run on real ARM hardware (Apple M5 Pro, NEON + dotprod).

[kiljoy001]

I'll look into the ptq1_0 performance and get back to you

…

On Thu, Sep 24, 2026, 11:39 AM bri-prism ***@***.***> wrote: *bri-prism* left a comment (PrismML-Eng/llama.cpp#265) <#265 (comment)> For context: the review above is the on-device follow-up to the earlier x86-only comment, run on real ARM hardware (Apple M5 Pro, NEON + dotprod). — Reply to this email directly, view it on GitHub <#265?email_source=notifications&email_token=ACD25T75BU3HCPTR5UEKNST5QVS47A5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTKOBRHE4TSMJSGEY2M4TFMFZW63VGMF2XI2DPOKSWK5TFNZ2KYZTPN52GK4S7MNWGSY3L#issuecomment-5819991211>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACD25T3C5QCRPZIMAOF3QY35QVS47AVCNFSNUABGKJSXA33TNF2G64TZHMYTCNRWHE3DKOJWGM5US43TOVSTWNJVG4YDKMJTGIZDNILWAI> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[kiljoy001]

Fixed in 22e70a2d0: all four sites (the two new ones plus the two pre-existing ones you also caught) now use ggml_vqtbl1q_u8.

One honest caveat: I don't have ARMv7 hardware or a cross-toolchain here (no sudo to install gcc-arm-linux-gnueabihf), so I couldn't independently re-verify this compiles there the way you did with clang. This is a pattern-match against your diagnosis and against how the codebase already does it correctly elsewhere in the same file (ggml_vec_dot_q2_0_q8_0), not an independent confirmation on real ARMv7. What I did re-verify: ggml_vqtbl1q_u8 is a plain #define to the raw intrinsic on __aarch64__, so the change is a no-op by construction on the hardware I can test, and test-quantize-fns still passes 0 failures on the Orange Pi after the change.

If you get a chance to re-run your clang -fsyntax-only check against 22e70a2d0, that would be the actual confirmation - happy to iterate again if it's still not clean.

Thanks for reviewing on real ARM hardware (M5 Pro) as a follow-up to the x86-only pass - that's exactly the kind of thing I can't do myself for this PR, and the scale-handling review of ggml_vec_dot_pq2_0_q8_K is a more careful check than I ran on my own code.

Still looking into the PTQ1_0 divergence question - will follow up separately.

[karusrus]

Some PTQ1_0 data from a phone, since the open question here is whether a NEON PTQ1_0 kernel can beat the generic loop.

Device: Motorola Edge 70, Snapdragon 7 Gen 4 (SM7750: 1× + 4× Cortex-A720, 3× Cortex-A520), NEON + dotprod + i8mm. Android 16, llama.cpp built on-device in Termux with clang 21, -DGGML_CPU_ARM_ARCH=armv8.6-a+dotprod+i8mm+bf16. Ternary Bonsai 2 27B PTQ1_0, CPU only, -t 5. Same model and flags for all three builds, runs alternated, two passes:

build	pp64 t/s	tg16 t/s
prism @ adfffbe (generic PTQ1_0)	1.22 / 1.22	1.09 / 1.09
this PR @ 22e70a2d0	1.19 / 1.07	1.07 / 0.96
alternative kernel below	2.33 / 2.33	1.31 / 1.31

So on this chip I see the same thing bri-prism saw on the M5 Pro: the PTQ1_0 kernel in this PR is slightly slower than the generic loop. The kernel below is ~1.9× on prompt and ~1.2× on generation against generic.

What it does differently (branch: https://github.com/karusrus/llama.cpp/tree/arm-neon-ptq1_0, commit f9078a625, 157 lines on top of adfffbe):

1. Trit decode stays in 8-bit lanes. ((q + (q >> 1)) >> 1) >> 6 is exactly (3q) >> 8 for every byte value 0..255 (checked exhaustively), the same trick the upstream TQ1_0 NEON kernel uses. No widening to u16. The whole 128-trit block is decoded into eight int8x16_t in element order (vhadd/vshr/vmul on 16 + 8 lanes, the 2-byte qh tail via one vmul_u8 against {1,1,3,3,9,9,27,27}).
2. One horizontal add per row, not four per block. Each 32-wide Q8_0 sub-block's sdot result is converted and accumulated with vmlaq_n_f32 into a float32x4; vaddvq_f32 only at the end.
3. i8mm 2×2 tile for prompt processing. With __ARM_FEATURE_MATMUL_INT8, PTQ1_0 gets .nrows = 2 (like Q8_0), and the nrc == 2 path decodes two weight rows once and multiplies them against two activation columns with vmmlaq_s32, so every decoded trit vector is used twice. This is where most of the prompt gain comes from; tg (nrc = 1) only benefits from 1 and 2.

Correctness: test-quantize-fns 0 failures (including the packed ternary check). Greedy 24-token completion on the 27B is byte-identical between this kernel and the generic path on the same device, with the prompt going through the nrc == 2 path. I have not run perplexity on it yet.

Micro-benchmark from test-backend-ops perf -b CPU -o MUL_MAT on the same phone, m=4096, k=14336: n=1 PTQ1_0 76 GFLOPS vs Q4_0 86; n=512 PTQ1_0 157 vs Q4_0 214.

How would you like to take it? I'm happy for you to fold it into this PR (replacing the PTQ1_0 kernel here), or to open it as a follow-up once the PQ2_0 part lands, as bri-prism suggested. I can run tests on this device on request.

Disclosure, per CONTRIBUTING.md: the kernel was written with Claude Code (AI). I built, tested and measured it on the device myself and I'm responsible for it.

[kiljoy001]

Id fold it in - the performance is clearly better. It might be a few days till I get to this, working on xmx prefil for sycl :)

…

On Sat, Sep 26, 2026, 3:05 PM Ruslan ***@***.***> wrote: *karusrus* left a comment (PrismML-Eng/llama.cpp#265) <#265 (comment)> Some PTQ1_0 data from a phone, since the open question here is whether a NEON PTQ1_0 kernel can beat the generic loop. *Device:* Motorola Edge 70, Snapdragon 7 Gen 4 (SM7750: 1× + 4× Cortex-A720, 3× Cortex-A520), NEON + dotprod + i8mm. Android 16, llama.cpp built on-device in Termux with clang 21, -DGGML_CPU_ARM_ARCH=armv8.6-a+dotprod+i8mm+bf16. Ternary Bonsai 2 27B PTQ1_0, CPU only, -t 5. Same model and flags for all three builds, runs alternated, two passes: build pp64 t/s tg16 t/s prism @ adfffbe (generic PTQ1_0) 1.22 / 1.22 1.09 / 1.09 this PR @ 22e70a2 1.19 / 1.07 1.07 / 0.96 alternative kernel below *2.33 / 2.33* *1.31 / 1.31* So on this chip I see the same thing bri-prism saw on the M5 Pro: the PTQ1_0 kernel in this PR is slightly slower than the generic loop. The kernel below is ~1.9× on prompt and ~1.2× on generation against generic. What it does differently (branch: https://github.com/karusrus/llama.cpp/tree/arm-neon-ptq1_0, commit f9078a6, 157 lines on top of adfffbe): 1. *Trit decode stays in 8-bit lanes.* ((q + (q >> 1)) >> 1) >> 6 is exactly (3q) >> 8 for every byte value 0..255 (checked exhaustively), the same trick the upstream TQ1_0 NEON kernel uses. No widening to u16. The whole 128-trit block is decoded into eight int8x16_t in element order ( vhadd/vshr/vmul on 16 + 8 lanes, the 2-byte qh tail via one vmul_u8 against {1,1,3,3,9,9,27,27}). 2. *One horizontal add per row, not four per block.* Each 32-wide Q8_0 sub-block's sdot result is converted and accumulated with vmlaq_n_f32 into a float32x4; vaddvq_f32 only at the end. 3. *i8mm 2×2 tile for prompt processing.* With __ARM_FEATURE_MATMUL_INT8, PTQ1_0 gets .nrows = 2 (like Q8_0), and the nrc == 2 path decodes two weight rows once and multiplies them against two activation columns with vmmlaq_s32, so every decoded trit vector is used twice. This is where most of the prompt gain comes from; tg (nrc = 1) only benefits from 1 and 2. *Correctness:* test-quantize-fns 0 failures (including the packed ternary check). Greedy 24-token completion on the 27B is byte-identical between this kernel and the generic path on the same device, with the prompt going through the nrc == 2 path. I have not run perplexity on it yet. Micro-benchmark from test-backend-ops perf -b CPU -o MUL_MAT on the same phone, m=4096, k=14336: n=1 PTQ1_0 76 GFLOPS vs Q4_0 86; n=512 PTQ1_0 157 vs Q4_0 214. How would you like to take it? I'm happy for you to fold it into this PR (replacing the PTQ1_0 kernel here), or to open it as a follow-up once the PQ2_0 part lands, as bri-prism suggested. I can run tests on this device on request. Disclosure, per CONTRIBUTING.md: the kernel was written with Claude Code (AI). I built, tested and measured it on the device myself and I'm responsible for it. — Reply to this email directly, view it on GitHub <#265?email_source=notifications&email_token=ACD25T3K7PK6HM2S43DPHMT5RA4SFA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTKOBVGAZDSNJZGM22M4TFMFZW63VGMF2XI2DPOKSWK5TFNZ2KYZTPN52GK4S7MNWGSY3L#issuecomment-5850295935>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACD25T5OM5B6RIKUJMUNXSD5RA4SFAVCNFSNUABGKJSXA33TNF2G64TZHMYTCNRWHE3DKOJWGM5US43TOVSTWNJVG4YDKMJTGIZDNILWAI> . You are receiving this because you authored the thread.Message ID: ***@***.***>