qwen35: apply the Hadamard inverse to the MTP token-embedding lookup

[zhaoyilun]

What

--spec-type draft-mtp cannot start at all on a Hadamard-folded model when the MTP block lives in the target file. The MTP graph (qwen35::graph_mtp) does its own ggml_get_rows() on the token embedding table but never restores the primal basis, while the trunk path (llm_graph_context::build_inp_embd) does exactly that two lines after its own lookup. llama_verify_hadamard_graph then (correctly) refuses the graph:

W llama_verify_hadamard_graph: latent lookup 'mtp_tok_embd-64' consumed by op=RMS_NORM name='norm-64' src0 hint=0
E llama_init_from_model: failed to initialize the context: Hadamard-latent table 'token_embd.weight' is read without the inverse transform
E common_speculative_init_result: failed to create MTP context
E srv load_model: failed to create MTP context

Why it is worth fixing

Packaging the MTP block inside the target GGUF is not just convenient, it is what makes MTP profitable here: -m <bundle> --spec-type draft-mtp (no -md) builds the draft context against the target, so the draft shares the target's token_embd/output instead of carrying its own 248320x5120 copies. A separate -md sidecar duplicates the vocabulary (~92% of its bytes), pushing the per-token draft cost ratio to rho ~ 0.43, which turns even 40% acceptance into a net loss (measured: 5.16 t/s vs 16.6 t/s no-spec). Sharing the vocabulary drops rho to ~0.06 and the same head becomes a win.

So this one guard currently gates the only cheap speculative configuration on folded models.

Repro (fails before, works after)

llama-server -m Ternary-Bonsai-2-27B-PQ2_0-MTP-Q8_0.gguf \
  -ngl 99 -fa on -c 32768 -ctk q4_0 -ctv q4_0 -np 1 --jinja \
  --spec-type draft-mtp --spec-draft-n-max 2

Reproduced on prism-b10683-d8f26ee (the newest release with published assets — see #193) and still present on prism-b10709-9a9394a; also reported for the same model on Metal in #203's environment. The file is ProCreations/Ternary-Bonsai-2-27B-MTP (the unchanged PQ2_0 target plus one MTP layer in the same file).

Fix

Apply the same rot + signs inverse the trunk uses, keyed on the table actually looked up (layer.nextn.embed_tokens when present, otherwise model.tok_embd). 14 lines, no new API.

Verification

Built for sm_89 (Ada) from this branch, then:

• context creation succeeds (creating MTP draft context against the target model), no hadamard error;
• draft acceptance is identical to the known-good community-patched build on the same three probes — 49/90, 59/71, 57/75 (54.4% / 83.1% / 76.0%), i.e. the graph semantics match exactly;
• 12-prompt A/B vs --spec-type none on RTX 4080 SUPER 16 GB: median 1.338x decode, aggregate acceptance 68.1% (1.23-1.70x; code/format 1.5-1.7x, free reasoning prose 1.25-1.3x), and the model still answers correctly.

Raw JSON, harness, build script, launch units and the long-context depth sweep: https://github.com/zhaoyilun/bonsai2-27b-mtp-repro (dataset mirror: https://huggingface.co/datasets/zhaokeqi/bonsai2-27b-mtp-repro), discussion: https://huggingface.co/prism-ml/Ternary-Bonsai-2-27B-gguf/discussions/23.

Credit

The same change ships as runtime/bonsai-mtp-embedding.patch in ProCreations/Ternary-Bonsai-2-27B-MTP, whose patched source archive already contains it. This PR is the equivalent against current prism (the guard and the trunk helper have moved slightly since b10683, e.g. the verifier now reports the consuming op). If you would rather take the fix from them or fold it into a wider pass over every latent table the draft graph reads, please do — the point is that in-file MTP is unusable on official binaries until this lands.

[azfarh95]

Independent verification on gfx1100 / ROCm 6.4 / HIP (RX 7900 XTX, Windows 11), 2026-09-19:

Without this patch, --spec-type draft-mtp on prism-ml/Ternary-Bonsai-2-27B-GGUF (PQ2_0 target with the ProCreations MTP module embedded) fails hard during context init:

llama_verify_hadamard_graph: latent lookup 'mtp_tok_embd-64' consumed by op=RMS_NORM name='norm-64' src0 hint=0
llama_init_from_model: failed to initialize the context: Hadamard-latent table 'token_embd.weight' is read without the inverse transform
common_speculative_init_result: failed to create MTP context

With the patch applied (ProCreations' bonsai-mtp-embedding.patch, which matches this PR's change), the same file loads and draft-mtp --spec-draft-n-max 2 runs clean: draft acceptance 0.82, 2.64 tokens committed per round, decode 75.9 t/s cold / 73.1 t/s warm vs 41.6 t/s with the drafter disabled — measured through a load/teardown-gated harness on :5802 with 32k context. Output correctness spot-checked on every run.

Also noted while testing: --spec-draft-n-max 4 is slower than 2 on this hardware (60.8/57.8 t/s), consistent with the per-round cost saturating. The patch is required for draft-mtp to work on prism GGUFs at all — it is not an optimization.

[zhaoyilun]

Thanks for the independent check on completely different hardware — RDNA3 / ROCm / Windows against our Ada / CUDA / WSL, same failure line, same fix. That is worth more than another data point from me.

The two speedups line up once you look at the no-spec baselines:

platform	no spec	with MTP	speedup	acceptance
RX 7900 XTX, gfx1100, ROCm 6.4	41.6 t/s	75.9 / 73.1	1.76-1.82x	0.82
RTX 4080 SUPER, sm_89, CUDA 13.3	~67 t/s	~90	1.34x	0.68 (12-prompt set)

Speculation pays more where the target decodes slower: at 41.6 t/s the ternary path is closer to instruction/compute bound, so the draft's roughly fixed cost is a smaller fraction of a round. Our long-context sweep shows the same shape for a different reason — as depth grows decode falls 87 -> 35 t/s while acceptance rises 66% -> 84%, so the MTP multiple holds at 1.16-1.38x instead of decaying with depth.

Your --spec-draft-n-max 4 being slower than 2 also matches Ada: 2 and 3 are a wash here (82.4 vs 83.0 t/s total, acceptance 69.9% vs 59.0%), i.e. past ~3 you are mostly trading acceptance for round count. And our 0.68 aggregate is dragged down by four free-form reasoning-prose prompts (48-58%); the code / JSON / list continuations sit at 77-94%, which is closer to your 0.82.

Harness, raw JSON and the depth sweep: https://github.com/zhaoyilun/bonsai2-27b-mtp-repro (per-prompt single runs, temp 0 / top_k 1 / seed 7 / n_predict 128).

[noonghunna]

Independent confirmation on Ampere / sm_86 (2× RTX 3090, CUDA 12.8, Linux) — a third architecture after the RDNA3/ROCm and Ada/CUDA reports above.

Built prism-b10709-9a9394a from source with this PR applied, -DCMAKE_CUDA_ARCHITECTURES=86;89;120 (matching the stock release matrix, verified with cuobjdump --list-elf). Target is the bundled Ternary-Bonsai-2-27B-PQ2_0-MTP-Q8_0.gguf, -ngl 99 -fa on -ctk q8_0 -ctv q8_0 -np 1 -c 32768.

Before this PR — identical failure to the one described:

llama_verify_hadamard_graph: latent lookup 'mtp_tok_embd-64' consumed by op=RMS_NORM name='norm-64' src0 hint=0
llama_init_from_model: failed to initialize the context: Hadamard-latent table 'token_embd.weight' is read without the inverse transform
common_speculative_init_result: failed to create MTP context

After: the MTP context creates and drafts.

Draft-depth sweep

/completion, temperature=0, top_k=1, seed=7, cache_prompt=false, n_predict=128, fresh boot per rung, acceptance from timings.draft_n / draft_n_accepted. Median across three prompts (code continuation, free-form prose, a markdown table):

--spec-draft-n-max	median t/s	speedup	acceptance
off	48.70	1.00x	—
1	74.96	1.54x	94%
2	79.78	1.64x	86%
3	82.57	1.70x	82%
4	89.58	1.84x	77%
5	73.21	1.50x	61%
6	69.68	1.43x	58%

Two notes that may be useful to others reproducing this:

The optimum is n=4 here, and the cliff past it is sharp — 89.58 → 73.21 is −18% for one extra draft token. We had initially measured only at n-max 2 (following the protocol in the Ada report) and concluded 1.65x; sweeping depth raised that to 1.84x. Worth sweeping rather than inheriting a depth.

The optimum is workload-dependent. Code and table/format continuation peak at n=4 (89.58 and 90.47 t/s); free-form prose peaks at n=2 (67.85) and by n=5 has fallen to 49.69 — within 2% of the no-speculation baseline, i.e. paying the full draft cost for nothing. A single default cannot serve both.

⚠️ Measurement caveats: one boot per rung, and these are /completion numbers rather than llama-bench, so the absolute t/s are not directly comparable to the tables above. The depth trend is monotone and the peak is 8.5% above its neighbour, so the shape is solid; the exact peak value is not.

⚠️ For completeness: our build also carries #216 (GDN warp-layout gate) and #189 (short-query MMA FA fallback). Neither should affect the MTP path at n=4 — #189 is gated Q->ne[1] <= 4 and n=4 gives Q=5 — but the environment isn't this PR in isolation.

Thanks for the fix.

[bri-prism]

Agent review: posted by the maintainer's coding agent at their request.

No findings in the embedding inverse-transform change. The translation unit passes a Clang C++17 syntax check against same-base headers, including without the extra include present in #217.

#217 contains the same functional fix. Please consolidate the two so only one lands. This addresses a clear correctness gap and is a priority merge candidate, subject to the normal model/build gates. Model-backed MTP execution was not rerun here.

Reviewed commit: 518ad108f0b72bac4f397a486695a5786e33a58d.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[Copilot]

Copilot review overview

🟢 Approval recommended

The implementation matches the established trunk transform, with only a non-blocking comment-style nit remaining.

Review effort: Balanced
Findings: 1

Open (1)

• Simplify overly verbose hard-wrapped comment · New

[zhaoyilun]

Consolidating with #217

I read #217 line by line against this one. The executable change is identical: the same find(tok_embd_w) on hadamard_inverses, the same llama_mul_mat_hadamard(ctx0, tok_embd, it->second.rot), the same conditional ggml_mul by it->second.signs, in the same place after the lookup. Only the comment wording differs, and #217 carries one extra #include "llama-impl.h" which @bri-prism's review already established this translation unit does not need.

So there is no functional difference to choose on, and the only question is which PR is the cheaper place to land it. I suggest this one:

• the evidence trail is already here — two independent reproductions on unrelated stacks (RDNA3 / ROCm / Windows, and Ampere / sm_86 / Linux) on top of the Ada numbers — so nothing has to be gathered again;
• no redundant include;
• the description covers the packaging upstream actually ships, the MTP block inside the target file, which is the case that makes draft-mtp profitable at all rather than a net loss.

I have asked @sudoingX on #217 to let this PR be the single landing point. So that nothing from #217 is lost, three things from it belong here and I am folding them in:

1. Other architectures do the same raw lookup. qwen3next, glm4-moe, deepseek2 and friends all read their embedding table in the MTP graph without the inverse transform; none of them has a Hadamard-folded export today, so scoping this to qwen35 is right — but it is worth recording that the guard is latent everywhere.

2. The long-term shape is dflash: apply the target's Hadamard transforms to borrowed embeddings and head #210's. As @sudoingX noted, once llm_graph_context::build_embd_rows() exists this collapses to one line at qwen35.cpp:636:

   tok_embd = build_embd_rows(tok_embd_w, inp->tokens);

   dflash: apply the target's Hadamard transforms to borrowed embeddings and head #210 does not touch src/models/qwen35.cpp, so the MTP graph still fails to build without this change — both are needed. I will rebase this onto build_embd_rows() the moment dflash: apply the target's Hadamard transforms to borrowed embeddings and head #210 lands.

3. Their low-VRAM measurements. On an RTX 3060 12 GB, with the fix the MTP head reads the trunk's own table instead of carrying a second unrotated copy: 9,956 MiB instead of 10,638 MiB at 131072 context, and 196608 context fits at 11,990 MiB where the fat file OOMs on the MTP compute buffer. Acceptance is unchanged (0.89 / 0.67 / 0.77 on code / prose / bash).

We are asking for this one to land, and #217's author has been told the same so a duplicate does not sit in the queue. You have the final call; our recommendation is this PR.

[zhaoyilun]

Thanks — this is the useful kind of report, because it corrects us rather than confirms us.

We stopped at n=2 and n=3 and concluded the curve had flattened (82.4 vs 83.0 t/s total, "past ~3 you are mostly trading acceptance for round count"). Two rungs is not a sweep. Your n=4 peak is what sweeping actually looks like, and "worth sweeping rather than inheriting a depth" lands on us — we inherited ours from the Ada protocol.

The two sweeps disagree, and the disagreement is the interesting part:

	your 3090 (median of 3)	our 4080 SUPER (12 prompts)
no-spec	48.70 t/s	~67 t/s
acceptance @ n=2	86%	69.9%
acceptance @ n=3	82%	59.0%
best rung	n=4, 1.84x	n=2-3, 1.34x

At n=3 our absolute throughput is nearly identical (82.57 vs 83.0 t/s) while acceptance differs by 23 points — which is your own caveat doing work: acceptance is a property of the prompt set as much as of the model, so it is not a number that crosses tables. Our 12-prompt set deliberately carries four free-form reasoning-prose prompts sitting at 48-58% on their own, which drags the aggregate to 0.68; your code / prose / table set has no such tail.

If where the curve turns is set by how fast acceptance decays with draft depth, then a set with a low-acceptance tail should peak earlier and fall off harder — which is exactly the asymmetry between our two tables. That is a sharper version of your "a single default cannot serve both": the default may depend not on the workload alone but on the mix a server actually serves.

The cliff we never saw, because we never went past 3 — 89.58 -> 73.21 for one more draft token is the number that makes the case for sweeping, and we did not have it.

Also for the axis we did sweep, so the two don't get conflated: ours is context length, not draft depth. Decode falls 87 -> 35 t/s from 4k to 131k while acceptance rises 66% -> 84%, so the MTP multiple holds at 1.16-1.38x rather than decaying.

Noted on #216 / #189: at n=4 you get Q=5, so #189's ne[1] <= 4 gate does not fire, and neither touches the MTP path. Agreed the environment is not this PR in isolation, but the before/after is unambiguous regardless.

[bri-prism]

Follow-up review and local testing at 518ad10: no findings. Recommend landing this fix once CI passes; #217 has the same executable change, so only one needs to land.

Validation performed on ARM CPU:

• Built the llama library from the exact PR head, including the modified translation unit.
• Executed six focused GGML graph cases using the transform block extracted from this head and its real Hadamard helper: shared and per-layer embedding tables, with and without signs, an unmapped per-layer table, and a null transform map. Repeated token IDs and blockwise transforms were included.
• Compared 122880 output values against an independent scalar Walsh-Hadamard reference; maximum absolute error 2.58e-7 (tolerance 2e-5).
• The current graph verifier accepted the patched graphs; all four corresponding unpatched folded-embedding graphs were rejected as expected.

These are local build, numerical and graph-coverage tests, not full-model generation or fresh CUDA/HIP measurements. Ubuntu and Windows CI are currently pending. #210 does not patch this MTP lookup, so merging #210 alone will not resolve it; after both land the duplicated lookup logic can use build_embd_rows().