qwen3_5: add a GEMM handle with one algorithm per weight shape

[twu3202]

Purpose

For #85: a backend step the posted plan didn't include, explained in this update. It builds on #97 and stays a draft until that is merged and the backend path is settled with #96. This PR's own change is the last commit, f095f07.

cuBLASLt's heuristic picks an algorithm per M, so a row's result depends on how many rows are in the call. In the 9B fused GDN input projection (synthetic inputs), row 0 differs in 6,768 of 12,352 outputs between M = 1 and M = 2. With prefix reuse, a prefix and its branch run as separate GEMMs, so they round differently from one pass over the same tokens. In a first shared-prefix executor, that alone moved the final hidden state by up to 1.1e-2 relative L2 against forward on 9B, above the plan's 1e-2 gate. That held even with chunk-aligned splits, where the conv, the gated delta rule and attention are bit-identical.

This PR adds cs1_gemm_create_fixed(workspace_bytes, reference_m). The handle keeps one algorithm per weight shape (N, K, ldy) for every M: the heuristic's first choice at reference_m rows, among algorithms without split-K. Each output row then takes the same path whatever the other rows, so its result depends neither on M nor on where the row sits. An M the algorithm can't serve returns a cuBLAS status instead of switching algorithms.

• cuBLASLt doesn't document this property. The new test checks it, and it has passed on sm_89 only (CUDA 13.2, driver 595.91.07), so it should be rerun on other GPUs.
• Split-K is excluded because a split-K choice may not serve every M. In an earlier experiment that allowed it (reference M 256), the pinned choice for the 9B down projection was rejected at every M tested from 1,030 rows up, while 1,024 passed.

The fixed handle moves the result away from the default handle's: a full pass with it differs from forward by up to 1.5e-2 relative L2 on 9B (#99). That's why #99 compares reuse with a full pass under the same handle, and why the update asks how to gate reuse against today's output.

cs1_gemm_create is unchanged, and so are the workers: nothing creates a fixed handle yet. #99 uses it only for the shared-prefix runs, so the plain forward keeps today's algorithms and speed. On a fixed handle, #64's vision GEMMs (FP32, and BF16 with bias) also keep one algorithm per shape; nothing uses them that way, and the test covers cs1_gemm only. The ABI moves from 6 to 7, including the note in src/backends/cuda/contract.md.

Test Plan

System1-Omni Version / Commit: f095f07 on #97 (4678e3d), on main 47eff9c. The kernel tests and the plain-forward comparison ran on the final code, in #99's tree. The timing ran at an earlier version, before the rebase onto #64. For these BF16 GEMMs it chose algorithms the same way and lacked only the split-K check.

CPU: the five Rust steps in CI (format, Clippy, frontend tests, workspace tests, release build), and mkdocs build --strict.

GPU, one RTX 6000 Ada (sm_89), CUDA 13.2, driver 595.91.07:

• all GPU kernel tests, including a new one covering every projection of the 4B, 9B and 27B backbones. With the output filled with NaN before each call, the fixed handle's results for fewer rows must match the matching rows of a 4,096-row call bit for bit. That covers rows at the start and rows shifted down: 2 rows from row 1, 64 from row 37, 65 from row 64, 200 from row 1,000 and 1,096 from row 3,000. The first and last rows must also be within 1e-2 of a float64 product, relative to its largest magnitude;
• GEMM timing, default against fixed handle (reference M 64), in one process, alternating, for every 4B, 9B and 27B projection;
• the plain forward on this branch against main: Cua-S1 and Open-Jev-9B workers side by side.

Test Result

CPU. All checks pass on macOS.

Kernel tests. 11 of 11 pass, the new one for all 15 shapes.

Plain forward unchanged, with the final code in #99's tree against main 47eff9c: Cua-S1 567 of 567 byte-identical (134 of them forward passes) in eager and CUDA Graph modes, and Open-Jev-9B 253 of 253 identical apart from timing.

GEMM time (microseconds per call, default / fixed; mean of two rounds, each the median of three runs):

9B	M = 1	32	107	300	1,063	3,109
GDN input	33 / 36	67 / 47	152 / 117	391 / 291	1111 / 1071	3110 / 3474
GDN output	20 / 38	15 / 30	27 / 31	74 / 76	193 / 211	809 / 824
Attention input	23 / 30	46 / 34	80 / 80	167 / 173	850 / 799	2324 / 2306
Attention output	19 / 36	15 / 30	27 / 30	72 / 76	189 / 206	803 / 826
MLP gate and up	229 / 256	242 / 260	309 / 293	769 / 936	2020 / 2575	5912 / 7346
MLP down	32 / 84	46 / 88	118 / 130	322 / 312	1239 / 792	2334 / 3319

4B and 27B

4B	M = 1	32	107	300	1,063	3,109
GDN input	21 / 20	29 / 21	61 / 53	143 / 122	511 / 526	1766 / 1705
GDN output	13 / 31	11 / 30	19 / 30	33 / 38	160 / 143	408 / 443
Attention input	21 / 21	27 / 20	47 / 48	102 / 97	423 / 413	1435 / 1438
Attention output	13 / 30	11 / 30	19 / 30	33 / 38	152 / 130	439 / 462
MLP gate and up	29 / 36	42 / 41	97 / 86	319 / 254	910 / 925	2748 / 2972
MLP down	17 / 64	29 / 64	30 / 64	94 / 81	418 / 356	1525 / 1251

27B	M = 1	32	107	300	1,063	3,109
GDN input	192 / 201	205 / 206	209 / 221	750 / 752	2249 / 2177	5813 / 5773
GDN output	18 / 24	24 / 25	57 / 74	159 / 157	577 / 645	2042 / 2003
Attention input	168 / 176	186 / 180	189 / 200	601 / 625	2025 / 2024	5056 / 5086
Attention output	18 / 24	24 / 25	55 / 70	168 / 167	582 / 663	2053 / 2013
MLP gate and up	403 / 418	428 / 428	711 / 714	1551 / 1527	4692 / 4686	12408 / 12834
MLP down	204 / 218	227 / 221	283 / 273	630 / 657	1860 / 1642	4216 / 5328

Without split-K, the projections with small N take the biggest losses at small M:

• the down projection, up to 3.8 times as long (4B at M = 1);
• the output projections, about 1.9 to 2.7 times on 4B and 9B, and up to 33% longer on 27B.

At large M, the 9B gate and up projection is 22 to 28% slower from 300 rows, and the 9B and 27B down projections are 42% and 26% slower at 3,109. Other cells are faster, such as the 9B GDN input, 23 to 30% faster from 32 to 300 rows. That mix is why the fixed handle isn't the default. In a whole 9B pass, it costs 8% at 1,024 tokens and 18% at 3,072 (#99).

Demo / evidence

No demo: nothing changes for a worker. Timing output and scripts are kept locally.

Self-review

• I have reviewed the full diff and addressed the issues I found.
• I have checked that the change follows the project's architecture and stays focused on the stated purpose.
• I have run the checks appropriate to this change and reported commands, results, and anything I could not verify above.
• I have checked that the PR description, documentation, and any accuracy or performance claims match the implementation and available evidence.