perf(cuda): the recurrent-state GET_ROWS folds into GATED_DELTA_NET's load, f32 or q8_0 cache (#220 plus -cts q8_0)

[marcospaulo]

What

build_rs gathers each Gated DeltaNet layer's recurrent state out of the cache with a GET_ROWS into a temp, and only
the GATED_DELTA_NET reads that temp. With a q8_0 state cache (-cts q8_0, the served head), that is a 786,432-element
dequantizing copy per layer, 5.2 µs at a decode step and 5.6 µs at a 3-row verify on the RTX 5070 Ti (Nsight,
research/gdn-q8-cache-fusion-2026-09-22), 48 times a step.

The CUDA evaluator now skips that GET_ROWS (ggml_cuda_try_gdn_gather_skip) and registers the gather for the GDN
node; the kernel reads the sequence's cache row ids[seq] itself, f32 or q8_0 dequantized as it loads with
get_rows' formula, so it sees the values the skipped GET_ROWS would have written. From PrismML-Eng#220 (f32),
plus the q8_0 cache.

Declined, keeping the GET_ROWS, when:

• more than one sequence is in the ubatch (a copied sequence's source row can be another's destination, which the
  GET_ROWS read before any write and the kernel's blocks would race on);
• anything else reads the temp;
• a node between the two writes the cache (build_rs's extra-states copy relocates rows there);
• a node up to and including the GDN sits on ids' memory (b534681). The kernel reads ids at the GDN, but the
  skipped GET_ROWS can be ids' last reader in graph order (the last recurrent layer's), after which ggml-alloc frees
  ids (an input copy is not an output with one GPU) and may place a later node on it. cuda: fold the recurrent-state gather into the GATED_DELTA_NET kernel, per-context registry (+3.8% TG on Ada) PrismML-Eng/llama.cpp#220 has the same exposure;
• the GDN takes the chunked prefill path (its pipeline reads a gathered s0).

Off switch: GGML_CUDA_GDN_STATE_GATHER_LEGACY=1.

Verified

• test-backend-ops GATED_DELTA_NET_CACHE_FUSION 36/36, with and without the switch; GATED_DELTA_NET 63/63
  (RTX PRO 4000 Blackwell, sm_120). New cases: the state gathered from row 1 or 4 of an f32 or q8_0 cache, 1 or 3
  tokens, and the chunked case, which declines.
• Mutants, each rebuilt and run against the fusion cases: the kernel ignoring ids (28/36 pass), a q8_0 row read
  without its block scale (32/36), q8_0 bytes read as f32 (32/36). Restored: 36/36.
• The real served graph (box-only instrumentation, never committed; served GDN flags, one greedy completion): the
  fold takes all 48 GDN layers at every capture; the ids guard never declines there.

Measured (RTX 5070 Ti, 595.71.05, e0da2d1: this branch plus perf/gdn-conv-state-update, one binary three ways by the two switches)

Nsight Systems, llama-perplexity over 1,024 tokens with the served cache flags, legs legacy / fold / new / new / fold / legacy:

rows	q8_0 gather launches / step	GDN kernel	GPU ms / step (legacy → fold)	PPL
1	48 → 0 (270 µs)	266 → 306 µs	12.698 → 12.338 (−2.8 %)	6.4770 all legs
3	48 → 0 (272 µs)	434 → 473 µs	14.774 → 14.427 (−2.3 %)	6.4690 all legs

Served, the head's flags (public MTP r2 pack, -cts q8_0, MTP n_max 2, one slot at 40,960), the 48 held-out eval requests
greedy, legs new / fold / legacy / legacy / fold / new, per request with leg order cancelled:

• the fold against legacy: +1.69 % tok/s (95 % CI +1.59 to +1.80), 48 of 48 faster;
• text identical in all six legs (48/48 each), draft acceptance 0.7274 in every leg.

Artifacts: libggml-cuda 24d7516a…, the pack 0e5524be…; rig local/research/gdn-conv-state-update-2026-09-25/served-ab-e0da2d116.
TORAD.md row added.

Where this goes next

perf/gdn-conv-state-update (stacked on this branch) folds the conv-state chain the same way.