Skip to content

Keep DeepSeek Engram residuals on their tensor-parallel row owners - #330

Draft
FujitsuPolycom wants to merge 7 commits into
NVIDIA:mainfrom
FujitsuPolycom:codex/deepseek-prefill-owner-pack
Draft

FujitsuPolycom wants to merge 7 commits into
NVIDIA:mainfrom
FujitsuPolycom:codex/deepseek-prefill-owner-pack

Conversation

@FujitsuPolycom

Copy link
Copy Markdown
Contributor

Builds on #282, #283, and #284, which are still under review. This PR adds three commits that keep DeepSeek prefill residuals on the GPUs that own their rows. Review just those three commits.

Why

In DeepSeek V4.1 Flash prefill on four GPUs, each GPU mixes a contiguous group of 2,048 token rows from an 8,192-row batch. At an Engram layer, the existing all-reduce gives every GPU the entire 400 MiB projection, even though each GPU needs only its own rows for residual mixing. At the layer exit, the feed-forward network (FFN) also combines outputs for every row before the next layer selects its local rows.

What

For a single request using the 8,192-row prefill graph, the Engram projection is split into five 80 MiB bands. Each reduce-scatter sums the contributions from all four GPUs and delivers each GPU its rows. The FFN uses the same row ownership for its output. Across attention, each GPU keeps its residual rows and FP32 mixing coefficients, gathering the normalized inputs that attention and the FFN consume for all rows. The smaller mixing operations use the same single-split TF32 reduction configuration as the full-row operations to preserve accumulation order.

This path requires compatible transport buffers and sharded layers on either side of Engram. The tested core was exported with --row-shard-prefill and --row-shard-attention. The prerequisites supply row-sharded residual mixing, attention on local query rows, and the exchange of selected expert IDs and weights, respectively. #282 · #283 · #284

Benefit

On four GB10 GPUs, keeping residuals local across attention reduced complete Layer 1 time from 120.18 to 94.29 ms with the Engram and FFN changes already enabled. Layer numbers below use the model’s zero-based indices.

Timed operation Before After
Engram reduction and mixing, including packing and the final gather 76.64 ms 52.91 ms
FFN output reduction, residual mixing and handoff to the next layer 22.17 ms 14.30 ms
Complete Layer 1, adding the attention change 120.18 ms 94.29 ms

Each value is the median of 20 replays on FlashInfer 7b35510b, using the slowest GPU for each replay. The first two measurements start after the projections have produced their partial results. Adding these savings would double-count work included in the layer timing.

Validation and limits

On FlashInfer 4fd836dc, the compiled Layer 14 and 15 tests continued from an 8,191-token prefix with text or encoded image features. The 8,192-row graph ran with all rows active, with 2,049 active rows crossing the first GPU’s ownership boundary, and with 8,191 active rows leaving one padding row.

The full-row and row-sharded paths produced different outputs when they chose sparse attention blocks independently. FlashInfer permits different block IDs when scores tie at the selection cutoff. Both selectors ran and passed its exact score check on every row. When both paths consumed the same verified selection, Layer 14’s FFN inputs, expert IDs, FP32 expert weights and residual mixing coefficients matched bit for bit. So did the attention window, pending values, gates and position, Engram history, and latent and index cache pages. Residuals after expert execution and some Layer 15 outputs still varied, including between repeats of the same path. Whole-model numerical equivalence remains open.

On four GB10 GPUs, flare run served DeepSeek V4.1 Flash with these model changes and additional scheduler and transport changes. For one request at each of 16k, 64k and 128k prompt tokens, prompt length divided by client time to first token gave 6,067, 6,165 and 6,308 tok/s, respectively. An earlier 16k request on the same server process measured 5,549 tok/s. Decode completed at all three prompt lengths with 1, 2, 4, 8 and 16 concurrent requests.

In a separate cache-pressure test, the runtime evicted two additional snapshots, restored the retained 8,195-token prefix byte for byte on all four GPUs, processed an eight-token continuation and generated 32 tokens. The allocation observer recorded no CUDA storage-management calls between runtime readiness and completion of queued GPU work.

FujitsuPolycom and others added 7 commits November 14, 2026 23:26
Full-row hyperconnections repeat the same prefill work on every tensor-parallel
rank. Reduce-scatter their branch outputs into contiguous rank-owned rows,
normalize there, and gather the branch inputs needed by full-row consumers.
Engram layers and decoder boundaries retain complete rows and continuation state.

Add the explicit preparation option, fixed-shape shared collective components,
row-transition scheduling, and preparation-only rebuild metadata. Regenerate the
Python bindings from the combined schema while retaining current main's fields.
The intended benefit is less repeated hyperconnection work without changing the
serving operation families or the disabled-option behavior.

Three focused CLI/planning/metadata round-trip tests pass. The broader Windows
CLI selection has 205 passes and 41 failures; the same 41 failures reproduce on
unmodified main in this environment, which lacks Heliopause and assumes different
configuration paths. Syntax and staged whitespace checks pass. Current-default
CUDA preparation and native serving have not been run on this head.

Design-Signed-Off-By: Codex (GPT-6); design-review=ed22
Signed-off-by: FujitsuPolycom <87842395+FujitsuPolycom@users.noreply.github.com>
Tensor-parallel prefill gathers normalized rows before attention and reduces its
output afterward. Compute all attention heads and output groups for each rank's
own rows, then gather the cache projections required by full-row consumers.
This removes those exchanges while preserving complete cache writes.

Load replicated query/output projections only for eligible prefill layers,
retain partitioned decode copies, and carry selection state across row
transitions. Account for values, both retained MXFP8 scale layouts and FP32
sinks: 17 layers add 2,121,273,600 numeric bytes per rank. Preserve main's
metadata and online-conversion options. Document the qualified hardware and
retained-core scope separately from unrun dependency/mode configurations.

The benefit is less attention communication at a retained-weight cost. Grouping
all eight output groups changes BF16 rounding association; comparisons use
operator-derived bounds rather than exact partitioned-output equality.

Validation includes accounting/layout tests and combined-stack GPU/native
checks. On 32 captured native rows, all 163,840 projection outputs satisfy the
unchanged operator budget and match native BF16 bits. Existing 7b35510b/ABI-14
cores pass checkpoint/Restore, inactive-state, replay and allocation checks;
4fd836dc component/compiler checks pass separately. Floating cache differences,
baseline-reproduced replay failures and measured dependency scopes remain
explicit; no fresh default-dependency whole-model export or isolated speedup
is claimed. Computational source is unchanged from the reviewed 8d38f298 tree.

Design-Signed-Off-By: Codex (GPT-6); design-review=ed22
Signed-off-by: FujitsuPolycom <87842395+FujitsuPolycom@users.noreply.github.com>
Row-sharded attention owns each rank's query rows, but gathering all 384
router logits repeats communication and top-k work. Gather six selected
expert IDs and their FP32 weight bits in a 48-byte record, then unpack
contiguous expert operands without numeric conversion.

Prepare the router image mask at the rank-local row count, including
nonzero ranks, while retaining the full layer mask and runtime selection.
Use sqrtsoftplus and larger-ID ties in the reference, require nonempty
margin coverage, and reject incorrect routes on ambiguous rows. Preserve
main's optional activation-mode forwarding and explicit scratch operands.

The change reduces logical gathered route payload by 32 times at top-k
six. It introduces no independent runtime policy or isolated service-gain
claim; route gathering follows opt-in row-sharded attention.

Validation: four-host GB10 GPU route-record and image-router preparation
regressions pass with FlashInfer 4fd836dc, including exact IDs/FP32 bits,
partial/padded rows, ties and compiled graph transitions. Combined-stack
existing-core native cache/replay checks retain their 7b35510b/ABI-14
artifact scope. CPU reference tests reject wrong routes, empty coverage
and mask shape errors. Model numerical limits and older baseline replay
failures remain documented; no fresh default-dependency whole-model
export is claimed. The separate tail-capacity/masking proposal is excluded.

Design-Signed-Off-By: Codex (GPT-6); design-review=ed22
Signed-off-by: FujitsuPolycom <87842395+FujitsuPolycom@users.noreply.github.com>
The three proposed prefill changes build on one another and were reviewed
on an earlier main. Merge their published tips into main 5fd4070 to
provide one source base for the additional model and runtime changes.

This retains the three original commits and current main's cache and
platform fixes. The merge tree has the published content in all 34 paths
changed only by those PRs. The CLI design and TODO file include edits
from both parents. The index has no conflicts and passes diff --check.
Hardware measurements are attached to the builds on which they ran.

Design-Signed-Off-By: Codex (GPT-6); design-review=ed22
Signed-off-by: Codex (GPT-6) <codex@localhost>
An 8192-row tensor-parallel prefill gathers the four-stream residual
before Engram and all-reduces its 400 MiB projected features on every
rank. Engram mixing is row-local, so neither full residuals nor reduced
features are needed on every rank until the following attention.

Add pipe-sized column reduce-scatter and use it at eligible TP4 Engram
transitions. Keep lookup and partitioned projection full-row, mix each
rank's 2048 rows, then gather the mixed residual for full-row attention.
The existing FP32 mix gather, mask, readiness and table requests remain.
Other capacities, scored execution and incompatible layouts keep their
existing paths; the runtime and core format do not change.

On four GB10 GPUs with FlashInfer 7b35510b, communication and mixing fell
from 76.643 to 52.913 ms, including packing and the output gather. Timing
started with the projection partials ready and used the slowest GPU in
each of 20 replays per variant. The figures are medians.

GPU checks on 7b35510b exercised the real table lookup and projection.
With FlashInfer 4fd836dc, 58 CPU tests covered row
layout, eligibility and the handoff to attention. Compiled text and image
tests on four GB10s used the combined Engram, FFN and attention changes.
The Engram lookup, projection and reduced rows matched the full-row
reference bit for bit on the tested inputs.

Signed-off-by: FujitsuPolycom <87842395+FujitsuPolycom@users.noreply.github.com>
Design-Signed-Off-By: Codex (GPT-6.2 Astra); design-review=ed22
Packed prefill completes an Engram layer's full-row FFN with an all-reduce
and full-row post-mix, then selects each rank's rows to restart sharded
execution. The next layer consumes only those owner rows, so the reduction
can deliver them directly without computing a discarded full-row post.

At eligible declared Engram-to-sharded transitions, reduce-scatter the FFN
partial and index-select the matching residual, post, combination and FP32
mix rows before the learned post. Pass these owner outputs directly to the
successor and omit its reseed. Preserve full-row attention and FFN inputs,
selection movement, and the paths for scoring, auxiliary consumers,
suffix boundaries and other capacities.

On four GB10 GPUs with FlashInfer 7b35510b, the FFN reduction, residual
mixing and handoff fell from 22.169 to 14.301 ms. These are medians
over 20 replays per variant, timing the slowest GPU from prepared FFN
partials through the complete local outputs.

On FlashInfer 7b35510b, GPU checks compared the complete owner outputs
and FP32 mixing coefficients bit for bit and checked replay allocations.
With FlashInfer 4fd836dc, 151 CPU tests covered eligibility, construction,
prefill transitions and buffer lifetimes. The native export and replay
tests on that revision of FlashInfer included all three model changes.
In the compiled layer comparison, post-expert residuals differed
between paths and also varied on repeats of a single path.

Signed-off-by: FujitsuPolycom <87842395+FujitsuPolycom@users.noreply.github.com>
Design-Signed-Off-By: Codex (GPT-6.2 Astra); design-review=ed22
Engram prefill widens the mixed four-stream residual before attention,
then computes mHC for rows discarded by the following owner transition.
Keeping mHC on owner rows avoids this residual transfer, but changing
its split reduction also changes the rounded FFN inputs and routing.

Keep eligible Engram residuals and FP32 mix on each rank's 2048 rows,
gather only normalized inputs for the unchanged full-row attention and
FFN, and use their reduce-scattered outputs with owner mHC. Retain the
attention schedule's original selection representation and unreplicated
weights. Pass the full-row TF32 mHC projection configurations through
the public preparation override before tuning; owner queries and scratch
remain 2048 rows. Other capacities and consumer boundaries are unchanged.

On four GB10 GPUs with FlashInfer 7b35510b, median Layer 1 time fell
from 120.184 to 94.288 ms. Each variant ran 20 replays from the same
Layer 0 inputs and cache history, timing the slowest GPU per replay. All seven observed consumer inputs, route IDs and FP32 weights
matched exactly, as did FP32 mix and saved window. Final BF16 outputs
were not bitwise stable in either arm, so this is diagnostic component
evidence rather than whole-model numerical acceptance or an isolated
serving gain.

With FlashInfer 4fd836dc, 207 CPU tests covered construction, fallback
shapes, reduction configuration and row ownership. In a four-GB10
Layer 1 comparison with attention precision held fixed, three inputs
produced identical saved window values, expert IDs and FP32 route weights.

With FlashInfer 4fd836dc, Ignition exported all three model changes
together and flare run served the resulting core on four GB10s. In compiled Layer 14 text and image
tests, each sparse selector passed the exact score check. With the same
valid block selection on both paths, FFN inputs, route IDs, FP32 route
weights and the compared cache regions matched bit for bit. Post-expert
residuals and some Layer 15 outputs still differed. Whole-model numerical
equivalence remains open.

Signed-off-by: FujitsuPolycom <87842395+FujitsuPolycom@users.noreply.github.com>
Design-Signed-Off-By: Codex (GPT-6.2 Astra); design-review=ed22
@original-el8 original-el8 added area:kernels GPU kernels and numerical execution. area:preparation Ignition checkpoint loading, tuning, compilation, capture, and export. enhancement New feature or request model:deepseek-v4.1 Affected recipe or documented reproducer: deepseek-v4.1. Impact may be broader. performance Performance, latency, throughput, or memory-efficiency investigation or improvement. triage:validation Proposed implementation needs acceptance evidence; no claim that a reported defect is fixed. labels Nov 18, 2026
@original-el8

Copy link
Copy Markdown
Collaborator

This extends the still-unmerged #282 → #283 → #284 stack. Main 5fd4070 already contains the general snapshot-planning optimization #309, but not those row-sharding changes. This head merges cleanly; clean mergeability does not resolve the predecessor reviews. Please keep the incremental Engram/FFN row-ownership change and its own state, numerical and retained-memory evidence distinct from the predecessor stack, and carry the final combined export/qualification revision through review. I have checked dependency status and mergeability, not independently qualified this GPU path.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:kernels GPU kernels and numerical execution. area:preparation Ignition checkpoint loading, tuning, compilation, capture, and export. enhancement New feature or request model:deepseek-v4.1 Affected recipe or documented reproducer: deepseek-v4.1. Impact may be broader. performance Performance, latency, throughput, or memory-efficiency investigation or improvement. triage:validation Proposed implementation needs acceptance evidence; no claim that a reported defect is fixed.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants