Repository navigation
Keep DeepSeek Engram residuals on their tensor-parallel row owners - #330
Draft
FujitsuPolycom wants to merge 7 commits into
Draft
FujitsuPolycom wants to merge 7 commits into
FujitsuPolycom wants to merge 7 commits into
Conversation
Full-row hyperconnections repeat the same prefill work on every tensor-parallel rank. Reduce-scatter their branch outputs into contiguous rank-owned rows, normalize there, and gather the branch inputs needed by full-row consumers. Engram layers and decoder boundaries retain complete rows and continuation state. Add the explicit preparation option, fixed-shape shared collective components, row-transition scheduling, and preparation-only rebuild metadata. Regenerate the Python bindings from the combined schema while retaining current main's fields. The intended benefit is less repeated hyperconnection work without changing the serving operation families or the disabled-option behavior. Three focused CLI/planning/metadata round-trip tests pass. The broader Windows CLI selection has 205 passes and 41 failures; the same 41 failures reproduce on unmodified main in this environment, which lacks Heliopause and assumes different configuration paths. Syntax and staged whitespace checks pass. Current-default CUDA preparation and native serving have not been run on this head. Design-Signed-Off-By: Codex (GPT-6); design-review=ed22 Signed-off-by: FujitsuPolycom <87842395+FujitsuPolycom@users.noreply.github.com>
Tensor-parallel prefill gathers normalized rows before attention and reduces its output afterward. Compute all attention heads and output groups for each rank's own rows, then gather the cache projections required by full-row consumers. This removes those exchanges while preserving complete cache writes. Load replicated query/output projections only for eligible prefill layers, retain partitioned decode copies, and carry selection state across row transitions. Account for values, both retained MXFP8 scale layouts and FP32 sinks: 17 layers add 2,121,273,600 numeric bytes per rank. Preserve main's metadata and online-conversion options. Document the qualified hardware and retained-core scope separately from unrun dependency/mode configurations. The benefit is less attention communication at a retained-weight cost. Grouping all eight output groups changes BF16 rounding association; comparisons use operator-derived bounds rather than exact partitioned-output equality. Validation includes accounting/layout tests and combined-stack GPU/native checks. On 32 captured native rows, all 163,840 projection outputs satisfy the unchanged operator budget and match native BF16 bits. Existing 7b35510b/ABI-14 cores pass checkpoint/Restore, inactive-state, replay and allocation checks; 4fd836dc component/compiler checks pass separately. Floating cache differences, baseline-reproduced replay failures and measured dependency scopes remain explicit; no fresh default-dependency whole-model export or isolated speedup is claimed. Computational source is unchanged from the reviewed 8d38f298 tree. Design-Signed-Off-By: Codex (GPT-6); design-review=ed22 Signed-off-by: FujitsuPolycom <87842395+FujitsuPolycom@users.noreply.github.com>
Row-sharded attention owns each rank's query rows, but gathering all 384 router logits repeats communication and top-k work. Gather six selected expert IDs and their FP32 weight bits in a 48-byte record, then unpack contiguous expert operands without numeric conversion. Prepare the router image mask at the rank-local row count, including nonzero ranks, while retaining the full layer mask and runtime selection. Use sqrtsoftplus and larger-ID ties in the reference, require nonempty margin coverage, and reject incorrect routes on ambiguous rows. Preserve main's optional activation-mode forwarding and explicit scratch operands. The change reduces logical gathered route payload by 32 times at top-k six. It introduces no independent runtime policy or isolated service-gain claim; route gathering follows opt-in row-sharded attention. Validation: four-host GB10 GPU route-record and image-router preparation regressions pass with FlashInfer 4fd836dc, including exact IDs/FP32 bits, partial/padded rows, ties and compiled graph transitions. Combined-stack existing-core native cache/replay checks retain their 7b35510b/ABI-14 artifact scope. CPU reference tests reject wrong routes, empty coverage and mask shape errors. Model numerical limits and older baseline replay failures remain documented; no fresh default-dependency whole-model export is claimed. The separate tail-capacity/masking proposal is excluded. Design-Signed-Off-By: Codex (GPT-6); design-review=ed22 Signed-off-by: FujitsuPolycom <87842395+FujitsuPolycom@users.noreply.github.com>
The three proposed prefill changes build on one another and were reviewed on an earlier main. Merge their published tips into main 5fd4070 to provide one source base for the additional model and runtime changes. This retains the three original commits and current main's cache and platform fixes. The merge tree has the published content in all 34 paths changed only by those PRs. The CLI design and TODO file include edits from both parents. The index has no conflicts and passes diff --check. Hardware measurements are attached to the builds on which they ran. Design-Signed-Off-By: Codex (GPT-6); design-review=ed22 Signed-off-by: Codex (GPT-6) <codex@localhost>
An 8192-row tensor-parallel prefill gathers the four-stream residual before Engram and all-reduces its 400 MiB projected features on every rank. Engram mixing is row-local, so neither full residuals nor reduced features are needed on every rank until the following attention. Add pipe-sized column reduce-scatter and use it at eligible TP4 Engram transitions. Keep lookup and partitioned projection full-row, mix each rank's 2048 rows, then gather the mixed residual for full-row attention. The existing FP32 mix gather, mask, readiness and table requests remain. Other capacities, scored execution and incompatible layouts keep their existing paths; the runtime and core format do not change. On four GB10 GPUs with FlashInfer 7b35510b, communication and mixing fell from 76.643 to 52.913 ms, including packing and the output gather. Timing started with the projection partials ready and used the slowest GPU in each of 20 replays per variant. The figures are medians. GPU checks on 7b35510b exercised the real table lookup and projection. With FlashInfer 4fd836dc, 58 CPU tests covered row layout, eligibility and the handoff to attention. Compiled text and image tests on four GB10s used the combined Engram, FFN and attention changes. The Engram lookup, projection and reduced rows matched the full-row reference bit for bit on the tested inputs. Signed-off-by: FujitsuPolycom <87842395+FujitsuPolycom@users.noreply.github.com> Design-Signed-Off-By: Codex (GPT-6.2 Astra); design-review=ed22
Packed prefill completes an Engram layer's full-row FFN with an all-reduce and full-row post-mix, then selects each rank's rows to restart sharded execution. The next layer consumes only those owner rows, so the reduction can deliver them directly without computing a discarded full-row post. At eligible declared Engram-to-sharded transitions, reduce-scatter the FFN partial and index-select the matching residual, post, combination and FP32 mix rows before the learned post. Pass these owner outputs directly to the successor and omit its reseed. Preserve full-row attention and FFN inputs, selection movement, and the paths for scoring, auxiliary consumers, suffix boundaries and other capacities. On four GB10 GPUs with FlashInfer 7b35510b, the FFN reduction, residual mixing and handoff fell from 22.169 to 14.301 ms. These are medians over 20 replays per variant, timing the slowest GPU from prepared FFN partials through the complete local outputs. On FlashInfer 7b35510b, GPU checks compared the complete owner outputs and FP32 mixing coefficients bit for bit and checked replay allocations. With FlashInfer 4fd836dc, 151 CPU tests covered eligibility, construction, prefill transitions and buffer lifetimes. The native export and replay tests on that revision of FlashInfer included all three model changes. In the compiled layer comparison, post-expert residuals differed between paths and also varied on repeats of a single path. Signed-off-by: FujitsuPolycom <87842395+FujitsuPolycom@users.noreply.github.com> Design-Signed-Off-By: Codex (GPT-6.2 Astra); design-review=ed22
Engram prefill widens the mixed four-stream residual before attention, then computes mHC for rows discarded by the following owner transition. Keeping mHC on owner rows avoids this residual transfer, but changing its split reduction also changes the rounded FFN inputs and routing. Keep eligible Engram residuals and FP32 mix on each rank's 2048 rows, gather only normalized inputs for the unchanged full-row attention and FFN, and use their reduce-scattered outputs with owner mHC. Retain the attention schedule's original selection representation and unreplicated weights. Pass the full-row TF32 mHC projection configurations through the public preparation override before tuning; owner queries and scratch remain 2048 rows. Other capacities and consumer boundaries are unchanged. On four GB10 GPUs with FlashInfer 7b35510b, median Layer 1 time fell from 120.184 to 94.288 ms. Each variant ran 20 replays from the same Layer 0 inputs and cache history, timing the slowest GPU per replay. All seven observed consumer inputs, route IDs and FP32 weights matched exactly, as did FP32 mix and saved window. Final BF16 outputs were not bitwise stable in either arm, so this is diagnostic component evidence rather than whole-model numerical acceptance or an isolated serving gain. With FlashInfer 4fd836dc, 207 CPU tests covered construction, fallback shapes, reduction configuration and row ownership. In a four-GB10 Layer 1 comparison with attention precision held fixed, three inputs produced identical saved window values, expert IDs and FP32 route weights. With FlashInfer 4fd836dc, Ignition exported all three model changes together and flare run served the resulting core on four GB10s. In compiled Layer 14 text and image tests, each sparse selector passed the exact score check. With the same valid block selection on both paths, FFN inputs, route IDs, FP32 route weights and the compared cache regions matched bit for bit. Post-expert residuals and some Layer 15 outputs still differed. Whole-model numerical equivalence remains open. Signed-off-by: FujitsuPolycom <87842395+FujitsuPolycom@users.noreply.github.com> Design-Signed-Off-By: Codex (GPT-6.2 Astra); design-review=ed22
Collaborator
|
This extends the still-unmerged #282 → #283 → #284 stack. Main 5fd4070 already contains the general snapshot-planning optimization #309, but not those row-sharding changes. This head merges cleanly; clean mergeability does not resolve the predecessor reviews. Please keep the incremental Engram/FFN row-ownership change and its own state, numerical and retained-memory evidence distinct from the predecessor stack, and carry the final combined export/qualification revision through review. I have checked dependency status and mergeability, not independently qualified this GPU path. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Builds on #282, #283, and #284, which are still under review. This PR adds three commits that keep DeepSeek prefill residuals on the GPUs that own their rows. Review just those three commits.
Why
In DeepSeek V4.1 Flash prefill on four GPUs, each GPU mixes a contiguous group of 2,048 token rows from an 8,192-row batch. At an Engram layer, the existing all-reduce gives every GPU the entire 400 MiB projection, even though each GPU needs only its own rows for residual mixing. At the layer exit, the feed-forward network (FFN) also combines outputs for every row before the next layer selects its local rows.
What
For a single request using the 8,192-row prefill graph, the Engram projection is split into five 80 MiB bands. Each reduce-scatter sums the contributions from all four GPUs and delivers each GPU its rows. The FFN uses the same row ownership for its output. Across attention, each GPU keeps its residual rows and FP32 mixing coefficients, gathering the normalized inputs that attention and the FFN consume for all rows. The smaller mixing operations use the same single-split TF32 reduction configuration as the full-row operations to preserve accumulation order.
This path requires compatible transport buffers and sharded layers on either side of Engram. The tested core was exported with
--row-shard-prefilland--row-shard-attention. The prerequisites supply row-sharded residual mixing, attention on local query rows, and the exchange of selected expert IDs and weights, respectively. #282 · #283 · #284Benefit
On four GB10 GPUs, keeping residuals local across attention reduced complete Layer 1 time from 120.18 to 94.29 ms with the Engram and FFN changes already enabled. Layer numbers below use the model’s zero-based indices.
Each value is the median of 20 replays on FlashInfer
7b35510b, using the slowest GPU for each replay. The first two measurements start after the projections have produced their partial results. Adding these savings would double-count work included in the layer timing.Validation and limits
On FlashInfer
4fd836dc, the compiled Layer 14 and 15 tests continued from an 8,191-token prefix with text or encoded image features. The 8,192-row graph ran with all rows active, with 2,049 active rows crossing the first GPU’s ownership boundary, and with 8,191 active rows leaving one padding row.The full-row and row-sharded paths produced different outputs when they chose sparse attention blocks independently. FlashInfer permits different block IDs when scores tie at the selection cutoff. Both selectors ran and passed its exact score check on every row. When both paths consumed the same verified selection, Layer 14’s FFN inputs, expert IDs, FP32 expert weights and residual mixing coefficients matched bit for bit. So did the attention window, pending values, gates and position, Engram history, and latent and index cache pages. Residuals after expert execution and some Layer 15 outputs still varied, including between repeats of the same path. Whole-model numerical equivalence remains open.
On four GB10 GPUs,
flare runserved DeepSeek V4.1 Flash with these model changes and additional scheduler and transport changes. For one request at each of 16k, 64k and 128k prompt tokens, prompt length divided by client time to first token gave 6,067, 6,165 and 6,308 tok/s, respectively. An earlier 16k request on the same server process measured 5,549 tok/s. Decode completed at all three prompt lengths with 1, 2, 4, 8 and 16 concurrent requests.In a separate cache-pressure test, the runtime evicted two additional snapshots, restored the retained 8,195-token prefix byte for byte on all four GPUs, processed an eight-token continuation and generated 32 tokens. The allocation observer recorded no CUDA storage-management calls between runtime readiness and completion of queued GPU work.