Skip to content

Keep the RoCE ring relay from forwarding a stripe's readiness flag ahead of its earlier packets - #325

Open
iein wants to merge 1 commit into
NVIDIA:mainfrom
iein:roce-relay-flag-order
Open

iein wants to merge 1 commit into
NVIDIA:mainfrom
iein:roce-relay-flag-order

Conversation

@iein

@iein iein commented Nov 17, 2026

Copy link
Copy Markdown

Fixes #324.

Why. The ring relay scan in ring_service reads mailbox serials one slot at a time. A packet that lands between two of those reads can be forwarded before an earlier packet from the same origin and rail. The packet that completes a stripe carries the destination's readiness flag. If it overtakes an earlier packet, the destination reduces stale data in the region that packet had not yet written. On DeepSeek V4.1 Flash and GLM-5.3 Flash at TP4, this shows up as NaN in multi-packet verification all-reduces: requests fail with device error 6, and on DeepSeek the server can also stop with tensor-parallel proposal confidence is invalid. #324 has the full trace.

What changed. For each origin and rail, ring_service now scans in two passes.

  1. The first pass records the newest serial that has arrived but has not been forwarded (the bound).
  2. The second pass reloads the serials and forwards, in serial order, only packets at or below that bound.

The relay receives an origin's packets on one rail through a single queue pair, in posting order. So once the first pass has seen the bound packet, every earlier packet has already landed, and the second pass sees all of them. Packets newer than the bound wait for the next scan. The cost is one extra pass over eight slots per origin and rail per scan. The wire format, proxy ABI and cores are unchanged; existing cores serve without re-export. FlashInfer's preparation copy of the proxy needs the same change and is not part of this PR.

Result. With the fix, DeepSeek V4.1 Flash and GLM-5.3 Flash no longer fail with device error 6, and the DeepSeek NaN proposal-confidence crash no longer occurs.

Model Proxy Batches Failed batches Correctness prompts
DeepSeek V4.1 Flash Unfixed, with a 20 µs busy-wait after each serial load 1 1: device error 6, then the server stopped not run
DeepSeek V4.1 Flash Fixed, with the same busy-wait 47 0 not run
DeepSeek V4.1 Flash Fixed, no busy-wait 120 0 544/544
DeepSeek V4.1 Flash Fixed, no busy-wait, one server process for about 2 hours 480 0 544/544
GLM-5.3 Flash Unfixed, with the busy-wait 50 50: 165 requests with device error 6 failed at the second prompt
GLM-5.3 Flash Fixed, with the busy-wait 30 0 160/160 before the time limit

The tests ran on four GB10 hosts in a switchless TP4 ring with two rails, using a DeepSeek V4.1 Flash core exported at the same base revision and an existing GLM-5.3 Flash core, both with default speculation. The busy-wait was local test instrumentation, not part of this change. It widens the window between serial loads. Runs with it are slower, so the fixed runs that used it stopped at their 45-minute limits, not at the batch target. Each batch had four concurrent streaming requests with distinct prompts of about 900 words and 512 output tokens, using seeded sampling.

The correctness column counts greedy prompts with known answers from a local prompt set. Without the fix and without the busy-wait, the batch workload failed in up to 3 of 60 batches on earlier DeepSeek V4.1 Flash TP4 cores. GLM's failure rate without the busy-wait has not been measured. No unit test covers the ordering; reproducing it needs the relay timing above.

…ead of its earlier packets

In a switchless ring, transfers between non-adjacent ranks pass through a
relay rank's proxy. A rail's payload above 64 KiB is split into several
relay packets, and only the packet that completes a stripe carries the
destination's readiness flag. ring_service loaded the eight mailbox serials
for an origin and rail one slot at a time, then forwarded every arrived
packet in serial order. Those loads are not one snapshot: if packet k lands
after its slot was read and packet k+1 lands before its slot is read, the
relay forwards k+1 without k. When k+1 completes a stripe, the destination
sees its flag and reduces stale bytes in k's region. On DeepSeek V4.1 Flash
and GLM-5.3 Flash at TP4, multi-packet verification all-reduces then
produce NaN, and requests fail with device error 6.

ring_service now scans each origin and rail in two passes. The first pass
records the newest serial that has arrived but has not been forwarded. The
second pass reloads the serials and forwards, in serial order, only packets
at or below that bound. The relay receives one origin's packets on one rail
through a single queue pair in posting order, so every packet older than
the bound had landed before the first pass observed the bound, and the
second pass sees each of them. Newer packets wait for the next scan. The
cost is one extra pass over eight slots per origin and rail per scan. The
wire format, proxy ABI, and cores are unchanged; existing cores serve
without re-export. FlashInfer's b12x copy of the proxy, used during
cross-host preparation, has the same scan and is not changed here.

With the change, the device error 6 failures above no longer occur.
Measured on four GB10 hosts in a switchless TP4 ring with two rails and
default speculation. Each batch sent four concurrent streaming requests
with distinct prompts of about 900 words and 512 output tokens. For local
testing only, a 20 microsecond busy-wait after each serial load in the
relay scan widened the race window.

- DeepSeek V4.1 Flash with the busy-wait: the unpatched proxy failed in the
  first batch, then the server stopped with NaN proposal confidence; the
  patched proxy ran 47 batches without a failure before its time limit.
- DeepSeek V4.1 Flash without the busy-wait: the patched proxy ran 120
  batches and then 480 batches in one server process without a failure,
  and answered 544 of 544 greedy prompts with known answers after each run.
  The unpatched proxy failed in up to 3 of 60 batches on earlier DeepSeek
  V4.1 Flash TP4 cores.
- GLM-5.3 Flash with the busy-wait: the unpatched proxy failed all 50
  batches it ran, and the prompt set failed at its second request; the
  patched proxy ran 30 batches without a failure and answered 160 of 160
  prompts before its time limit.

GLM's failure rate without the busy-wait has not been measured. No unit
test or TLA+ model covers relay packet ordering.

Design-Signed-Off-By: Claude Code (Claude Opus 5.5); design-review=ed22
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Iein Valdez <ivaldez@gmail.com>
@original-el8 original-el8 added area:rocenante RoCEnante cross-host GPU collectives and RoCE transport. bug Something isn't working priority:P1 High attention: unresolved correctness, serving failure, or model-data deletion risk. triage:review Proposed change needs code review; this does not imply merge readiness. labels Nov 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:rocenante RoCEnante cross-host GPU collectives and RoCE transport. bug Something isn't working priority:P1 High attention: unresolved correctness, serving failure, or model-data deletion risk. triage:review Proposed change needs code review; this does not imply merge readiness.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

RoCE ring relay can forward a stripe's flag-carrying packet before an earlier packet, so the far rank reduces stale bytes

2 participants