Repository navigation
Conversation
…ead of its earlier packets In a switchless ring, transfers between non-adjacent ranks pass through a relay rank's proxy. A rail's payload above 64 KiB is split into several relay packets, and only the packet that completes a stripe carries the destination's readiness flag. ring_service loaded the eight mailbox serials for an origin and rail one slot at a time, then forwarded every arrived packet in serial order. Those loads are not one snapshot: if packet k lands after its slot was read and packet k+1 lands before its slot is read, the relay forwards k+1 without k. When k+1 completes a stripe, the destination sees its flag and reduces stale bytes in k's region. On DeepSeek V4.1 Flash and GLM-5.3 Flash at TP4, multi-packet verification all-reduces then produce NaN, and requests fail with device error 6. ring_service now scans each origin and rail in two passes. The first pass records the newest serial that has arrived but has not been forwarded. The second pass reloads the serials and forwards, in serial order, only packets at or below that bound. The relay receives one origin's packets on one rail through a single queue pair in posting order, so every packet older than the bound had landed before the first pass observed the bound, and the second pass sees each of them. Newer packets wait for the next scan. The cost is one extra pass over eight slots per origin and rail per scan. The wire format, proxy ABI, and cores are unchanged; existing cores serve without re-export. FlashInfer's b12x copy of the proxy, used during cross-host preparation, has the same scan and is not changed here. With the change, the device error 6 failures above no longer occur. Measured on four GB10 hosts in a switchless TP4 ring with two rails and default speculation. Each batch sent four concurrent streaming requests with distinct prompts of about 900 words and 512 output tokens. For local testing only, a 20 microsecond busy-wait after each serial load in the relay scan widened the race window. - DeepSeek V4.1 Flash with the busy-wait: the unpatched proxy failed in the first batch, then the server stopped with NaN proposal confidence; the patched proxy ran 47 batches without a failure before its time limit. - DeepSeek V4.1 Flash without the busy-wait: the patched proxy ran 120 batches and then 480 batches in one server process without a failure, and answered 544 of 544 greedy prompts with known answers after each run. The unpatched proxy failed in up to 3 of 60 batches on earlier DeepSeek V4.1 Flash TP4 cores. - GLM-5.3 Flash with the busy-wait: the unpatched proxy failed all 50 batches it ran, and the prompt set failed at its second request; the patched proxy ran 30 batches without a failure and answered 160 of 160 prompts before its time limit. GLM's failure rate without the busy-wait has not been measured. No unit test or TLA+ model covers relay packet ordering. Design-Signed-Off-By: Claude Code (Claude Opus 5.5); design-review=ed22 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Iein Valdez <ivaldez@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #324.
Why. The ring relay scan in
ring_servicereads mailbox serials one slot at a time. A packet that lands between two of those reads can be forwarded before an earlier packet from the same origin and rail. The packet that completes a stripe carries the destination's readiness flag. If it overtakes an earlier packet, the destination reduces stale data in the region that packet had not yet written. On DeepSeek V4.1 Flash and GLM-5.3 Flash at TP4, this shows up as NaN in multi-packet verification all-reduces: requests fail with device error 6, and on DeepSeek the server can also stop withtensor-parallel proposal confidence is invalid. #324 has the full trace.What changed. For each origin and rail,
ring_servicenow scans in two passes.The relay receives an origin's packets on one rail through a single queue pair, in posting order. So once the first pass has seen the bound packet, every earlier packet has already landed, and the second pass sees all of them. Packets newer than the bound wait for the next scan. The cost is one extra pass over eight slots per origin and rail per scan. The wire format, proxy ABI and cores are unchanged; existing cores serve without re-export. FlashInfer's preparation copy of the proxy needs the same change and is not part of this PR.
Result. With the fix, DeepSeek V4.1 Flash and GLM-5.3 Flash no longer fail with device error 6, and the DeepSeek NaN proposal-confidence crash no longer occurs.
The tests ran on four GB10 hosts in a switchless TP4 ring with two rails, using a DeepSeek V4.1 Flash core exported at the same base revision and an existing GLM-5.3 Flash core, both with default speculation. The busy-wait was local test instrumentation, not part of this change. It widens the window between serial loads. Runs with it are slower, so the fixed runs that used it stopped at their 45-minute limits, not at the batch target. Each batch had four concurrent streaming requests with distinct prompts of about 900 words and 512 output tokens, using seeded sampling.
The correctness column counts greedy prompts with known answers from a local prompt set. Without the fix and without the busy-wait, the batch workload failed in up to 3 of 60 batches on earlier DeepSeek V4.1 Flash TP4 cores. GLM's failure rate without the busy-wait has not been measured. No unit test covers the ordering; reproducing it needs the relay timing above.