softnpu: Don't hold pipeline mutex across a blocking operation - #1201
Closed
sion42x wants to merge 1 commit into
Closed
softnpu: Don't hold pipeline mutex across a blocking operation#1201sion42x wants to merge 1 commit into
sion42x wants to merge 1 commit into
Conversation
On the external ingress path, the lock will only cover `process_packet`. A syscall inside the critical section descheduled the holder and every packet has wait during scheduler wakeup.
Author
|
@zeeshanlakhani I'd appreciate your take on this. The change in performance was dramatic. While I agree with the TODO that we shouldn't be using a mutex at all, this is a good half measure whilst I work on a possible solution there. |
This was referenced Aug 28, 2026
Author
|
Closed for now with the changes made by Ry for #1204 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Each softnpu port has an ingress that held a lock through multiple stages, and since sending is a syscall the lock holder was descheduled inside the critical section on every forwarded packet. This resulted in every waiting port thread to go to sleep and every packet handoff to go through a scheduler wakeup, which in turn results in a 6.6k pps rate on the whole switch.
On a voxel test, the emulated fabric topped out at 6.2k pps with a cross-sled TCP throughput of 100-340KB/s (loopback is 118MB/s on the same endpoint).
This change scopes the lock to
process_packetand forwards it after it drops on both external and guest paths, and the management handler stops holding the lock while writing replies over UART. Single flow throughput go from 0.1-0.34 -> 5 MB/s, and retransmits droped from 346% -> 29% of transmitted bytes. Per flow ordering remains unchanged.