Optimize SFTP writes - #1603
Open
ddpasa wants to merge 2 commits into
Open
Conversation
jsvk
added a commit
to jsvk/MaterialFiles
that referenced
this pull request
Aug 22, 2026
Squash-merge of upstream PR zhanghai#1603. Replaces synchronous, RTT-bound SFTP writes with pipelined async writes (up to 8 in-flight requests via a new PipelinedFileOutputStream), giving a ~10x throughput improvement on high-latency links. Adds Client.openOutputStream / SftpFileSystemProvider.newOutputStream, exposes RemoteFile.asyncWrite through RemoteFileAccessor, and reorders the copy/close/delete-on-failure logic in SftpCopyMove and ForeignCopyMove so errors surfaced at close still trigger cleanup. The PR's dav4jvm build tweak was dropped since this fork already uses the com.github.zhanghai coordinate. Original work by ddpasa (https://github.com/ddpasa). Co-authored-by: ddpasa <112642920+ddpasa@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
I noticed that SFTP writes were extremely slow with MaterialFiles, especially over wireguard networks. After some digging, it turned out that they were bottlenecked by syncronous writes capped by RTT latency. This PR allows for pipelined writes, which leads to massively improved performance. Over my cell phone connection, writes went up from around 150-200 KB/s to over 1.5MB/s.
I think they can be optimized further by being smarter, but this alone gives x10 improvement over the current state.