Repository navigation
asyncio: large bytes write is quadratic when queued behind pending data #159503
Copy link
Copy link
Open
Labels
performancePerformance or resource usagePerformance or resource usagestdlibStandard Library Python modules in the Lib/ directoryStandard Library Python modules in the Lib/ directorytopic-asynciotype-bugAn unexpected behavior, bug, or errorAn unexpected behavior, bug, or error
Description
Bug report
Bug report
Bug description:
When a bytes object is written to a selector socket transport while data is already buffered, each partial send re-queues the leftover as b[nbytes:] (_adjust_leftover_buffer) or buffer[n:] (_write_send), copying the whole remainder every time. One large write is therefore O(n**2). write() already avoids this on its first-send path with memoryview(data)[n:].
One write queued behind a 2 MiB prefix on a socketpair (Linux, main, best of 3):
size before after
16 MiB 205 ms 12.3 ms
48 MiB 1633 ms 14.4 ms
bench_write_copy.py
# Time one large bytes write that has to queue behind data already buffered
# in an asyncio socket transport.
# Usage: python bench_write_copy.py [size in MiB, default 48]
import asyncio
import socket
import sys
import time
SIZE = int(sys.argv[1] if len(sys.argv) > 1 else 48) * 1024 * 1024
PREFIX = 2 * 1024 * 1024
class Reader(asyncio.Protocol):
"""Counts received bytes and resolves `done` once all have arrived."""
def __init__(self, expected, done):
self.received = 0
self.expected = expected
self.done = done
def data_received(self, data):
self.received += len(data)
if self.received == self.expected:
self.done.set_result(None)
async def main():
loop = asyncio.get_running_loop()
sock_a, sock_b = socket.socketpair()
done = loop.create_future()
reader, _ = await loop.connect_accepted_socket(
lambda: Reader(PREFIX + SIZE, done), sock_b)
writer, _ = await loop.connect_accepted_socket(asyncio.Protocol, sock_a)
payload = b"x" * SIZE
start = time.perf_counter()
writer.write(b"p" * PREFIX) # fills the socket, so the next write is buffered
writer.write(payload) # sent in many partial chunks
await done
elapsed = time.perf_counter() - start
writer.close()
reader.close()
await asyncio.sleep(0)
print(f"{SIZE >> 20} MiB write: {elapsed * 1000:.1f} ms")
asyncio.run(main())CPython versions tested on:
3.14, CPython main branch
Operating systems tested on:
Linux
Linked PRs
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
performancePerformance or resource usagePerformance or resource usagestdlibStandard Library Python modules in the Lib/ directoryStandard Library Python modules in the Lib/ directorytopic-asynciotype-bugAn unexpected behavior, bug, or errorAn unexpected behavior, bug, or error
Projects
- StatusShow more project fieldsTodo
Activity