Skip to content

[fix] Do not retain the read buffer via the frame mask - #2350

Merged
lpinca merged 1 commit into
websockets:masterfrom
sdrsdr:fix/receiver-mask-retention
Nov 3, 2026
Merged

lpinca merged 1 commit into
websockets:masterfrom
sdrsdr:fix/receiver-mask-retention

Conversation

@sdrsdr

@sdrsdr sdrsdr commented Nov 2, 2026 •

Copy link
Copy Markdown
Contributor

The problem

Receiver#getMask stores the frame mask as a view into the buffer the stream delivered:

this._mask = this.consume(4);

consume(n) returns new FastBuffer(buf.buffer, buf.byteOffset, n) when n is smaller than
the head buffer, and getData is the only reader. Nothing clears it afterwards, so the view
survives until the next frame's getMask overwrites it — holding that buffer alive for as
long as the connection is idle.

Where the 64 KiB comes from

(An earlier revision of this description said node allocates socket read buffers at 64 KB.
That was wrong — see the discussion below. The correction:)

Over plain TCP the read buffer is sized to what arrived, capped at 64 KiB, so a 25-byte write
yields a 25-byte ArrayBuffer. Over TLS it is different: TLSWrap decrypts into a 64 KiB
pool and carves every chunk out of it at an advancing offset. Measured on node v26.13.0,
same 25-byte message:

transport chunk.buffer.byteLength
plain TCP 25
TLS 65536

And the pool is shared across reads — six messages written 40 ms apart, arriving as six separate
reads, all landed in the same 65536-byte ArrayBuffer:

msg 1: mask offset=    2 in a 65536-byte ArrayBuffer  sameBufferAsFirst=true
msg 2: mask offset=   27 in a 65536-byte ArrayBuffer  sameBufferAsFirst=true
msg 3: mask offset=   52 in a 65536-byte ArrayBuffer  sameBufferAsFirst=true
...

So on a wss:// server the retention is size-independent: a 4-byte _mask pins 64 KiB holding
the accumulated plaintext of many unrelated messages, whatever the message size. Over plain TCP
the same thing needs a large chunk to arrive and the connection to then go quiet.

How much it costs

Measured on a production wss:// server holding 37.1k concurrent sockets. Heap snapshot,
grouping native / system / JSArrayBufferData nodes by their single retaining field — 1.29 GB
of backing stores across 36007 nodes:

retaining field slabs bytes share per slab
_mask on Receiver 16502 1031 MB 79.4% 64.0 KiB
ALPNProtocols on TLSSocket._tlsOptions 2809 175 MB 13.5% 63.8 KiB
geoip reader 1 61 MB 4.7% —

Every slab a full 64 KiB, while a message on that service is a few hundred bytes — which is
the pool above, not per-message allocation.

A/B across 12 production hosts, matched within 0.4 h of uptime and 0.9% of connection count,
read as avg_over_time(...[1h]) because the raw series sawtooths ~42%:

ArrayBuffers/conn RSS minor GC event-loop p99
stock 67.3 KB 5.19 GB 102.0/min 96.9 ms
mask cleared 31.6 KB 4.20 GB 84.5/min 70.2 ms

−0.99 GB of RSS per process, and it is cheaper on GC and event-loop latency rather than
dearer
— clearing a property allocates nothing, and a gigabyte less live external memory is a
gigabyte the collector does not keep walking.

Two notes for anyone trying to reproduce this. A load generator speaking plain ws will not
see it at all
, because the TLS pool is never allocated. And a burst-connect rig
understates the _mask share even over TLS: pool waste needs sparse retention, so a rig that
opens every socket at once leaves each small ALPNProtocols buffer alone in its own pooled
slab and ALPN dominates (325 slabs vs 127 for _mask locally). Production's arrivals are
spread out, so those buffers pack ~13 per slab — while a _mask cannot pack at any arrival
rate, because its slab is the per-socket TLS read pool and there is exactly one per socket.

The change

Clear the mask once the payload has been unmasked. getData is its only reader, and the state
machine always reassigns it before the next read — getInfo sets _state = GET_MASK for a
masked frame and getMask sets GET_DATA — so nothing can observe it afterwards.

One property store per frame, no allocation.

bench/parser.benchmark.js, mean of two runs each:

baseline this PR
ping frame (no payload) 4,996k ops/s 5,127k
ping frame (5 B) 4,122k 4,209k
text frame (20 B) 3,206k 3,314k
binary frame (125 B) 3,184k 3,224k
binary frame (64 KiB) 558k 560k
binary frame (1 MiB) 42.7k 41.4k

Every row is within run-to-run variance — the 1 MiB row alone moves 3.7% between two baseline
runs.

I first wrote this as a copy into a preallocated 4-byte buffer, mirroring
Sender._maskBuffer, which is the tidier symmetry. It benchmarked 7.5–8.7% slower on small
frames
(.copy() is too much call overhead for four bytes), so I dropped it. Happy to switch
if you prefer the symmetry and consider that acceptable, but clearing is both the smaller diff
and the faster one.

Test

test/receiver.test.js gains one case asserting that _mask is no longer set once the message
has been delivered. Against the current master receiver it fails with Buffer(4) where
undefined is expected; with the fix it passes. 440 tests pass and receiver.js stays at 100%
coverage.

Prior art in this repo

2b2abd4 ("[security] Limit retained message parts") bounded retained message parts for
essentially this reason — retention scaling with something other than payload size. This is the
same class of problem one level down: a single 4-byte field holding a whole read buffer.

Workaround for anyone hitting this before it lands

ws.on('message', () => { if (ws._receiver) ws._receiver._mask = undefined; });

It reaches into private fields, so pin the ws version or guard it with a test. It is the same
one-line clear, applied from outside — which is how the fix became obvious, after that
workaround had been running in production for a few days.

@lpinca

lpinca commented Nov 2, 2026 •

Copy link
Copy Markdown
Member

Node allocates socket read buffers at 64 KB

I don't understand.

const net = require('net');

const server = net.createServer();

server.on('listening', function () {
  const socket = net.createConnection(server.address().port, 'localhost');

  socket.on('connect', function () {
    const frame = Buffer.from(
      '81933483a86801b992524fa1c60959e68a5216e6cb005ba1d5',
      'hex'
    );

    socket.write(frame);
  });
});

server.on('connection', function (socket) {
  socket.on('data', function (chunk) {
    console.log(chunk.buffer.byteLength);
  });
});

server.listen(0);

In the example 25 bytes are received on the server. If a 64 KiB chunk is received and it contains multiple frames, the whole chunk will be held in memory until it is completely read regardless of this._mask, no?

Comment thread lib/receiver.js Outdated

//
// The mask is a view into the buffer the stream delivered, which is commonly
// 64 KB, and it is not needed once the payload above has been unmasked.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
// 64 KB, and it is not needed once the payload above has been unmasked.
// 64 KiB, and it is not needed once the payload above has been unmasked.

Comment thread test/receiver.test.js Outdated
Comment on lines +68 to +79
const frame = Buffer.from(
'81933483a86801b992524fa1c60959e68a5216e6cb005ba1d5',
'hex'
);

//
// Put the frame at the start of a chunk the size the stream layer commonly
// delivers. `consume()` returns views into it, so a mask left set after the
// frame is parsed keeps the whole thing alive until the next one arrives.
//
const chunk = Buffer.alloc(65536);
frame.copy(chunk);

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
const frame = Buffer.from(
'81933483a86801b992524fa1c60959e68a5216e6cb005ba1d5',
'hex'
);
//
// Put the frame at the start of a chunk the size the stream layer commonly
// delivers. `consume()` returns views into it, so a mask left set after the
// frame is parsed keeps the whole thing alive until the next one arrives.
//
const chunk = Buffer.alloc(65536);
frame.copy(chunk);

It is not needed for the test, just check that receiver._mask is nulled when it is no longer needed.

Comment thread test/receiver.test.js Outdated
});

receiver.write(
new FastBuffer(chunk.buffer, chunk.byteOffset, frame.length)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
new FastBuffer(chunk.buffer, chunk.byteOffset, frame.length)
Buffer.from('81933483a86801b992524fa1c60959e68a5216e6cb005ba1d5', 'hex')

@lpinca

lpinca commented Nov 2, 2026

Copy link
Copy Markdown
Member

Ok, I see the point when the mask is at the end of big chunk of data and connections are idle.

Comment thread test/receiver.test.js Outdated
Comment on lines 12 to 14
const FastBuffer = Buffer[Symbol.species];

describe('Receiver', () => {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
const FastBuffer = Buffer[Symbol.species];
describe('Receiver', () => {
describe('Receiver', () => {

`getMask()` stores the frame mask as a view into the buffer the stream
delivered, and `getData()` is the only reader. The view is left in place
after the frame is parsed, so it keeps that whole buffer alive until the
next frame overwrites it.

Over TLS that buffer is a 64 KiB pool: `TLSWrap` decrypts into it and
carves every chunk out at an advancing offset, so successive reads share
one ArrayBuffer. A 25-byte message therefore leaves a 4-byte `_mask`
pinning 64 KiB, and that 64 KiB holds the accumulated plaintext of many
unrelated messages. On a connection that is then idle between messages,
that is the steady state rather than a transient. Over plain TCP the
read buffer is sized to what arrived (capped at 64 KiB), so the same
retention appears when a large chunk lands and the connection goes quiet.

Measured on a production server holding 37.1k concurrent WSS sockets: of
1.29 GB of retained array buffers, grouping by retaining field put
1031 MB across 16502 slabs on `_mask` alone -- 79.4%, each slab a full
64 KiB. Clearing it took RSS from 5.19 GB to 4.20 GB across a 12-host
A/B, and was *cheaper* on minor GC (84.5/min vs 102.0) and event-loop
p99 (70 ms vs 97 ms), since a gigabyte less live external memory is a
gigabyte the collector does not keep walking.

Clearing after the payload is unmasked costs one property store per
frame and allocates nothing. `bench/parser.benchmark.js` is unchanged
within run-to-run variance. Copying the mask into a preallocated 4-byte
buffer instead -- mirroring what `Sender` does -- also fixes the
retention, but measured 7.5-8.7% slower on small frames, so it is not
worth it.
@sdrsdr
sdrsdr force-pushed the fix/receiver-mask-retention branch from 15036ed to 9ebd070 Compare November 3, 2026 08:06
@sdrsdr

sdrsdr commented Nov 3, 2026

Copy link
Copy Markdown
Contributor Author

You're right, and my sentence was wrong — node does not allocate 64 KB per read. Your script is accurate: over plain TCP the read buffer is sized to what arrived (capped at 64 KiB), so a 25-byte write gives a 25-byte ArrayBuffer.

The 64 KiB comes from TLS. Your script with net → tls and nothing else changed:

const tls = require('tls');
const fs = require('fs');

const server = tls.createServer({
  key: fs.readFileSync('test/fixtures/key.pem'),
  cert: fs.readFileSync('test/fixtures/certificate.pem')
});

server.on('listening', function () {
  const socket = tls.connect({
    port: server.address().port,
    host: 'localhost',
    rejectUnauthorized: false
  });

  socket.on('secureConnect', function () {
    const frame = Buffer.from(
      '81933483a86801b992524fa1c60959e68a5216e6cb005ba1d5',
      'hex'
    );

    socket.write(frame);
  });
});

server.on('secureConnection', function (socket) {
  socket.on('data', function (chunk) {
    console.log(chunk.buffer.byteLength);
  });
});

server.listen(0);

prints 65536 where your version prints 25, for the same 25-byte write.

TLSWrap decrypts into a 64 KiB pool rather than a per-read allocation, and successive reads carve out of the same ArrayBuffer at an advancing offset. Six messages written 40 ms apart, arriving as six separate reads, with Receiver attached:

msg 1: mask offset=    2 in a 65536-byte ArrayBuffer  sameBufferAsFirst=true
msg 2: mask offset=   27 in a 65536-byte ArrayBuffer  sameBufferAsFirst=true
msg 3: mask offset=   52 in a 65536-byte ArrayBuffer  sameBufferAsFirst=true
...

So on a wss:// server a 4-byte _mask does not pin one message's buffer — it pins a pool holding the accumulated plaintext of many unrelated messages.

On "the whole chunk will be held in memory until it is completely read regardless of this._mask": agreed, and that part is fine. The problem is the retention after it has been completely read — _buffers is empty, every frame has been delivered, and the only thing keeping the pool alive is a mask nothing will read again until the next frame arrives. On a connection that is idle between messages, that is the steady state.

I've pushed your suggestions and corrected the claim in the commit message. Thanks for pushing back on it — the sentence was wrong as written and it was load-bearing for the whole argument.

@lpinca
lpinca merged commit 335a610 into websockets:master Nov 3, 2026
62 checks passed
@lpinca

lpinca commented Nov 3, 2026

Copy link
Copy Markdown
Member

Thank you.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants