Repository navigation
[fix] Do not retain the read buffer via the frame mask - #2350
Conversation
I don't understand. const net = require('net');
const server = net.createServer();
server.on('listening', function () {
const socket = net.createConnection(server.address().port, 'localhost');
socket.on('connect', function () {
const frame = Buffer.from(
'81933483a86801b992524fa1c60959e68a5216e6cb005ba1d5',
'hex'
);
socket.write(frame);
});
});
server.on('connection', function (socket) {
socket.on('data', function (chunk) {
console.log(chunk.buffer.byteLength);
});
});
server.listen(0);In the example 25 bytes are received on the server. If a 64 KiB chunk is received and it contains multiple frames, the whole chunk will be held in memory until it is completely read regardless of |
|
|
||
| // | ||
| // The mask is a view into the buffer the stream delivered, which is commonly | ||
| // 64 KB, and it is not needed once the payload above has been unmasked. |
There was a problem hiding this comment.
| // 64 KB, and it is not needed once the payload above has been unmasked. | |
| // 64 KiB, and it is not needed once the payload above has been unmasked. |
| const frame = Buffer.from( | ||
| '81933483a86801b992524fa1c60959e68a5216e6cb005ba1d5', | ||
| 'hex' | ||
| ); | ||
|
|
||
| // | ||
| // Put the frame at the start of a chunk the size the stream layer commonly | ||
| // delivers. `consume()` returns views into it, so a mask left set after the | ||
| // frame is parsed keeps the whole thing alive until the next one arrives. | ||
| // | ||
| const chunk = Buffer.alloc(65536); | ||
| frame.copy(chunk); |
There was a problem hiding this comment.
| const frame = Buffer.from( | |
| '81933483a86801b992524fa1c60959e68a5216e6cb005ba1d5', | |
| 'hex' | |
| ); | |
| // | |
| // Put the frame at the start of a chunk the size the stream layer commonly | |
| // delivers. `consume()` returns views into it, so a mask left set after the | |
| // frame is parsed keeps the whole thing alive until the next one arrives. | |
| // | |
| const chunk = Buffer.alloc(65536); | |
| frame.copy(chunk); |
It is not needed for the test, just check that receiver._mask is nulled when it is no longer needed.
| }); | ||
|
|
||
| receiver.write( | ||
| new FastBuffer(chunk.buffer, chunk.byteOffset, frame.length) |
There was a problem hiding this comment.
| new FastBuffer(chunk.buffer, chunk.byteOffset, frame.length) | |
| Buffer.from('81933483a86801b992524fa1c60959e68a5216e6cb005ba1d5', 'hex') |
|
Ok, I see the point when the mask is at the end of big chunk of data and connections are idle. |
| const FastBuffer = Buffer[Symbol.species]; | ||
|
|
||
| describe('Receiver', () => { |
There was a problem hiding this comment.
| const FastBuffer = Buffer[Symbol.species]; | |
| describe('Receiver', () => { | |
| describe('Receiver', () => { |
`getMask()` stores the frame mask as a view into the buffer the stream delivered, and `getData()` is the only reader. The view is left in place after the frame is parsed, so it keeps that whole buffer alive until the next frame overwrites it. Over TLS that buffer is a 64 KiB pool: `TLSWrap` decrypts into it and carves every chunk out at an advancing offset, so successive reads share one ArrayBuffer. A 25-byte message therefore leaves a 4-byte `_mask` pinning 64 KiB, and that 64 KiB holds the accumulated plaintext of many unrelated messages. On a connection that is then idle between messages, that is the steady state rather than a transient. Over plain TCP the read buffer is sized to what arrived (capped at 64 KiB), so the same retention appears when a large chunk lands and the connection goes quiet. Measured on a production server holding 37.1k concurrent WSS sockets: of 1.29 GB of retained array buffers, grouping by retaining field put 1031 MB across 16502 slabs on `_mask` alone -- 79.4%, each slab a full 64 KiB. Clearing it took RSS from 5.19 GB to 4.20 GB across a 12-host A/B, and was *cheaper* on minor GC (84.5/min vs 102.0) and event-loop p99 (70 ms vs 97 ms), since a gigabyte less live external memory is a gigabyte the collector does not keep walking. Clearing after the payload is unmasked costs one property store per frame and allocates nothing. `bench/parser.benchmark.js` is unchanged within run-to-run variance. Copying the mask into a preallocated 4-byte buffer instead -- mirroring what `Sender` does -- also fixes the retention, but measured 7.5-8.7% slower on small frames, so it is not worth it.
15036ed to
9ebd070
Compare
|
You're right, and my sentence was wrong — node does not allocate 64 KB per read. Your script is accurate: over plain TCP the read buffer is sized to what arrived (capped at 64 KiB), so a 25-byte write gives a 25-byte The 64 KiB comes from TLS. Your script with const tls = require('tls');
const fs = require('fs');
const server = tls.createServer({
key: fs.readFileSync('test/fixtures/key.pem'),
cert: fs.readFileSync('test/fixtures/certificate.pem')
});
server.on('listening', function () {
const socket = tls.connect({
port: server.address().port,
host: 'localhost',
rejectUnauthorized: false
});
socket.on('secureConnect', function () {
const frame = Buffer.from(
'81933483a86801b992524fa1c60959e68a5216e6cb005ba1d5',
'hex'
);
socket.write(frame);
});
});
server.on('secureConnection', function (socket) {
socket.on('data', function (chunk) {
console.log(chunk.buffer.byteLength);
});
});
server.listen(0);prints
So on a On "the whole chunk will be held in memory until it is completely read regardless of I've pushed your suggestions and corrected the claim in the commit message. Thanks for pushing back on it — the sentence was wrong as written and it was load-bearing for the whole argument. |
|
Thank you. |
The problem
Receiver#getMaskstores the frame mask as a view into the buffer the stream delivered:consume(n)returnsnew FastBuffer(buf.buffer, buf.byteOffset, n)whennis smaller thanthe head buffer, and
getDatais the only reader. Nothing clears it afterwards, so the viewsurvives until the next frame's
getMaskoverwrites it — holding that buffer alive for aslong as the connection is idle.
Where the 64 KiB comes from
(An earlier revision of this description said node allocates socket read buffers at 64 KB.
That was wrong — see the discussion below. The correction:)
Over plain TCP the read buffer is sized to what arrived, capped at 64 KiB, so a 25-byte write
yields a 25-byte
ArrayBuffer. Over TLS it is different:TLSWrapdecrypts into a 64 KiBpool and carves every chunk out of it at an advancing offset. Measured on node v26.13.0,
same 25-byte message:
chunk.buffer.byteLengthAnd the pool is shared across reads — six messages written 40 ms apart, arriving as six separate
reads, all landed in the same 65536-byte
ArrayBuffer:So on a
wss://server the retention is size-independent: a 4-byte_maskpins 64 KiB holdingthe accumulated plaintext of many unrelated messages, whatever the message size. Over plain TCP
the same thing needs a large chunk to arrive and the connection to then go quiet.
How much it costs
Measured on a production
wss://server holding 37.1k concurrent sockets. Heap snapshot,grouping
native / system / JSArrayBufferDatanodes by their single retaining field — 1.29 GBof backing stores across 36007 nodes:
_maskonReceiverALPNProtocolsonTLSSocket._tlsOptionsEvery slab a full 64 KiB, while a message on that service is a few hundred bytes — which is
the pool above, not per-message allocation.
A/B across 12 production hosts, matched within 0.4 h of uptime and 0.9% of connection count,
read as
avg_over_time(...[1h])because the raw series sawtooths ~42%:−0.99 GB of RSS per process, and it is cheaper on GC and event-loop latency rather than
dearer — clearing a property allocates nothing, and a gigabyte less live external memory is a
gigabyte the collector does not keep walking.
The change
Clear the mask once the payload has been unmasked.
getDatais its only reader, and the statemachine always reassigns it before the next read —
getInfosets_state = GET_MASKfor amasked frame and
getMasksetsGET_DATA— so nothing can observe it afterwards.One property store per frame, no allocation.
bench/parser.benchmark.js, mean of two runs each:Every row is within run-to-run variance — the 1 MiB row alone moves 3.7% between two baseline
runs.
Test
test/receiver.test.jsgains one case asserting that_maskis no longer set once the messagehas been delivered. Against the current
masterreceiver it fails withBuffer(4)whereundefinedis expected; with the fix it passes. 440 tests pass andreceiver.jsstays at 100%coverage.
Prior art in this repo
2b2abd4("[security] Limit retained message parts") bounded retained message parts foressentially this reason — retention scaling with something other than payload size. This is the
same class of problem one level down: a single 4-byte field holding a whole read buffer.
Workaround for anyone hitting this before it lands
It reaches into private fields, so pin the
wsversion or guard it with a test. It is the sameone-line clear, applied from outside — which is how the fix became obvious, after that
workaround had been running in production for a few days.