Fix excessive memory allocation from crafted container headers

[Watson1978]

A container header carries an element count that the unpacker uses to pre-allocate the Array or Hash. The count is capped at SHRT_MAX per container, but nested containers each pre-allocate up to that cap, so a small crafted message of deeply nested headers can still force a large allocation. A streaming Unpacker (feed_each/each) keeps the stack on EOF, so it holds that memory until the connection ends, which makes it reachable for anything decoding untrusted data off the wire.

An element needs at least one byte (a map pair at least two), so a container cannot hold more elements than there are bytes left to read. This caps the initial size by the buffered size as well; the container still grows as more data arrives, so fully-buffered decodes are unaffected.

Reproduction

require "msgpack"

# 128 nested map16 headers, each declaring 65535 pairs, with no bodies.
# Each header is 3 bytes, so the whole message is only 384 bytes.
payload = ("\xde\xff\xff".b) * 128

base = `ps -o rss= -p #{Process.pid}`.strip.to_i
MessagePack::Unpacker.new.feed_each(payload) {}
after = `ps -o rss= -p #{Process.pid}`.strip.to_i

puts "input size: #{payload.bytesize} bytes"
puts "RSS growth: #{after - base} KB"

On my environment the 384-byte message grows RSS by about 16.9 MB before this change and about 0.4 MB after it.

🤖 Generated with Claude Code

[Watson1978]

Thanks