Don't size writeTo() output from the parsed wire size

[merlimat]

Problem

writeTo() reserves getSerializedSize() bytes in the target, then writes the message in place through the target's backing array. Nothing checks those writes against the reservation.

For a parsed message, getSerializedSize() returned the number of bytes that parseFrom() consumed. That is the serialized size only if LightProto writes the message back exactly as it arrived. #24, #25, #28, #30 and #35 stop the caching for some inputs: unknown fields, unknown enum values, bools other than 0 or 1, map entries with a missing or repeated key or value, and nested messages that hit one of these. Other valid input still breaks it.

Written back larger than it arrived (writeTo() writes past its reservation):

• a negative int32 or enum in fewer than 10 bytes;
• a packed chunk for a field declared unpacked (proto3 producers pack by default), or a single unpacked element for a packed field;
• a uint32 >= 2^31 in protobuf-java's own 5-byte encoding (LightProto writes 10);
• any of these inside a nested message;
• a nested message whose fields run past its declared length, when written on its own;
• a nested message changed through its getter right after parsing.

Written back smaller (part of the reserved bytes is left unwritten):

• overlong varints, tags or lengths;
• a singular field, message field or oneof present more than once;
• proto3 default values on the wire;
• empty or split packed chunks.

Pooled heap buffers share one backing array per chunk, so the extra bytes overwrite the next buffer, and writeTo() returns normally with a truncated message. Direct buffers throw or get a truncated message, and toByteArray() throws. Code that writes back a parsed message is affected, e.g. a proxy, or a gRPC service forwarding a request: the gRPC marshaller sizes its buffer with getSerializedSize().

Example

Repeated.x_int32 is declared unpacked in repeated_numbers.proto. This input sends it as one packed chunk, as a proto3 producer would:

byte[] wire = new byte[102];
wire[0] = 0x0A;                          // field 1, length-delimited: packed encoding
wire[1] = 100;                           // 100 one-byte elements
Arrays.fill(wire, 2, 102, (byte) 1);     // x_int32 = [1, 1, ..., 1]

Repeated m = new Repeated();
m.parseFrom(wire);
m.getSerializedSize();                   // 102: the wire size. Written back unpacked it takes 200 (08 01 per element)

PooledByteBufAllocator alloc = new PooledByteBufAllocator(false);
ByteBuf target = alloc.heapBuffer(102, 102);
ByteBuf neighbour = alloc.heapBuffer(102, 102);  // same backing array, 112 bytes further
m.writeTo(target);                       // returns 102, writerIndex 102
// neighbour now starts with 08 01 08 01 ...: the last 88 bytes of m landed there

The current generated code:

// parseFrom()
if (!_hasUnknownFields) {
    _cachedSize = _size;                 // bytes consumed from the wire
}
// getSerializedSize()
if (_cachedSize > -1) {
    return _cachedSize;                  // trusted as the serialized size
}

Flagging each remaining case at parse time would take a check on nearly every field: varint lengths, presence before overwrite, default values, and the end of every length-delimited field.

Change

getSerializedSize() only trusts the sizes it computed itself. It caches them with the sign bit set, and returns a cached size only when that bit is set:

public int getSerializedSize() {
    if (_cachedSize < -1) {                       // computed by an earlier call
        return _cachedSize & Integer.MAX_VALUE;
    }
    int _size = 0;
    ...                                           // field walk, unchanged
    _cachedSize = _size | Integer.MIN_VALUE;
    return _size;
}

• parseFrom() is unchanged. It is byte-identical in every generated message. The plain wire size (>= 0) it leaves is no longer returned as the serialized size, so for a parsed message the size is computed from the fields, as for a built one. writeTo(), toByteArray() and the length prefixes in parent messages all go through getSerializedSize(), so they now get the right size whatever the input encoding.
• The clear() release gate still uses the wire size as its estimate of how much data the instance retains. The original unsigned compare settles a parsed size on its own. Computed sizes and -1 both have the sign bit set, so they go on to a second compare with that bit masked off:

  if (Integer.compareUnsigned(_cachedSize, LightProtoCodec.CLEAR_RETAIN_MAX) > 0
          && (_cachedSize & Integer.MAX_VALUE) > LightProtoCodec.CLEAR_RETAIN_MAX) {
      return _clearAndRelease();
  }

• _isSizeCached() becomes _cachedSize != -1. Right after a parse it returns the same result as before, so the propagation from Don't cache the wire size after dropping nested unknown fields #28 is unchanged, and it stays true once a computed size is cached.
• The existing _hasUnknownFields flags keep working. They no longer matter for size safety: they only make clear() take the release path.

The cost is one size computation when a parsed message is written back, the same computation a built message already pays. Measured with JMH, interleaved A/B, one fork per variant per round, on an earlier version of this change that used the single masked compare in clear():

• Parse then writeTo() (the cost above): 3-8% slower; AddressBook was 21% slower on JDK 21, over 4 rounds.
• Other deserialize and serialize benchmarks: below 5%, and some flipped sign between JDK 21 and 26. The split gate above was added to keep master's single compare on the parse path, and has not been re-measured.

Testing

NonCanonicalInputTest covers every case listed above, 32 in total. Each case compares getSerializedSize(), toByteArray() and writeTo() against a copyFrom() copy, which computes the size from the fields. The writeTo() checks use heap targets, direct targets (both the scratch and the NIO path), and a pooled heap buffer of exactly getSerializedSize() bytes. For the pooled buffer, the test also checks that no byte of its backing array outside the buffer changed. 23 of the 32 fail on master; the other 9 are the cases already fixed by #24/#25/#28/#30.

ClearReleaseTest checks that a small parsed message still keeps the O(1) clear. It fails if the gate does not mask the sign bit.

Full mvn verify passes.

The sint32 writer overrun is fixed separately in #42.