Skip to content

Write uint32 values as varints of at most 5 bytes - #44

Merged
merlimat merged 3 commits into
streamnative:masterfrom
merlimat:fix-uint32-varint-size
Oct 2, 2026
Merged

merlimat merged 3 commits into
streamnative:masterfrom
merlimat:fix-uint32-varint-size

Conversation

@merlimat

@merlimat merlimat commented Oct 2, 2026 •

Copy link
Copy Markdown
Collaborator

Follow-up to #42: reuses the writeRawVarUInt() it added.

Problem

LightProto sized and wrote a uint32 like an int32. A value >= 2^31, which is negative as a Java int, was therefore written as the 10-byte sign extension. protobuf writes a uint32 as an unsigned 32-bit varint, at most 5 bytes.

Every reader decodes both encodings to the same value, so this costs 5 extra bytes per such value. It also means protobuf-java's own encoding of these values re-serializes to a different size than it was parsed from (#43 covers the size cache side of that).

Example

NumbersOuterClass.Numbers.newBuilder().setXUint32(0x80000000).build().getSerializedSize();  // 6
new Numbers().setXUint32(0x80000000).getSerializedSize();                                  // 11

The current code (LightProtoNumberField):

} else if (field.getProtoType().equals("int32") || field.getProtoType().equals("uint32")) {
    writer = "writeRawVarInt";                                   // a negative int takes 10 bytes
...
} else if (field.getProtoType().equals("uint32")) {
    return String.format("LightProtoCodec.computeVarIntSize(%s)", name);   // 10 for a negative int

Change

uint32 is now sized with computeVarUIntSize() and written with writeRawVarUInt():

} else if (field.getProtoType().equals("int32")) {
    writer = "writeRawVarInt";
} else if (field.getProtoType().equals("uint32")) {
    writer = "writeRawVarUInt";
...
} else if (field.getProtoType().equals("uint32")) {
    return String.format("LightProtoCodec.computeVarUIntSize(%s)", name);

The output now matches protobuf-java for singular, repeated, packed and map uint32 fields. This changes LightProto's output bytes for values >= 2^31. Parsing is unchanged, and both encodings decode the same everywhere.

Testing

NumbersTest and RepeatedNumbersTest cover 2^31 and 2^32 - 1. For singular, repeated and packed fields, sizes and bytes match protobuf-java. These tests fail without the change. Full mvn verify passes.

getSerializedSize() counts a sint32 as its zigzag value encoded as an
unsigned 32-bit varint, at most 5 bytes, as protobuf does. But
writeRawSignedVarInt() went through writeRawVarInt(), the int32 writer,
which writes a negative int as a 10-byte varint. The zigzag value has its
top bit set for every v >= 2^30 and every v < -2^30, so for each such
value writeTo() wrote 5 bytes more than it had reserved, with no parsing
involved: toByteArray() threw ArrayIndexOutOfBoundsException, and a
pooled heap target got the extra bytes written into the neighbouring
buffer of its chunk.

Write the zigzag value with a new writeRawVarUInt() (byte[] and NIO
sinks), which never writes more than 5 bytes. The output now matches
protobuf-java for singular, repeated and packed sint32 fields.
A uint32 >= 2^31 was sized and written like an int32, as the 10-byte
sign extension of a negative int, while protobuf writes a uint32 as an
unsigned 32-bit varint of at most 5 bytes. Every reader, LightProto's
included, decodes both to the same value, so the only effects were 5
extra bytes per such value, and that protobuf-java's encoding was
re-serialized to a different size than it was parsed from.

Size uint32 with computeVarUIntSize() and write it with
writeRawVarUInt(), so the output matches protobuf-java for singular,
repeated, packed and map uint32 fields.
# Conflicts:
#	code-generator/src/main/resources/io/streamnative/lightproto/generator/LightProtoCodec.java
#	tests/src/test/java/io/streamnative/lightproto/tests/NumbersTest.java
#	tests/src/test/java/io/streamnative/lightproto/tests/RepeatedNumbersTest.java
@merlimat
merlimat merged commit 9f7549c into streamnative:master Oct 2, 2026
1 check passed
@merlimat
merlimat deleted the fix-uint32-varint-size branch October 2, 2026 22:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant