Repository navigation
Write uint32 values as varints of at most 5 bytes - #44
Merged
Merged
Conversation
getSerializedSize() counts a sint32 as its zigzag value encoded as an unsigned 32-bit varint, at most 5 bytes, as protobuf does. But writeRawSignedVarInt() went through writeRawVarInt(), the int32 writer, which writes a negative int as a 10-byte varint. The zigzag value has its top bit set for every v >= 2^30 and every v < -2^30, so for each such value writeTo() wrote 5 bytes more than it had reserved, with no parsing involved: toByteArray() threw ArrayIndexOutOfBoundsException, and a pooled heap target got the extra bytes written into the neighbouring buffer of its chunk. Write the zigzag value with a new writeRawVarUInt() (byte[] and NIO sinks), which never writes more than 5 bytes. The output now matches protobuf-java for singular, repeated and packed sint32 fields.
A uint32 >= 2^31 was sized and written like an int32, as the 10-byte sign extension of a negative int, while protobuf writes a uint32 as an unsigned 32-bit varint of at most 5 bytes. Every reader, LightProto's included, decodes both to the same value, so the only effects were 5 extra bytes per such value, and that protobuf-java's encoding was re-serialized to a different size than it was parsed from. Size uint32 with computeVarUIntSize() and write it with writeRawVarUInt(), so the output matches protobuf-java for singular, repeated, packed and map uint32 fields.
# Conflicts: # code-generator/src/main/resources/io/streamnative/lightproto/generator/LightProtoCodec.java # tests/src/test/java/io/streamnative/lightproto/tests/NumbersTest.java # tests/src/test/java/io/streamnative/lightproto/tests/RepeatedNumbersTest.java
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #42: reuses the
writeRawVarUInt()it added.Problem
LightProto sized and wrote a
uint32like anint32. A value >= 2^31, which is negative as a Java int, was therefore written as the 10-byte sign extension. protobuf writes auint32as an unsigned 32-bit varint, at most 5 bytes.Every reader decodes both encodings to the same value, so this costs 5 extra bytes per such value. It also means protobuf-java's own encoding of these values re-serializes to a different size than it was parsed from (#43 covers the size cache side of that).
Example
The current code (
LightProtoNumberField):Change
uint32is now sized withcomputeVarUIntSize()and written withwriteRawVarUInt():The output now matches protobuf-java for singular, repeated, packed and map
uint32fields. This changes LightProto's output bytes for values >= 2^31. Parsing is unchanged, and both encodings decode the same everywhere.Testing
NumbersTestandRepeatedNumbersTestcover 2^31 and 2^32 - 1. For singular, repeated and packed fields, sizes and bytes match protobuf-java. These tests fail without the change. Fullmvn verifypasses.