Repository navigation
feat: add standard encryption key metadata - #991
Merged
Merged
Conversation
4 of 17 tasks
wgtmac
reviewed
Oct 15, 2026
StandardKeyMetadata is the per-file key metadata of the standard Iceberg encryption scheme: the data encryption key, the AAD prefix (file ID) and, for AES GCM streams, the encrypted file length. It is stored in the key_metadata field of data, delete and manifest files, and (encrypted) in the table metadata encryption-keys for manifest lists. The serialized form is Java's: a schema version byte (1) followed by an Avro binary record. Tests check byte-for-byte compatibility with vectors written by Java (checked in under test/resources/encryption/vectors, excluded from the text-fixing pre-commit hooks). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Round trip the extreme and negative file lengths, distinguish an empty AAD prefix from an absent one, and reject negative byte lengths, truncated varints and varints longer than ten bytes (as Java's Avro decoder does). Also clarify why trailing bytes are ignored.
Embed the two key metadata vectors serialized by Iceberg Java in the test instead of checking in binary files. The other vectors belong to later steps; this also drops the pre-commit exclusion that was only needed for binary files.
Avro encodes a union branch index as an int, and Java's BinaryDecoder.readIndex delegates to readInt, which caps the varint at five bytes. ReadLong accepted up to ten, so a six-byte encoding of branch zero parsed here but is rejected by Java.
shangxinli
force-pushed
the
pr/03-key-metadata
branch
from
October 15, 2026 14:25
95056bf to
2aff707
Compare
wgtmac
approved these changes
Oct 16, 2026
5 tasks done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #988.
Adds
StandardKeyMetadata, the per-file key metadata of Iceberg table encryption (data key, AAD prefix, file length), serialized byte-compatible with Java. Tests check it against vectors written by Java.