You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[Feature] Support BLOB values in top-level MAP fields #321
I searched in the issues and found nothing similar.
Motivation
The Parquet read path now handles both timestamp conversion and compatible BLOB representations,
so the name ParquetTimestampConverter and its timestamp-specific method names no longer describe
its responsibility accurately.
Parquet ARROW:schema may restore the physical binary column of a Paimon BLOB as large_binary. Identifying this as a compatible BLOB representation requires the target field
metadata. The existing recursive compatibility check passes only DataType below the root STRUCT,
so that metadata is lost when traversing MAP values.
Paimon C++ also rejects every nested BLOB field, although a BLOB can be the direct value of a
top-level MAP field. The compatibility fixtures currently group ARRAY<BLOB> and MAP<K, BLOB> together, preventing the supported MAP case from being tested independently.
The Rust fixture stores raw BLOB bytes inline as Parquet binary. Java cannot interpret those bytes
as a Paimon BlobDescriptor, so C++ should reject this non-standard representation instead of
adding a special conversion for it.
Solution
Rename ParquetTimestampConverter to ParquetReadTypeAdapter while retaining its static utility
structure.
Keep the existing timestamp conversion implementation.
Retain Arrow Field metadata during recursive compatibility checks so BLOB can be identified
inside MAP values.
Treat large_binary restored through ARROW:schema as compatible with a logical Paimon BLOB
field.
Move the pre-conversion and post-conversion type checks into AdaptArray.
Allow BLOB as the direct value of a top-level MAP field while keeping ARRAY<BLOB> unsupported.
Split ARRAY BLOB and MAP BLOB fixtures.
Verify Python and Java standard MAP<K, BLOB> data.
Explicitly reject Rust raw-binary MAP<K, BLOB> data because it is not encoded as a Paimon BlobDescriptor.
Anything else?
No public API, storage format, or protocol changes are proposed.
changed the title [-]Refactor Parquet read type adaptation for nested schemas[/-][+][Feature] Refactor Parquet read type adaptation for nested schemas[/+]on Sep 10, 2026
changed the title [-][Feature] Refactor Parquet read type adaptation for nested schemas[/-][+][Feature] Support BLOB values in top-level MAP fields[/+]on Sep 11, 2026
Activity
[-]Refactor Parquet read type adaptation for nested schemas[/-][+][Feature] Refactor Parquet read type adaptation for nested schemas[/+][-][Feature] Refactor Parquet read type adaptation for nested schemas[/-][+][Feature] Support BLOB values in top-level MAP fields[/+]