Skip to content

[Feature] Support BLOB values in top-level MAP fields #321

Closed

Description

@zjw1111

Search before asking

  • I searched in the issues and found nothing similar.

Motivation

The Parquet read path now handles both timestamp conversion and compatible BLOB representations,
so the name ParquetTimestampConverter and its timestamp-specific method names no longer describe
its responsibility accurately.

Parquet ARROW:schema may restore the physical binary column of a Paimon BLOB as
large_binary. Identifying this as a compatible BLOB representation requires the target field
metadata. The existing recursive compatibility check passes only DataType below the root STRUCT,
so that metadata is lost when traversing MAP values.

Paimon C++ also rejects every nested BLOB field, although a BLOB can be the direct value of a
top-level MAP field. The compatibility fixtures currently group ARRAY<BLOB> and
MAP<K, BLOB> together, preventing the supported MAP case from being tested independently.

The Rust fixture stores raw BLOB bytes inline as Parquet binary. Java cannot interpret those bytes
as a Paimon BlobDescriptor, so C++ should reject this non-standard representation instead of
adding a special conversion for it.

Solution

  • Rename ParquetTimestampConverter to ParquetReadTypeAdapter while retaining its static utility
    structure.
  • Keep the existing timestamp conversion implementation.
  • Retain Arrow Field metadata during recursive compatibility checks so BLOB can be identified
    inside MAP values.
  • Treat large_binary restored through ARROW:schema as compatible with a logical Paimon BLOB
    field.
  • Move the pre-conversion and post-conversion type checks into AdaptArray.
  • Allow BLOB as the direct value of a top-level MAP field while keeping ARRAY<BLOB> unsupported.
  • Split ARRAY BLOB and MAP BLOB fixtures.
  • Verify Python and Java standard MAP<K, BLOB> data.
  • Explicitly reject Rust raw-binary MAP<K, BLOB> data because it is not encoded as a Paimon
    BlobDescriptor.

Anything else?

No public API, storage format, or protocol changes are proposed.

Implementation PR: #322

Are you willing to submit a PR?

  • I'm willing to submit a PR!

Activity

  1. changed the title [-]Refactor Parquet read type adaptation for nested schemas[/-] [+][Feature] Refactor Parquet read type adaptation for nested schemas[/+] on Sep 10, 2026
  2. changed the title [-][Feature] Refactor Parquet read type adaptation for nested schemas[/-] [+][Feature] Support BLOB values in top-level MAP fields[/+] on Sep 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions