[Feature] Support writing MAP<K, BLOB> fields #415

Search before asking

• I searched in the issues and found nothing similar.

Motivation

Java Paimon (apache/paimon#8635, with key types extended in apache/paimon#8963 and apache/paimon#9057) and PyPaimon can write top-level MAP<K, BLOB> columns to dedicated .blob files. #283 (PR #278) adds read-only MAP<K, BLOB> support to Paimon C++, and #389 (PR #392) adds write support for ARRAY<BLOB>, but C++ table creation (TableSchema::Create) and writes (FileStoreWrite::Create) still reject MAP<K, BLOB> fields through BlobUtils::ValidateContainerBlobWriteSchema. As a result, C++ users cannot write keyed collections of multimodal objects, such as the images of one sample keyed by camera name, to a Paimon table.

Solution

Add Java-compatible write support for top-level MAP<K, BLOB> fields declared with blob-field:

• Schema validation: Keep the rules consistent with Java and with ARRAY<BLOB> (feat(blob): support writing ARRAY<BLOB> fields #392):
  • allow MAP<K, BLOB> only as a top-level field declared with blob-field;
  • reject it for blob-descriptor-field, blob-view-field, partition keys, and tables without data evolution;
  • reject unsupported key types and BLOB values nested more deeply, such as MAP<K, ARRAY<BLOB>> or ARRAY<MAP<K, BLOB>>.
• Blob format writer: Extend BlobFormatWriter to encode each MAP<K, BLOB> row as the nested payload defined in the BLOB file spec and decoded by the C++ reader from feat(blob): support reading map blob values #278. The payload contains:
  • map magic 1296188226;
  • version 1;
  • entry count;
  • concatenated key data;
  • concatenated BLOB data;
  • Delta-Varint key length index;
  • Delta-Varint BLOB length index;
  • key index length and BLOB index length.
• Key encoding: Match the Java key encodings for the key types supported by the C++ reader:
  • TINYINT, SMALLINT, INT, BIGINT: fixed-width little-endian signed integers;
  • BOOLEAN: one byte, 0 or 1;
  • DATE: four-byte little-endian days since epoch;
  • DECIMAL(p, s): an eight-byte little-endian unscaled value for p <= 18, and the minimal-length big-endian two's-complement unscaled value for p > 18;
  • STRING: UTF-8 bytes; BINARY: raw bytes.
• Null handling: Distinguish null maps (outer length -1), empty maps (entry count 0), null values (BLOB length -1), and empty values (BLOB length 0). Reject duplicate keys, as Java does (apache/paimon#9107). Also reject null keys, which Arrow map keys cannot hold; this differs from Java, which allows one null key per map.
• Value sources: Support both inline raw bytes and BlobDescriptor values. Descriptor values are materialized from their referenced storage when written, as Java does and as the ARRAY<BLOB> writer does.
• Placeholder: Support data-evolution partial updates of MAP<K, BLOB> by recognizing an internal placeholder row on the write channel, aligned with Java BlobMapPlaceholder, and persisting it as a -2 entry. The fallback reader from feat(blob): support reading map blob values #278 then resolves these rows from older sequence layers.
• Write path: Route MAP<K, BLOB> fields to dedicated blob files in the data-evolution write path, and remove the write-side rejection in BlobUtils::ValidateContainerBlobWriteSchema.
• Documentation: Update data_types.rst and write.rst, which currently state that Paimon C++ can read but not write MAP<kt, BLOB> columns.
• Tests:
  • add unit tests for payload encoding of every supported key type, null/empty cases, null/duplicate key rejection, and error handling;
  • add round-trip tests with the C++ reader from feat(blob): support reading map blob values #278, including placeholder fallback across sequence layers;
  • add Java/Python cross-compatibility tests, with C++ writing and Java/Python reading.

Anything else?

• Depends on Support reading MAP<K, BLOB> blob files #283 / feat(blob): support reading map blob values #278 for the reader side and builds on the ARRAY<BLOB> writer from [Feature] Support writing ARRAY<BLOB> fields #389 / feat(blob): support writing ARRAY<BLOB> fields #392.
• TIME keys (apache/paimon#9009), compaction of data-evolution tables, paimon.map.selected-keys for MAP<K, BLOB> reads, and MAP<K, BLOB> in primary-key managed BLOB storage ([Feature] Support managed BLOBs in primary-key tables and normal-file compaction across data-evolution field groups #204) are out of scope and can be tracked separately.
• No public API changes are expected. The storage format follows the existing Java spec.

Are you willing to submit a PR?

• I'm willing to submit a PR!