GH-3817: Content defined chunking of data pages

[kszucs]

Rationale for this change

Parquet C++ and parquet-rs can end data pages at content-defined boundaries. Files that share a run of values then share byte-identical pages, which content-addressable storage (e.g. Hugging Face Xet) can deduplicate. This PR adds the same opt-in feature to parquet-java. Iceberg Java needs it to honor the write.parquet.content-defined-chunking.* table properties.

What changes are included in this PR?

• ContentDefinedChunker, a port of the Arrow C++ / arrow-rs chunker with the same gear tables, mask derivation, defaults and cut rules, so that for the same physical values the page boundaries match Arrow C++. Arrow types that Arrow hashes at their own width (narrow integers, decimals, dictionary arrays) chunk differently.
• CdcOptions: min/max chunk size and normalization level, defaulting to 256 KiB / 1 MiB / 0 as in Arrow.
• ParquetProperties.Builder and ParquetWriter.Builder: withContentDefinedChunking(CdcOptions), where null, the default, disables it.
• ParquetOutputFormat: parquet.page.content-defined-chunking.{enabled,min.size,max.size,norm.level}.
• Chunking continues across the row groups of a file, as in arrow-rs and Arrow C++ (GH-51684: [C++][Parquet] Keep the CDC chunking state across row groups arrow#51685). InternalParquetRecordWriter keeps one ContentDefinedChunkers per file and passes it to each row group's store through the new ParquetProperties.newColumnWriteStore(schema, pageStore, bloomFilterWriteStore, chunkers). With chunking enabled, a store made without the file's chunkers fails fast instead of silently restarting the chunking, so integrations that build a column write store per row group, such as Iceberg's ParquetWriter, must keep one ContentDefinedChunkers per data file and use that overload.
• ChunkingColumnWriter wraps each column writer only when chunking is enabled, so the default write path is unchanged. The page size, value count and row count limits still apply, counted from the page start, so that they cut in the same places after an edit.
• With chunking enabled, a dictionary falls back to plain encoding only when it outgrows its page size limit, as in Arrow C++ and arrow-rs, rather than on parquet-java's first-page compression check, which an edit in the first page could flip for the whole column chunk. Column chunks of nulls alone get an empty dictionary page, as Arrow C++ writes.
• ContentDefinedChunkingWriteBenchmarks measures the write cost. In local runs, chunking added about 12–30% to write time, depending on the data.

Are these changes tested?

Yes:

• Page boundaries are pinned to the ones Arrow C++ writes through pyarrow for required and optional int64, optional string, optional list<int64> and a maximum-size-dominated envelope, with V1 and V2 pages and through the record writer.
• Edits (insert, delete, update, prepend, append, once and twice) across seven column shapes change only the pages around them, and an insertion into one of several row groups leaves the other row groups deduplicating.
• Chunking continues across row groups, each column has its own chunker, and the page limits, dictionary encoding and fallback, null columns, empty files and configuration keys are covered; data of every physical type reads back with V1 and V2 pages.
• The tests were checked to fail with chunking disabled, with a chunker that never cuts, with cuts at the maximum size only, with one chunker shared by all columns, and with chunking restarted in every row group.

Are there any user-facing changes?

Yes: new writer options and configuration keys, and new public API (CdcOptions, ContentDefinedChunkers, the newColumnWriteStore overload and the ColumnWriteStoreV1/V2 constructors that take the chunkers), all marked EXPERIMENTAL and off by default. Written files are ordinary Parquet files and need no reader support.

Closes #3817

🤖 Generated with Claude Code