[EPIC] Nested types in the native Parquet and Iceberg writers: correctness and performance follow-ups #7051

What is the problem the feature request solves?

Nested types (structs, arrays and maps) in the native Parquet writer and the native Iceberg writer were audited on main at c230bf4, as #7040 did for the scans. The audit looked at NULL structs, arrays and maps, NULLs inside them, field ids at every level, Parquet statistics and Iceberg manifest metrics. Each write was made twice, once natively and once by Spark with Comet disabled, and the two were compared on the values read back, the Parquet schema and footer, the column statistics, the manifest metrics and the files pruned for nested predicates. Parquet writes ran on Spark 3.4.3, 3.5.9 and 4.1.3, and Iceberg writes on Iceberg 1.5.2, 1.8.1 and 1.11.0. A code review of both writers, arrow-rs 59.3 and iceberg-rust at the current pin ran alongside, and each finding below was reproduced before it was listed.

Both writers write nested data correctly. What remains are Spark settings the native Parquet writer ignores, a fallback that keeps writes into NOT NULL Iceberg columns off the native path, and one wrong statement in the docs. This epic collects the new issues and the related ones that were already open. Items are listed by priority within each section.

Correctness: native Parquet writer

• Native Parquet writer ignores the datetime rebase modes, so Spark rejects or shifts its pre-1582 dates #7047 The datetime rebase modes are ignored. Under the Spark 3.4 and 3.5 defaults, the native writer writes pre-1582 dates and timestamps that Spark refuses to write, and Spark then can't read them back. LEGACY reads shift them, on every version.
• Native Parquet writer ignores outputTimestampType and writeLegacyFormat, and omits Spark's schema metadata #7048: outputTimestampType and writeLegacyFormat are ignored, and Spark's schema footer key is missing. With TIMESTAMP_MILLIS, timestamps read back with the microseconds Spark drops.
• [COMET NATIVE WRITER] Write Spark version into Parquet metadata #3427: write org.apache.spark.version into the footer. Native Parquet writer ignores the datetime rebase modes, so Spark rejects or shifts its pre-1582 dates #7047 needs it.
• [COMET NATIVE WRITER] parquet timestamp conversion (timestamp96) #3425: INT96 timestamps, the Spark default.

Performance and coverage: native Iceberg writer

• Native Iceberg writes into NOT NULL columns fall back on Spark 3.5+ because Comet has no native AssertNotNull #7049: writes into NOT NULL columns, top-level or nested, fall back on Spark 3.5+ when no exchange sits in front of the writer, because Comet has no native AssertNotNull. Native Iceberg writes are opt-in (feat: make native Iceberg writes opt-in for 1.2.0 #6962).
• Native Iceberg writes: support uuid and unknown columns #6959: uuid and unknown columns fall back, nested ones included.

Performance: both writers

• Native Parquet writer ignores Spark's Parquet writer properties (block size, page size, dictionary, writer version, statistics) #5304: Spark's Parquet writer properties are ignored. For nested rows, the row-count cap gives one row group per file, 239 MB for 100,000 rows of 100-element arrays where Spark writes row groups of about 128 MB (measured there).
• Faster Parquet level encoding for nested columns (parquet-rs RleEncoder) #6918, Faster string min/max statistics in the parquet-rs writer #6919 and Reduce string dictionary interning and level computation cost in the parquet-rs writer #6920: per-value costs in the parquet-rs writer for nested columns (level encoding, string statistics, dictionary interning and level computation), from [EPIC] Performance follow-ups from profiling the end-to-end Iceberg benchmark #6911.
• Make the native Iceberg writer's DictionaryChooser pre-pass cheaper #6914: a cheaper DictionaryChooser pre-pass for the native Iceberg writer.

Documentation

• Iceberg write docs say the float/double null-count difference can't change IS NULL pruning, but it can #7050: iceberg-writes.md says the float/double null-count difference can't change IS NULL pruning. It can, and the native counts are the correct ones (Core, Parquet: Null counts are incorrect for child fields under a null struct iceberg#17561).

Related epics

• [EPIC] Nested types in the native Parquet and Iceberg scans: correctness and performance follow-ups #7040: nested types in the native Parquet and Iceberg scans.
• [EPIC] Add support for native Parquet writes #1625: native Parquet writes.
• [EPIC] Production-quality native Iceberg writes #5649: production-quality native Iceberg writes.
• [EPIC] Performance follow-ups from profiling the end-to-end Iceberg benchmark #6911: performance follow-ups from profiling the end-to-end Iceberg benchmark.
• [EPIC] Consistent handling of invalid UTF-8 in native StringType (ingress policy) #4764: invalid UTF-8 in native StringType, including the writers' input from the JVM.

Describe the potential solution

Each item is a linked issue. Check it here when it closes.

Additional context

Differences that are not Comet bugs, listed so they are not refiled:

• On Iceberg 1.9 and later, iceberg-java's value_counts and null_value_counts for a float or double field under a NULL struct leave out the NULL-struct rows. The native writer takes its counts from the Parquet footer, which includes them (Native Iceberg writer's value and null counts for a float under a NULL struct differ from iceberg-java 1.9+ #6659, closed). iceberg-java's counts are the wrong ones, because they let IS NULL prune files that hold matching rows (Core, Parquet: Null counts are incorrect for child fields under a null struct iceberg#17561).
• parquet-rs normalizes a float or double zero bound to -0.0 or +0.0. That is documented and never prunes more than iceberg-java's bounds.
• In a copy-on-write UPDATE and in a partitioned write, Spark's AssertNotNull projection runs in the JVM in front of a columnar shuffle. The NOT NULL check still applies there, and the write stays native.

What matched: values, Parquet repetition, field ids and column statistics for NULL structs, arrays and maps, empty arrays and maps, and NULLs inside them at every level, in deep and wide schemas. That includes input sliced by LIMIT and OFFSET, filtered input, and nested values computed by expressions before the write. Parquet field ids at every level, with spark.sql.parquet.fieldId.write.enabled on and off. Nested timestamp, decimal, boolean and binary leaves. Iceberg manifest metrics (value, null and NaN counts, lower and upper bounds) under the default and full modes on all three Iceberg versions, and on 1.11.0 also under counts, truncate(4), per-column overrides on nested fields, max-inferred-column-defaults and tiny pages and row groups. Nested partition sources (identity on s.a, bucket, truncate and days(ts.t)), a sort order on a nested field, and copy-on-write DELETE, UPDATE and MERGE. Required nested fields under NULL parents written straight from a native scan, and fixed fields under NULL structs. Natively written files with pre-1582 values read back correctly by Comet's own scan.

Not covered: Variant (#5433), partitioned and bucketed native Parquet writes (#3193, #3194), object-store and HDFS outputs, Iceberg format v3 types, encryption, and write throughput. The probes ran against a debug build, and #6911 profiles a release build.