[ump45nose]

Which issue does this PR close?

Closes #7214.

Rationale for this change

Spark's vectorized reader has no updater for these reads and fails with PARQUET_COLUMN_DATA_TYPE_MISMATCH, but the native scan casts the values instead:

• an INT96 or INT64 timestamp column read as string, byte/short/int, float/double or decimal (returns text, raw numbers or NULL);
• an INT96 column read as bigint (returns microseconds since the epoch). An INT64 timestamp read as bigint is fine in Spark and stays allowed.
• a DATE column read as string, float or decimal (returns text or numbers). Spark's INT32 case has no string or float branch, and isDecimalTypeMatched refuses a date annotation. Reading a DATE as int stays allowed. This one isn't in Native Parquet scan reads timestamp columns as string, int, double or decimal, and INT96 as bigint, where Spark rejects them #7214; it came up in review.

What changes are included in this PR?

• spark_has_updater rejects these reads through its existing rules: Date32 and Timestamp join the primitive types that can't be read as a string or binary, and the (Timestamp, Int8 | Int16 | Int32 | Float32 | Float64 | Decimal*) and (Date32, Float32 | Decimal*) pairs join the is_spark_rejected_conversion table. These are deferred like the other getUpdater rejections (RejectOnNonEmpty).
• INT96 and INT64 timestamps have the same Arrow type by the time the check runs, so the INT96 bigint rule needs the Parquet type. Like TIMESTAMP(NANOS) in fix: reject TIMESTAMP(NANOS) Parquet columns read as timestamp #7089, the footer check (mark_parquet_leaves) now marks INT96 leaves with a private INT96_TIMESTAMP_KEY, and files with INT96 leaves take the deferred check path even without row filters. To carry both markers, the nanos_timestamp: bool parameter became a small ParquetLeaf enum (Other, NanosTimestamp, Int96).

How are these changes tested?

• parquet_timestamp_read_as_non_timestamp_errors: a TIMESTAMP(MICROS) column, with and without a timezone, read as string, int, double and decimal(20,0) fails with the Spark-shaped error; read as bigint it still works.
• parquet_date_read_as_string_float_or_decimal_errors: a DATE column read as string, float and decimal(20,0) fails with the Spark-shaped error; read as int it is still accepted.
• int96_read_as_bigint_errors: an INT96 file read as bigint fails, with row filters on and off.
• int96_bigint_rejection_keys_on_the_parquet_type: a marked INT96 leaf is rejected as bigint and still accepted as timestamp; an unmarked timestamp leaf is still accepted as bigint.

All of these fail without the change. I ran the parquet:: lib tests in the datafusion-comet crate (332 passed, 1 ignored) and cargo clippy -p datafusion-comet --all-targets -- -D warnings. In the full lib run, only the native UDF tests failed, because I hadn't built comet-test-udfs. I did not run the Spark/JVM suites.

AI disclosure: I wrote the change, the tests and this description with help from an AI coding assistant. The part most worth checking is the INT96 marker, which uses the same footer-marking path as TIMESTAMP(NANOS).

[comphead]

I checked the new rules against ParquetVectorUpdaterFactory in 3.4.3, 3.5.9, 4.0.4 and 4.1.3. An INT64 timestamp has only the long, timestamp and day-time interval updaters, and isDecimalTypeMatched refuses a timestamp annotation, so the rejected targets match. One point on where the rule lives, which also covers a DATE gap of the same kind.

[rich7420]

@ump45nose, thanks for the patch.

[rich7420]

ArrowWriter stores UTC timestamp seconds as unannotated INT64. Spark accepts decimal(20,0) reads. Validate footer and runtime paths against Parquet types.

[ump45nose]

Good catch — addressed in 5714377.

ArrowWriter writes Timestamp(Second, _) as unannotated INT64 (Parquet has no second unit), and an ARROW:schema hint can restore Timestamp. Spark still reads the Parquet type and accepts decimal(20,0) under the INT64 canReadAsDecimal rule.

Changes:

• Timestamp → Decimal* rejection now applies only to annotated units (millis/micros/nanos); Timestamp(Second, _) follows the INT64 precision floor (>= 20).
• Footer mark_parquet_leaves rewrites an ARROW-restored Timestamp over unannotated INT64 back to Int64, so the conversion check keys on the Parquet type.
• Unit test covers Second→decimal(20,0) accept / decimal(19,0) reject, micros still rejected, and Int64 after rewrite; runtime roundtrip via ArrowWriter Second+UTC → decimal(20,0) checks the values.

(AI-assisted reply.)