Native Parquet writes on Spark 3.4/3.5 ignore maxRecordsPerFile #6751

Describe the bug

On Spark 3.4/3.5, a native Parquet write ignores maxRecordsPerFile, both the write option and spark.sql.files.maxRecordsPerFile. Spark rolls to a new file every N rows. CometNativeWriteExec writes one file per task, so the output has fewer, larger files than Spark writes. The rows read back are the same.

CometWriteFiles on Spark 4.0+ declines these writes, so they fall back to Spark. CometDataWritingCommand.getSupportLevel (the 3.x serde) has no such check. The test write with maxRecordsPerFile falls back to Spark in CometParquetWriterSuite is gated to Spark 4.0+ for that reason.

Steps to reproduce

On Spark 3.5, with spark.comet.write.parquet.enabled=true and spark.comet.operator.DataWritingCommandExec.allowIncompatible=true, write ten rows from a native scan with .option("maxRecordsPerFile", 1). Spark writes ten files. Comet writes one.

Expected behavior

Decline the write on 3.x as the 4.0+ serde does, using the same effective value Spark uses (FileFormatWriter reads the option first, then the session conf), and drop the 4.0+ gate on the test.

Additional context

Noted in the #4746 reviews (2026-09-28). It predates #4746.