COPY … PARTITIONED BY writes partition values containing = unescaped (p0=x=y); DuckDB and Polars cannot read the dataset or silently drop the partition column #26492

Describe the bug

COPY … TO 'dir/' PARTITIONED BY (p0) names the directory p0=<value>. The value is not escaped with Hive's
escaping rules. It goes through object_store::path::Path::join, which percent-encodes only the characters that
object stores advise against (/, %, #, ?, *, \, ", [, ], {, }, ^, |, <, >, ~, `, control
and non-ASCII characters). The key/value separator = is written as is, and so are : and ', which Hive's
escapePathName escapes, and spaces, which pyarrow, DuckDB and Polars escape.
A value x=y gives the directory p0=x=y.

DataFusion and pyarrow split at the first = and read the dataset back. DuckDB and Polars do not:

• next to other partitions, DuckDB fails with Hive partition mismatch … key "p0" not found, and Polars fails as well;
• when p0=x=y is the only partition, both return the rows without the p0 column, with no error.

For the same table, pyarrow, DuckDB and Polars write p0=x%3Dy (and p0=a%20b), which every reader, including
DataFusion, reads back.

The path is built in compute_hive_style_file_path (datafusion/datasource/src/write/demux.rs:636):
file_path.join(format!("{}={}", partition_by[j].0, part_key[j])). Escaping the value with the Hive rules (at least
= and %, which ListingTable already decodes on read since #23226) would make the output portable.

To Reproduce

import pyarrow as pa, pyarrow.parquet as pq
pq.write_table(pa.table({'rid': [1, 2, 3, 4], 'p0': ['x=y', 'a b', 'é', 'a/b']}), 'src.parquet')

CREATE EXTERNAL TABLE src STORED AS PARQUET LOCATION 'src.parquet';
COPY (SELECT * FROM src) TO 'd/' STORED AS PARQUET PARTITIONED BY (p0);

$ ls d
'p0=%C3%A9'  'p0=a b'  'p0=a%2Fb'  'p0=x=y'

import duckdb, polars as pl
duckdb.sql("SELECT rid, p0 FROM read_parquet('d/**/*.parquet', hive_partitioning = true)")
# Binder Error: Hive partition mismatch between file "d/p0=%C3%A9/….parquet" and "d/p0=x=y/….parquet": key "p0" not found
pl.scan_parquet('d/**/*.parquet', hive_partitioning=True, hive_schema={'p0': pl.String}).collect()
# could not create a new DataFrame: height of column 'p0' (3) does not match DataFrame height (4)

Only the x=y partition:

COPY (SELECT 1 AS rid, 'x=y' AS p0) TO 'one/' STORED AS PARQUET PARTITIONED BY (p0);   -- one/p0=x=y/…

duckdb.sql("SELECT * FROM read_parquet('one/**/*.parquet', hive_partitioning = true)").columns   # ['rid'] — p0 is gone
pl.scan_parquet('one/**/*.parquet', hive_partitioning=True).collect()                            # only column rid

Expected behavior

Partition values are escaped as Hive, pyarrow, DuckDB and Polars do (p0=x%3Dy; the last three also write
p0=a%20b), so that DuckDB, Polars, Spark and Hive read the
dataset back with the right p0 values.

Additional context

• DataFusion main 290ad15 (2026-10-18), built from source; also 91cd7ff (2026-10-15).
• Related (read side, merged): Decode Hive partition values in listing tables #23226 "Decode Hive partition values in listing tables". Open: fix: read Hive's null partition placeholder as NULL #26310 (Hive's NULL
  placeholder), Panic when reading from partitioned datasets with columns that have ' in them #9269 (panic with ' in partition values).
• Values with = are common: key-value strings, URL query strings, base64 with padding.

Willingness to contribute

None