[jorisvandenbossche]

Rationale for this change

This might not strictly speaking be a valid "minor" issue since it changes code, but it should keep the logic the same and (for me) clarify the intent of it.

Are these changes tested?

By existing tests

Are there any user-facing changes?

No

[jorisvandenbossche]

If metadata would come from metadata = self.schema.metadata, it means it already had a "pandas" key, and there is no need to add it back here (to_table() does not (or no longer) loose the metadata)

[Copilot]

🟢 Approval recommended

The change is a small, localized refactor that preserves intended behavior while making the metadata source/restore path more explicit.

Pull request overview

This PR refactors ParquetDataset.read() to make the pandas-metadata restoration logic clearer, specifically distinguishing the case where pandas metadata is sourced from _common_metadata / _metadata rather than from the dataset schema itself.

Changes:

• Introduces common_metadata to explicitly track when pandas metadata is obtained from common metadata files for backwards compatibility.
• Narrows the schema-metadata restoration step to only run when pandas metadata was sourced from common metadata (so it won’t already be present on the resulting table’s schema metadata).

File summaries

File	Description
python/pyarrow/parquet/core.py	Clarifies pandas metadata handling by tracking and conditionally re-attaching pandas schema metadata only when it was obtained from common metadata files.

Review details

• Files reviewed: 1/1 changed files
• Comments generated: 0
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[raulcd]

LGTM

[TomAugspurger]

FYI @jorisvandenbossche, this did cause a behavior change in pandas when reading zero columns. With pyarrow==25.x, that returned a (0, 0) shape dataframe. With pyarrow==26, it returns a (nrows, 0) shape dataframe

import io

import pandas as pd

df = pd.DataFrame({"a": [1, 2, 3]})
b = io.BytesIO()
df.to_parquet(b)
empty = pd.read_parquet(b, columns=[])
print(empty.shape)  # (0, 0) for pyarrow=25.x; (3, 0) for pyarrow=26.x

I think that (nrows, 0) is the shape I'd expect, but leaving a note here in case anyone else stumbles across this (xref NVIDIA/cudf#24520).

[jorisvandenbossche]

Damn, I should not have called it "minor" .. But relieved that the behaviour change was not too bad. Agreed that the new behaviour seems fine (and we should maybe add a test for that in pandas and pyarrow as well, to cover that case)

I suppose when the dataframe has a non-range index, it already gave the non-zero shape.

[jorisvandenbossche]

I was wondering how this could actually have caused such behaviour change, because essentially before this PR it was replacing identical pandas metadata.
But it seems replacing the schema metadata at the moment impacts the table:

>>> table = pa.table({"a": [1, 2, 3]}).select([])
>>> table.num_rows
3
>>> table = table.replace_schema_metadata({"test": "value"})
>>> table.num_rows
0

That seems like an actual bug in pyarrow.