Skip to content

fix(bigquery): handle overlapping dataframe index and column names - #18834

Open
AliAbouelazm wants to merge 1 commit into
googleapis:mainfrom
AliAbouelazm:fix/bigquery-index-column-collision
Open

AliAbouelazm wants to merge 1 commit into
googleapis:mainfrom
AliAbouelazm:fix/bigquery-index-column-collision

Conversation

@AliAbouelazm

Copy link
Copy Markdown
Contributor

Fixes #15726.

Schema inference resets the full DataFrame index before selecting values, which raises when an index and column share a name. Select the column or named index level directly instead, preserving the documented column precedence and resetting only the selected Series labels.

This fixes both the BigQuery fallback and pandas-gbq inference. Regression tests cover object/numeric values, duplicate row labels, mixed MultiIndexes, null/empty data, schema overrides, index exclusion, and unchanged input.

Validation:

  • Full unit suites on Python 3.12: BigQuery 2,949 passed, 2 skipped; pandas-gbq 268 passed.
  • Focused helper/schema tests on Python 3.10 through 3.15, including pandas 1.5.3, 2.3.3, and 3.x.
  • Both package linters and mypy pass. All changed executable lines and branches are covered, with no package coverage decrease.
  • Offline integration checks pass. Cloud system tests were not run.

The delegated BigQuery path needs a pandas-gbq release containing this fix. No dependency bump is included here.

AI assistance was used for implementation, testing, and review.

@AliAbouelazm
AliAbouelazm requested review from a team as code owners October 23, 2026 18:48
@AliAbouelazm
AliAbouelazm requested review from tswast and removed request for a team October 23, 2026 18:48

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request updates the schema inference logic for pandas DataFrames in both google-cloud-bigquery and pandas-gbq to handle cases where an index shares a name with a column. Instead of resetting the index, columns and index levels are now retrieved directly, with columns taking precedence. However, a critical bug was identified in _pandas_helpers.py where the variable values is defined conditionally inside an if block but used unconditionally later, which would lead to an UnboundLocalError if the condition is not met.

Comment on lines 519 to +522
bq_type = _PANDAS_DTYPE_TO_BQ.get(dtype.name)
if bq_type is None:
sample_data = _first_valid(dataframe_reset_index[column])
values = get_column_or_index(dataframe, column)
sample_data = _first_valid(values)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

critical

If bq_type is not None, the if bq_type is None: block is skipped, which means values is never defined. When the execution falls through to Step 3, calling _get_schema_by_pyarrow(column, values) will raise an UnboundLocalError. To fix this, define values unconditionally before checking bq_type.

Suggested change
bq_type = _PANDAS_DTYPE_TO_BQ.get(dtype.name)
if bq_type is None:
sample_data = _first_valid(dataframe_reset_index[column])
values = get_column_or_index(dataframe, column)
sample_data = _first_valid(values)
values = get_column_or_index(dataframe, column)
bq_type = _PANDAS_DTYPE_TO_BQ.get(dtype.name)
if bq_type is None:
sample_data = _first_valid(values)

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Unable to load table from dataframe with overlapping index/column name

1 participant