[Python] Update pandas->pyarrow conversion for the .values deprecation in pandas

Pandas has deprecated certain aspects of .values (when it looses information), which our pandas->pyarrow conversion currently relies upon, so in the tests with pandas nightly we are seeing a lot of warnings because of that (https://github.com/ursacomputing/crossbow/actions/runs/34548523730/job/103106277981).

Small illustration:

>>> import pandas as pd
>>> import pyarrow as pa
>>> ser = pd.Series(pd.date_range("2025-01-01", periods=3, tz="UTC"))
>>> pa.array(ser)
<python-input-4>:1: Pandas4Warning: Series.values returning an ndarray that drops timezone information for DatetimeTZDtype is deprecated. In a future version, this will return the underlying DatetimeArray instead. Use 'Series.to_numpy()' to get a NumPy array, or 'Series.array' to get the ExtensionArray.
<pyarrow.lib.TimestampArray object at 0x7ff624fb1a20>
[
  2025-01-01 00:00:00.000000Z,
  2025-01-02 00:00:00.000000Z,
  2025-01-03 00:00:00.000000Z
]

This is caused by our handling of a Series object in pa.array(..):

arrow/python/pyarrow/array.pxi

Lines 5396 to 5407 in 07be48c

    
           cdef object get_values(object obj, bint* is_series): 
        
               if pandas_api.is_series(obj) or pandas_api.is_index(obj): 
        
                   result = pandas_api.get_values(obj) 
        
                   is_series[0] = True 
        
               elif isinstance(obj, np.ndarray): 
        
                   result = obj 
        
                   is_series[0] = False 
        
               else: 
        
                   result = pandas_api.series(obj, copy=False).values 
        
                   is_series[0] = False 
        
               return result

and

arrow/python/pyarrow/pandas-shim.pxi

Lines 231 to 242 in 07be48c

    
               cpdef get_values(self, obj): 
        
                   """ 
        
                   Get the underlying array values of a pandas Series or Index in the 
        
                   format (np.ndarray or pandas ExtensionArray) as we need them. 
        
                   Assumes obj is a pandas Series or Index. 
        
                   """ 
        
                   self._check_import() 
        
                   if isinstance(obj.dtype, (self.pd.api.types.IntervalDtype, 
        
                                             self.pd.api.types.PeriodDtype)): 
        
                       return obj.array 
        
                   return obj.values

So we already have some custom handling for Period/Interval, but that will also be needed for DatetimeTZDtype

GH-51302: [Python] Avoid deprecated .values call in pandas->pyarrow c…

Issue resolved by pull request 51484
#51484

Navigation Menu

Repository navigation

[Python] Update pandas->pyarrow conversion for the .values deprecation in pandas #51302

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

	cdef object get_values(object obj, bint* is_series):
	if pandas_api.is_series(obj) or pandas_api.is_index(obj):
	result = pandas_api.get_values(obj)
	is_series[0] = True
	elif isinstance(obj, np.ndarray):
	result = obj
	is_series[0] = False
	else:
	result = pandas_api.series(obj, copy=False).values
	is_series[0] = False

	return result

	cpdef get_values(self, obj):
	"""
	Get the underlying array values of a pandas Series or Index in the
	format (np.ndarray or pandas ExtensionArray) as we need them.

	Assumes obj is a pandas Series or Index.
	"""
	self._check_import()
	if isinstance(obj.dtype, (self.pd.api.types.IntervalDtype,
	self.pd.api.types.PeriodDtype)):
	return obj.array
	return obj.values

[Python] Update pandas->pyarrow conversion for the .values deprecation in pandas #51302

Description

Activity

AlenkaF commented on Oct 1, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions