GH-51652: [Python] Add NumPy StringDType to Arrow conversion

[ngoldbaum]

Rationale for this change

NumPy 2.0 added StringDType, a variable-width UTF-8 string dtype. pa.array currently rejects it with Unsupported numpy type 2056. Now that pyarrow requires NumPy 2.0 the conversion can use the public NpyString_* C API.

This supersedes #50951. The first commit is @alippai's first commit from that PR, unchanged. His later commits added batching and new bulk builder APIs in Arrow C++; those can come later in their own PR with benchmarks, as discussed there. Nothing outside the NumPy conversion code changes here.

What changes are included in this PR?

• NumPyDtypeToArrow maps StringDType to string, so it is also the inferred type for pa.array, pa.infer_type, pa.from_numpy_dtype and lists of such arrays. large_string and string_view can be requested with type. Any other type raises a TypeError naming the requested type.
• The conversion holds the dtype's allocator lock for the whole array and never holds the GIL while taking it.
• A null entry in a StringDType array becomes an Arrow null unless the dtype's na_object is a str. NumPy reports such entries as that string from __getitem__, in ufuncs and comparisons, and in casts, and there is no way to tell them apart from a regular entry holding the same string, so the conversion writes the string too. NaN-like sentinels such as np.nan and pd.NA, None, and arbitrary objects all become arrow nulls.
• A short section in docs/source/python/numpy.rst describing the support and the semantics for converting missing data.

If left unfixed, #51156 also becomes possible to trigger from Python by creating a StringDType array with a very large entry.

Are these changes tested?

Yes. Tests cover the three target types, strings crossing NumPy's short, medium and long storage thresholds, non-ASCII input, strided input, mask, None, NaN and string sentinels, the rejected target type, and list inference. I also ran a threaded test that converts an array while other threads hit NumPy's null-comparison error path on the same array to check the lock ordering.

Are there any user-facing changes?

Yes. pa.array and the inference functions accept StringDType arrays.

AI disclosure: I used an AI model to work on the follow-up commits on top of @alippai's first commit and for code review.

• GitHub Issue: [Python] Conversion to/from numpy 2.0+ new StringDType #42018

• GitHub Issue: [Python] Conversion from numpy StringDtype to Arrow string #51652

[ngoldbaum]

@jorisvandenbossche gentle ping for a review pass on this

[github-actions]

⚠️ GitHub issue #51652 has been automatically assigned in GitHub to PR creator.

[jorisvandenbossche]

This looks good to me!

One small additional test item:

• Can you add a new case for StringDtype to test_from_numpy_dtype

[jorisvandenbossche]

Is this needed? (we don't seem to have it here for other types)

If you leave it out, we get a less good error message? (which feels as something to improve in general, so fine to leave this here as is, and consider cleaning this up / generalizing this as another issue)

[jorisvandenbossche]

Opened #51678 to keep track of this in general

[jorisvandenbossche]

One remaining behaviour question I have, is what to do about NaN, which we explicitly disallow for object dtype when converting to string (or to any type other than float, in general), unless you explicitly specify from_pandas=True:

# converting None to null -> fine
>>> pa.array(np.array(["a", None], dtype=object))
<pyarrow.lib.StringArray object at 0x7f813a761300>
[
  "a",
  null
]

# converting NaN -> complains about not being a string or null (error msg could be better, though ;))
>>> pa.array(np.array(["a", np.nan], dtype=object))
...
ArrowTypeError: Expected bytes, got a 'float' object

# unless specifying `from_pandas=True`, and then NaN also gets recognized as null
>>> pa.array(np.array(["a", np.nan], dtype=object), from_pandas=True)
<pyarrow.lib.StringArray object at 0x7f813bd98ac0>
[
  "a",
  null
]

So on the one hand, we could argue for consistency that also here NaN should only work if from_pandas=True is specified. But on the other hand, here it is numpy that explicitly recognizes and models it as "missing" (since you have the has_nan_na vs has_string_na on the dtype descr, and the goal of the sentinel is for missing values (if not a string sentinel)). So I assume that the current behaviour to always treat a non-string sentinel as missing is fine.

[jorisvandenbossche]

Thanks @alippai and @ngoldbaum!

[ngoldbaum]

Thanks Joris! I'm on vacation this week and am participating in a Quansight all-hands next week but am planning to get to the followups you pointed out in your comments soon.

[alippai]

Thanks for landing this, @ngoldbaum!