Repository navigation
[C++] Add a "list_contains" kernel #33295
Description
Assume you have a list array:
arr = pa.array([["a", "b"], ["a", "c"], ["b", "c", "d"]])And you want to know for each list if it contains a certain value (of the same type as the list's values). A "list_contains" function (or other name) would be useful for that:
pc.list_contains(arr, "a")
# -> True, True FalseThe current workaround that I found was flattening, checking equality, and then reducing again with groupby, but this is quite tedious:
>>> temp = pa.table({'index': pc.list_parent_indices(arr), 'contains_value': pc.equal(pc.list_flatten(arr), "a")})
>>> temp.group_by('index').aggregate([('contains_value', 'any')])['contains_value_any'].chunk(0)
<pyarrow.lib.BooleanArray object at 0x7ffaf3f8de20>
[
true,
true,
false
]But this also only works if there are no empty or missing list values.
Reporter: Joris Van den Bossche / @jorisvandenbossche
Related issues:
Note: This issue was originally created as ARROW-18097. Please see the migration documentation for further details.
Activity
asfimport commentedon Oct 19, 2022
Antoine Pitrou / @pitrou:
Then there probably should be a "list_index" function as well, similar to "is_in" vs. "index_in" ?
asfimport commentedon Nov 2, 2022
Jacek Pliszka / @JacekPliszka:
Maybe no need for new name but is_in can be reused:
asfimport commentedon Nov 3, 2022
Joris Van den Bossche / @jorisvandenbossche:
[~jacek.pliszka] That would give a different signature; "is_in" is currently a scalar kernel that returns a result of the same shape as the first input argument. So that generally wouldn't work here (for lists, it would the second argument that we want to check element-by-element in your example).
While it might be technically possible in C++ to have kernels with different input types, I am not sure it is a good idea to reuse the same name for this.
thisisnic commentedon Feb 18, 2026
This issue has been marked as stale because it has had no activity in the past 2 years. Please remove the stale label or comment below, or this issue will be closed in 14 days.
alexshpilkin commentedon Feb 18, 2026
Mere passage of time does not cause a feature to appear. Not stale.
thisisnic commentedon Feb 18, 2026
Thanks @alexshpilkin, will mark as not stale. Are you interested in submitting a PR?
alexshpilkin commentedon Feb 18, 2026
@thisisnic This was one of the issue that pushed me to Polars (not the issue, by any means, just one of the things that contributed to the decision). So it’s less than immediately relevant to me at the moment. Still, I’m not against giving it a shot in principle, if you or someone else could point me in the right direction. I just don’t promise the kernel will be particularly good :)
thisisnic commentedon Feb 18, 2026
If you make a start and draft a PR, folks around here are pretty friendly and happy to help, just pop some comments in explaining where you got to and where you aren't sure :) Perfection is not a requirement, for sure!
jonasdedden commentedon Sep 28, 2026
@pitrou @jorisvandenbossche @thisisnic please have a look at #51611, where I implement a list.contains compute kernel.
apacheGH-33295: [C++] Support arrays of values in list_contains
apacheGH-33295: [C++] Type-check null values in list_contains
apacheGH-33295: [C++] Match float16 NaN values in list_contains
apacheGH-33295: [C++] Only compare child values referenced by list vi…
apacheGH-33295: [C++] Stop at the first match in list_contains
apacheGH-33295: [C++] Share one code path for all list types and valu…
apacheGH-33295: [C++] Condense list_contains tests
apacheGH-33295: [C++] Speed up the list_contains scan for short lists
apacheGH-33295: [C++] Search each list for its query in list_contains
apacheGH-33295: [C++] Look up queries in a scalar list with is_in in …
apacheGH-33295: [C++] Check null queries like other queries in list_c…
apacheGH-33295: [C++] Fix -Wbitwise-instead-of-logical in list_contains