Arrow filter pushdown: comparing a dictionary-encoded binary, decimal or TIMESTAMPTZ column raises an InternalException #664

What happens?

A pushed-down comparison on a dictionary-encoded pyarrow column whose values are binary raises InternalException: INTERNAL Error: Attempted to dereference unique_ptr that is NULL! and invalidates the database. Every later query on the connection and its cursors fails with FatalException: FATAL Error: Failed: database has been invalidated because of a previous fatal error. The database must be restarted prior to being used again. A filter on another column of the same table works, and the same filter on a plain (not dictionary-encoded) column of the same value type works.

The same exception occurs for dictionary values of large_binary, binary_view, fixed_size_binary, decimal32, decimal64, decimal128 (both 18 digits or fewer and more than 18) and timestamp with a time zone, and for a dictionary-encoded field of a struct (s.d = ...). Dictionaries of utf8, large_utf8, integers, floats, date32, bool and timestamp without a time zone filter correctly. (string_view dictionaries fail differently; see #663.)

Likely cause, on main and v2.0-cyanoptera (1.5.6 has the same pattern in GetScalar): PyArrowFilterPushdown::TransformFilter passes the column's own ArrowType, through TransformFilter and TransformExpression, to ResolveColumn (src/arrow/filter_pushdown_visitor.cpp), which returns it unchanged as the leaf type, for a bare column and for a struct_extract child. For a dictionary-encoded column that ArrowType describes the indices: its type_info is null and the value type is in GetDictionary(). MakePyArrowScalar then calls arrow_type->GetTypeInfo<...>() in its BLOB, DECIMAL and TIMESTAMP_TZ branches, which dereferences the null type_info. The branches for the types that work never read arrow_type.

Suggested fix

Resolve a dictionary-encoded leaf to its dictionary's ArrowType:

diff --git a/src/arrow/filter_pushdown_visitor.cpp b/src/arrow/filter_pushdown_visitor.cpp
--- a/src/arrow/filter_pushdown_visitor.cpp
+++ b/src/arrow/filter_pushdown_visitor.cpp
@@ -23,8 +23,18 @@ bool ValueIsNan(const Value &value) {
 	return false;
 }
 
+// The ArrowType of the values a filter compares. For a dictionary-encoded column that is
+// the dictionary's type: the column's own ArrowType describes its indices.
+const ArrowType *ValueType(const ArrowType *type) {
+	if (type && type->HasDictionary()) {
+		return &type->GetDictionary();
+	}
+	return type;
+}
+
 // ResolveColumn walks a column-side expression to extract the (full path, leaf
-// ArrowType) pair. Accepts a bare BoundReferenceExpression and (nested)
+// ArrowType) pair, the leaf ArrowType being that of the leaf's values (see ValueType).
+// Accepts a bare BoundReferenceExpression and (nested)
 // `struct_extract` chains. Anything else throws NotImplementedException —
 // that gives the OPTIONAL_FILTER catch point a chance to swallow it.
 struct ResolvedColumn {
@@ -34,7 +44,7 @@ struct ResolvedColumn {
 
 ResolvedColumn ResolveColumn(const Expression &expr, const vector<Identifier> &root_path, const ArrowType *root_type) {
 	if (expr.GetExpressionClass() == ExpressionClass::BOUND_REF) {
-		return {root_path, root_type};
+		return {root_path, ValueType(root_type)};
 	}
 	if (expr.GetExpressionClass() != ExpressionClass::BOUND_FUNCTION) {
 		throw NotImplementedException("Cannot push down arrow scan filter on column-side expression: %s",
@@ -50,7 +60,7 @@ ResolvedColumn ResolveColumn(const Expression &expr, const vector<Identifier> &r
 	auto inner = ResolveColumn(*func.GetChildren()[0], root_path, root_type);
 	inner.path.push_back(StructType::GetChildName(func.GetChildren()[0]->GetReturnType(), child_idx));
 	if (inner.leaf_type) {
-		inner.leaf_type = &inner.leaf_type->GetTypeInfo<ArrowStructInfo>().GetChild(child_idx);
+		inner.leaf_type = ValueType(&inner.leaf_type->GetTypeInfo<ArrowStructInfo>().GetChild(child_idx));
 	}
 	return inner;
 }

With this change the reproducer below returns [(1,), (3,)]. A test for tests/fast/arrow/test_filter_pushdown.py is below. It covers utf8, binary, large_binary, fixed_size_binary, decimal128(30, 2) and timestamp[us, tz=UTC] dictionaries, each as a column and as a struct field, and asserts that the filter was pushed down. The five non-utf8 cases fail without the change and pass with it, and the tests/fast/arrow tests pass with TZ=UTC (test_8522 fails under a non-UTC local zone with or without this change). The patch applies to both main and v2.0-cyanoptera.

The test sets the session TimeZone to the column's zone. When the session zone differs from the column's, a dictionary of millisecond or microsecond TIMESTAMPTZ still fails after this change, with pyarrow's ArrowNotImplementedError: Function 'equal' has no kernel matching input types (timestamp[us, tz=UTC], timestamp[s, tz=America/New_York]) (session TimeZone set to America/New_York), because the pushed scalar carries the session zone. Dictionaries of binary_view still fail with ArrowNotImplementedError: Unsupported cast from dictionary<values=binary_view, indices=int8, ordered=0> to binary_view using function cast_binary_view, and dictionaries of decimal32 and decimal64 with ArrowNotImplementedError: Function 'equal' has no kernel matching input types (decimal32(5, 2), decimal32(5, 2)) and (decimal64(15, 2), decimal64(15, 2)). These raise duckdb.Error and leave the database usable. The core change suggested in #663 stops pushing these filters down.

diff --git a/tests/fast/arrow/test_filter_pushdown.py b/tests/fast/arrow/test_filter_pushdown.py
--- a/tests/fast/arrow/test_filter_pushdown.py
+++ b/tests/fast/arrow/test_filter_pushdown.py
@@ -31,6 +31,7 @@ from __future__ import annotations
 
 import datetime as dt
 import re
+from decimal import Decimal
 
 import pytest
 from _pushdown_helpers import (
@@ -122,6 +123,40 @@ def test_integer_max_value(duckdb_cursor, factory, data_type, max_value):
     assert duckdb_cursor.execute("SELECT * FROM arrow_table WHERE i = ?", (max_value,)).fetchall() == expected
 
 
+# A filter on a dictionary-encoded column compares the dictionary's values, so the
+# pyarrow scalar is built from the value type, not from the index type.
+@pytest.mark.parametrize(
+    ("value_type", "values", "literal"),
+    [
+        pytest.param(pa.utf8(), ["red", "blue"], "'red'", id="utf8"),
+        pytest.param(pa.binary(), [b"red", b"blue"], "'red'::BLOB", id="binary"),
+        pytest.param(pa.large_binary(), [b"red", b"blue"], "'red'::BLOB", id="large_binary"),
+        pytest.param(pa.binary(4), [b"red_", b"blue"], "'red_'::BLOB", id="fixed_size_binary"),
+        pytest.param(
+            pa.decimal128(30, 2), [Decimal("1.25"), Decimal("2.50")], "1.25::DECIMAL(30,2)", id="decimal128_30_2"
+        ),
+        pytest.param(
+            pa.timestamp("us", tz="UTC"),
+            [dt.datetime(2020, 1, 1, tzinfo=dt.timezone.utc), dt.datetime(2021, 1, 1, tzinfo=dt.timezone.utc)],
+            "TIMESTAMPTZ '2020-01-01 00:00:00+00'",
+            id="timestamptz",
+        ),
+    ],
+)
+def test_dictionary_encoded_comparison(duckdb_cursor, value_type, values, literal):
+    """A pushed-down comparison on a dictionary-encoded column matches on its values."""
+    # The pushed TIMESTAMPTZ scalar carries the session time zone, and pyarrow cannot compare a
+    # dictionary<timestamp[tz=UTC]> with a timestamp in another zone, so match the column's zone.
+    duckdb_cursor.execute("SET TimeZone = 'UTC'")
+    dict_array = pa.DictionaryArray.from_arrays(pa.array([0, 1, 0], pa.int8()), pa.array(values, value_type))
+    arrow_table = pa.table({"x": [1, 2, 3], "d": dict_array, "s": pa.StructArray.from_arrays([dict_array], ["d"])})
+    duckdb_cursor.register("arrow_table", arrow_table)
+    for column in ("d", "s.d"):
+        query = f"SELECT x FROM arrow_table WHERE {column} = {literal} ORDER BY x"
+        assert _was_pushed(duckdb_cursor, query)
+        assert duckdb_cursor.execute(query).fetchall() == [(1,), (3,)]
+
+
 # ===========================================================================
 # 2. OR pushdown decisions
 #

To Reproduce

import duckdb
import pyarrow as pa

d = pa.DictionaryArray.from_arrays(pa.array([0, 1, 0], pa.int8()), pa.array([b"red", b"blue"], pa.binary()))
c = duckdb.connect()
c.register("t", pa.table({"x": [1, 2, 3], "d": d}))
print(c.sql("SELECT x FROM t WHERE x > 1").fetchall())  # [(2,), (3,)]
print(c.sql("SELECT x FROM t WHERE d = 'red'::BLOB").fetchall())  # InternalException

Expected: [(1,), (3,)] from the second query.

OS:

macOS 26.5.1, arm64

DuckDB Package Version:

1.5.6 and 2.0.0.dev2610011535

Python Version:

3.14.7 (pyarrow 26.0.0)

Full Name:

Will Frey

Affiliation:

dataplor

What is the latest build you tested with? If possible, we recommend testing with the latest nightly build.

I have tested with a nightly build

Did you include all relevant data sets for reproducing the issue?

Not applicable - the reproduction does not require a data set

Did you include all code required to reproduce the issue?

• Yes, I have

Did you include all relevant configuration to reproduce the issue?

• Yes, I have