Skip to content

Join filters get pushed into Arrow scans that can't handle them (string_view / binary_view tables) #665

Closed

Description

@willfrey

What happens?

Joining against a pyarrow table that has a string_view column fails:

ArrowNotImplementedError: Function 'equal' has no kernel matching input types (string_view, string)

It fails even when the join is on a different column of that table, just with a different error from pyarrow's array_filter (no kernel matching input types (string_view, bool)). binary_view columns hit the array_filter error too. The same comparisons as WHERE filters work fine, since filter pushdown is already turned off for these tables (#187, #227).

I think the join filter pushdown just never asks the scan whether it can take the filter. The Arrow scan says no through supports_pushdown_type (ArrowPushdownType), and the planner respects that for normal table filters by keeping a FILTER above the scan (plan_get.cpp). JoinFilterPushdownOptimizer::GetPushdownFilterTargets only checks filter_pushdown on the LOGICAL_GET, so the join's dynamic filters still end up in the scan and get handed to pyarrow.

Tables with a list<string_view> column fail the same way (in array_take), and so do tables with a run-end encoded column, which also need the fix in #666. SET disabled_optimizers = 'join_filter_pushdown' works around it.

Suggested fix

The fix would go in duckdb/duckdb: skip the columns the scan refuses, like the planner does.

diff --git a/src/optimizer/join_filter_pushdown_optimizer.cpp b/src/optimizer/join_filter_pushdown_optimizer.cpp
--- a/src/optimizer/join_filter_pushdown_optimizer.cpp
+++ b/src/optimizer/join_filter_pushdown_optimizer.cpp
@@ -233,10 +233,18 @@
 		}
 		get.ResolveOperatorTypes();
 		const auto bindings = get.GetColumnBindings();
+		vector<JoinFilterPushdownColumn> pushable_columns;
+		pushable_columns.reserve(columns.size());
 		for (auto &filter : columns) {
 			if (filter.probe_column_index.table_index != get.table_index) {
 				// the filter does not apply to the probe side here - bail-out
 				return;
+			}
+			if (get.function.supports_pushdown_type &&
+			    !get.function.supports_pushdown_type(*get.bind_data,
+			                                         get.GetColumnIndex(filter.probe_column_index).GetPrimaryIndex())) {
+				// the scan cannot evaluate a filter on this column - skip it, as the planner does for table filters
+				continue;
 			}
 			// add storage type to columns
 			for (idx_t i = 0; i < bindings.size(); i++) {
@@ -254,8 +262,12 @@
 			    reconstructed_type.id() == LogicalTypeId::VARIANT) {
 				return;
 			}
-		}
-		targets.emplace_back(get, std::move(columns));
+			pushable_columns.push_back(filter);
+		}
+		if (pushable_columns.empty()) {
+			return;
+		}
+		targets.emplace_back(get, std::move(pushable_columns));
 		break;
 	}
 	case LogicalOperatorType::LOGICAL_PROJECTION: {

The Top-N, build-side and row-group pruning optimizers also call GetPushdownFilterTargets, so they'd skip those columns too. The early return when no columns are left is there for RowGroupPruner::FindLogicalGet, which reads pushdown_targets[0].columns[0].

With the patch the reproducer below returns the right rows, and so do joins on binary_view and list<string_view> tables. Below are a Catch test for test/arrow/arrow_filter_pushdown.cpp and a pytest case for tests/fast/arrow/test_filter_pushdown.py. Both fail without the patch and pass with it, and [arrow], test/optimizer/* and tests/fast/arrow still pass. It applies cleanly to main (a809d3c) and to the core v2.0-cyanoptera pins (939d6af).

Core test
diff --git a/test/arrow/arrow_filter_pushdown.cpp b/test/arrow/arrow_filter_pushdown.cpp
--- a/test/arrow/arrow_filter_pushdown.cpp
+++ b/test/arrow/arrow_filter_pushdown.cpp
@@ -86,6 +86,34 @@
 	}
 }
 
+// Helper: get the EXPLAIN ANALYZE output for a hash join that probes an arrow_scan on id with a one-row build side
+static string GetExplainAnalyzeForJoin(Connection &con, ArrowTestFactory &factory) {
+	con.TableFunction("arrow_scan", {}, {}, ArrowTestHelper::ConstructArrowScan(factory))->CreateView("probe");
+	const auto result =
+	    con.Query("EXPLAIN ANALYZE SELECT count(*) FROM probe JOIN (SELECT 5 AS k) v ON probe.id = v.k");
+	REQUIRE(!result->HasError());
+	return result->Collection().GetValue(1, 0).ToString();
+}
+
+TEST_CASE("Arrow filter pushdown - join filters follow the pushdown types", "[arrow]") {
+	DuckDB db;
+	Connection con(db);
+
+	REQUIRE(!con.Query("CREATE TABLE src AS SELECT i AS id, i::VARCHAR AS name FROM range(10) tbl(i)")->HasError());
+
+	SECTION("Table with a string view column: no join filter in scan") {
+		auto factory = MakeArrowFactory(con, "SELECT * FROM src", true);
+		auto explain_str = GetExplainAnalyzeForJoin(con, *factory);
+		REQUIRE(explain_str.find("Dynamic Filters") == string::npos);
+	}
+
+	SECTION("Table with a regular string column: join filter in scan") {
+		auto factory = MakeArrowFactory(con, "SELECT * FROM src", false);
+		auto explain_str = GetExplainAnalyzeForJoin(con, *factory);
+		REQUIRE(explain_str.find("Dynamic Filters") != string::npos);
+	}
+}
+
 TEST_CASE("Arrow filter pushdown - nested view types disable pushdown", "[arrow]") {
 	DuckDB db;
 	Connection con(db);
Python test
diff --git a/tests/fast/arrow/test_filter_pushdown.py b/tests/fast/arrow/test_filter_pushdown.py
--- a/tests/fast/arrow/test_filter_pushdown.py
+++ b/tests/fast/arrow/test_filter_pushdown.py
@@ -647,6 +647,20 @@
         res = duckdb.sql("SELECT * FROM dset WHERE col = 'abc'").fetchall()
         assert res == [("abc",)]
 
+    @pytest.mark.parametrize(
+        ("view_type", "values", "key"),
+        [
+            pytest.param(pa.string_view(), ["abc", "efg"], "'abc'", id="string_view"),
+            pytest.param(pa.binary_view(), [b"abc", b"efg"], "'abc'::BLOB", id="binary_view"),
+        ],
+    )
+    def test_view_join_does_not_crash(self, view_type, values, key):
+        """A join's runtime filters cannot be pushed into a scan of a table with a view column either."""
+        con = duckdb.connect()
+        con.register("t", pa.table({"x": [1, 2], "col": pa.array(values, type=view_type)}))
+        assert con.execute(f"SELECT x FROM t JOIN (SELECT {key} AS k) b ON t.col = b.k").fetchall() == [(1,)]
+        assert con.execute("SELECT x FROM t JOIN (SELECT 2 AS k) b ON t.x = b.k").fetchall() == [(2,)]
+
 
 # ===========================================================================
 # 11. Projection / scanner-path interactions

To Reproduce

import duckdb
import pyarrow as pa

c = duckdb.connect()
c.register("t", pa.table({"x": [1, 2], "col": pa.array(["abc", "efg"], type=pa.string_view())}))
print(c.sql("SELECT x FROM t WHERE col = 'abc'").fetchall())  # [(1,)]
print(c.sql("SELECT x FROM t JOIN (SELECT 'abc' AS k) b ON t.col = b.k").fetchall())  # duckdb.Error

The join should return [(1,)]. Joining on x instead (SELECT x FROM t JOIN (SELECT 2 AS k) b ON t.x = b.k) fails with the array_filter error, and should return [(2,)].

OS:

macOS 26.5.1, arm64

DuckDB Package Version:

1.5.6 and 2.0.0.dev2610011535, and a source build of v2.0-cyanoptera at 8bbac25

Python Version:

3.14.7 (pyarrow 26.0.0)

Full Name:

Will Frey

Affiliation:

dataplor

What is the latest build you tested with? If possible, we recommend testing with the latest nightly build.

I have tested with a nightly build

Did you include all relevant data sets for reproducing the issue?

Not applicable - the reproduction does not require a data set

Did you include all code required to reproduce the issue?

  • Yes, I have

Did you include all relevant configuration to reproduce the issue?

  • Yes, I have

Activity

  1. evertlammerts commented on Oct 27, 2026

    @evertlammerts
    Member

    Thanks for tracking this down to the join filter optimizer. I reran your joins on string_view, binary_view and list of string_view tables against the released 2.0.0 and against main, as a table, a reader and a Dataset. They all return the right rows now. In 2.0.0 the Arrow scan was replaced (#693) and no longer accepts the engine's join or Top-N filters at all, so this cannot come back through that path. Closing as fixed in 2.0.0; please reopen if you hit it there.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions