Repository navigation
Join filters get pushed into Arrow scans that can't handle them (string_view / binary_view tables) #665
Description
What happens?
Joining against a pyarrow table that has a string_view column fails:
ArrowNotImplementedError: Function 'equal' has no kernel matching input types (string_view, string)
It fails even when the join is on a different column of that table, just with a different error from pyarrow's array_filter (no kernel matching input types (string_view, bool)). binary_view columns hit the array_filter error too. The same comparisons as WHERE filters work fine, since filter pushdown is already turned off for these tables (#187, #227).
I think the join filter pushdown just never asks the scan whether it can take the filter. The Arrow scan says no through supports_pushdown_type (ArrowPushdownType), and the planner respects that for normal table filters by keeping a FILTER above the scan (plan_get.cpp). JoinFilterPushdownOptimizer::GetPushdownFilterTargets only checks filter_pushdown on the LOGICAL_GET, so the join's dynamic filters still end up in the scan and get handed to pyarrow.
Tables with a list<string_view> column fail the same way (in array_take), and so do tables with a run-end encoded column, which also need the fix in #666. SET disabled_optimizers = 'join_filter_pushdown' works around it.
Suggested fix
The fix would go in duckdb/duckdb: skip the columns the scan refuses, like the planner does.
diff --git a/src/optimizer/join_filter_pushdown_optimizer.cpp b/src/optimizer/join_filter_pushdown_optimizer.cpp
--- a/src/optimizer/join_filter_pushdown_optimizer.cpp
+++ b/src/optimizer/join_filter_pushdown_optimizer.cpp
@@ -233,10 +233,18 @@
}
get.ResolveOperatorTypes();
const auto bindings = get.GetColumnBindings();
+ vector<JoinFilterPushdownColumn> pushable_columns;
+ pushable_columns.reserve(columns.size());
for (auto &filter : columns) {
if (filter.probe_column_index.table_index != get.table_index) {
// the filter does not apply to the probe side here - bail-out
return;
+ }
+ if (get.function.supports_pushdown_type &&
+ !get.function.supports_pushdown_type(*get.bind_data,
+ get.GetColumnIndex(filter.probe_column_index).GetPrimaryIndex())) {
+ // the scan cannot evaluate a filter on this column - skip it, as the planner does for table filters
+ continue;
}
// add storage type to columns
for (idx_t i = 0; i < bindings.size(); i++) {
@@ -254,8 +262,12 @@
reconstructed_type.id() == LogicalTypeId::VARIANT) {
return;
}
- }
- targets.emplace_back(get, std::move(columns));
+ pushable_columns.push_back(filter);
+ }
+ if (pushable_columns.empty()) {
+ return;
+ }
+ targets.emplace_back(get, std::move(pushable_columns));
break;
}
case LogicalOperatorType::LOGICAL_PROJECTION: {The Top-N, build-side and row-group pruning optimizers also call GetPushdownFilterTargets, so they'd skip those columns too. The early return when no columns are left is there for RowGroupPruner::FindLogicalGet, which reads pushdown_targets[0].columns[0].
With the patch the reproducer below returns the right rows, and so do joins on binary_view and list<string_view> tables. Below are a Catch test for test/arrow/arrow_filter_pushdown.cpp and a pytest case for tests/fast/arrow/test_filter_pushdown.py. Both fail without the patch and pass with it, and [arrow], test/optimizer/* and tests/fast/arrow still pass. It applies cleanly to main (a809d3c) and to the core v2.0-cyanoptera pins (939d6af).
Core test
diff --git a/test/arrow/arrow_filter_pushdown.cpp b/test/arrow/arrow_filter_pushdown.cpp
--- a/test/arrow/arrow_filter_pushdown.cpp
+++ b/test/arrow/arrow_filter_pushdown.cpp
@@ -86,6 +86,34 @@
}
}
+// Helper: get the EXPLAIN ANALYZE output for a hash join that probes an arrow_scan on id with a one-row build side
+static string GetExplainAnalyzeForJoin(Connection &con, ArrowTestFactory &factory) {
+ con.TableFunction("arrow_scan", {}, {}, ArrowTestHelper::ConstructArrowScan(factory))->CreateView("probe");
+ const auto result =
+ con.Query("EXPLAIN ANALYZE SELECT count(*) FROM probe JOIN (SELECT 5 AS k) v ON probe.id = v.k");
+ REQUIRE(!result->HasError());
+ return result->Collection().GetValue(1, 0).ToString();
+}
+
+TEST_CASE("Arrow filter pushdown - join filters follow the pushdown types", "[arrow]") {
+ DuckDB db;
+ Connection con(db);
+
+ REQUIRE(!con.Query("CREATE TABLE src AS SELECT i AS id, i::VARCHAR AS name FROM range(10) tbl(i)")->HasError());
+
+ SECTION("Table with a string view column: no join filter in scan") {
+ auto factory = MakeArrowFactory(con, "SELECT * FROM src", true);
+ auto explain_str = GetExplainAnalyzeForJoin(con, *factory);
+ REQUIRE(explain_str.find("Dynamic Filters") == string::npos);
+ }
+
+ SECTION("Table with a regular string column: join filter in scan") {
+ auto factory = MakeArrowFactory(con, "SELECT * FROM src", false);
+ auto explain_str = GetExplainAnalyzeForJoin(con, *factory);
+ REQUIRE(explain_str.find("Dynamic Filters") != string::npos);
+ }
+}
+
TEST_CASE("Arrow filter pushdown - nested view types disable pushdown", "[arrow]") {
DuckDB db;
Connection con(db);Python test
diff --git a/tests/fast/arrow/test_filter_pushdown.py b/tests/fast/arrow/test_filter_pushdown.py
--- a/tests/fast/arrow/test_filter_pushdown.py
+++ b/tests/fast/arrow/test_filter_pushdown.py
@@ -647,6 +647,20 @@
res = duckdb.sql("SELECT * FROM dset WHERE col = 'abc'").fetchall()
assert res == [("abc",)]
+ @pytest.mark.parametrize(
+ ("view_type", "values", "key"),
+ [
+ pytest.param(pa.string_view(), ["abc", "efg"], "'abc'", id="string_view"),
+ pytest.param(pa.binary_view(), [b"abc", b"efg"], "'abc'::BLOB", id="binary_view"),
+ ],
+ )
+ def test_view_join_does_not_crash(self, view_type, values, key):
+ """A join's runtime filters cannot be pushed into a scan of a table with a view column either."""
+ con = duckdb.connect()
+ con.register("t", pa.table({"x": [1, 2], "col": pa.array(values, type=view_type)}))
+ assert con.execute(f"SELECT x FROM t JOIN (SELECT {key} AS k) b ON t.col = b.k").fetchall() == [(1,)]
+ assert con.execute("SELECT x FROM t JOIN (SELECT 2 AS k) b ON t.x = b.k").fetchall() == [(2,)]
+
# ===========================================================================
# 11. Projection / scanner-path interactionsTo Reproduce
import duckdb
import pyarrow as pa
c = duckdb.connect()
c.register("t", pa.table({"x": [1, 2], "col": pa.array(["abc", "efg"], type=pa.string_view())}))
print(c.sql("SELECT x FROM t WHERE col = 'abc'").fetchall()) # [(1,)]
print(c.sql("SELECT x FROM t JOIN (SELECT 'abc' AS k) b ON t.col = b.k").fetchall()) # duckdb.ErrorThe join should return [(1,)]. Joining on x instead (SELECT x FROM t JOIN (SELECT 2 AS k) b ON t.x = b.k) fails with the array_filter error, and should return [(2,)].
OS:
macOS 26.5.1, arm64
DuckDB Package Version:
1.5.6 and 2.0.0.dev2610011535, and a source build of v2.0-cyanoptera at 8bbac25
Python Version:
3.14.7 (pyarrow 26.0.0)
Full Name:
Will Frey
Affiliation:
dataplor
What is the latest build you tested with? If possible, we recommend testing with the latest nightly build.
I have tested with a nightly build
Did you include all relevant data sets for reproducing the issue?
Not applicable - the reproduction does not require a data set
Did you include all code required to reproduce the issue?
- Yes, I have
Did you include all relevant configuration to reproduce the issue?
- Yes, I have
Activity
[arrow] Fix pushdown errors (duckdb#27319)
evertlammerts commentedon Oct 27, 2026
Thanks for tracking this down to the join filter optimizer. I reran your joins on string_view, binary_view and list of string_view tables against the released 2.0.0 and against main, as a table, a reader and a Dataset. They all return the right rows now. In 2.0.0 the Arrow scan was replaced (#693) and no longer accepts the engine's join or Top-N filters at all, so this cannot come back through that path. Closing as fixed in 2.0.0; please reopen if you hit it there.