[adriangb]

Which issue does this PR close?

• Closes Null-aware join: an equality correlation makes a correlated NOT IN slower, not faster #25438.

Rationale for this change

An equality correlation made a correlated NOT IN slower, not faster. The cost grew with the square of the scope size. This is the reproduction from the issue, with outer_t and inner_t at 100,000 rows each:

-- (a) non-equality correlation only
SELECT count(*) FROM outer_t o
WHERE o.id_n50 NOT IN (SELECT i.id_n0 FROM inner_t i WHERE i.z < o.z);

-- (b) the same, plus an equality correlation on k
SELECT count(*) FROM outer_t o
WHERE o.z > 900
   OR o.id_n50 NOT IN (SELECT i.id_n0 FROM inner_t i WHERE i.k = o.k AND i.z < o.z);

Release datafusion-cli, Apple M4 Pro, median of 10 runs:

Query	Before	After
(a)	13 ms	13 ms
(b)	415 ms	10 ms

Query (b) is now as fast as query (a). Both queries return the same results as before.

The null_aware_join benchmark suite, default sizes, median of 3 interleaved runs of each side (CARGO_PROFILE_RELEASE_LTO=thin for both sides):

Query	Before (ms)	After (ms)	Change
Q01	3.71	3.70	1.00x
Q02	2.63	2.66	1.01x
Q03	2.27	2.27	1.00x
Q04	0.30	0.31	1.02x
Q05	0.54	0.55	1.02x
Q06	0.54	0.55	1.02x
Q07	0.49	0.49	1.00x
Q08	12.85	0.90	0.07x
Q09	0.41	0.42	1.01x

Only Q08 is a null-aware join with a correlation scope key. The other queries do not reach the changed code, and their changes are noise.

What changes are included in this PR?

A correlated null-aware join marks a build row UNKNOWN when a candidate pair passes the join filter. A marked build row stays marked. With a scope key, the candidate pairs come from a hash lookup on the scope key. Before this PR, each probe row found all the build rows in its scope, and the marked build rows were removed only after the lookup made the pairs. A marked build row was thus paired again with every later probe row in its scope.

The change is in UnmarkedPairs::scope_matches in hash_join/not_in.rs:

1. The lookup drops the matches whose build row is marked, before it makes pairs. The check is one bit per match.
2. That alone is not sufficient. Each probe row still walks its whole scope, so (b) went only from 1.8 s to 1.0 s of CPU. So the join counts the matches it skipped in a probe batch. When they cost more than a sort of the probe rows, the join sorts the rest of the probe rows by scope hash. Then each hash is looked up once, and its group of probe rows is paired like the cross product of the path without scope keys: the marked build rows are dropped after each chunk of pairs, and the pairing of a group stops when no unmarked build row is left.
3. A build row belongs to one scope hash only. So the join refreshes the build rows of a group only after a chunk that can contain that group's pairs. It does not take the bitmap lock once per group.

UnmarkedPairs also serves the RightAnti path, which marks probe rows. A probe row is looked up only once, so that path never switches to grouping.

The PR has three commits, for review one at a time:

1. refactor: no behavior change. The cross product of the path without scope keys and the scope-key lookup share one struct, UnmarkedPairs, which holds the chunk buffers and the refresh logic. The scope-key lookup yields full row indices, and the join checks the scope-key equality per chunk, so NullValueBuildRows drops its copy of the scope values.
2. test: a unit test that pins the current behavior. The pairs made for one build row grow tenfold when the probe rows grow tenfold.
3. perf: the fix. The test diff shows that the pairs no longer grow.

Why not the alternative in the issue. The issue also proposes a choice per probe batch between the lookup and the cross product, with the scope equality applied to the cross-product pairs. That choice must be made before any build row is marked, so the estimate is wrong exactly in the case that matters: in Q08 the cross product is 16 times larger than the lookup, but it stops after a few probe rows. The switch in this PR is made during the batch, from the matches the join actually skipped. The cost before the switch is bounded, and shapes that never re-pair a marked build row never pay for a sort.

Why not a sort in every batch. I measured that first. The pairs of the plain lookup are cheap, about 5 ns each, so a sort of every probe batch costs the same as a few pairs per probe row. With a unique scope key, that made the join 40–70% slower in CPU. A fixed threshold on the average scope size did not work either: 8 build rows per scope, with one probe row each, still cost 70% more.

Cost on shapes that do not gain. I measured shapes with a scope key that is unique, or nearly unique, on the build side (2M × 2M rows, 1–50% NULL, no filter or a filter that is never or always true, 1 or 2 probe rows per scope). These never switch to grouping. CPU is 3–15% higher and wall time 0–10% higher. This comes from the bit check at lookup time, which runs in addition to the check before the join filter.

What is the testing strategy for this PR?

• No result changes. null_aware_anti_join.slt, null_aware_mark_join.slt, subquery.slt, the full sqllogictest suite and the hash_join unit tests pass.
• I compared eight scope-key queries against DuckDB 1.5.2: anti and mark joins, NULLs on both sides, NULL scope keys, one and two scope keys, residual filters, and 7 to 9 scope groups. I ran them at batch sizes 8192, 7 and 1, with 1 and 4 partitions, and on a larger variant (30k × 25k rows) where each scope holds hundreds of build rows. All results match.
• New unit test unmarked_pairs_do_not_grow_with_probe_rows in hash_join/not_in.rs. It gives 100 build rows in two scopes to UnmarkedPairs::scope_matches, with 1,000 and then 10,000 probe rows, and marks every pair that it gets. It checks that all build rows are marked, and that the pairs made for one build row do not grow with the number of probe rows, at batch sizes 1, 16 and 8192. With the old behavior (no marked build row skipped at the lookup), a build row gets 500 pairs with 1,000 probe rows and 5,000 with 10,000, and the test fails. I added no sqllogictest, because the change does not change any result, so a new .slt case would pass before and after the change.

Are there any user-facing changes?

No. Correlated NOT IN queries with an equality correlation are faster.

[codecov-commenter]

Codecov Report

❌ Patch coverage is 90.30303% with 32 lines in your changes missing coverage. Please review.
✅ Project coverage is 82.49%. Comparing base (95bb0a0) to head (0bbdfa9).

Files with missing lines	Patch %	Lines
...fusion/physical-plan/src/joins/hash_join/stream.rs	91.60%	10 Missing and 11 partials ⚠️
...on/optimizer/src/decorrelate_predicate_subquery.rs	86.07%	3 Missing and 8 partials ⚠️

Additional details and impacted files

@@           Coverage Diff            @@
##             main   #25679    +/-   ##
========================================
  Coverage   82.48%   82.49%            
========================================
  Files        1140     1140            
  Lines      437614   437727   +113     
  Branches   437614   437727   +113     
========================================
+ Hits       360986   361092   +106     
- Misses      54834    54849    +15     
+ Partials    21794    21786     -8     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[viirya]

Thanks for this — the grouping idea fixes the pathology nicely. I rebased the last commit onto current main to check it against the repro sunchao raised on #26214 (#26214 (comment)). Conflicts are small: the multi-column mark closure from #19857, MapOffset → ProbeOffset (#26203), and equal_rows_arr now needs the comparator slot (the build scope values are normalized, so equal_rows_arr_with_normalized_left fits). With 8192 × 8192 rows and a constant scope key, a NOT IN (SELECT x FROM inner_t i WHERE i.y = o.b) goes from ~206 ms to ~1 ms, including the variant where one outer row sits in a scope that is never marked. The issue's Q08 at 100k goes from ~367 ms to ~10 ms, and the unique-scope 2M × 2M shapes cost +2–6% wall, in line with your numbers.

Now that #25560 has landed, do you plan to take this out of draft? #26214 makes NOT NULL tuple elements of a multi-column NOT IN into scope keys, which routes more queries into this path, so I'd like to stack it on top of this. If you don't plan to continue with it, we'd be happy to open a PR that builds on your commit.

[viirya]

Following up on my earlier comment: #26214 is on hold until the scope-key marking stops re-pairing already-marked rows, which is what this PR does. Would you have a chance to pick this up again soon? If you're busy with other things, I'd be glad to help move it forward: I could open a PR that builds on your commit and rebases it onto current main (the null-aware join code has since moved to hash_join/not_in.rs), with you credited as co-author. Maybe sometime next week, if that works for you? Happy to hand it back anytime if you'd rather finish it yourself.

[adriangb]

Sorry for the delay @viirya. Marked as ready now!

[Copilot]

🟡 Changes recommended

Regression coverage does not validate grouped pairing with remaining unmarked rows or detect loss of the hash-chain traversal optimization.

1 open issue

• Add selective grouped-pairing regression and bound hash-chain traversal · New

What changed in this PR

This PR addresses #25438 by reducing repeated work in DataFusion’s scope-key null-aware joins, improving correlated NOT IN performance without intended result changes.

Changes:

• Skip marked rows during scope lookup and adaptively group remaining probe rows by hash.
• Share chunked pairing logic through UnmarkedPairs and remove retained copies of NULL-build scope values.
• Add a regression test for emitted-pair growth.

File	Description
datafusion/​physical-plan/​src/​joins/​hash_join/​not_in.rs	Optimizes scope matching, consolidates pairing logic, reduces retained state, and adds regression coverage.

_{Review effort: Balanced}