[viirya]

Which issue does this PR close?

• Closes Narrow multi-column NOT IN UNKNOWN checks by treating NOT NULL tuple elements as scope keys #26088.

Rationale for this change

A multi-column (a, b) NOT IN (SELECT x, y ...) plans as a null-aware hash join that decides UNKNOWN per build row. Without correlation keys, every row with a NULL in any tuple element is paired with every row on the other side, so the NULL path costs O(rows x NULL-valued rows).

A tuple element that cannot be NULL in scope on either side never makes the tuple comparison UNKNOWN: against a definite value it is either equal, which leaves the other elements to decide, or a definite mismatch, which makes the comparison FALSE. It only selects which rows a NULL in another element is compared with, which is exactly what a correlation scope key does. For example, with b and y NOT NULL, a NULL a only needs to be compared with the subquery rows whose y equals its b.

What changes are included in this PR?

• Decorrelation (build_join): for a null-aware tuple, the elements are partitioned by may_be_null_in_scope, keeping their order within each group. The elements that can be NULL come first and are the NOT IN value keys (null_aware_value_keys counts only them); the others follow as scope keys, before any correlation keys. The "value keys first" layout is unchanged, so the hash join needs no change: it already narrows the NULL candidate pairs through its scope hash map.
  • All elements NOT NULL: the join is not null-aware, as before.
  • Exactly one nullable element: V = 1, a scalar NOT IN on that element correlated by the others.
  • LeftMark (projection / OR) is reordered the same way and stays three-valued exact.
  • A NOT NULL element that is not a hashable equi-join key (e.g. RunEndEncoded) now stays in the join filter like any correlation instead of being rejected. A nullable one is still rejected.
• Docs: Join::null_aware_value_keys, HashJoinExec::null_aware_value_keys and the 56.0.0 upgrade guide describe the new key layout.
• The Q11 / Q12 comments of the null_aware_join benchmark now describe the scope-key path: their z column (value % 1000) is NOT NULL.

What is the testing strategy for this PR?

• null_aware_anti_join.slt: EXPLAIN plans for the reordering, with a correlation, with two nullable elements and one NOT NULL element, a constant element, NOT NULL on the outer side only (not reordered), and a mark join.
• null_aware_multi_column.slt (batch size matrix): results with NOT NULL elements for LeftAnti, LeftMark and OR, equivalence with the scalar NOT IN shape, correlated subqueries, an empty subquery, a constant element and a non-hashable NOT NULL element.
• null_aware_join_fuzz.rs: NOT NULL columns on both sides and seven mixed shapes, compared with the NOT EXISTS ... IS NOT FALSE reference. Placing the NOT NULL elements first fails this fuzz on its first seed.
• roundtrip_logical_plan.rs: the existing multi-column case had only NOT NULL VALUES columns for b/y, so it now expects one value key; a variant with a nullable b keeps covering two value keys.

Benchmark null_aware_join against main (mean of 3 alternating runs):

NAJ_ROWS	Q10 (no NULL)	Q11 (1% NULL in subquery)	Q12 (1% NULL in outer)
10000	0.34 ms -> 0.34 ms	3.55 ms -> 0.35 ms	0.72 ms -> 0.36 ms
20000	0.42 ms -> 0.47 ms	7.85 ms -> 0.49 ms	0.77 ms -> 0.51 ms

Q01 to Q09 are unchanged. Q10 (20000 rows, one outlier run dropped) is slightly slower: with a scope key, the build side eagerly builds a scope hash map over all build rows, which is only probed for NULL-valued probe rows. This is the existing behavior of correlated scalar NOT IN; building that map lazily could be a follow-up.

Are there any user-facing changes?

Faster multi-column NOT IN when some tuple elements are NOT NULL. Plans show fewer NOT IN value keys (null_aware(value_keys=N)) for such tuples. No API change.

[codecov-commenter]

Codecov Report

❌ Patch coverage is 95.83333% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 82.80%. Comparing base (454ca08) to head (06c0885).
⚠️ Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
...on/optimizer/src/decorrelate_predicate_subquery.rs	95.83%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #26214      +/-   ##
==========================================
- Coverage   82.80%   82.80%   -0.01%     
==========================================
  Files        1149     1149              
  Lines      452213   452226      +13     
  Branches   452213   452226      +13     
==========================================
  Hits       374473   374473              
- Misses      54945    54953       +8     
- Partials    22795    22800       +5     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[viirya]

@sunchao @comphead Can you take a look?

[sunchao]

Found one performance regression in the scope-key promotion; details inline. The local reproduction checks candidate enumeration using extracted executor helpers with minimal storage shims, rather than end-to-end query timings.

[sunchao]

[P2] Preserve early termination when promoting low-cardinality scope keys

This promotion can substantially increase work when the new scope key has few distinct values. For example, consider materialized tables with these schemas and contents:

-- outer_t(a INT, b INT NOT NULL): 8,192 rows of (NULL, 1)
-- inner_t(x INT, y INT NOT NULL): 8,192 rows of (2, 1)
SELECT *
FROM outer_t
WHERE (a, b) NOT IN (SELECT x, y FROM inner_t);

Previously, both elements were value keys, so for_each_unmarked_cross_product stopped once all outer rows were marked UNKNOWN. With b = y promoted to a scope key, for_each_scope_match enumerates every matching pair; retain_unmarked discards already-resolved rows only after the lookup has produced them. Both plans return the same result, but the new path keeps traversing the constant-key bucket.

A focused executable harness using the actual traversal/iteration helper bodies with minimal storage shims confirmed 8,192 candidate pairs before versus 67,108,864 afterward, with batch size 8,192 and both paths marking exactly 8,192 rows. These are helper-level work counts, not end-to-end query timings. The scalar early-exit path does not apply because the join still has two keys, and NULL marking runs before the empty full-key-map shortcut.

Could we preserve early termination or prune resolved scope entries before applying this transformation unconditionally, and add a constant/low-cardinality scope benchmark?

[viirya]

Thanks, confirmed. On the example from your comment, the scope-key path takes ~300 ms versus ~1 ms when b = y stays a value key. With #26213 merged it now plans as RightAnti and regresses the same way, and LeftMark too (218 ms vs 1 ms).

The root cause is not specific to this PR: the scope-key marking walks every pair in a bucket and drops already-marked rows only afterwards, so a correlated scalar NOT IN with a low-cardinality equality correlation already hits it on main (a NOT IN (SELECT x FROM inner_t i WHERE i.y = o.b) takes ~300 ms on the same data). #25679 works on this by skipping marked rows in the scope lookup and grouping probe rows by scope hash.

Since this PR only makes that path more common, I'll keep it on hold until a fix like #25679 lands, and then add a constant/low-cardinality scope-key query to the null_aware_join benchmark here.