Skip to content

[core] Negate predicates like Java so NULL rows stay excluded #1145

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
JeonDaehong wants to merge 4 commits into
base: main
Choose a base branch
from
Open

[core] Negate predicates like Java so NULL rows stay excluded #1145

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
fc40d40
[core] Negate predicates like Java so NULL rows stay excluded
JeonDaehong Oct 17, 2026
dfc1319
[core] Keep NaN rows under negation and refine floating bitmap candid…
JeonDaehong Oct 18, 2026
00d863a
[core] Keep exact BTree answers for floating filters
JeonDaehong Oct 19, 2026
189d400
[core] Guard nested negation and skip pruned bitmap entries in exactness
JeonDaehong Oct 22, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions crates/integrations/datafusion/src/filter_pushdown.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1350,7 +1350,7 @@ mod tests {
.pushed_predicate
.expect("supported conjunct should still translate")
.to_string(),
"(dt = '2024-01-01' AND NOT (hr = 10))"
"(dt = '2024-01-01' AND hr != 10)"
);
assert!(analysis.requires_residual);
}
Expand All @@ -1368,7 +1368,7 @@ mod tests {
.pushed_predicate
.expect("NOT partition predicate should translate inexactly")
.to_string(),
"NOT (dt = '2024-01-01')"
"dt != '2024-01-01'"
);
assert!(analysis.requires_residual);
}
Expand Down Expand Up @@ -1530,7 +1530,7 @@ mod tests {
build_pushed_predicate(&[filter], &fields)
.expect("NOT should translate as an inexact pushed predicate")
.to_string(),
"NOT (dt = '2024-01-01')"
"dt != '2024-01-01'"
);
}

Expand Down
16 changes: 8 additions & 8 deletions crates/integrations/datafusion/src/physical_plan/scan.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -452,11 +452,10 @@ fn predicate_covers_candidate(predicate: &Predicate, candidate: &Predicate) -> b
op: PredicateOperator::NotBetween,
literals,
},
Predicate::Not(inner),
// `Predicate::negate` turns NOT(x >= lo AND x <= hi) into
// x < lo OR x > hi.
Predicate::Or(children),
) if literals.len() == 2 => {
let Predicate::And(children) = inner.as_ref() else {
return false;
};
let matches_bound = |candidate: &Predicate,
expected_op: PredicateOperator,
expected_literal: &Datum| {
Expand All @@ -476,12 +475,13 @@ fn predicate_covers_candidate(predicate: &Predicate, candidate: &Predicate) -> b
&& candidate_literals[0] == *expected_literal
)
};
children
.iter()
.any(|child| matches_bound(child, PredicateOperator::GtEq, &literals[0]))
children.len() == 2
&& children
.iter()
.any(|child| matches_bound(child, PredicateOperator::Lt, &literals[0]))
&& children
.iter()
.any(|child| matches_bound(child, PredicateOperator::LtEq, &literals[1]))
.any(|child| matches_bound(child, PredicateOperator::Gt, &literals[1]))
}
_ => false,
}
Expand Down
29 changes: 26 additions & 3 deletions crates/paimon/src/arrow/residual.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -166,9 +166,16 @@ pub(crate) fn evaluate_predicate_mask(
else {
return Ok(None);
};
Ok(Some(boolean_mask_from_predicate(mask.len(), |row_index| {
!mask.value(row_index)
})))
let negated =
boolean_mask_from_predicate(mask.len(), |row_index| !mask.value(row_index));
if let Some(guard) = inner.not_null_guard() {

@JingsongLi JingsongLi Oct 22, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Keep raw double negation from selecting NULL rows

The new guard makes an inner Not(leaf) false for NULL, but an outer Not has no guard when its child is another Not, so it turns that false back into true. Public Predicate::Not is a logical-negation variant and existing code accepts raw nested Not trees. A public read probe writes value=[NULL,1,2], persists a supported INT-to-BIGINT schema evolution, and reads with Not(Not(value = 1)): this head returns ids [0,1], while exact base cea0caa returns only [1]. This is a newly introduced double-NOT regression, separate from the pre-existing raw Not(AND/OR) limitation. Normalize/eliminate double negation before evaluation, or preserve NULL semantics through nested Not consistently in both residual and BinaryRow evaluation.

@JeonDaehong JeonDaehong Oct 22, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right, the outer Not had no guard when its child was another Not. not_null_guard now passes through a nested Not, so Not(Not(v = 1)) takes v's guard in both the residual and row evaluation. The new test reads and deletes with it, with and without the INT→BIGINT widening, and returns [1] again ([0, 1] before).

Side note: clippy on the rebased branch fails on an unused BinaryRow import in table_upsert.rs from #1198, which this PR doesn't touch.

let Some(guard) = evaluate_predicate_mask(batch, &guard, file_fields, scan_fields)?
else {
return Ok(None);
};
return Ok(Some(combine_filter_masks(&negated, &guard, false)));
}
Ok(Some(negated))
}
Predicate::Leaf {
column,
Expand Down Expand Up @@ -2125,6 +2132,22 @@ mod tests {
assert_eq!(int_values(&out), vec![10, 20]);
}

#[test]
fn test_not_excludes_null_row() {
// NOT(age > 25) on [10, null, 30] -> [10]; NOT(unknown) is unknown.
let f = int_field(0, "age");
let b = int_batch("age", vec![Some(10), None, Some(30)]);
let pred = Predicate::Not(Box::new(leaf(
0,
DataType::Int(IntType::new()),
PredicateOperator::Gt,
vec![Datum::Int(25)],
)));
let fp = file_predicates(vec![pred], vec![f.clone()]);
let out = filter_record_batch_by_predicates(b, &fp, &[f]).unwrap();
assert_eq!(int_values(&out), vec![10]);
}

#[test]
fn test_null_row_excluded() {
// age = [10, null, 30]; age > 5 -> [10, 30] (null → false).
Expand Down
5 changes: 3 additions & 2 deletions crates/paimon/src/predicate_stats.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1054,13 +1054,14 @@ mod tests {
let dt = DataType::Int(IntType::new());
let fields = vec![DataField::new(0, "id".to_string(), dt.clone())];
let mapping = vec![Some(0)];
let predicate = Predicate::negate(Predicate::Leaf {
// Built directly: `Predicate::negate` would flip the leaf to `<=`.
let predicate = Predicate::Not(Box::new(Predicate::Leaf {
column: "id".to_string(),
index: 0,
data_type: dt,
op: PredicateOperator::Gt,
literals: vec![Datum::Int(5)],
});
}));

let with_nulls = MockStats {
row_count: 5,
Expand Down
Loading
Loading