Conversation
`NewColumnTransform::Reader` / `NewColumnTransform::Stream` wrote new-column values straight to the fragment, skipping the file format null check that the hash-join based merge path applies via `HashJoiner::check_lance_support_null`. On file format versions that cannot store nulls for a type (e.g. integer columns on `Legacy`), an explicit `NULL` in the stream was silently written as the default value: adding `[1, NULL, 3]` succeeded and read back as `[1, 0, 3]`. Apply the same check to every column of each batch before writing it, so such writes fail with an error instead.
There was a problem hiding this comment.
✅ Gate recommendation: approve.
The stream/reader path now enforces the established format-specific null contract immediately before updating a fragment, preventing Legacy integer nulls from being silently materialized as defaults while leaving supported V2 nulls unchanged. Keeping the validation in the lazy consumption path avoids pre-scanning or buffering and lets the existing failure cleanup cover partial fragment work.
yanghua
left a comment
There was a problem hiding this comment.
The fix looks correct and brings the stream/reader path in line with the existing hash-join validation. I did not find a blocking correctness, compatibility, security, or memory-regression issue.
Two non-blocking test suggestions:
- Add a V2.1+ positive case verifying that nullable integer values are accepted and round-trip with the null bitmap intact.
- Track nested-array validation separately:
validate_nullsappears to inspect only the top-level type andnull_count(), so a non-null Struct/List containing child nulls may bypass the Legacy-format restriction. This is pre-existing and shared with the hash-join path, so I would not block this PR on it.
| // Reject nulls the dataset's file format cannot store (e.g. integer | ||
| // nulls on Legacy), matching the hash-join based merge path, instead | ||
| // of silently writing them as default values. | ||
| for column in new_batch.columns() { |
There was a problem hiding this comment.
check_lance_support_null currently surfaces the error from validate_nulls, whose message says "Join produced null values...". There is no join in this stream/reader path, so the message is misleading. It also does not identify the offending column, which makes failures ambiguous when multiple columns are added.
Could we either wrap the error here with the operation and column name, or update the validator to accept that context? For example:
for (i, column) in new_batch.columns().iter().enumerate() {
HashJoiner::check_lance_support_null(column, updater.dataset())
.map_err(|err| Error::invalid_input(format!(
"add_columns: column '{}': {}",
new_batch.schema().field(i).name(),
err
)))?;
}
| } | ||
|
|
||
| #[tokio::test] | ||
| async fn test_add_columns_via_reader_rejects_unsupported_nulls() { |
There was a problem hiding this comment.
This test directly covers only NewColumnTransform::Reader. Reader and Stream currently share add_columns_from_stream, so the implementation is exercised indirectly, but a future refactor could make the two paths diverge.
Could we parameterize this regression test over both variants, or add a focused Stream case?
`validate_nulls` reported "Join produced null values ...", which is wrong for the `add_columns` stream/reader path (there is no join) and never identified which column carried the unsupported null. Pass the column name through `HashJoiner::check_lance_support_null` and make the message operation-neutral. Also cover both `NewColumnTransform::Reader` and `NewColumnTransform::Stream` in the regression test.
7e8534b to
c27eb65
Compare
NewColumnTransform::Reader/NewColumnTransform::Streamwrote new-column values straight to the fragment, skipping the file format null check that the hash-join based merge path applies viaHashJoiner::check_lance_support_null. On file format versions that cannot store nulls for a type (e.g. integer columns onLegacy), an explicitNULLin the stream was silently written as the default value: adding[1, NULL, 3]succeeded and read back as[1, 0, 3].Apply the same check to every column of each batch before writing it, so such writes fail with an error instead.