Skip to content

fix: reject unsupported nulls in the add_columns stream path - #9239

Merged
yanghua merged 2 commits into
lance-format:mainfrom
zhangyue19921010:add-columns-stream-legacy-null-check
Sep 17, 2026
Merged

yanghua merged 2 commits into
lance-format:mainfrom
zhangyue19921010:add-columns-stream-legacy-null-check

Conversation

@zhangyue19921010

Copy link
Copy Markdown
Collaborator

NewColumnTransform::Reader / NewColumnTransform::Stream wrote new-column values straight to the fragment, skipping the file format null check that the hash-join based merge path applies via HashJoiner::check_lance_support_null. On file format versions that cannot store nulls for a type (e.g. integer columns on Legacy), an explicit NULL in the stream was silently written as the default value: adding [1, NULL, 3] succeeded and read back as [1, 0, 3].

Apply the same check to every column of each batch before writing it, so such writes fail with an error instead.

`NewColumnTransform::Reader` / `NewColumnTransform::Stream` wrote new-column
values straight to the fragment, skipping the file format null check that the
hash-join based merge path applies via `HashJoiner::check_lance_support_null`.
On file format versions that cannot store nulls for a type (e.g. integer
columns on `Legacy`), an explicit `NULL` in the stream was silently written as
the default value: adding `[1, NULL, 3]` succeeded and read back as `[1, 0, 3]`.

Apply the same check to every column of each batch before writing it, so such
writes fail with an error instead.
@github-actions github-actions Bot added the bug Something isn't working label Sep 15, 2026

@lance-gatekeeper lance-gatekeeper Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Gate recommendation: approve.

The stream/reader path now enforces the established format-specific null contract immediately before updating a fragment, preventing Legacy integer nulls from being silently materialized as defaults while leaving supported V2 nulls unchanged. Keeping the validation in the lazy consumption path avoids pre-scanning or buffering and lets the existing failure cleanup cover partial fragment work.

@lance-gatekeeper lance-gatekeeper Bot added the K-approved Latest Gatekeeper recommendation permits acceptance. label Sep 15, 2026

@yanghua yanghua left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The fix looks correct and brings the stream/reader path in line with the existing hash-join validation. I did not find a blocking correctness, compatibility, security, or memory-regression issue.

Two non-blocking test suggestions:

  1. Add a V2.1+ positive case verifying that nullable integer values are accepted and round-trip with the null bitmap intact.
  2. Track nested-array validation separately: validate_nulls appears to inspect only the top-level type and null_count(), so a non-null Struct/List containing child nulls may bypass the Legacy-format restriction. This is pre-existing and shared with the hash-join path, so I would not block this PR on it.

// Reject nulls the dataset's file format cannot store (e.g. integer
// nulls on Legacy), matching the hash-join based merge path, instead
// of silently writing them as default values.
for column in new_batch.columns() {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

check_lance_support_null currently surfaces the error from validate_nulls, whose message says "Join produced null values...". There is no join in this stream/reader path, so the message is misleading. It also does not identify the offending column, which makes failures ambiguous when multiple columns are added.

Could we either wrap the error here with the operation and column name, or update the validator to accept that context? For example:

for (i, column) in new_batch.columns().iter().enumerate() {
    HashJoiner::check_lance_support_null(column, updater.dataset())
        .map_err(|err| Error::invalid_input(format!(
            "add_columns: column '{}': {}",
            new_batch.schema().field(i).name(),
            err
        )))?;
}

}

#[tokio::test]
async fn test_add_columns_via_reader_rejects_unsupported_nulls() {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This test directly covers only NewColumnTransform::Reader. Reader and Stream currently share add_columns_from_stream, so the implementation is exercised indirectly, but a future refactor could make the two paths diverge.

Could we parameterize this regression test over both variants, or add a focused Stream case?

@github-actions github-actions Bot added the A-python Python bindings label Sep 16, 2026
@lance-gatekeeper lance-gatekeeper Bot removed the K-approved Latest Gatekeeper recommendation permits acceptance. label Sep 16, 2026
`validate_nulls` reported "Join produced null values ...", which is wrong for
the `add_columns` stream/reader path (there is no join) and never identified
which column carried the unsupported null. Pass the column name through
`HashJoiner::check_lance_support_null` and make the message operation-neutral.

Also cover both `NewColumnTransform::Reader` and `NewColumnTransform::Stream`
in the regression test.
@zhangyue19921010
zhangyue19921010 force-pushed the add-columns-stream-legacy-null-check branch from 7e8534b to c27eb65 Compare September 16, 2026 08:34
@yanghua
yanghua merged commit 9957629 into lance-format:main Sep 17, 2026
37 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

A-python Python bindings bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants