Children of nullable Struct may have null values propagated to zone maps

[myrrc]

All Vortex files which have somewhere a nullable Struct with children face the following bug: children's values which are NULL may have their (unspecified) values propagated to zone maps. This also influences null count statistic.
What's worse, we can reproduce this simpler with FlatLayoutStrategy. As we use null count for falsifying zones, we may reject a valid zone.

See the following example: is_null filter returns 0 and not 1.

use vortex::VortexSessionDefault;
use vortex::array::IntoArray;
use vortex::array::arrays::BoolArray;
use vortex::array::arrays::StructArray;
use vortex::array::expr::get_item;
use vortex::array::expr::is_null;
use vortex::array::expr::root;
use vortex::array::stream::ArrayStreamExt;
use vortex::array::validity::Validity;
use vortex::buffer::ByteBufferMut;
use vortex::buffer::buffer;
use vortex::dtype::FieldNames;
use vortex::error::VortexResult;
use vortex::file::OpenOptionsSessionExt;
use vortex::file::WriteOptionsSessionExt;
use vortex::session::VortexSession;

#[tokio::main]
async fn main() -> VortexResult<()> {
    let session = VortexSession::default();

    let nullable_struct = StructArray::try_new(
        FieldNames::from(["a"]),
        vec![buffer![1i32, 2].into_array()],
        2,
        Validity::Array(BoolArray::from_iter([true, false]).into_array()),
    )?;
    let st = StructArray::try_new(
        FieldNames::from(["s"]),
        vec![nullable_struct.into_array()],
        2,
        Validity::NonNullable,
    )?
    .into_array();

    let expr = is_null(get_item("a", get_item("s", root())));
    println!(
        "in-memory is_null(s.a): {}",
        st.clone().apply(&expr)?.display_values()
    );

    let mut buf = ByteBufferMut::empty();
    session
        .write_options()
        .write(&mut buf, st.to_array_stream())
        .await?;
    let file = session.open_options().open_buffer(buf)?;

    let projected = file
        .scan()?
        .with_projection(get_item("a", get_item("s", root())))
        .into_array_stream()?
        .read_all()
        .await?;
    println!("file scan, projected s.a: {}", projected.display_values());

    let filtered = file
        .scan()?
        .with_filter(expr)
        .into_array_stream()?
        .read_all()
        .await?;
    println!(
        "file scan, filter is_null(s.a): {} rows (expected 1)",
        filtered.len()
    );

    Ok(())
}

Our writer rejects top-level nullable structs but doesn't reject such structs which are nested.

Arrow allows children of nullable structs to hold unspecified values, so if we reject this, we'll be rejecting valid Arrow inputs.

[myrrc]

The reason for this bug is we write fields unmasked (i.e. not applying validity) and storing struct validity separately, and our filters evaluate column's validity only, not taking parent validity into account.