Skip to content

Native Parquet scan builds an invalid StringArray for BINARY map keys read as STRING #7360

Open

Description

@andygrove

Describe the bug

When a Parquet BINARY column without the UTF8 annotation is read as STRING (through a read schema or spark.sql.parquet.binaryAsString=true), #7106 decodes nested values with Spark's lossy UTF-8 decoding, except for map keys. Spark tells map keys apart by their bytes: decoding would turn the keys 41 FF 42 and 41 EF BF BD 42 into the same string, so a lookup of one returns the other's value and map_from_entries fails with DUPLICATED_MAP_KEY. To avoid that, the map-key arm of parquet_convert_array_impl (native/core/src/parquet/parquet_support.rs) keeps the raw bytes and wraps them with StringArray::new_unchecked.

If a key is not valid UTF-8, that breaks the constructor's precondition. With Arrow 59.3.0, StringArray::try_new with the same offsets, bytes and validity rejects the key 41 FF 42 at byte index 1. new_unchecked requires that the checked constructor would succeed (contract), so a later string operation on the key can produce an invalid Rust &str. The fixture in the #7106 test "native scan decodes invalid UTF-8 in nested BINARY read as string" builds such a key. Its byte-level assertions pass because they don't check that the array is valid.

Reported by @sunchao in #7340 (comment).

This is the same unsound state that top-level values already reach (#7032, #4764). It is not a regression: before #7106 these keys became NULL through Arrow's safe cast. Affected: main, branch-1.2 (including 1.2.0), and branch-1.1 once #7340 merges.

Steps to reproduce

Build the key array from the #7106 fixture with StringArray::try_new instead of new_unchecked; it returns an invalid UTF-8 error. In a debug build, any operation that validates the array fails the same way.

Expected behavior

The native scan neither admits invalid UTF-8 into a Utf8 map key array nor merges keys that Spark keeps distinct. Options suggested in the review:

  1. Keep the keys as binary until a boundary that preserves the bytes.
  2. Fall back to Spark for a scan that reads a BINARY map key as STRING.
  3. Validate with the checked constructor and return an error instead of building the array. This is sound but fails queries that Spark answers.

Additional context

The general policy for invalid UTF-8 entering native string arrays is tracked in #4764. A fix there for the top-level path should cover this one too.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions