You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Native Parquet scan builds an invalid StringArray for BINARY map keys read as STRING #7360
When a Parquet BINARY column without the UTF8 annotation is read as STRING (through a read schema or spark.sql.parquet.binaryAsString=true), #7106 decodes nested values with Spark's lossy UTF-8 decoding, except for map keys. Spark tells map keys apart by their bytes: decoding would turn the keys 41 FF 42 and 41 EF BF BD 42 into the same string, so a lookup of one returns the other's value and map_from_entries fails with DUPLICATED_MAP_KEY. To avoid that, the map-key arm of parquet_convert_array_impl (native/core/src/parquet/parquet_support.rs) keeps the raw bytes and wraps them with StringArray::new_unchecked.
If a key is not valid UTF-8, that breaks the constructor's precondition. With Arrow 59.3.0, StringArray::try_new with the same offsets, bytes and validity rejects the key 41 FF 42 at byte index 1. new_unchecked requires that the checked constructor would succeed (contract), so a later string operation on the key can produce an invalid Rust &str. The fixture in the #7106 test "native scan decodes invalid UTF-8 in nested BINARY read as string" builds such a key. Its byte-level assertions pass because they don't check that the array is valid.
This is the same unsound state that top-level values already reach (#7032, #4764). It is not a regression: before #7106 these keys became NULL through Arrow's safe cast. Affected: main, branch-1.2 (including 1.2.0), and branch-1.1 once #7340 merges.
Steps to reproduce
Build the key array from the #7106 fixture with StringArray::try_new instead of new_unchecked; it returns an invalid UTF-8 error. In a debug build, any operation that validates the array fails the same way.
Expected behavior
The native scan neither admits invalid UTF-8 into a Utf8 map key array nor merges keys that Spark keeps distinct. Options suggested in the review:
Keep the keys as binary until a boundary that preserves the bytes.
Fall back to Spark for a scan that reads a BINARY map key as STRING.
Validate with the checked constructor and return an error instead of building the array. This is sound but fails queries that Spark answers.
Additional context
The general policy for invalid UTF-8 entering native string arrays is tracked in #4764. A fix there for the top-level path should cover this one too.
Activity