[client-v2] Native format: a LowCardinality column is read as plain values, desynchronizes the stream ("Non-empty typeName is required")

[polyglotAI-bot]

Description

NativeFormatReader cannot read a top-level LowCardinality(...) column. Over HTTP, the server sends LowCardinality in the Native format with its dictionary encoding. The reader decodes it as if each row held one plain value (the RowBinary layout). It reads the wrong bytes, the stream loses synchronization, and the reader then parses data bytes as the next column header.

The result depends on the schema:

• If another column follows the LowCardinality column, or the result has more than one row, client.newBinaryFormatReader(response) throws IllegalArgumentException: Non-empty typeName is required. The message does not identify the column or the cause.
• For a single-row result with only the LowCardinality column (SELECT toLowCardinality('x') AS a), reader.next() returns 12 empty records ({}) before the same exception occurs. The expected row {a=x} is never returned.

Every LowCardinality shape I tested fails: LowCardinality(String), LowCardinality(Nullable(String)), LowCardinality(FixedString(1)), LowCardinality(UInt32), a table column, a column in the middle of the schema, and a multi-block result. The same queries read correctly with RowBinaryWithNamesAndTypes.

This is separate from #3156. That issue covers Array(LowCardinality(T)) and other nested shapes, where the element sub-column is decoded row-wise. Here the column is top-level and the cause is that no LowCardinality dictionary decoder exists.

Steps to reproduce

1. Create a client-v2 Client for an HTTP endpoint.
2. Run SELECT toLowCardinality(toString(number)) AS a, toInt32(number + 100) AS b FROM numbers(2) with QuerySettings.setFormat(ClickHouseFormat.Native).
3. Call client.newBinaryFormatReader(response).

Error Log or Exception StackTrace

java.lang.IllegalArgumentException: Non-empty typeName is required
	at com.clickhouse.data.ClickHouseDataType.of(ClickHouseDataType.java:488)
	at com.clickhouse.data.ClickHouseColumn.readColumn(ClickHouseColumn.java:714)
	at com.clickhouse.data.ClickHouseColumn.of(ClickHouseColumn.java:763)
	at com.clickhouse.client.api.data_formats.NativeFormatReader.readBlock(NativeFormatReader.java:87)
	at com.clickhouse.client.api.data_formats.NativeFormatReader.<init>(NativeFormatReader.java:34)
	at com.clickhouse.client.api.Client.newBinaryFormatReader(Client.java:2465)
	at com.clickhouse.client.api.Client.newBinaryFormatReader(Client.java:2485)

Expected Behaviour

The reader returns the same rows as the server and as the RowBinaryWithNamesAndTypes path. Server output (FORMAT JSONEachRow):

{"a":"0","b":100}
{"a":"1","b":101}

Results of the reproduction, client-v2 main @ 67a6b9e90:

Query	RowBinaryWithNamesAndTypes	Native
SELECT toLowCardinality(toString(number)) AS a, toInt32(number + 100) AS b FROM numbers(2)	{a=0, b=100}, {a=1, b=101}	IllegalArgumentException
SELECT toLowCardinality(if(number = 1, NULL, toNullable(toString(number)))) AS a, toInt32(number + 100) AS b FROM numbers(3)	{a=0, b=100}, {a=null, b=101}, {a=2, b=102}	IllegalArgumentException
SELECT toLowCardinality(toFixedString(toString(number), 1)) AS a, toInt32(number + 100) AS b FROM numbers(2)	correct	IllegalArgumentException
SELECT toInt32(number) AS id, toLowCardinality(toString(number)) AS a, toInt32(number + 100) AS b FROM numbers(2)	correct	IllegalArgumentException
SELECT toLowCardinality(toString(number % 3)) AS a, toInt32(number + 100) AS b FROM numbers(7) SETTINGS max_block_size=3	correct (7 rows)	IllegalArgumentException
SELECT a, b FROM t where a LowCardinality(String)	correct	IllegalArgumentException
SELECT toLowCardinality('x') AS a	{a=x}	12 empty records, then IllegalArgumentException
SELECT CAST(a AS String) AS a, b FROM t (workaround)	correct	correct

The server setting low_cardinality_allow_in_native_format=0 is not a workaround over HTTP: on 26.9 the server still sends the LowCardinality type and the dictionary encoding, and the reader fails the same way.

Code Example

QuerySettings settings = new QuerySettings().setFormat(ClickHouseFormat.Native);
try (QueryResponse response = client.query(
        "SELECT toLowCardinality(toString(number)) AS a, toInt32(number + 100) AS b FROM numbers(2)",
        settings).get()) {
    ClickHouseBinaryFormatReader reader = client.newBinaryFormatReader(response); // throws here
    Map<String, Object> row;
    while ((row = reader.next()) != null) {
        System.out.println(row.get("a") + " " + row.get("b"));
    }
}

Root cause

NativeFormatReader.readBlock has no branch for LowCardinality:

• client-v2/src/main/java/com/clickhouse/client/api/data_formats/NativeFormatReader.java:130 excludes LowCardinality from the Nullable null-map branch (correct, because LowCardinality(Nullable(T)) has no null map in Native).
• The column then reaches the generic branch at NativeFormatReader.java:153-158, which calls binaryStreamReader.readValue(column) once per row. That is the RowBinary decoder. In RowBinary the server sends LowCardinality(T) as plain T values, so the decoder reads a String (or UInt32, …) per row.

What the server sends in Native for one column in one block (raw bytes of SELECT toLowCardinality(toString(number)) AS a FROM numbers(2) FORMAT Native, after the header):

01 00 00 00 00 00 00 00   key serialization version (UInt64) = 1
00 06 00 00 00 00 00 00   index type + flags (UInt64) = 0x0600: UInt8 indexes, HasAdditionalKeys | NeedUpdateDictionary
03 00 00 00 00 00 00 00   dictionary size (UInt64) = 3
00 | 01 30 | 01 31        dictionary values as a Native String column: "", "0", "1"
02 00 00 00 00 00 00 00   number of indexes (UInt64) = 2
01 02                     indexes (UInt8): rows -> "0", "1"

The RowBinary decoder reads 01 00 as a 1-byte string and 00 as an empty string, stops there, and then reads the rest of the column as the name and type of the next column. The type name is empty, so ClickHouseColumn.of throws at NativeFormatReader.java:87.

Suggested fix

Add a LowCardinality Native decoder and route top-level LowCardinality columns to it in readBlock (before the generic branch). Per column per block:

1. Read the UInt64 key serialization version (expect 1, shared dictionaries with additional keys).
2. Read the UInt64 index type + flags. The low 8 bits give the index width: 0 = UInt8, 1 = UInt16, 2 = UInt32, 3 = UInt64 (for example, 300 distinct keys give 0x0601, UInt16).
3. Read the UInt64 dictionary size, then the dictionary as one Native column of the inner type. For LowCardinality(Nullable(T)) the dictionary is plain T with no null map, and index 0 means NULL. Raw bytes for (0, NULL, 2): dictionary "", "", "0", "2", indexes 02 00 03.
4. Read the UInt64 number of indexes, then the indexes, and map each index to its dictionary value.

Each block repeats steps 1-4 with its own dictionary (verified with max_block_size=2), so no state carries across blocks.

Contrast cases that must keep their current behavior:

• The RowBinary formats: there the server sends LowCardinality(T) as plain T, and readValue is correct.
• LowCardinality inside Array/Map/Tuple: that is part of [client-v2] Native format: nested columns (Tuple/Map/Nested/Variant, Array of Nullable/Array/LowCardinality) are decoded row-wise and misread #3156. If the fix does not cover nested shapes, they should fail with a clear ClientException and not desynchronize the stream.

If a decoder is out of scope, a smaller fix is to reject LowCardinality columns in readBlock with a clear ClientException that names the column and suggests a RowBinary format, as the reader already does for unsupported QBit shapes. That at least removes the misleading exception and the empty records.

Workaround until then: use RowBinaryWithNamesAndTypes (the default format), or CAST(col AS T) in the query.

Provenance

Found by automated analysis of client-v2 while investigating #3156. Verified against a live server with an integration test through Client.query + newBinaryFormatReader, not by inspection.

Configuration

Client Configuration

new Client.Builder()
        .addEndpoint(Protocol.HTTP, host, 8123, false)
        .setUsername("default")
        .setPassword(password)
        .build();

Environment

• Cloud
• Client version: client-v2 0.12.0-rc1-SNAPSHOT (main @ 67a6b9e90)
• Language version: Java 17
• OS: Linux (Docker)

ClickHouse Server

• ClickHouse Server version: 26.9.1.1629
• ClickHouse Server non-default settings, if any: none (low_cardinality_allow_in_native_format default 1)
• CREATE TABLE statements for tables involved: CREATE TABLE t (a LowCardinality(String), b Int32) ENGINE = MergeTree ORDER BY b (only for the table case; the other cases use numbers())
• Sample data for all these tables: INSERT INTO t VALUES ('x', 1), ('y', 2), ('x', 3)