Skip to content

Commit 6e9a401

Browse files
docs: clarify decimal/timestamp arrow-native flags (Thrift), verified live
Both `_use_arrow_native_decimals` and `_use_arrow_native_timestamps` look like twin knobs, but behave oppositely when off — confirmed against a live warehouse: - `_use_arrow_native_decimals=False`: no value-level effect. The wire encoding becomes an Arrow string, but the connector unconditionally re-casts it back to `decimal128` (`convert_decimals_in_arrow_table`), so fetches always yield `Decimal`. - `_use_arrow_native_timestamps=False`: genuinely returns Python `str` (Arrow `string`) — there is no re-cast on the Arrow path — while `cursor.description` still reports `'timestamp'`. The `timestampAsArrow=False` flag also wins over the always-sent `timestampAsString=false` conf. Doc-only change. Co-authored-by: Isaac
1 parent 44f6eba commit 6e9a401

1 file changed

Lines changed: 2 additions & 2 deletions

File tree

CONNECTION_PARAMETERS.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -139,8 +139,8 @@ to change without notice.
139139
| `enable_query_result_lz4_compression` | `bool` ||| `True` | LZ4-compress result payloads. Not forwarded; the kernel handles compression internally. |
140140
| `_disable_pandas` | `bool` ||| `False` | Skip the pandas-based Arrow→row deserialization and materialize rows directly with PyArrow. This is a **Python-side** result-conversion toggle, not a wire option: the kernel returns results as Arrow (`RecordBatch`es) and the connector runs the *same* `_convert_arrow_table` for both backends, so the flag is honored on the kernel path too. Affects only row fetches (`fetchone`/`fetchmany`/`fetchall`); the `fetch*_arrow` methods return the Arrow table unchanged regardless of this flag. |
141141
| `_use_arrow_native_complex_types` | `bool` ||| `True` | Return `ARRAY`/`MAP`/`STRUCT` as native Arrow types instead of JSON strings. Forwarded to the kernel. |
142-
| `_use_arrow_native_decimals` | `bool` ||| `True` | Return `DECIMAL` as a native Arrow type instead of a string. Thrift-only. |
143-
| `_use_arrow_native_timestamps` | `bool` ||| `True` | Return `TIMESTAMP` as a native Arrow type instead of a string. Thrift-only. |
142+
| `_use_arrow_native_decimals` | `bool` ||| `True` | Thrift wire encoding for `DECIMAL`: `True` native Arrow `decimal128`, `False` → Arrow string. **No value-level effect**, though: the connector unconditionally re-casts the column back to `decimal128` (`convert_decimals_in_arrow_table`, `thrift_backend.py`), so both `fetchall()` and `fetchall_arrow()` yield `Decimal` / `decimal128(p,s)` either way (verified live). Not forwarded to the kernel, which always returns native Arrow decimals. |
143+
| `_use_arrow_native_timestamps` | `bool` ||| `True` | Thrift wire encoding for `TIMESTAMP`: `True` native Arrow timestamp (→ Python `datetime`), `False` → Arrow string (→ Python **`str`**). **Unlike decimals there is no re-cast**, so `False` genuinely surfaces strings — and `cursor.description` still reports the type code as `'timestamp'`, a mismatch to watch for (verified live). Note the connector always also sends the `spark.thriftserver.arrowBasedRowSet.timestampAsString=false` conf, but the `timestampAsArrow=False` flag wins. Not forwarded to the kernel, which always returns native Arrow timestamps. |
144144

145145
## Session defaults & transactions
146146

0 commit comments

Comments
 (0)