Conversation
Lakehouse//RT warehouses only speak the Statement Execution API (SEA) and reject the Thrift protocol, so Rill (pinned to databricks-sql-go v1.10.0) could not connect to them at all. Bump databricks-sql-go v1.10.0 -> v1.15.1 (adds the SEA/kernel backend) and auto-detect RT: on first connect the connector probes once and, if the warehouse rejects Thrift, transparently switches to the SEA backend. The decision is cached and shared by both the OLAP path and the warehouse ingest path (effectiveDSN), so an RT warehouse works with no configuration. DBSQL warehouses accept Thrift and are unchanged. A `use_kernel` connector property (default false) is available as an explicit override. The SEA backend exports Arrow C Data rather than IPC streams, so the bulk ingest path in warehouse.go falls back from GetArrowIPCStreams to GetArrowBatches (re-serialized to a self-contained Arrow IPC stream via the driver's Arrow v12 writer, which the existing v18 ipc.Reader consumes) when the driver returns ErrNotSupportedByKernel. The OLAP path already worked over SEA. information_schema.Lookup previously JOINed information_schema.tables and columns; that join forces a shuffle that RT's Photon rejects (PHOTON_INTERNAL_ERROR, retry unsupported), breaking the schema browser. Split it into two filtered point-lookups (no shuffle); equivalent on DBSQL. Adds unit tests for DSN resolution, the RT-detection predicate, and the useKernel DSN helper, and documents the behavior. Co-authored-by: Isaac <no-reply@databricks.com>
# Conflicts: # go.mod # go.sum
Adds TestOLAP_LakehouseRT, a live integration test that drives the full Rill Databricks OLAP path against a Lakehouse//RT (Reyden) SQL warehouse over the auto-detected SEA backend. The connector config passes only the DSN (no use_kernel), so a passing query proves the Thrift->SEA autodetection end to end. Because Lakehouse//RT speaks only SEA, the test requires the SEA-via-kernel backend and carries //go:build databricks_kernel (CGO), excluding it from the default Thrift-only build and the standard `go test -short ./...` CI job. It is otherwise gated like the existing Databricks/Snowflake live tests: a leading t.Skip disables it by default, testmode.Expensive keeps it out of normal runs, and it needs RILL_RUNTIME_DATABRICKS_RT_TEST_DSN to point at an RT warehouse. Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
INSERT DESCRIPTION HERE
Checklist: