Skip to content

Commit c95ca3f

Browse files
feat(kernel): route Azure Entra auth (azure-sp-m2m + azure-oauth) through the auth bridge (#919)
* feat(kernel): thread Azure Entra OAuth (U2M + SP M2M) through the auth bridge The kernel auth bridge rejected azure-oauth and had no azure-sp-m2m path. Route both Azure auth types onto the kernel's generic OAuth flows (the kernel needs no Azure-specific code; PR databricks/databricks-sql-kernel#263 added the token_url/scope override plumbing this relies on): - azure-oauth (Azure AD U2M) -> oauth-u2m with the Azure app client id (96eecda7-...), redirect port 8030, and the {app_id}/user_impersonation offline_access delegated scope (via AzureOAuthEndpointCollection, honoring DATABRICKS_AZURE_TENANT_ID). The kernel discovers endpoints via the workspace /oidc redirector. (PECOBLR-4120) - azure-sp-m2m (Azure service principal) -> oauth-m2m with the Azure creds, an Entra v2.0 token_url, and the {effective_app_id}/.default scope. Requires an explicit azure_tenant_id (the kernel path does not auto-discover it). The management-token header / azure_workspace_resource_id are not applied on the kernel path -- no SQL connector uses them, matching Go and Node. (PECOBLR-4141) kernel_auth_kwargs now takes hostname (for the effective Azure app id); the client passes self._server_hostname. TDD: replaced the azure-oauth NotSupportedError test with routing tests and added a TestKernelAzureSpM2M suite (routing, required tenant/creds, federation client id). 50 bridge tests pass; black clean. Co-authored-by: Isaac Signed-off-by: eric-wang-1990 <e.wang@databricks.com> * docs: update CONNECTION_PARAMETERS.md for kernel Azure OAuth support The auth table marked the azure_* fields as Kernel-unsupported and claimed azure-oauth 'still works on the kernel' (it was actually rejected). Reflect the new routing: azure-sp-m2m + azure-oauth now work on the kernel path; azure_tenant_id is required there; the management token / azure_workspace_resource_id are not applied (matching Go/Node). Co-authored-by: Isaac Signed-off-by: eric-wang-1990 <e.wang@databricks.com> * refactor(kernel): thin azure-oauth — kernel owns Azure resolution The kernel is the auth core now: for azure-oauth the bridge forwards only auth_type='azure-oauth' (+ optional client_id/redirect_port passthrough), and the kernel pins the workspace v2.0 authorize/token endpoints, the Azure app client id, port 8030, and the user_impersonation scope. Drops the connector-side endpoint/scope construction (and the AzureOAuthEndpointCollection / PYSQL_OAUTH_AZURE_* imports) from the kernel path. Live-verified end-to-end against an Azure workspace. azure-sp-m2m still routes to oauth-m2m here pending the kernel's dedicated azure-sp-m2m variant. Co-authored-by: Isaac Signed-off-by: eric-wang-1990 <e.wang@databricks.com> * refactor(kernel): thin azure-sp-m2m — kernel owns Azure resolution + optional mgmt token Make the `azure-sp-m2m` bridge thin, matching the kernel becoming the Azure-aware auth core. The connector now forwards `auth_type='azure-sp-m2m'` + `azure_client_id` / `azure_client_secret` (and optional `azure_tenant_id` / `azure_workspace_resource_id`) straight to the kernel Session, instead of constructing the Entra token endpoint and `{app_id}/.default` scope itself. Behavior changes on the kernel path (Thrift parity): - `azure_tenant_id` is now OPTIONAL — the kernel auto-discovers the tenant from the workspace's `/aad/auth` redirect when omitted, exactly as the Thrift backend does. (Previously the kernel path required it.) - `azure_workspace_resource_id` is now honored as an optional add-on: forward it and the kernel fetches an Azure-management token and sends the `X-Databricks-Azure-SP-Management-Token` + `X-Databricks-Azure-Workspace-Resource-Id` pair, so an SP with only an Azure RBAC role (not a workspace member) can authenticate. (Previously it was dropped with a warning.) Also thread the `azure_*` connection kwargs into `kernel_auth_options` in session.py — without this the bridge never received them and `azure-sp-m2m` failed at session-open with "requires azure_client_id". Adds a regression test for that threading, and rewrites the bridge tests for thin forwarding (tenant optional, resource id forwarded). Drops the now-unused `get_effective_azure_login_app_id` import and `_AZURE_AAD_LOGIN_HOST` constant. Co-authored-by: Isaac Signed-off-by: eric-wang-1990 <e.wang@databricks.com> * refactor(kernel): drop azure-sp-m2m management token on the kernel path Mirror the kernel-side simplification: the kernel's `azure-sp-m2m` no longer supports the Azure management-token flow (RBAC-only SPs), so stop forwarding `azure_workspace_resource_id` to the kernel. The SP must be a workspace member on `use_kernel=True`. `azure_workspace_resource_id` is still a valid connection parameter for the Thrift path (unchanged). On the kernel path, setting it now logs a warning (rather than silently dropping a security-relevant auth parameter and failing later with an opaque 403) and the SP authenticates with the Databricks-audience data token alone. Drops the threading through `kernel_auth_options` in session.py. RBAC-only-SP support is a documented follow-up. Co-authored-by: Isaac Signed-off-by: eric-wang-1990 <e.wang@databricks.com> * refactor(kernel): reject azure-oauth on the kernel path; point to databricks-oauth The kernel dropped its dedicated Azure U2M flow, so `auth_type="azure-oauth"` on `use_kernel=True` is now rejected with a clear pointer to `databricks-oauth` — whose in-house OAuth browser flow works against Azure workspaces (the workspace federates login to Entra). This replaces the thin forward-the-selector branch. `azure-sp-m2m` is unchanged; `azure-oauth` on the Thrift path is unaffected. Rejecting (rather than silently remapping) keeps the user's selected flow explicit. Co-authored-by: Isaac Signed-off-by: eric-wang-1990 <e.wang@databricks.com> * feat(kernel): forward azure_workspace_resource_id for complete Azure SP M2M The kernel's azure-sp-m2m now always sends the Azure SP management token and, when a workspace resource id is set, the X-Databricks-Azure-Workspace-Resource-Id header — matching the Thrift connector. So the bridge forwards azure_workspace_resource_id to the kernel (previously warned-and-ignored), and session.py threads it into kernel_auth_options. This completes Azure SP M2M parity: a service principal with an Azure RBAC role but no workspace membership now authenticates on the kernel path too. Co-authored-by: Isaac Signed-off-by: eric-wang-1990 <e.wang@databricks.com> * docs(kernel): azure-sp-m2m management token is gated on azure_workspace_resource_id Correct the wording (bridge docstring/comments, CHANGELOG, CONNECTION_PARAMETERS) to match the kernel: the Azure SP management token + X-Databricks-Azure-Workspace-Resource-Id header are sent only when azure_workspace_resource_id is set (JDBC parity), not always. A workspace-member SP authenticates with the data token alone; the management token is only needed for an RBAC-only SP (Azure role, not a workspace member), which is what the resource id signals. Co-authored-by: Isaac Signed-off-by: eric-wang-1990 <e.wang@databricks.com> * feat(kernel): route azure-oauth (Azure AD U2M) to the kernel OAuth U2M flow On use_kernel=True, auth_type='azure-oauth' was rejected with a pointer to databricks-oauth. Instead route it to the kernel's oauth-u2m, identically to databricks-oauth: the kernel runs the in-house workspace-federated browser flow, which Azure Databricks workspaces support (the workspace federates the login to Microsoft Entra). It forwards the connector's databricks-sql-python app bundle, NOT the Thrift Azure app (96eecda7 / redirect port 8030): that app is registered for Thrift's direct-Entra flow (login.microsoftonline.com config + {tenant}/user_impersonation scopes), which the kernel does not perform. So on the kernel path azure-oauth and databricks-oauth are the same flow. The U2M-vs-M2M ambiguity guard now also covers azure-oauth, so azure-oauth + oauth_client_secret is rejected as ambiguous rather than silently falling through to the M2M branch. Connector-only; the kernel already exposes oauth-u2m. (PECOBLR-4120.) Co-authored-by: Isaac Signed-off-by: eric-wang-1990 <e.wang@databricks.com> * ai: apply changes for #919 (2 review threads) Addresses: - #3811121476 at src/databricks/sql/backend/kernel/auth_bridge.py:158 - #3824120741 at tests/unit/test_kernel_auth_bridge.py:36 Signed-off-by: peco-engineer-bot[bot] <peco-engineer-bot[bot]@users.noreply.github.com> * ai: apply changes for #919 (1 review thread) Addresses: - #3828326882 at src/databricks/sql/backend/kernel/auth_bridge.py:200 Signed-off-by: peco-engineer-bot[bot] <peco-engineer-bot[bot]@users.noreply.github.com> * ai: apply changes for #919 (1 review thread) Addresses: - #3828367785 at src/databricks/sql/backend/kernel/auth_bridge.py:265 Signed-off-by: peco-engineer-bot[bot] <peco-engineer-bot[bot]@users.noreply.github.com> * ai: apply changes for #919 (1 review thread) Addresses: - #3828404026 at src/databricks/sql/backend/kernel/auth_bridge.py:279 Signed-off-by: peco-engineer-bot[bot] <peco-engineer-bot[bot]@users.noreply.github.com> * ai: apply changes for #919 (1 review thread) Addresses: - #3828445075 at src/databricks/sql/backend/kernel/auth_bridge.py:285 Signed-off-by: peco-engineer-bot[bot] <peco-engineer-bot[bot]@users.noreply.github.com> * ci(kernel): bump KERNEL_REV to the build carrying the pyo3 Azure SP surface Point kernel-e2e's KERNEL_REV at the kernel commit that adds the pyo3 azure-sp-m2m surface (databricks-sql-kernel#263), so the connector's kernel-e2e builds a kernel wheel that can accept auth_type='azure-sp-m2m'. Temporary pin to the unmerged #263 branch tip; re-point to a kernel main SHA once #263 merges and a release is cut. Customer-facing pin (pyproject databricks-sql-kernel ^0.2.0) still needs a release bump to a published kernel with #263. Co-authored-by: Isaac Signed-off-by: eric-wang-1990 <e.wang@databricks.com> * ai: apply changes for #919 (1 review thread) Addresses: - #3828603835 at src/databricks/sql/backend/kernel/auth_bridge.py:18 Signed-off-by: peco-engineer-bot[bot] <peco-engineer-bot[bot]@users.noreply.github.com> * fix(kernel): make auth_bridge.py pass black + mypy Two quality-check regressions from the azure-sp-m2m addition: - mypy [no-redef]: the new azure-sp-m2m branch's `kwargs` assignment now precedes the JWT branch's annotated `kwargs: Dict[str, Any]`. Move the annotation onto the first (azure-sp-m2m) assignment so the later ones are plain reassignments to the same annotated name. - black: split the over-long `ignored_signals` tuple. Verified locally: `black --check src` clean, `mypy src` reports no issues. Co-authored-by: Isaac Signed-off-by: eric-wang-1990 <e.wang@databricks.com> --------- Signed-off-by: eric-wang-1990 <e.wang@databricks.com> Signed-off-by: peco-engineer-bot[bot] <peco-engineer-bot[bot]@users.noreply.github.com> Co-authored-by: peco-engineer-bot[bot] <peco-engineer-bot[bot]@users.noreply.github.com>
1 parent e550cb7 commit c95ca3f

8 files changed

Lines changed: 386 additions & 83 deletions

File tree

CHANGELOG.md

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,8 @@
22

33
# Unreleased
44
- Kernel backend (`use_kernel=True`): OAuth **M2M with a JWT private-key client assertion** (RFC 7523) is now supported. Pass `oauth_client_id` + `oauth_jwt_key_file` + `oauth_jwt_kid` (with optional `oauth_jwt_passphrase` for an encrypted PKCS#8 key, `oauth_jwt_algorithm` defaulting to `RS256`, `oauth_scopes`, and `token_url` for the IdP token endpoint) and the connector routes them to the kernel's `auth_type="oauth-m2m-jwt"`, which signs a short-lived assertion with the private key instead of sending a client secret. The kernel owns the token lifecycle. A private-key file is treated as unambiguous JWT M2M intent and is mutually exclusive with `oauth_client_secret` / `credentials_provider` (both raise `NotSupportedError`). Verified end-to-end against an Azure Databricks workspace with the service principal's public certificate registered on its Entra ID app registration. Requires `databricks-sql-kernel >= 0.2.0` with JWT support.
5-
- Kernel backend (`use_kernel=True`): OAuth U2M with `auth_type="databricks-oauth"` now forwards the connector's `databricks-sql-python` OAuth-app bundle (`client_id` + `sql offline_access` scopes + redirect port) into the kernel, so a bare U2M connection authenticates as `databricks-sql-python` — parity with the Thrift path — instead of inheriting the kernel's own `databricks-sql-connector` default. A caller-supplied `oauth_client_id` (with its coupled `oauth_redirect_port`) is honored, as is a caller-supplied `oauth_scopes`; absent one, the connector default (`sql offline_access`) is forwarded. Note: the kernel binds a single U2M redirect port, so unlike the Thrift path (which tries the full `8020..8024` range) the kernel path uses only one port and does not fall back to the next port if it is already bound — pass `oauth_redirect_port` (with `oauth_client_id`) to pick a free one on a port collision. `auth_type="azure-oauth"` (Azure AD) is not yet supported on the kernel path and raises `NotSupportedError` — use the Thrift backend for it (PECOBLR-4040; Azure tracked by PECOBLR-4120)
5+
- Kernel backend (`use_kernel=True`): OAuth U2M with `auth_type="databricks-oauth"` now forwards the connector's `databricks-sql-python` OAuth-app bundle (`client_id` + `sql offline_access` scopes + redirect port) into the kernel, so a bare U2M connection authenticates as `databricks-sql-python` — parity with the Thrift path — instead of inheriting the kernel's own `databricks-sql-connector` default. A caller-supplied `oauth_client_id` (with its coupled `oauth_redirect_port`) is honored, as is a caller-supplied `oauth_scopes`; absent one, the connector default (`sql offline_access`) is forwarded. Note: the kernel binds a single U2M redirect port, so unlike the Thrift path (which tries the full `8020..8024` range) the kernel path uses only one port and does not fall back to the next port if it is already bound — pass `oauth_redirect_port` (with `oauth_client_id`) to pick a free one on a port collision (PECOBLR-4040)
6+
- Kernel backend (`use_kernel=True`): **Azure Entra (Azure AD) service-principal M2M is now supported.** `auth_type="azure-sp-m2m"` forwards `azure_client_id` / `azure_client_secret`; the kernel is the Azure-aware auth core — it builds the Entra v2.0 token endpoint and the `{app_id}/.default` scope, and **auto-discovers the tenant** from the workspace's `/aad/auth` redirect when `azure_tenant_id` is omitted (matching Thrift). The `Authorization` bearer is the Databricks-audience data token, which alone authenticates a workspace-member SP. Set `azure_workspace_resource_id` and the kernel also sends the Azure SP management token (`X-Databricks-Azure-SP-Management-Token`) + `X-Databricks-Azure-Workspace-Resource-Id` header (matching the JDBC driver), so a service principal with an Azure RBAC role but no workspace membership can authenticate; omit it and no ARM management-scope token is fetched. Azure AD **U2M** (`auth_type="azure-oauth"`) now routes to the kernel's OAuth U2M flow, identically to `auth_type="databricks-oauth"`: the kernel runs the in-house workspace-federated browser flow, which Azure workspaces support (the workspace federates login to Entra). It forwards the connector's `databricks-sql-python` OAuth app, not the Thrift Azure app (`96eecda7` / port 8030), which is registered for Thrift's direct-Entra flow the kernel does not perform (PECOBLR-4141; PECOBLR-4120)
67

78
# 4.4.0 (2026-07-22)
89
- Raised the minimum supported Python version to 3.10, dropping the end-of-life 3.8/3.9, to update the lockfile and clear CVE-flagged dependencies in the repo (databricks/databricks-sql-python#798)

CONNECTION_PARAMETERS.md

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -69,7 +69,7 @@ to change without notice.
6969
| Option | Type | Thrift | Kernel | Default Value | Note |
7070
| --------------------------------------------------- | -------------------- | :----: | :----: | ------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------- |
7171
| `access_token` (PAT) | `str` ||| `None` | Personal Access Token / bearer token. The default auth mode when set; otherwise auth falls back to OAuth. |
72-
| `auth_type` | `str` ||| `None` ⇒ Databricks OAuth | `databricks-oauth` or `azure-oauth`. |
72+
| `auth_type` | `str` ||| `None` ⇒ Databricks OAuth | `databricks-oauth` (U2M), `azure-oauth` (Azure AD U2M), or `azure-sp-m2m` (Azure service-principal M2M). All three are supported on the kernel path (#919): `azure-oauth` routes to the kernel's OAuth U2M flow identically to `databricks-oauth` (the in-house workspace-federated browser flow, which Azure workspaces support), so it uses the `databricks-sql-python` app rather than Thrift's direct-Entra Azure app. All three work on the Thrift path. |
7373
| `oauth_client_id` (U2M) | `str` ||| built-in client id | Custom U2M client id. Forwarded on both; when absent, each path applies its own built-in default. |
7474
| `oauth_redirect_port` (U2M) | `int` ||| `None` | Localhost redirect port for the browser flow. On **both** backends it is only honored when a custom `oauth_client_id` is also supplied — then that single port becomes the redirect URI. With the built-in client id (or when omitted) the connector uses the full registered range 8020–8024 and binds the first free port, so a bare `oauth_redirect_port` has no effect. (Thrift: `auth.py` `oauth_redirect_port_range`; Kernel: same logic, forwarded as `redirect_ports`.) |
7575
| `oauth_client_secret` (OAuth M2M) | `str` ||| `None` | **Kernel-only in practice.** The Thrift auth path never reads `oauth_client_secret`; use `credentials_provider` or an Azure service principal for M2M on Thrift. |
@@ -82,7 +82,8 @@ to change without notice.
8282
| `credentials_provider` | `CredentialsProvider`||| `None` | Custom external credentials provider. **Rejected on the kernel path** (`NotSupportedError`) — it is an opaque token source, so the kernel cannot own the token lifecycle; use `oauth_client_id` + `oauth_client_secret` for M2M, or the Thrift backend. |
8383
| `identity_federation_client_id` | `str` ||| `None` | Workload identity / token-federation client id (kernel support added in #910). |
8484
| `experimental_oauth_persistence` | `OAuthPersistence` ||| `None` | **Thrift-only.** The kernel owns its own token lifecycle and does not accept a persistence store. |
85-
| `azure_client_id` / `azure_client_secret` / `azure_tenant_id` / `azure_workspace_resource_id` | `str` ||| `None` | **Thrift-only.** The Azure service-principal (Entra ID M2M) fields are not forwarded to the kernel. (Azure *U2M* still works on the kernel via `auth_type="azure-oauth"`, the browser flow.) |
85+
| `azure_client_id` / `azure_client_secret` / `azure_tenant_id` | `str` ||| `None` | Azure service-principal (Entra ID M2M), selected by `auth_type="azure-sp-m2m"`. On the kernel path the connector forwards these to the kernel, which owns Azure resolution (Entra v2.0 token endpoint + the Databricks-resource `.default` scope) (#919). **`azure_tenant_id` is optional on the kernel path too** — like Thrift, the kernel auto-discovers it from the workspace's `/aad/auth` redirect when omitted. |
86+
| `azure_workspace_resource_id` | `str` ||| `None` | For `azure-sp-m2m`. When set, the SP **management token** (`X-Databricks-Azure-SP-Management-Token`) + `X-Databricks-Azure-Workspace-Resource-Id` header are sent, to authorize an SP that has an Azure RBAC role but is not a workspace member. Omit it for a workspace-member SP (the data token authenticates alone; no management token is fetched). Works on both the kernel and Thrift paths. |
8687
| `_use_cert_as_auth` (+ `_tls_client_cert_file`) | `bool` ||| `False` | Authenticate with a TLS client certificate instead of a token. Thrift-only. |
8788
| `username` / `password` | `str` ||| `None` | **Removed.** Basic auth is no longer supported; passing either raises `ValueError`. |
8889

KERNEL_REV

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1 +1 @@
1-
9e3dbf9c40733b176151e001c9a15202030b967a
1+
ad78a5be3dc8bb7fc78ec574492515ab24e23d4c

0 commit comments

Comments
 (0)