Skip to content

fix(sqlalchemy-spanner): reflect vector columns and unrecognized column types - #18844

Open
rocksnow1942 wants to merge 2 commits into
googleapis:mainfrom
rocksnow1942:fix-18839-unknown-column-types
Open

rocksnow1942 wants to merge 2 commits into
googleapis:mainfrom
rocksnow1942:fix-18839-unknown-column-types

Conversation

@rocksnow1942

Copy link
Copy Markdown

_designate_type() raised KeyError for column types it doesn't map. Since get_multi_columns() reads all the tables of a schema in one query, one such column breaks the reflection of the whole schema. These column types fail on the emulator (1.5.58):

Column SPANNER_TYPE Error
vector ARRAY<FLOAT32>(vector_length=>3) KeyError: 'FLOAT32>(vector_length=>3'
proto `my.package.Message` KeyError: '`my.package.Message`'
enum `my.package.Status` KeyError: '`my.package.Status`'

(The issue also mentions INTERVAL and STRUCT, but neither can be a column type.)

This PR:

  • reads the element type of an array up to the first >, so vector columns are reflected as ARRAY(REAL) or ARRAY(Float);
  • reflects types the dialect doesn't recognize as NullType with an SAWarning, as SQLAlchemy's built-in dialects do. Arrays of such types are reflected as ARRAY(NullType()).

The NullType fallback was first proposed by @waiho-gumloop in #16622, which was moved to draft after #17911 fixed its TOKENLIST part. This PR supersedes it and adds the vector, proto and enum cases.

Tested with the unit and mockserver tests and the compliance suite against the emulator.

  • Make sure to open an issue as a bug/issue before writing your code! That way we can discuss the change, evaluate designs, and agree on the general idea
  • Ensure the tests and linter pass
  • Code coverage does not decrease (if any source code was changed)
  • Appropriate docs were updated (if necessary)

Fixes #18839

@google-cla

google-cla Bot commented Oct 24, 2026

Copy link
Copy Markdown

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

…mn types

_designate_type() raised KeyError for vector columns, whose type ends with
the vector length (ARRAY<FLOAT32>(vector_length=>128)), and for types it
doesn't map, like PROTO and ENUM columns, which broke the reflection of the
whole schema. Parse the element type of vector columns, and reflect
unrecognized types as NullType with a warning.

Fixes googleapis#18839
@rocksnow1942
rocksnow1942 force-pushed the fix-18839-unknown-column-types branch from 06d25f6 to 21375ae Compare October 24, 2026 08:44

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request enhances the Spanner dialect's type designation to support vector columns and gracefully handle unrecognized types like PROTO and ENUM by reflecting them as NullType with a warning. Unit and integration tests have been added to verify these changes. Feedback on the changes highlights a potential robustness issue where using str_repr.index('>') could raise a ValueError and crash schema reflection if the character is missing; using find with a fallback is recommended.

… NullType

str_repr.index(">") raised ValueError for them, which would break the
reflection of the whole schema.
@rocksnow1942
rocksnow1942 marked this pull request as ready for review October 24, 2026 09:53
@rocksnow1942
rocksnow1942 requested a review from a team as a code owner October 24, 2026 09:53

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[sqlalchemy-spanner]: _designate_type() raises KeyError on unrecognized Spanner column types instead of falling back to NullType

1 participant