Skip to content

feat: keep extension semantic types and describe views in schema mappings - #9496

Draft
Xuanwo wants to merge 2 commits into
xuanwo/semantic-type-upgradefrom
xuanwo/semantic-type-extensions
Draft

Xuanwo wants to merge 2 commits into
xuanwo/semantic-type-upgradefrom
xuanwo/semantic-type-extensions

Conversation

@Xuanwo

@Xuanwo Xuanwo commented Sep 23, 2026 •

Copy link
Copy Markdown
Member

Stacked on #9495; like the rest of the stack it should wait for #9485.

#7073 lets applications define their own semantic types as Arrow extension types over a core type. Readers that do not know an extension return the storage type, and implementations must keep the extension name and metadata through every rewrite. This checks that promise and brings the two type mappings that serialize schemas in line with tables whose output layouts include views.

An extension Lance does not know keeps its name and metadata through appends in another layout, add_columns, drop_columns, alter_columns, and compaction in both modes. This holds on new 2.3 tables, on migrated tables, and on legacy tables. No code needed to change; the test pins it.

  • The namespace JSON schema describes a view by its values, like a dictionary. Utf8View maps to utf8 and BinaryView to binary, so describing a table read as views no longer fails.
  • lance::arrow::json serializes views as string_view and binary_view and reads the canonical decimal:<p>:<s>.
  • The data types guide explains semantic types, output encodings, scan overrides, migration, and extension types for users.

The discussion's open decision 2, a Lance-owned lance-schema:extension-name key for layering extensions on Blob v2 or json, is not implemented, because it is still open.

@github-actions github-actions Bot added A-namespace Namespace impls A-docs Documentation enhancement New feature or request labels Sep 23, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

A-docs Documentation A-namespace Namespace impls enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant