Skip to content

feat: migrate existing tables to the semantic type contract - #9495

Draft
Xuanwo wants to merge 2 commits into
xuanwo/semantic-type-readfrom
xuanwo/semantic-type-upgrade
Draft

Xuanwo wants to merge 2 commits into
xuanwo/semantic-type-readfrom
xuanwo/semantic-type-upgrade

Conversation

@Xuanwo

@Xuanwo Xuanwo commented Sep 23, 2026 •

Copy link
Copy Markdown
Member

Do not merge before #9485, the format specification this implements, is voted and merged. Stacked on #9494.

Tables created before the semantic type contract of #7073, and every table on data storage 2.2 or earlier, stay legacy tables: they reject appends in another layout and cannot use output encodings. This adds the explicit upgrade the design calls for.

Dataset::migrate_to_semantic_types() commits one metadata-only version. It rewrites each legacy alias to its canonical type with the output encoding that keeps the current read type (large_string becomes string with large_utf8, decimal:256:10:2 becomes decimal:10:2 with decimal256), and sets FLAG_SEMANTIC_TYPES. No data file is rewritten. Reads return the same Arrow types, values, and field IDs, and appends then accept any layout of a column's type. It is idempotent, there is no downgrade, and restoring an earlier version keeps the flag. Python exposes LanceDataset.migrate_to_semantic_types() and Java Dataset.migrateToSemanticTypes().

The migration refuses, and commits nothing, in three cases:

  • The table references data files of version 2.0 or earlier. Flagged tables need 2.1 or later.
  • A column has no semantic type, meaning a dictionary whose values are not strings or bytes. The admission rule gives such a column no canonical form, so it cannot move without changing its read type.
  • A field already carries a lance-schema:output-encoding entry. Legacy tables ignore it, so adopting it would change what reads return.

The commit follows migrate_to_stable_row_ids: it projects the current schema with a commit option that sets the flag, and it does not retry past a concurrent write. This keeps the transaction format unchanged.

@github-actions github-actions Bot added the enhancement New feature or request label Sep 23, 2026
@github-actions github-actions Bot added A-python Python bindings A-java Java bindings + JNI labels Sep 23, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

A-java Java bindings + JNI A-python Python bindings enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant