Skip to content

feat: parse logical types into semantic types and output encodings - #9486

Draft
Xuanwo wants to merge 1 commit into
mainfrom
xuanwo/semantic-type-model
Draft

Xuanwo wants to merge 1 commit into
mainfrom
xuanwo/semantic-type-model

Conversation

@Xuanwo

@Xuanwo Xuanwo commented Sep 23, 2026

Copy link
Copy Markdown
Member

Part of implementing #7073 (spec in #9485). This adds the internal model the semantic type contract needs and routes schema compatibility checks through one entry point, without changing the behavior of any existing table.

LogicalType::semantic() reads a logical_type string as a semantic type (string, binary, list, decimal:<p>:<s>, json, blob, or an unchanged type) plus the output encoding a legacy alias implies: large_string is string with large_utf8, dict:string:int16:false is string with dictionary:int16:utf8, and decimal:256:10:2 is decimal:10:2 with decimal256. OutputEncoding parses and prints the values of the lance-schema:output-encoding field entry and checks them against a semantic type. Schema::to_canonical_types() produces the form a table under the contract records, keeping field IDs, nullability, metadata, and children; nothing calls it yet. The model stays internal to the Rust core; there is no plugin trait.

Compatibility checks now decide field types through Field::type_matches with a TypeComparison. SchemaCompareOptions carries it, and every check against a dataset takes it from Dataset::type_comparison(), as does the Merge commit check for field bindings. Every table currently compares exactly, so Field::data_type(), compatibility results, and error messages stay the same. The follow-up derives TypeComparison::Semantic from FLAG_SEMANTIC_TYPES.

The only parsing change is that the canonical decimal:<p>:<s> now parses, to Decimal128 when p ≤ 38 and Decimal256 otherwise. Nothing writes it yet.

Checks that compare Arrow DataTypes directly (project_by_field, do_intersection, Field::merge, the partial-schema check in merge_insert, index copying and index segment merges) are left unchanged. They compare physical layouts on both sides, and the write path in the follow-up decides how they apply to tables with mixed layouts.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant