feat(format): specify semantic types for table schemas

[Xuanwo]

This specifies Revision 2 of #7073: in a table that sets the new FLAG_SEMANTIC_TYPES, Field.logical_type names a semantic type instead of an Arrow layout.

Today each logical_type string maps to exactly one Arrow type, so it mixes semantics (string, decimal precision and scale) with layout (large_string, dict:string:int16:false, decimal:256:10:2). Schema compatibility therefore compares layouts: appending LargeUtf8 data to a string column fails even though every value fits, and every new Arrow layout, such as the view types, would need a new logical type that non-Arrow engines have to learn.

With the flag set:

• A semantic type is distinct exactly when it changes the value domain, the precision, or the computation semantics. string, binary, list, and decimal:<p>:<s> are representation-only: writers encode any listed physical layout unchanged, and data files of one column may differ. json and blob are the only value-transforming types. All other types keep their current strings.
• The field metadata entry lance-schema:output-encoding names the Arrow layout that reads return. A caller's override comes first, then the entry, then the type's default. Writers record the input layout when a column is created; appends never change it. The entry is advisory: an invalid value is rejected when set, and an unrecognized value falls back to the default on read.
• Schema compatibility compares semantic types and parameters. Decimal256(10, 2) appends to decimal:10:2; Decimal128(12, 2) does not.
• Data file schemas keep the existing Arrow-mapped strings and record the exact layout each file encodes. Encodings, footers, pages, and LanceFileVersion are unchanged; the table schema remains the only source of semantics.
• Applications can define extension semantic types over a core type. Readers that do not know an extension read its storage type; the lance. prefix is reserved.
• Legacy aliases map to a canonical type plus an implied output encoding, which is what a later metadata-only upgrade commit will rewrite.

FLAG_SEMANTIC_TYPES takes bit 2048 in both reader and writer flags, the next free bit on main. Several open format PRs also propose 2048; whichever lands later renumbers. The flag is set only on new tables with data storage version 2.3 or later, never implicitly on an existing table, and it stays set across later versions and restores. Because 2.3 data files already require new readers, enabling it by default there adds no new compatibility break, while older clients see one clear "unsupported" error instead of misreading types. Legacy tables keep their exact behavior.

The implementation stack (#9486, #9488, #9494–#9497) pinned down three rules the discussion leaves open, and the spec states them. A table under the contract references only data files of version 2.1 or later, whose pages describe their own layout, so readers decode each file straight into the output layout. The upgrade commit applies only when every field has a semantic type and no field already carries an output encoding entry. Writers keep an output encoding entry they do not recognize.

Two points go slightly beyond the discussion text. For a legacy alias that also carries an explicit lance-schema:output-encoding entry, the entry wins; writers never produce that combination, but readers need a defined order. The legacy alias table limits dict:* to string and binary values, because dictionary encoding of other types has no canonical form under the admission rule.

The protobuf messages do not change: the contract uses the existing logical_type string and field metadata.

[github-actions]

Important

Format specification vote

This PR modifies the Lance format specification, so it requires 3 binding +1 votes from PMC members (excluding the proposer), at least one of them on the latest commit, and a minimum 72-hour voting period, weekends excluded, before it can merge. Vote by approving this PR (+1) or requesting changes (−1, a veto). See the voting process.

Approvals carry over across pushes, so a rebase or a typo fix does not send everyone back to re-vote. Whoever approves the latest commit is vouching that nothing substantive has changed since the earlier approvals; if something has, ask for fresh votes.

Status: ❌ Blocked — 0 of 3 required approvals

Approvals	none (0/3)
Latest commit approved by	none — one PMC member must approve the latest commit
Vetoes	none
Voting period	ends Mon 2026-09-28 14:48 UTC (07:48 PDT)

_{Updated automatically by the format-spec vote gate, which re-checks every 15 minutes — just voted? Re-check now (press Run workflow; leave the input blank to re-check every open format PR). A PMC member may apply the format-waived label to waive the vote for a trivial edit (typo, wording, formatting).}

[lance-gatekeeper]

❌ Gate recommendation: request changes.

The semantic type split is worthwhile, but the format contract needs to define extension identity at the schema compatibility boundary. Require append, merge, and update to preserve an extension's meaning across equal storage types; this closes a silent reinterpretation path while retaining the representation-only conversions.

[lance-gatekeeper]

Please make extension identity part of this compatibility rule. ARROW:extension:name and ARROW:extension:metadata are outside logical_type, and the current schema comparison ignores metadata by default. A stored example.bbox and incoming example.rgba can both have fixed_size_list:float:4; comparing only the core type would append color values that readers later interpret as coordinates, without a read error or reliable recovery. Require exact extension name and metadata equality, or an explicit checked conversion, for append, merge, and update. Output-encoding hints can remain excluded.