Skip to content

feat: request output encodings when scanning - #9494

Draft
Xuanwo wants to merge 3 commits into
xuanwo/semantic-type-writefrom
xuanwo/semantic-type-read
Draft

Xuanwo wants to merge 3 commits into
xuanwo/semantic-type-writefrom
xuanwo/semantic-type-read

Conversation

@Xuanwo

@Xuanwo Xuanwo commented Sep 23, 2026

Copy link
Copy Markdown
Member

Do not merge before #9485, the format specification this implements, is voted and merged. Stacked on #9488.

Under the semantic type contract of #7073, a read picks each field's Arrow layout in this order: a request override, then the field's lance-schema:output-encoding entry, then the type's default. #9488 implements the last two. This adds the override, so a caller can read a string column as Utf8View or a decimal:10:2 column as Decimal256 without changing the table.

Scanner::output_encodings takes field paths and output encoding names, such as {"name": "utf8_view", "price": "decimal256", "doc": "lance.json"}, and applies them ahead of the recorded entries. The layouts are produced by the decoders, like the recorded ones. It rejects a path that names no field, an encoding that is not one of the field type's output encodings, and any use on a legacy table, which names one Arrow type per column. The Python scanner and to_table take output_encodings= as a dict, and Java takes ScanOptions.Builder.outputEncodings(Map). Validation stays in the Rust core.

JSON columns now honor their output encoding too. A column recorded or requested as lance.json reads as JSONB; the default arrow.json still converts to text. The choice rides on the returned Arrow field as the same metadata entry, so nested JSON fields follow it as well.

The override lives on Projection, next to BlobHandling, so every read that uses the scan's projection applies it. Dataset::take does not take an override yet.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

A-java Java bindings + JNI A-python Python bindings enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant