Conversation
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Do not merge before #9485, the format specification this implements, is voted and merged. Stacked on #9488.
Under the semantic type contract of #7073, a read picks each field's Arrow layout in this order: a request override, then the field's
lance-schema:output-encodingentry, then the type's default. #9488 implements the last two. This adds the override, so a caller can read astringcolumn asUtf8Viewor adecimal:10:2column asDecimal256without changing the table.Scanner::output_encodingstakes field paths and output encoding names, such as{"name": "utf8_view", "price": "decimal256", "doc": "lance.json"}, and applies them ahead of the recorded entries. The layouts are produced by the decoders, like the recorded ones. It rejects a path that names no field, an encoding that is not one of the field type's output encodings, and any use on a legacy table, which names one Arrow type per column. The Python scanner andto_tabletakeoutput_encodings=as a dict, and Java takesScanOptions.Builder.outputEncodings(Map). Validation stays in the Rust core.JSON columns now honor their output encoding too. A column recorded or requested as
lance.jsonreads as JSONB; the defaultarrow.jsonstill converts to text. The choice rides on the returned Arrow field as the same metadata entry, so nested JSON fields follow it as well.The override lives on
Projection, next toBlobHandling, so every read that uses the scan's projection applies it.Dataset::takedoes not take an override yet.