Add custom_metadata to IPC DictionaryEncoding

[rustyconover]

Which issue does this PR close?

• Closes Support DictionaryEncoding.custom_metadata (apache/arrow#49704) #11621.
• Related to Field metadata for dictionary-encoded extension types is lost #7982.

Rationale for this change

In Arrow IPC, a dictionary whose values are an extension type and an extension type whose storage is a dictionary serialize identically, because Field.custom_metadata is the only metadata slot. apache/arrow#49704 adds an optional custom_metadata: [KeyValue] member to DictionaryEncoding holding metadata for the dictionary value type. That change has lazy consensus on dev@arrow, targets format version 1.6, and is drafted in apache/arrow#53994, with a Go implementation in apache/arrow-go#1381 and a nanoarrow implementation in apache/arrow-nanoarrow#972. arrow-rs needs to read schemas that use it.

This PR is a draft until the format change is voted on in apache/arrow.

What changes are included in this PR?

• Schema: format/Schema.fbs gains DictionaryEncoding.custom_metadata and the version 1.6 history line. arrow-ipc/src/gen/Schema.rs is regenerated; only the DictionaryEncoding additions change.
• IPC readers: DataType::Dictionary cannot carry metadata for its value type (Field metadata for dictionary-encoded extension types is lost #7982). Until it can, the reader copies the value type metadata into the Field's metadata when the field has no ARROW:extension:name of its own; keys already in the field metadata win. Otherwise the value metadata is dropped. This keeps today's behaviour for arrow-rs users, who see e.g. arrow.uuid on dictionary fields, and that metadata survives read-then-write. The mapping is documented in the arrow_ipc::reader module docs and on the schema conversion functions.
• Integration JSON: field_from_json validates the optional "dictionary"."metadata" attribute and maps it the same way, so the JSON and IPC readings of an integration file agree.
• C Data Interface import (arrow-schema): Field::try_from(&FFI_ArrowSchema) maps the metadata of an imported dictionary schema's dictionary (its value type) the same way, so the IPC, integration JSON and C Data readers agree. The C Data export is unchanged.
• IPC writer: unchanged. It never writes the new member, and dictionary field metadata is written to Field.custom_metadata, the pre-1.6 position.

Are these changes tested?

Yes. Unit tests build schema flatbuffers containing the new member with the generated builders. They cover:

• copying the value metadata to the field;
• field keys taking precedence;
• dropping the value metadata when the field has its own extension type;
• an empty but present vector;
• nesting in list and struct;
• rejection of non-UTF-8 strings;
• reading through StreamReader and through the FileReader footer.

Integration JSON tests cover the same mapping and the validation errors, and C Data import tests cover the mapping, including nesting in list and struct. With the C++ and Go drafts, archery reads their dictionary_extension output correctly in arrow-rs, over IPC and C Data.

Are there any significant user-facing changes?

• Copied metadata: value type metadata is copied into arrow-rs Field metadata as described above. This applies to the arrow-ipc readers and to the arrow-schema C Data import (Field::try_from(&FFI_ArrowSchema)), which previously dropped the metadata of an imported schema's dictionary. Malformed metadata on that dictionary is now an import error.
• Output position: files and schemas written or exported by arrow-rs keep dictionary field metadata on the field itself, the pre-1.6 position. Readers implementing format 1.6 read it as metadata of the field's own type, so they do not read a dictionary of an extension type written by arrow-rs back as one.
• UTF-8 check: the flatbuffers verifier now also checks that the strings in DictionaryEncoding.custom_metadata are valid UTF-8. A schema with non-UTF-8 bytes there, which Arrow C++ accepts, is now rejected; previously the member was ignored as an unknown field.
• Breaking API change: arrow_ipc::DictionaryEncodingArgs gains a public field, custom_metadata. Code that builds it with a struct literal must add the field or use ..Default::default(). This needs the api-change label.

The code and this description were written by AI (Claude Code). The generated flatbuffers code was verified byte-identical against flatc 25.2.10 output, and the change was reviewed by a human maintainer and an AI reviewer.

🤖 Generated with Claude Code