feat: support reading dictionary-encoded columns over Arrow Flight

[owencorrigan]

What's Changed

The C# Arrow Flight reader (RecordBatchReaderImplementation) threw
NotImplementedException on any dictionary batch, so a Flight stream containing a
dictionary-encoded column could not be read.

• ReadNextRecordBatchAsync now loops over the incoming Flight messages, feeding
  each through the base CreateArrowObjectFromMessage (which reads dictionary
  batches into the DictionaryMemo and returns null for them) until it yields a
  record batch, instead of throwing on any non-record-batch message. Handles both
  single and replacement dictionaries.
• The schema is decoded using the reader's DictionaryMemo via a new base helper
  ReadSchemaFromMessage, so dictionary-encoded fields are registered
  (FlightMessageSerializer previously discarded them).

The change routes dictionary batches to the existing ArrowReaderImplementation
decoder, so it is not specific to any particular index or value type. Validated
against a pyarrow RecordBatchStream serving a dictionary<int16, string> column
(single + replacement dictionaries); the current reader throws on the same stream.

Refs #180 (receive side).

Generated-By: Claude Opus 4.8

[Copilot]

🟡 Changes recommended

Preserve cancellation in the Flight read loop and add dictionary-stream regression coverage.

Get a fresh assessment by requesting another Copilot review.

Pull request overview

This PR adds Arrow Flight support for reading dictionary-encoded columns, including replacement dictionaries.

Changes:

• Decode schemas using the reader’s DictionaryMemo.
• Process dictionary messages before record batches.

File summaries

File	Summary
src/Apache.Arrow/Ipc/ArrowReaderImplementation.cs	Adds dictionary-aware schema decoding.
src/Apache.Arrow.Flight/Internal/RecordBatchReaderImplementation.cs	Processes dictionary and record-batch messages; needs cancellation propagation and Flight regression tests.

Review details

• Files reviewed: 2/2 changed files
• Comments generated: 2
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[CurtHagenlocher]

Thanks! This would definitely benefit from some tests.

[owencorrigan]

I've added support for cancellation. I have also added support for sending dictionary encoded columns over flight so that we could do a round trip test
Let me know if this is OK

[CurtHagenlocher]

A round-trip test sounds like a good idea. Can you also please rebase and then resolve the conflicts?

[Copilot]

🟡 Changes recommended

A critical issue remains in dictionary emission for later Flight batches.

Get a fresh assessment by requesting another Copilot review.

Review details

Suppressed comments (1)

test/Apache.Arrow.Flight.Tests/FlightTests.cs:94

• This regression test only sends one record batch, so it covers a single dictionary message but not the new loop's multi-message/replacement behavior described by the PR. Add a Flight-level fixture with an initial dictionary and a subsequent replacement (and assert both batches), otherwise a regression in dictionary replacement handling can still pass the test suite.

        [Fact]
        public async Task TestGetRecordBatchWithDictionaryColumn()
        {
            var flightDescriptor = FlightDescriptor.CreatePathDescriptor("test");
            var expectedBatch = CreateTestBatchWithDictionaryColumn(0, 100);
            GivenStoreBatches(flightDescriptor, new RecordBatchWithMetadata(expectedBatch));

• Files reviewed: 4/4 changed files
• Comments generated: 1
• Review effort level: Lite