Skip to content

fix: Write one zero offset for empty variable-size arrays - #978

Merged
paleolimbot merged 1 commit into
apache:mainfrom
rustyconover:fix/ipc-encoder-empty-offsets
Oct 20, 2026
Merged

paleolimbot merged 1 commit into
apache:mainfrom
rustyconover:fix/ipc-encoder-empty-offsets

Conversation

@rustyconover

Copy link
Copy Markdown
Contributor

An offsets buffer holds length + 1 offsets, so even an empty string, binary, list or map array has one: offsets[0] = 0. The array view of an empty array has a zero-length offsets buffer, though (ArrowArrayViewSetArray() deliberately doesn't require one), and ArrowIpcEncoderCollectArray() wrote that buffer as is. So every empty variable-size array nanoarrow writes has a 0-byte offsets buffer.

IPC permits that, and Arrow C++ reads it, but not every reader does. With nanoarrow 0.9.0 (Python) writing struct<s: string, l: list<int32>> with zero rows, arrow-go v18.8.0 reads the batch and then panics on String.ValueOffsets():

panic: runtime error: slice bounds out of range [:1] with capacity 0

The same batch written by pyarrow 25.0.1 has a 4-byte offsets buffer (8 bytes for large_string), and arrow-go reads it. Apache.Arrow for .NET had the same problem (apache/arrow-dotnet#462, now fixed there, but released versions still throw).

The encoder now writes one zero offset (4 or 8 bytes) in place of a zero-length offsets buffer of an empty array, as Arrow C++ does. Non-empty arrays and other buffer types are unchanged.

Tests:

  • NanoarrowIpcEncoderWritesOffsetForEmptyArrays encodes an empty struct<s: string, b: binary, ls: large_string, l: list<int32>> built the usual way. It decodes the message and checks that each child's offsets buffer holds one zero offset of the right width. On main the test fails: the buffers are 0 bytes.
  • AssertArrayViewIdentical(), used by the decoder round-trip tests, compared buffer sizes exactly. It now expects that one zero offset where the original empty array's view has a zero-length offsets buffer.

ctest passes (Linux aarch64, Debug, 868 tests with NANOARROW_BUILD_TESTS_WITH_ARROW), and 393 tests under ASan/UBSan with clang. The changed files are formatted with clang-format 16.0.6, the pre-commit version.

This PR's code and description were written by AI (Claude Code) and reviewed by a human maintainer.

🤖 Generated with Claude Code

The array view of an empty string, binary, list or map array has a zero-length
offsets buffer, and the encoder wrote that buffer as is. IPC permits it, but
arrow-go and Apache.Arrow for .NET read the first offset unconditionally.
Arrow C++ writes one zero offset; the encoder now does too.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@codecov-commenter

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 80.00000% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 78.30%. Comparing base (10885c5) to head (f23b5c8).

Files with missing lines Patch % Lines
src/nanoarrow/ipc/encoder.c 80.00% 0 Missing and 1 partial ⚠️
Additional details and impacted files
@@           Coverage Diff           @@
##             main     #978   +/-   ##
=======================================
  Coverage   78.30%   78.30%           
=======================================
  Files         106      106           
  Lines       16919    16924    +5     
  Branches     2016     2018    +2     
=======================================
+ Hits        13248    13252    +4     
  Misses       2429     2429           
- Partials     1242     1243    +1     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@paleolimbot paleolimbot left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you!

@paleolimbot paleolimbot changed the title fix(ipc): Write one zero offset for empty variable-size arrays fix: Write one zero offset for empty variable-size arrays Oct 20, 2026
@paleolimbot
paleolimbot merged commit f751f8c into apache:main Oct 20, 2026
43 checks passed
@rustyconover
rustyconover deleted the fix/ipc-encoder-empty-offsets branch October 20, 2026 03:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants