Skip to content

docs: document how to register IPC compression codecs - #515

Open
ump45nose wants to merge 1 commit into
apache:mainfrom
ump45nose:docs/298-ipc-compression-codecs
Open

ump45nose wants to merge 1 commit into
apache:mainfrom
ump45nose:docs/298-ipc-compression-codecs

Conversation

@ump45nose

Copy link
Copy Markdown

What's Changed

Arrow JS can read and write compressed IPC record batches, but nothing outside the tests explained that you have to register codecs yourself, so this adds TSDoc covering it:

  • Codec and compressionRegistry: no codecs are bundled or registered by default; a codec needs decode to read and encode to write; encode has to produce an LZ4 frame or Zstandard frame and is checked against the magic number when registered; codecs are process-wide; includes a short LZ4 example.
  • tableFromIPC: compressed data is decompressed only if a codec was registered first, otherwise it throws.
  • tableToIPC and the writer compressionType option: a codec with encode must be registered, buffers that don't shrink are written uncompressed, and the legacy IPC format isn't supported.

Docs only, no behavior change. eslint src/ipc passes and typedoc reports the same warning count as on main. I also checked the example and the error cases with a throwaway jest test (writing without a codec throws, reading without a decode throws "Record batch is compressed but codec not found", and the round trip works once both are registered).

This PR was prepared with AI assistance and reviewed by me.

Closes #298.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Documentation for how to use IPC RecordBatch compression

1 participant