Repository navigation
[C++] Allow filesystems to be implemented in separate library (+ move remote filesystems out of libarrow into their own shared libraries) #49399
Description
Describe the enhancement requested
This is a follow up of the original issue with the same title which shouldn't have been closed originally:
This issue should be an issue covering different sub-issues to track the overall goal. Please do not close this issue for interim steps and keep it to cover the full feature.
Currently all our supported file systems in C++ (local, hdfs, s3, google cloud (gs/gcs), azure (abfs)) are incorporated in the core libarrow library.
There is a desire to be able to separate those filesystems in their own libraries, such that they can be installed separately. The remote filesystems each come with their own (potentially quite large) dependencies, and one typically doesn't need all of them at the same time.
There has been work towards achieving this goal in the past:
- Creating a FileSystem Registry and factory where users can add their own filesystems (GH-38309: [C++] build filesystems as separate modules #39067)
- Moving the Local Filesystem to the registry: [C++] Move LocalFileSystem to a separate module #40342
- Allowing to move s3 to the registry as it's own module (even though this does not contain the same functionality as s3 built as part of libarrow): [C++] Move S3FileSystem to a separate module #40343
In order to move all the functionality for S3 to be part of it's own module see the following issue and discussion
This is also in-line with some work that we've done in order to break down libarrow.so into smaller submodules like:
Related issue on conda feedstocks that spawned the creation of this one:
Component(s)
C++
Activity