GH-50194: [C++] Move S3 and AWS-SDK to its own libarrow_s3.so by raulcd · Pull Request #50195 · apache/arrow

raulcd · 2026-06-16T15:10:27Z

Rationale for this change

Trying to reduce the size of libarrow.so and remove AWS SDK on some builds. Allow for users to plug and play based on requirements and divide our functionality into cleaner modules.

What changes are included in this PR?

Unconditionally build S3 and the AWS SDK into a different module libarrow_s3.so outside of libarrow.so.
Update bindings to link against the new libarrow_s3.so library.
Update the Linux Package jobs to have the new module into a different package.

Are these changes tested?

Yes via CI

Are there any user-facing changes?

Yes, users will need to either link against libarrow_s3.so or register using LoadFileSystemFactories

raulcd · 2026-06-22T18:03:40Z

@pitrou @kou I've been working on splitting the S3 library (and the AWS SDK) outside libarrow.so into its own library libarrow_s3.so.
On this PR I am just moving the AWS-SDK and the s3 filesystem related source into its own library. Any user trying to leverage it would require linking against it in order to use it (as we do with bindings) or dlopen via LoadFileSystemFactories (path), which would register at load time, no link dependency.
On this PR I am not planning on moving our existing bindings to the FileSystemFromUriAndOptions and LoadFileSystemFactories path. I am also not sure we should do that. I think that path is good for a user that doesn't want to link against libarrow_s3.so and have the same functionality but probably not what the majority of users (and our internal bindings) should do? What are your thoughts on that?
As per the size of the artifacts with the new code the size of libarrow.so and libarrow_s3.so:

$ ls -lhL libarrow.so libarrow_s3.so
-rwxrwxr-x 1 raulcd raulcd  59M Jun 22 19:30 libarrow_s3.so
-rwxrwxr-x 1 raulcd raulcd 317M Jun 22 19:29 libarrow.so

And we can see AWS symbols aren't present on libarrow.so

$  nm -C libarrow.so | grep -c "Aws::"
0
$ nm -C libarrow_s3.so | grep -c "Aws::"
33991

With current main libarrow.so size and it contains AWS SDK symbols:

$ ls -lhL libarrow.so
-rwxrwxr-x 1 raulcd raulcd 368M Jun 22 19:45 libarrow.so
$ ls -lhL libarrow_s3.so
ls: cannot access 'libarrow_s3.so': No such file or directory
$ nm -C libarrow.so | grep -c "Aws::"
33991

Those are debug builds but as a summary:
libarrow.so goes from 368M to 317M (~51M smaller), and AWS (33,991 symbols) move entirely into the new 59M libarrow_s3.so.

kou · 2026-06-23T01:10:23Z

I think that we should use LoadFileSystemFactories() for bindings to avoid loading the S3 module for users who don't need S3. For example, PyArrow users who also want to use the S3 module, they will install pyarrow_s3 (or something) explicitly.

I think that bindings can provide convenient API to use the S3 module even if we use LoadFileSystemFactories().

raulcd · 2026-06-23T09:14:36Z

I think that we should use LoadFileSystemFactories() for bindings to avoid loading the S3 module for users who don't need S3. For example, PyArrow users who also want to use the S3 module, they will install pyarrow_s3 (or something)` explicitly.

With conda this isn't necessary, we already ship all the .so as different packages allowing users to pick and choose what to install. This won't change, if a user installs pyarrow-core and installs libarrow-s3 will have S3 capabilities, same as if we install today pyarrow-core with libarrow-flight. If libarrow-s3 is not installed PyArrow would just ImportError when not finding the corresponding DLL. Basically we build with all capabilities turned on but install only the necessary .so and if not found they ImportError . I'll validate pyarrow fails with ImportError if the .so isn't present but this should behave as the other modules.

With wheels this is another different beast and I have to explore a little further. A related issue:

[Python] usability improvements for a "minimal" pyarrow #38536

The original problem we had with wheels is that there's no mechanism to share dependencies between wheels. Auditwheel/delvewheel/delocate mangle the .so name to avoid other wheels clashing with other dependencies symbols. The problem is that libarrow_s3.so requires libarrow symbols and it's not clear how this pyarrow_s3 would be shipped. Should it include its own libarrow using the mangled symbols for the new wheel? Should it use the libarrow library coming from the main pyarrow wheel? What happens with different versions of pyarrow and pyarrow_s3 installed?

As a note, I've just validated we don't mangle libarrow (or any of our .so) on the wheels. I am going to start exploring this a little further to see if I can come up with something even though I am still unclear about some of the questions above, like version matching to avoid ABI problems.

Related: @amol- who worked on consolidatewheels in the past:

https://github.com/amol-/consolidatewheels

And some Python PEP attempts to define some external dependencies for wheels are on discussion:
https://discuss.python.org/t/pep-725-specifying-external-dependencies-in-pyproject-toml-round-2/103890

What I am saying is that using LoadFileSystemFactories() isn't solving the real problem which in my opinion is: how do we share a single libarrow between several extra wheels and coordinate versioning?

cc @h-vetinari who knows this space and might shed some light

raulcd · 2026-06-23T09:16:03Z

cc @jorisvandenbossche

h-vetinari · 2026-06-23T10:02:12Z

For the PyPI side, you might be able to do something similar to what numpy/scipy are doing with openblas as a wheel.

pitrou · 2026-06-23T12:04:02Z

-      target_link_libraries(arrow_s3fs PRIVATE ${AWSSDK_LINK_LIBRARIES} arrow_shared)
-      set_source_files_properties(filesystem/s3fs.cc filesystem/s3fs_module.cc
-                                  PROPERTIES SKIP_UNITY_BUILD_INCLUSION ON)
+    if(ARROW_BUILD_STATIC AND WIN32)


The AND WIN32 isn't useful, right?

We use the same pattern on other places:

if(ARROW_BUILD_STATIC AND WIN32) target_compile_definitions(arrow_compute_static PUBLIC ARROW_COMPUTE_STATIC) endif()

or

if(ARROW_BUILD_STATIC AND WIN32) target_compile_definitions(arrow_static PUBLIC ARROW_STATIC) endif()

Taking a look at the definition on visibility.h of ARROW_S3_STATIC is already guarded for WIN32:

#if defined(_WIN32) || defined(__CYGWIN__)

So it will only be used on WIN32, it does not seem necessary on others so I would say the AND WIN32 does nothing but it's hygiene?

pitrou · 2026-06-23T12:07:12Z

So, this is as if ARROW_S3_MODULE was always enabled, right?

raulcd · 2026-06-23T12:48:22Z

So, this is as if ARROW_S3_MODULE was always enabled, right?

Yes but with a small caveat. ARROW_S3_MODULE bundled both into libarrow and generated a separate .so. This removes ARROW_S3_MODULE and makes ARROW_S3 unconditionally generate a new libarrow_s3.so and remove AWS SDK and s3fs.cc from libarrow.so. The bindings link against this new library. I have also validated the size reduction on libarrow.so and that no symbols for AWS are included in it.

pitrou · 2026-06-23T13:20:34Z

Oh, great, thank you!

…nd add required ARROW_EXPORTs

…covered on s3fs_module_test and s3fs_module_test doesn't make sense on static builds

and Ruby builds

…row and static build

Copilot

🔵 Needs a closer look

A critical static-build pkg-config link-order issue remains unresolved across the R configuration paths.

Review details

Files reviewed: 23/23 changed files
Comments generated: 1
Review effort level: Lite

kou · 2026-09-15T11:31:04Z

Could you try the following to debug reproducible builds failure?

diff --git a/.github/workflows/package_linux.yml b/.github/workflows/package_linux.yml
index dd5d5a4eaf..1627d3286b 100644
--- a/.github/workflows/package_linux.yml
+++ b/.github/workflows/package_linux.yml
@@ -353,7 +353,6 @@ jobs:
           reprotest \
             --build-command \
               "rake -C dev/tasks/linux-packages ${TASK_NAMESPACE}:build" \
-            --no-diffoscope \
             --vary=-fileordering \
             "${PWD}" \
             "dev/tasks/linux-packages/*/apt/repositories/${DISTRIBUTION}/pool/${DISTRIBUTION_CODE_NAME}/*/*/*/*.deb"

https://github.com/apache/arrow/actions/runs/34948510691/job/104313673313?pr=50195#step:21:15883

[882](https://github.com/apache/arrow/actions/runs/34948510691/job/104313673313?pr=50195#step:21:15883)
Binary files /tmp/tmp50jhndcd/control/source-root/dev/tasks/linux-packages/apache-arrow/apt/repositories/debian/pool/trixie/main/a/apache-arrow/arrow-tools-dbgsym_26.0.0~dev20260915-1_amd64.deb and /tmp/tmp50jhndcd/experiment-1/source-root/dev/tasks/linux-packages/apache-arrow/apt/repositories/debian/pool/trixie/main/a/apache-arrow/arrow-tools-dbgsym_26.0.0~dev20260915-1_amd64.deb differ
Binary files /tmp/tmp50jhndcd/control/source-root/dev/tasks/linux-packages/apache-arrow/apt/repositories/debian/pool/trixie/main/a/apache-arrow/arrow-tools_26.0.0~dev20260915-1_amd64.deb and /tmp/tmp50jhndcd/experiment-1/source-root/dev/tasks/linux-packages/apache-arrow/apt/repositories/debian/pool/trixie/main/a/apache-arrow/arrow-tools_26.0.0~dev20260915-1_amd64.deb differ
Binary files /tmp/tmp50jhndcd/control/source-root/dev/tasks/linux-packages/apache-arrow/apt/repositories/debian/pool/trixie/main/a/apache-arrow/libarrow2600-dbgsym_26.0.0~dev20260915-1_amd64.deb and /tmp/tmp50jhndcd/experiment-1/source-root/dev/tasks/linux-packages/apache-arrow/apt/repositories/debian/pool/trixie/main/a/apache-arrow/libarrow2600-dbgsym_26.0.0~dev20260915-1_amd64.deb differ
Binary files /tmp/tmp50jhndcd/control/source-root/dev/tasks/linux-packages/apache-arrow/apt/repositories/debian/pool/trixie/main/a/apache-arrow/libarrow2600_26.0.0~dev20260915-1_amd64.deb and /tmp/tmp50jhndcd/experiment-1/source-root/dev/tasks/linux-packages/apache-arrow/apt/repositories/debian/pool/trixie/main/a/apache-arrow/libarrow2600_26.0.0~dev20260915-1_amd64.deb differ
Binary files /tmp/tmp50jhndcd/control/source-root/dev/tasks/linux-packages/apache-arrow/apt/repositories/debian/pool/trixie/main/a/apache-arrow/libparquet2600-dbgsym_26.0.0~dev20260915-1_amd64.deb and /tmp/tmp50jhndcd/experiment-1/source-root/dev/tasks/linux-packages/apache-arrow/apt/repositories/debian/pool/trixie/main/a/apache-arrow/libparquet2600-dbgsym_26.0.0~dev20260915-1_amd64.deb differ
Binary files /tmp/tmp50jhndcd/control/source-root/dev/tasks/linux-packages/apache-arrow/apt/repositories/debian/pool/trixie/main/a/apache-arrow/libparquet2600_26.0.0~dev20260915-1_amd64.deb and /tmp/tmp50jhndcd/experiment-1/source-root/dev/tasks/linux-packages/apache-arrow/apt/repositories/debian/pool/trixie/main/a/apache-arrow/libparquet2600_26.0.0~dev20260915-1_amd64.deb differ

raulcd · 2026-09-15T12:19:43Z

Could you try the following to debug reproducible builds failure?

I will try but on a different PR as this is also failing on main (nightlies). I've opened:

[CI][Packaging] Debian-trixie nightly jobs are currently failing on main due to reproducible build differences #51343

raulcd · 2026-09-15T14:11:39Z

Weirdly enough I can't reproduce on the standalone PR, I'll temporary remove the --no-diffoscope here.

…ible builds issue

Copilot

🔵 Needs a closer look

The changes span native libraries, bindings, build systems, packaging, and CI, requiring final human review.

Review details

Files reviewed: 24/24 changed files
Comments generated: 0 new
Review effort level: Lite

kou · 2026-09-15T21:09:33Z

Oh, sorry. I missed it...

Let's debug it in #51344 . "The operation was canceled" means that this is reproduced. :-)

I'll try something in #51344 .

This comment was marked as off-topic.

Sign in to view

github-actions Bot added Component: C++ Component: Python awaiting committer review Awaiting committer review labels Jun 16, 2026

raulcd force-pushed the GH-50194 branch 3 times, most recently from 74a0d20 to 07fa9cc Compare June 18, 2026 07:44

github-actions Bot added the CI: Extra: C++ Run extra C++ CI label Jun 18, 2026

raulcd force-pushed the GH-50194 branch from 2ba103a to 80359ae Compare June 22, 2026 07:56

github-actions Bot added Component: R Component: GLib labels Jun 22, 2026

raulcd force-pushed the GH-50194 branch from d8fb928 to f7d7f58 Compare June 22, 2026 11:28

github-actions Bot removed the CI: Extra: C++ Run extra C++ CI label Jun 22, 2026

raulcd added CI: Extra: R Run extra R CI CI: Extra: Package: Linux Run extra Linux Packages CI labels Jun 22, 2026

aboderinsamuel mentioned this pull request Jun 22, 2026

GH-49644: [Python] Support converting list of multi-dimensional arrays to FixedShapeTensor #50203

Merged

pitrou reviewed Jun 23, 2026

View reviewed changes

github-actions Bot added awaiting changes Awaiting changes and removed awaiting committer review Awaiting committer review labels Jun 23, 2026

h-vetinari mentioned this pull request Jun 23, 2026

Update/relax the hard pin of aws-sdk-cpp conda-forge/arrow-cpp-feedstock#2021

Closed

raulcd force-pushed the GH-50194 branch from 2f065c9 to 4e3bca9 Compare July 23, 2026 13:08

github-actions Bot removed the awaiting changes Awaiting changes label Jul 23, 2026

github-actions Bot added awaiting changes Awaiting changes and removed awaiting change review Awaiting change review labels Sep 14, 2026

tadeja reviewed Sep 14, 2026

View reviewed changes

Comment thread cpp/src/arrow/CMakeLists.txt

tadeja reviewed Sep 14, 2026

View reviewed changes

Comment thread cpp/src/arrow/filesystem/CMakeLists.txt

raulcd added 12 commits September 15, 2026 10:43

apacheGH-50194: [C++] Move S3 and AWS-SDK to its own libarrow_s3.so a…

688e05f

…nd add required ARROW_EXPORTs

arrow-filesystem-test shouldn't require s3fs_module_test.cc as it is …

a30b5e3

…covered on s3fs_module_test and s3fs_module_test doesn't make sense on static builds

Add the new S3 library to R configure and configure.win and to C GLib

4c3ed95

and Ruby builds

Attempt to fix debian packages

a2317c0

Update debian versions after rebase and new release

3ab82c8

Fix Pyarrow S3 with ARROW static build

805d760

Fix Arrow S3 only being built on R Windows for Rtools >= 4

a883873

More fixes for static builds

7abc306

Extra fixes for static builds on R Windows

c1d8b5d

Fix AWS SDK system dependencies being pulled by Arrow S3 insead of Ar…

6eb4a6e

…row and static build

Add suggestion for unnecessary bundled code

aaacbd3

Several fixes as discussed on PR review

c24caae

Copilot AI review requested due to automatic review settings September 15, 2026 08:43

raulcd force-pushed the GH-50194 branch from c84b6c9 to c24caae Compare September 15, 2026 08:43

Copilot started reviewing on behalf of raulcd September 15, 2026 08:43 View session

github-actions Bot added awaiting change review Awaiting change review awaiting review Awaiting review and removed awaiting changes Awaiting changes awaiting review Awaiting review labels Sep 15, 2026

Copilot AI reviewed Sep 15, 2026

View reviewed changes

Comment thread r/configure

Temporarily remove --no-diffoscope to try and find potential reproduc…

3ca0def

…ible builds issue

Copilot AI reviewed Sep 15, 2026

View reviewed changes

Uh oh!

Conversation

raulcd commented Jun 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

This comment was marked as off-topic.

raulcd commented Jun 22, 2026

Uh oh!

kou commented Jun 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

raulcd commented Jun 23, 2026

Uh oh!

raulcd commented Jun 23, 2026

Uh oh!

h-vetinari commented Jun 23, 2026

Uh oh!

pitrou Jun 23, 2026

Choose a reason for hiding this comment

Uh oh!

raulcd Jun 23, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

pitrou commented Jun 23, 2026

Uh oh!

raulcd commented Jun 23, 2026

Uh oh!

pitrou commented Jun 23, 2026

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🔵 Needs a closer look

Uh oh!

Uh oh!

kou commented Sep 15, 2026

Uh oh!

raulcd commented Sep 15, 2026

Uh oh!

raulcd commented Sep 15, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🔵 Needs a closer look

Uh oh!

kou commented Sep 15, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

raulcd commented Jun 16, 2026 •

edited

Loading

kou commented Jun 23, 2026 •

edited

Loading