Skip to content

Disable s3 multipart download option #12413

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
janvanmansum wants to merge 12 commits into
base: develop
Choose a base branch
from
Open

Disable s3 multipart download option #12413

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
5718ae6
DD-2271 separate clients for upload and download
janvanmansum May 19, 2026
afb1766
Added extension mime type definition for css to git rid of warnings i…
janvanmansum May 20, 2026
859285f
DD-2283 No Accept/Cancel buttons when downloading dataset in Archival…
janvanmansum May 26, 2026
3b347a4
wip
janvanmansum May 29, 2026
ed4f4e1
reformat
janvanmansum May 29, 2026
a1420c0
docs
janvanmansum May 29, 2026
05a25e7
Release note
janvanmansum May 29, 2026
a786378
Update release notes for S3 multipart download setting
qqmyers May 29, 2026
4432154
Update S3-compatible storage documentation
qqmyers Jul 7, 2026
9668f11
Merge branch 'develop' into disable-s3-multipart-download-option
pdurbin Jul 22, 2026
000e1f6
Merge branch 'develop' into disable-s3-multipart-download-option
janvanmansum Jul 27, 2026
e8987ea
Merge branch 'develop' into disable-s3-multipart-download-option
pdurbin Aug 28, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions doc/release-notes/12413-disable-multipart-download-for-indirect-download.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
A new configuration setting has been introduced for S3 compatible storage drivers that addresses an incompatibility between the AWS S3 library used in Dataverse and certain S3 implementations such as the Ceph Object Gateway:

@qqmyers qqmyers May 29, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@janvanmansum - I made some edits - feel free to revert. I wanted to make it clear this new setting is mainly to address a problem, not something you'd want to set true otherwise. I also guessed that you're talking about the Ceph Object Gateway.

Beyond that, I might suggest adding Ceph to the list at https://github.com/IQSS/dataverse/blob/develop/doc/sphinx-guides/source/installation/config.rst#reported-working-s3-compatible-storage with a note about this setting.

`dataverse.files.<id>.disable-multipart-download-for-indirect-download` (default: `false`).

When set to `true`, multipart download is disabled for the specified S3 driver, forcing the server to handle part reassembly and avoiding the incompatible
headers. This is recommended for Ceph-backed S3 storage if `412` errors are encountered during download when `dataverse.files.<id>.download-redirect` is set to `false`.
49 changes: 27 additions & 22 deletions doc/sphinx-guides/source/installation/config.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1417,28 +1417,29 @@ List of S3 Storage Options
.. table::
:align: left

@pdurbin pdurbin Aug 28, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm just reporting this here, randomly, so I can resolve it later.

On my laptop I configured opened the dropdown for storage...

Image

... and picked LocalStack:

Image

When I tried to upload a file I saw this in the logs:

dev_localstack> 2026-08-28T18:59:51.195 INFO --- [et.reactor-0] localstack.request.aws : AWS s3.HeadObject => 404 (NoSuchKey)

And in the UI the file I was uploading flashed and then disappeared:

Image

In CI, API test are passing with LocalStack so I'm not sure what's going on. I must admit that locally, on my laptop, I never change it from the default! 😅

@pdurbin pdurbin Aug 28, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I just tried again on develop (82e84a1) and got the same error. So maybe something's wrong with LocalStack in our dev env. 🤔

@pdurbin pdurbin Aug 28, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On develop I switched to "LocalStackNoDirect" and I was able to upload the file.

Screenshot 2026-08-28 at 3 34 42 PM

It was stored here:

"storageIdentifier": "localstack_noredirect://mybucket-noredirect:1a049dd7a79-b009fc090014",

From our dev compose file, here's the config:

    -Ddataverse.files.localstack_noredirect.label=LocalStackNoRedirect
    -Ddataverse.files.localstack_noredirect.custom-endpoint-url=http://localstack:4566
    -Ddataverse.files.localstack_noredirect.custom-endpoint-region=us-east-2
    -Ddataverse.files.localstack_noredirect.bucket-name=mybucket-noredirect
    -Ddataverse.files.localstack_noredirect.path-style-access=true
    -Ddataverse.files.localstack_noredirect.upload-redirect=false
    -Ddataverse.files.localstack_noredirect.download-redirect=false
    -Ddataverse.files.localstack_noredirect.access-key=default
    -Ddataverse.files.localstack_noredirect.secret-key=default

@qqmyers qqmyers Aug 28, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR should only affect download, so hopefully unrelated. The one thing I can think of that may be relevant - we do a HEAD to get the size of files being direct uploaded and I believe we saw issues before with 'eventual consistency' - the file had uploaded successfully but HEAD returned 404. We may have added a retry or delay for that - I don't recall without digging, but if LocalStack can't immediately find the file (waiting for some indexing etc.) it might look like what you're seeing.

@pdurbin pdurbin Aug 28, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, thanks. I threw this back in "ready for QA" and unassigned myself.

Per Slack, it sounds like we should probably update internal or the QA server to the newest Payara and test S3 stuff there.

Additional S3 PRs we want to get in 6.12:

=========================================== ================== =================================================================================== =============
JVM Option Value Description Default value
=========================================== ================== =================================================================================== =============
dataverse.files.storage-driver-id <id> Enable <id> as the default storage driver. ``file``
dataverse.files.<id>.type ``s3`` **Required** to mark this storage as S3 based. (none)
dataverse.files.<id>.label <?> **Required** label to be shown in the UI for this storage (none)
dataverse.files.<id>.bucket-name <?> The bucket name. See above. (none)
dataverse.files.<id>.download-redirect ``true``/``false`` Enable direct download or proxy through Dataverse. ``false``
dataverse.files.<id>.upload-redirect ``true``/``false`` Enable direct upload of files added to a dataset in the S3 store. ``false``
dataverse.files.<id>.upload-out-of-band ``true``/``false`` Allow upload of files by out-of-band methods (using some tool other than Dataverse) ``false``
dataverse.files.<id>.ingestsizelimit <size in bytes> Maximum size of directupload files that should be ingested (none)
dataverse.files.<id>.url-expiration-minutes <?> If direct uploads/downloads: time until links expire. Optional. 60
dataverse.files.<id>.min-part-size <?> Multipart direct uploads will occur for files larger than this. Optional. ``1024**3``
dataverse.files.<id>.custom-endpoint-url <?> Use custom S3 endpoint. Needs URL either with or without protocol. (none)
dataverse.files.<id>.custom-endpoint-region <?> Only used when using custom endpoint. Optional. ``dataverse``
dataverse.files.<id>.profile <?> Allows the use of AWS profiles for storage spanning multiple AWS accounts. (none)
dataverse.files.<id>.proxy-url <?> URL of a proxy protecting the S3 store. Optional. (none)
dataverse.files.<id>.path-style-access ``true``/``false`` Use path style buckets instead of subdomains. Optional. ``false``
dataverse.files.<id>.chunked-encoding ``true``/``false`` Disable chunked encoding. Optional ``true``
dataverse.files.<id>.connection-pool-size <?> The maximum number of open connections to the S3 server ``256``
dataverse.files.<id>.disable-tagging ``true``/``false`` Do not place the ``temp`` tag when redirecting the upload to the S3 server. ``false``
=========================================== ================== =================================================================================== =============
===================================================================== ================== =================================================================================== =============
JVM Option Value Description Default value
===================================================================== ================== =================================================================================== =============
dataverse.files.storage-driver-id <id> Enable <id> as the default storage driver. ``file``
dataverse.files.<id>.type ``s3`` **Required** to mark this storage as S3 based. (none)
dataverse.files.<id>.label <?> **Required** label to be shown in the UI for this storage (none)
dataverse.files.<id>.bucket-name <?> The bucket name. See above. (none)
dataverse.files.<id>.download-redirect ``true``/``false`` Enable direct download or proxy through Dataverse. ``false``
dataverse.files.<id>.upload-redirect ``true``/``false`` Enable direct upload of files added to a dataset in the S3 store. ``false``
dataverse.files.<id>.upload-out-of-band ``true``/``false`` Allow upload of files by out-of-band methods (using some tool other than Dataverse) ``false``
dataverse.files.<id>.ingestsizelimit <size in bytes> Maximum size of directupload files that should be ingested (none)
dataverse.files.<id>.url-expiration-minutes <?> If direct uploads/downloads: time until links expire. Optional. 60
dataverse.files.<id>.min-part-size <?> Multipart direct uploads will occur for files larger than this. Optional. ``1024**3``
dataverse.files.<id>.custom-endpoint-url <?> Use custom S3 endpoint. Needs URL either with or without protocol. (none)
dataverse.files.<id>.custom-endpoint-region <?> Only used when using custom endpoint. Optional. ``dataverse``
dataverse.files.<id>.profile <?> Allows the use of AWS profiles for storage spanning multiple AWS accounts. (none)
dataverse.files.<id>.proxy-url <?> URL of a proxy protecting the S3 store. Optional. (none)
dataverse.files.<id>.path-style-access ``true``/``false`` Use path style buckets instead of subdomains. Optional. ``false``
dataverse.files.<id>.chunked-encoding ``true``/``false`` Disable chunked encoding. Optional ``true``
dataverse.files.<id>.connection-pool-size <?> The maximum number of open connections to the S3 server ``256``
dataverse.files.<id>.disable-tagging ``true``/``false`` Do not place the ``temp`` tag when redirecting the upload to the S3 server. ``false``
dataverse.files.<id>.disable-multipart-download-for-indirect-download ``true``/``false`` Disable multipart download for indirect downloads from S3. ``false``
===================================================================== ================== =================================================================================== =============

.. table::
:align: left
Expand Down Expand Up @@ -1479,6 +1480,10 @@ You may provide the values for these via any `supported MicroProfile Config API
Reported Working S3-Compatible Storage
######################################

`Ceph Object Gateway <https://docs.ceph.com/en/reef/radosgw/#ceph-object-gateway>`_ (added July 2026/Dataverse v6.12)
Set ``dataverse.files.<id>.disable-multipart-download-for-indirect-download=true`` if not using direct download.
(This forces the S3 server to handle part reassembly and avoid incompatible headers that cause `412` errors from the Ceph Gateway.)

@pdurbin pdurbin Jul 22, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a heads up that there was a merge conflict here that I resolved in 9668f11

The conflict was because of this PR we merged:

`StorJ Object Store <https://www.storj.io>`_
StorJ is a distributed object store that can be configured with an S3 gateway. Per the S3 Storage instructions above, you'll first set up the StorJ S3 store by defining the id, type, and label. After following the general installation, set the following configuration to use a StorJ object store: ``dataverse.files.<id>.chunked-encoding=false``. For step-by-step instructions see https://docs.storj.io/dcs/how-tos/dataverse-integration-guide/

Expand Down
Loading
Loading