ci: check upload quota before release

[jpopesculian]

Rationale for this change

Prevent hitting upload quota

What changes are included in this PR?

A script to check the current size of the project and check whether a new release would surpass the upload quota (configured currently to 80 GiB)

Are these changes tested?

Locally with current 40GiB limit

Are there any user-facing changes?

No

[github-actions]

Thanks for opening a pull request!

If this is not a minor PR. Could you open an issue for this pull request on GitHub? https://github.com/apache/arrow/issues/new/choose

Opening GitHub issues ahead of time contributes to the Openness of the Apache Arrow project.

Then could you also rename the pull request title in the following format?

GH-${GITHUB_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}

or

MINOR: [${COMPONENT}] ${SUMMARY}

See also:

• Other pull requests
• Contribution Guidelines - Contributing Overview

[raulcd]

Thanks for this. I agree we should add a check, could you create an issue to track this?

The verify RC job isn't the best place for this in my opinion. That workflow is to track the status of the binary artifacts before voting for the release. This job failing shouldn't really make us drop a Release Candidate or stop the vote for the release.
It should stop uploading to PyPI. I think we should integrate the check before uploading on 09-post-python.sh but we should specifically run the script. I'd probably either integrate into archery or add a small curl + shell check validation.
We probably could request PyPI whether there's an API where we can know the remaining quota to provide the validation directly.