Add support for GCS.

[lesaux]

Description of the Change

Adding support for GCS buckets.

Implementation trying to follow how it was done for s3.

Checklist

• allow Maintainers to edit PR (rebase, run coverage, help with tests, ...)
• unit-test added (if change is algorithm)
• functional test added/updated (if change is functional)
• man page updated (if applicable)
• bash completion updated (if applicable)
• documentation updated
• [] author name in AUTHORS

[neolynx]

Hi !

great work :-)

I tried to fix the pipeline issue on master, but was not able to rebase the branch on your repo. could you give maintainer access to your PR ?

[lesaux]

Hi !

great work :-)

I tried to fix the pipeline issue on master, but was not able to rebase the branch on your repo. could you give maintainer access to your PR ?

Thank you!

I added you as a maintainer on my fork.

[codecov]

Codecov Report

❌ Patch coverage is 79.38596% with 47 lines in your changes missing coverage. Please review.
✅ Project coverage is 76.98%. Comparing base (809ab47) to head (f42e5f1).
⚠️ Report is 1 commits behind head on master.

Files with missing lines	Patch %	Lines
gcs/public.go	82.93%	20 Missing and 16 partials ⚠️
context/context.go	0.00%	10 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #1550      +/-   ##
==========================================
- Coverage   77.22%   76.98%   -0.25%     
==========================================
  Files         161      163       +2     
  Lines       15080    15308     +228     
==========================================
+ Hits        11646    11785     +139     
- Misses       2291     2367      +76     
- Partials     1143     1156      +13     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[neolynx]

I tried to rebase and push:

$ git push linuxdataflow -f feat/pls/gcs-support
ERROR: Permission to linuxdataflow/aptly.git denied to neolynx.
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

I think you need to click a ckechbox in the PR itself, "Allow maintainers to acess and edit" or similar...

[lesaux]

This is an org so I think I have to invite you to the repo.
I had invited you last week but I see the invitation expired.
I just invited you again. Let me know!

[neolynx]

oh, I missed that... accepted now, thanks !

[neolynx]

rebased on master

[neolynx]

@lesaux I wonder how we can get the coverage up... is there a GCS simulation we could run in CI ?

[lesaux]

@lesaux I wonder how we can get the coverage up... is there a GCS simulation we could run in CI ?

I will try to implement with fsouza/fake-gcs-server.

[lesaux]

To be able to use the fake-gcs-server I needed to change two things:

• NewPublishedStorage gains an endpoint parameter (mirrors S3PublishRoot.
  Endpoint), exposed via GCSPublishRoot.Endpoint. Real production knob for
  pointing at non-default GCS endpoints in dev/staging.

• When an emulator endpoint is in play (explicit endpoint or the GCS-Go-
  client-native STORAGE_EMULATOR_HOST), force storage.WithJSONReads(). The
  default XML download API uses virtual-host-style URLs that emulators
  can't serve under a single listener.

This allowed me to get the coverage much higher.

Let me know what you think!