CORE-1489: ci: replace MinIO with RustFS in the Spark, Trino and Dremio test stacks

[EladBarkay]

Why

The spark, trino and dremio CI jobs fail at image pull:

pull access denied for minio/minio, repository does not exist or may require 'docker login'

MinIO deleted minio/minio and minio/mc from Docker Hub on Sept 11, and since about Sept 24 the quay.io/minio/* copies reject anonymous pulls with 401 unauthorized (IBM notice, registry-stack#1402). That also breaks the quay.io switch in elementary-data/elementary#2360: a rerun of its dremio job today fails in seconds.

What

Replace MinIO with RustFS, an Apache-2.0 S3-compatible server, in the three test stacks:

• Server: rustfs/rustfs:1.0.0, credentials via RUSTFS_ACCESS_KEY / RUSTFS_SECRET_KEY, healthcheck on /health.
• Bucket setup: the mc containers become rustfs/rc:v0.1.36 (the RustFS client) and use rc alias set + rc bucket create --ignore-existing.
• Renames: minio services, hostnames, containers, volumes, credentials and seeder env vars (MINIO_* → RUSTFS_*) are now rustfs. The S3 endpoints in spark-defaults.conf, the Trino catalog and Hive metastore, and the Dremio S3 source, plus the Spark wait step in test-warehouse.yml, are updated to match.
• Reliability: images are pinned to exact versions. The setup containers wait on the storage healthcheck instead of sleep 5 (Trino) or a TCP probe (Dremio), and bucket creation can be rerun safely.

Dremio's MINIO_DOMAIN / region settings are dropped. Dremio uses path-style access, so they aren't needed.

Testing

Run locally on Docker (arm64 Mac, Dremio emulated as amd64), using a subset that exercises the storage paths: volume, column, freshness, dimension and schema-change anomalies, plus sampling.

Warehouse	Result
Trino	79 passed, 3 skipped
Spark	79 passed, 3 skipped; 84 seed CSVs uploaded to and read from RustFS
Dremio	81 passed, 1 skipped

Pre-commit hooks pass, and all three compose files validate. The full suites run in CI.

🤖 Generated with Claude Code

Summary by CodeRabbit

• Integration Updates
  • Integration test environments now use RustFS for object storage across Dremio, Spark, and Trino.
  • Storage services are checked for readiness before setup continues, and required buckets are created automatically.
  • Data seeding now uses RustFS connection settings.

[github-actions]

👋 @EladBarkay
Thank you for raising your pull request.
Please make sure to add tests and document all user-facing changes.
You can do this by editing the docs files in the elementary repository.

[coderabbitai]

Navigate logical layers of code changes, visualize relationships, and explore their blast radius.

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Advanced

Run ID: 47752113-25d2-483c-8382-a2a3b7ba5aec

📥 Commits

Reviewing files that changed from the base of the PR and between 5d22e8d and 03a1dd6.

📒 Files selected for processing (8)

• .github/workflows/test-warehouse.yml
• integration_tests/docker-compose-dremio.yml
• integration_tests/docker-compose-spark.yml
• integration_tests/docker-compose-trino.yml
• integration_tests/docker/dremio/dremio-setup.sh
• integration_tests/docker/spark/spark-defaults.conf
• integration_tests/docker/trino/catalog/iceberg.properties
• integration_tests/tests/data_seeder.py

Included review availability: This review used your included allowance. Your plan provides up to 4 included reviews per hour; 3 remain after this review.

---

📝 Walkthrough

Walkthrough

The integration-test storage services and setup jobs now use RustFS instead of MinIO. Dremio, Spark, Trino, and the Spark S3 seeder use RustFS endpoints and credentials.

Changes

RustFS integration-test storage

Layer / File(s)	Summary
RustFS services and bucket setup .github/workflows/test-warehouse.yml, integration_tests/docker-compose-dremio.yml, integration_tests/docker-compose-spark.yml, integration_tests/docker-compose-trino.yml	The Compose stacks replace MinIO with RustFS services and setup jobs. The setup jobs wait for RustFS health before configuring aliases and creating buckets. Workflow startup steps refer to RustFS setup containers.
RustFS client settings and seeder integration_tests/docker/dremio/dremio-setup.sh, integration_tests/docker/spark/spark-defaults.conf, integration_tests/docker/trino/catalog/iceberg.properties, integration_tests/docker-compose-trino.yml, integration_tests/tests/data_seeder.py	Dremio, Spark, Trino, and the Spark S3 seeder use RustFS endpoints and credentials. The seeder reads settings from RUSTFS_* environment variables.

Priority: ⬆️ High

Estimated code review effort: 2 (Simple) | ~12 minutes

Change: Bug fix

Merge Risk: ⚪ Minimal · up to 03a1d

The integration-test stacks appear ready to merge after normal checks; no storage configuration mismatch or blocking health-check issue was found.

Security Architecture Review

Security architecture risk: 🔵 Low · up to 03a1d

The change is confined to integration-test storage. The clients and setup jobs appear to agree on their new endpoints, credentials, and buckets, and the review found no demonstrated expansion of access or production exposure. Runtime behavior and the security properties of the replacement images remain unverified.

Retained concerns
No architecture-level concerns identified.

Security review details

Security Blast Radius

• inferred — If a test host's published storage ports are reachable, knowledge of a stack's static credentials can affect that stack's test objects. The same port-publication and static-credential pattern existed before this PR; no production or cross-tenant reachability is established.

Trust Boundaries and Controls

• inferred — The Spark seeder still takes its destination and credentials from its process environment or local defaults, then passes them to boto3. Its dbt caller does not add a new injection path; who controls that environment is not established here.

Resilience and Maintainability Implications

• inferred — Health-gated, repeat-safe bucket provisioning reduces the chance that an interrupted setup strands a missing test bucket. It does not make the seeder's upload-and-table transition atomic; that limitation predates the provider change.

Hardening Proposals

• proposed — Where test runners do not need remote storage access, consider restricting the published storage ports to the test host. This would harden an existing exposure pattern, not remedy an exposure introduced by this PR.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 3 functions across 2 files. (6 skipped: 6 …
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly and concisely summarizes the main change: replacing MinIO with RustFS in the Spark, Trino, and Dremio test stacks.

✨ Finishing Touches 💡 1

📝 Generate docstrings

• Commit to this branch
• Create a new PR

🧪 Generate unit tests (beta)

• Commit to this branch
• Create a new PR

🛠️ Fix failing CI checks 💡

• Commit to this branch
• Create a new PR

---

_{Comment @coderabbitai help to get the list of available commands.}

[linear]

CORE-1489