CloudStore — File Processing Platform (Laravel · MongoDB · AWS S3)

A backend REST API for file ingestion and processing built on Laravel 11, with AWS S3 as the object store and MongoDB as a schemaless metadata layer. It covers the operations real file platforms need: image transformation pipelines, resumable multipart uploads, async batch processing, spreadsheet import and server-side PDF generation.

---

The problem it solves

Handling user files in production is more than $file->store(). You need to transform images into responsive variants without bloating storage, accept multi-gigabyte uploads that survive a dropped connection, import large spreadsheets without timing out the request, run heavy work off the request thread, and generate documents on demand — all while keeping object storage and metadata in sync. CloudStore implements each of these as a clean, self-contained vertical slice over S3 + MongoDB, so the patterns are easy to lift into any media- or document-heavy product.

Senior backend skills demonstrated

• AWS S3 at the SDK level — not just Storage::put(). Direct S3Client usage for multipart uploads (createMultipartUpload → uploadPart → completeMultipartUpload / abortMultipartUpload) with per-part ETag tracking and progress.
• Presigned (temporary) URLs — time-limited download links generated server-side, so the bucket stays private.
• Image processing pipeline — single source image fanned out into an original + thumbnail/medium/large WebP variant set, each streamed straight to S3.
• Polyglot persistence — S3 for bytes, MongoDB (via mongodb/laravel-mongodb) for flexible, nested metadata (variant arrays, part lists, tags) that would be awkward in a relational schema.
• Asynchronous processing — heavy jobs dispatched to a Redis-backed queue with a persisted job-status model clients can poll.
• Batch spreadsheet ingestion — CSV/Excel parsed and imported under a batch ID with per-record status.
• Server-side document generation — order invoices and monthly reports rendered to PDF.
• DRY infrastructure code — a single UsesS3 trait centralises every S3 concern (client, keys, put/get/delete, public + temporary URLs) so services never repeat storage plumbing.
• API documentation — OpenAPI/Swagger annotations served interactively.

---

Tech stack

Layer	Technology
Language / Framework	PHP 8.2+, Laravel 11
Object storage	AWS S3 (aws/aws-sdk-php, league/flysystem-aws-s3-v3)
Metadata store	MongoDB (mongodb/laravel-mongodb 4.x)
Queue	Redis
Image processing	Intervention Image 3 (WebP encoding, resize/cover)
Spreadsheets	league/csv, phpoffice/phpspreadsheet
PDF	barryvdh/laravel-dompdf
API docs	OpenAPI / Swagger (darkaonline/l5-swagger)

---

Architecture

                       ┌──────────────────────────────┐
   HTTP (REST/JSON)    │  Controllers                 │
  ───────────────────► │  Image · Import · Job ·       │
                       │  Report · Upload             │
                       └──────────────┬───────────────┘
                                      │  thin controllers, logic in services
                       ┌──────────────▼───────────────┐
                       │  Services                    │
                       │  ImageService   (WebP fan-out)│
                       │  MultipartService (chunked)  │
                       │  ImportService  (CSV/Excel)  │
                       │  PdfService     (DomPDF)     │
                       └───────┬───────────────┬──────┘
                  UsesS3 trait │               │  Eloquent (Mongo)
                       ┌───────▼──────┐  ┌──────▼───────────────┐
                       │   AWS S3     │  │     MongoDB          │
                       │  (file bytes)│  │ (metadata/documents) │
                       └──────────────┘  └──────────────────────┘
                              ▲
              ProcessUploadJob │  (Redis queue — async heavy work)

Separation of concerns: controllers stay thin; every storage operation flows through the shared UsesS3 trait; bytes live in S3 while richly-nested metadata (image variants, multipart part lists, import records) lives in MongoDB.

---

Key features

1. Image upload & transformation — one upload produces an original.webp plus thumbnail (150×150 cover), medium (800w), and large (1920w) WebP variants, each pushed to S3; dimensions, byte sizes, keys and public URLs recorded per variant in MongoDB. Deleting an image purges every S3 object.
2. Resumable multipart upload — initiate a session, stream 5 MB parts in any order, then complete (or abort). Part ETags and uploaded-byte progress are persisted, so a client can resume after a failure.
3. CSV / Excel import — bulk import a spreadsheet under a batchId; query batch status and the resulting per-row records.
4. Async job processing — submit work to a Redis queue (ProcessUploadJob) and poll a persisted job-status document.
5. PDF generation — create an order, render its invoice to PDF on demand, and produce aggregated monthly reports.

---

API endpoints

Method	Endpoint	Description
GET	/api/images	List image documents (filterable by tags / uploader)
POST	/api/images	Upload an image → WebP variants → S3 + MongoDB metadata
GET	/api/images/{id}	Single image with all variants
DELETE	/api/images/{id}	Delete image + every S3 variant
POST	/api/import	Import a CSV/Excel file (returns a batchId)
GET	/api/import/batch/{batchId}	Batch import status
GET	/api/import/records	Imported records
POST	/api/jobs	Dispatch an async processing job
GET	/api/jobs · /api/jobs/{id}	List / poll job status
POST	/api/orders	Create an order
GET	/api/orders/{id}/pdf	Render the order invoice as PDF
GET	/api/reports/monthly	Aggregated monthly PDF report
POST	/api/uploads/initiate	Start a multipart upload session
POST	/api/uploads/{id}/parts	Upload a 5 MB part (returns ETag + progress)
POST	/api/uploads/{id}/complete	Assemble parts into the final S3 object
GET	/api/uploads/{id}	Upload session status
DELETE	/api/uploads/{id}	Abort the multipart upload

Interactive OpenAPI docs are served at /api/documentation (Swagger UI via l5-swagger).

---

Installation & setup

Requirements: PHP 8.2+, Composer 2, MongoDB (+ the mongodb PHP extension), Redis, and an AWS S3 bucket with credentials.

composer install
cp .env.example .env
php artisan key:generate

# Configure MongoDB, Redis and AWS S3 credentials in .env (see below)

# Run a queue worker for async jobs / processing
php artisan queue:work

The mongodb PHP extension is required: pecl install mongodb (then enable it in php.ini).

---

Environment variables (.env.example)

Variable	Example	Description
MONGODB_URI	mongodb://localhost:27017	MongoDB connection DSN
MONGODB_DATABASE	file_platform	Metadata database name
AWS_ACCESS_KEY_ID	your_key	AWS credentials for the S3 client
AWS_SECRET_ACCESS_KEY	your_secret	AWS secret
AWS_DEFAULT_REGION	eu-central-1	Bucket region
AWS_BUCKET	your-bucket	Target S3 bucket
FILESYSTEM_DISK	s3	Default Laravel disk → S3
QUEUE_CONNECTION	redis	Async jobs run on Redis
REDIS_URL	redis://localhost:6379	Redis connection
FRONTEND_URL	http://localhost:5173	SPA origin allowed by CORS

.env (and all .env.* except .env.example) is git-ignored — credentials never enter version control.

---

Advanced concepts demonstrated

AWS S3 multipart / chunked / resumable uploads · presigned (temporary) download URLs · S3 object lifecycle management · Intervention Image WebP transformation pipeline (cover/scale variants) · MongoDB document modelling with nested arrays via Laravel Eloquent · polyglot persistence (S3 bytes + Mongo metadata) · Redis queue async processing with pollable job status · CSV/Excel batch import (league/csv, PhpSpreadsheet) · server-side PDF generation (DomPDF) · trait-based DRY S3 infrastructure · service-oriented architecture with thin controllers · OpenAPI documentation.

License

MIT