diff --git a/AGENTS.md b/AGENTS.md index 092694c6..89cbb4a9 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -190,6 +190,11 @@ To maintain a clear separation between fast, offline unit tests and containerize - Use `thisObjectLock.readLock()` for fast, concurrent reads, and `thisObjectLock.writeLock()` for state mutations, lazy pattern compilation, or feature capability degradation. - Always acquire and release locks using `try ... finally` blocks (e.g. `thisObjectLock.readLock().lock(); try { return field; } finally { thisObjectLock.readLock().unlock(); }`). +### 24. UploadLock AutoCloseable Lifecycle & Try-With-Resources +- `UploadLock` implements `java.lang.AutoCloseable`. +- Whenever acquiring an `UploadLock` (e.g. via `lockingService.lockUpload(uploadId)` or `lockingService.lockUploadByUri(uri)`), always manage the lock lifecycle using a `try-with-resources` block (e.g. `try (UploadLock lock = lockingService.lockUpload(id)) { ... }`). +- This guarantees timely and reliable lock release upon normal completion or abrupt exceptions without requiring verbose and error-prone `try ... finally { if (lock != null) lock.release(); }` boilerplates. + ## IETF Resumable Uploads for HTTP (RUFH) Spec Maintenance & Update Playbook diff --git a/CHANGELOG.md b/CHANGELOG.md index d063cf3e..90e46ee0 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -21,6 +21,7 @@ See [docs/UPGRADING.md](docs/UPGRADING.md) for details and code examples. - **Download extension (breaking)**: no longer returns `204` for uploads still in progress and no longer sets the `Tus-Resumable` header (needed for dual-protocol support). The `UploadInProgressException` class was removed. - **Jackson bundled**: `jackson-databind`, `jackson-annotations` and `jackson-core` are now compile-scope dependencies; no separate declaration needed. - **Protocol detection**: `ProtocolVersion.AUTO` is the default. Restrict with `withSupportedProtocolVersions(...)`, see [docs/PROTOCOLS.md](docs/PROTOCOLS.md). +- **Checksum Deduplication Index**: The checksum index format has been updated to support reference tracking, parent tombstoning, and full cleanup upon upload deletion across Disk, S3, and Azure Blob storage. See [docs/DEDUPLICATION.md](docs/DEDUPLICATION.md) for migration instructions (clearing the index directory or running the migration script). ### New - **IETF Resumable Uploads for HTTP (RUFH)**: full draft-12 support with automatic protocol detection, RFC 9651 structured headers (`Upload-Offset`, `Upload-Complete`, `Upload-Length`, `Upload-Limit`), RFC 7807 `application/problem+json` errors, RFC 9530 HTTP digests, `104` interim responses and a spec-quoted conformity suite (`scripts/rufh_conformity_test.py`). See [docs/PROTOCOLS.md](docs/PROTOCOLS.md), [docs/MIGRATION.md](docs/MIGRATION.md), [docs/INTERIM_RESPONSES.md](docs/INTERIM_RESPONSES.md) and [docs/CONFORMITY_TESTING.md](docs/CONFORMITY_TESTING.md). @@ -32,12 +33,17 @@ See [docs/UPGRADING.md](docs/UPGRADING.md) for details and code examples. - **Automatic lock cleanup** on completion, termination (`DELETE`) and expiration across all locking services (`cleanupLock(...)`), plus a fallback sweeper for abandoned locks. - **JSON serialization**: store `UploadInfo` as JSON with `withJsonSerialization(true)`. - **Absolute base URLs**: `withUploadUri(String)` accepts e.g. `https://upload.example.com/files`, returning full `Location` URLs for Tus and RUFH; relative paths keep working. +- **Multi-Backend Upload Deduplication & Lifecycle Cleanup**: Content-based upload deduplication with `withUploadDeduplication(true)` across Disk, S3, and Azure Blob storage. Features child reference markers, parent tombstoning, and complete index directory cleanup when uploads expire or are deleted. See [docs/DEDUPLICATION.md](docs/DEDUPLICATION.md). +- **`NoOpUploadLockingService`**: Zero-overhead Null Object implementation of `UploadLockingService` for unit tests and custom environments, making locking service usage mandatory and eliminating defensive null checks across all storage backends, request handlers, and deduplication logic. ### Fixed - **Content-Type matching**: media types are matched per RFC 9110 and tolerate parameters such as `;charset=UTF-8` (added by Spring `CharacterEncodingFilter`, proxies or clients) instead of returning `406`. - **Undertow & other containers**: `Content-Length` is cleared before `sendError(...)` ([#40](https://github.com/tomdesair/tus-java-server/issues/40)). - **Disk**: truncate underflow guard in `DiskStorageService`; `FileChannel` leak on lock acquisition errors fixed in `FileBasedLock`. +### Deprecated +- **`UploadStorageService.cleanupExpiredUploads(UploadLockingService)`**: Deprecated in favor of the zero-argument `cleanupExpiredUploads()`, as storage services coordinate locking through `setUploadLockingService(UploadLockingService)`. + ### Security - **Path traversal protection**: hardened `LeaseFileLockingService` against directory traversal via crafted upload identifiers ([#141](https://github.com/tomdesair/tus-java-server/pull/141)). - **CORS origin validation**: prevented arbitrary `Origin` header reflection in CORS preflight responses ([#132](https://github.com/tomdesair/tus-java-server/pull/132)). diff --git a/README.md b/README.md index 02886496..c73da7e6 100644 --- a/README.md +++ b/README.md @@ -282,6 +282,7 @@ Detailed Tus and RUFH flow diagrams: [Protocols](docs/PROTOCOLS.md). | Tus vs. IETF RUFH, protocol version selection, extensions, HTTP digests, 104 responses | [Protocols](docs/PROTOCOLS.md), [Migrating clients to RUFH](docs/MIGRATION.md), [Interim responses](docs/INTERIM_RESPONSES.md) | | Multi-replica, Kubernetes, reverse proxies | [Deployment Guide](docs/DEPLOYMENT.md), [Locking](docs/LOCKING.md) | | Storage backends | [Disk & network (NFS/EFS/SMB)](docs/DISK_BASED_LOCKING.md), [S3-compatible](docs/S3_STORAGE.md), [Azure Blob](docs/AZURE_BLOB_STORAGE.md) | +| Upload deduplication | [Upload Deduplication & Migration](docs/DEDUPLICATION.md) | | Problems | [Troubleshooting](docs/TROUBLESHOOTING.md) | | Upgrading from 1.x | [Upgrade Guide](docs/UPGRADING.md), [CHANGELOG](CHANGELOG.md) | diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md index 80ebd339..e3a25c26 100644 --- a/docs/ARCHITECTURE.md +++ b/docs/ARCHITECTURE.md @@ -177,13 +177,46 @@ For local filesystems and shared network volumes, `tus-java-server` provides two - **Upload Deduplication**: Enabled via `.withUploadDeduplication(true)`. Completed uploads are indexed by checksum. Duplicate uploads are linked via `duplicatesUploadId`. Read operations dynamically resolve to the parent upload, while write and truncation operations strictly operate on the target upload to prevent corrupting parent data. - **Thread-Local Request Caching (`ThreadLocalCachedStorageAndLockingService`)**: Enabled via `.withThreadLocalCache(true)`. Caches `UploadInfo` in memory for the duration of a single HTTP request to eliminate redundant network roundtrips to S3 or Azure Blob Storage. +### Implementing a Custom Storage Backend + +Custom storage engines (e.g. Google Cloud Storage, relational database BLOB stores, or custom object systems) implement the [`UploadStorageService`](../src/main/java/me/desair/tus/server/upload/UploadStorageService.java) interface: + +1. **Upload Lifecycle**: + - `create(UploadInfo, String)`: Allocate initial storage resources and persist the upload metadata record. + - `append(UploadInfo, InputStream)`: Stream and append incoming payload chunks to storage. Persist partial progress if interrupted. + - `getUploadInfo(UploadId)` / `update(UploadInfo)`: Retrieve and update upload metadata. + - `getUploadedBytes(UploadId)` / `copyUploadTo(UploadInfo, OutputStream)`: Stream completed file bytes to consumers. +2. **Resource Termination & Lock Cleanup Contract (`terminateUpload`)**: + `terminateUpload(UploadInfo)` is the single authoritative method responsible for freeing all resources when an upload is deleted, cancelled, or expired. Custom implementations must: + - Free physical committed data bytes and any temporary chunk buffers. + - Delete the metadata record (`.info`). + - Coordinate deduplication index removal or tombstoning via `deduplicationService.removeUploadFromIndex(uploadInfo)` when deduplication is active. + - **Delegate Lock Cleanup**: Call `uploadLockingService.cleanupLock(uploadInfo.getId())`, ensuring no orphaned lock files, stop signals, or lease objects remain in the locking layer. +3. **Mandatory Locking Service**: + An `UploadLockingService` is mandatory across all storage backends. Storage services receive their `UploadLockingService` reference via `setUploadLockingService(UploadLockingService)` (enforcing non-null). For unit tests or specialized embedding environments where locking is bypassed, [`NoOpUploadLockingService`](../src/main/java/me/desair/tus/server/upload/NoOpUploadLockingService.java) provides a zero-overhead Null Object implementation. +4. **Supporting Checksum Deduplication**: + When upload deduplication is enabled (`.withUploadDeduplication(true)`), storage backends must implement the following critical interface methods: + - `deduplicate(UploadInfo, ChecksumKey)`: Coordinate cluster-safe deduplication upon upload completion. Storage backends typically instantiate a [`DeduplicationService`](../src/main/java/me/desair/tus/server/upload/checksum/DeduplicationService.java) backed by a custom [`ChecksumIndex`](../src/main/java/me/desair/tus/server/upload/checksum/ChecksumIndex.java) implementation to check for an existing parent, link the child upload, or register a new parent. + - `dataExists(UploadId)`: Return `true` if physical committed data exists for the upload ID. The deduplication service invokes this method during parent lookup to detect out-of-band data deletions and automatically self-heal stale checksum index entries. *(The default interface method returns `false`, which disables parent reuse if not overridden.)* + - `deleteData(UploadId)`: Delete physical committed data bytes without removing the upload's `.info` metadata record. This is invoked to free physical storage when a duplicate upload is linked to an existing parent, and to purge retained parent data when the last active child terminates. *(The default interface method is a no-op, which leaks storage if not overridden.)* + - `setUploadDeduplicationEnabled(boolean)` / `isUploadDeduplicationEnabled()`: Store and expose whether deduplication is active. + - **Read/Write Safety Rules**: + - *Read operations* (`getUploadedBytes`, `copyUploadTo`) MUST check `uploadInfo.getDuplicatesUploadId()` and dynamically stream bytes from the parent upload if set. + - *Write and mutation operations* (`append`, `removeLastNumberOfBytes`) MUST NEVER resolve `duplicatesUploadId` to the parent; they must operate strictly on child storage to prevent mutating parent data. +5. **Optional Features & Extensions**: + - `UploadConcatenationService`: Implement concatenation to support merging parallel chunk uploads into a single final resource. + - Size limits: Implement `setMinAppendSize`, `setMaxAppendSize`, `setMinSize`, and `setMaxUploadSize` to enforce upload payload constraints. + - Cloud streaming: Implement `setCloudUploadThreadPoolSize` and `setDrainTimeout` for asynchronous chunk uploaders. + - Cleanup: Implement `close()` to release underlying network clients, thread pools, or file descriptors. + + --- ## 6. Distributed Locking & Lock Contention Resolution Resumable uploads require exclusive concurrency control per upload resource to prevent race conditions (e.g., overlapping byte appends or simultaneous resume and delete requests). -All locking backends implement [`UploadLockingService`](../src/main/java/me/desair/tus/server/upload/UploadLockingService.java) and return [`UploadLock`](../src/main/java/me/desair/tus/server/upload/UploadLock.java) handles. +All locking backends implement [`UploadLockingService`](../src/main/java/me/desair/tus/server/upload/UploadLockingService.java) and return [`UploadLock`](../src/main/java/me/desair/tus/server/upload/UploadLock.java) handles. When no concurrency control is needed, [`NoOpUploadLockingService`](../src/main/java/me/desair/tus/server/upload/NoOpUploadLockingService.java) acts as the null-safe implementation. ### Building Lease-Based Lock Providers: `AbstractLeaseLockingService` @@ -305,5 +338,5 @@ For detailed testing instructions, see [`docs/TESTING.md`](TESTING.md) and [`doc | **Add a Request Validation Rule** | Create a class implementing `RequestValidator` in the target extension's `validation` package. | Keep validation side-effect-free. Throw typed `TusException` subclasses. Register in `extension.initValidators()`. | | **Add a Request Processing Action** | Create a class implementing `RequestHandler` (or extending `AbstractRequestHandler`). | Wrap streaming payload inputs in `InterruptibleInputStream`. Set headers on `TusServletResponse`. Register in `extension.initRequestHandlers()`. | | **Add a New Protocol Extension** | Create a package under `me.desair.tus.server.` and extend `AbstractTusExtension`. | Implement `getName()`, `isApplicable(method, version)`, `initValidators()`, and `initRequestHandlers()`. Register in `TusFileUploadService.initFeatures()`. | -| **Add a Storage Backend** | Implement `UploadStorageService` (and optionally `UploadConcatenationService`). | Support deduplication (`duplicatesUploadId`). Implement offline unit tests (`*Test.java`) and Testcontainers integration tests (`IT*`). | +| **Add a Storage Backend** | Implement `UploadStorageService` (and optionally `UploadConcatenationService`). | Implement `terminateUpload` (free data, metadata `.info`, and delegate `cleanupLock`). If supporting deduplication, implement `deduplicate`, `dataExists`, and `deleteData`, follow read/write safety rules (`duplicatesUploadId`), and implement a `ChecksumIndex`. Implement offline unit tests (`*Test.java`) and Testcontainers integration tests (`IT*`). | | **Add a Distributed Locking Provider** | Extend `AbstractLeaseLockingService` and `AbstractLeaseLock`. | Implement the template hooks (`tryAcquireLock`, `isLockExpired`, `evictExpiredLock`, `writeStopSignal`, `checkStopSignals`). Inherit NTP safety margins and stream interruption automatically. | diff --git a/docs/AZURE_BLOB_STORAGE.md b/docs/AZURE_BLOB_STORAGE.md index b0bb3bfd..83c242ec 100644 --- a/docs/AZURE_BLOB_STORAGE.md +++ b/docs/AZURE_BLOB_STORAGE.md @@ -89,11 +89,13 @@ TusFileUploadService tusService = new TusFileUploadService() ``` / -├── uploads/ # Final upload data (Block Blob) -├── metadata/.info # JSON-serialized UploadInfo -├── checksums// # Deduplication checksum index object -├── locks/.lock # Distributed lock target blob (Blob Lease) -└── locks/.stop # Cross-replica contention interrupt signal +├── uploads/ # Final upload data (Block Blob) +├── metadata/.info # JSON-serialized UploadInfo +├── checksums///parent # Deduplication parent pointer blob +├── checksums///parent-deleted # Tombstone marker blob if parent deleted +├── checksums///children/ # Active duplicate child reference blob +├── locks/.lock # Distributed lock target blob (Blob Lease) +└── locks/.stop # Cross-replica contention interrupt signal ``` ### Key Prefix Defaults diff --git a/docs/CONFIGURATION.md b/docs/CONFIGURATION.md index a3d5cdb3..8f34aa7d 100644 --- a/docs/CONFIGURATION.md +++ b/docs/CONFIGURATION.md @@ -49,7 +49,7 @@ For Azure Blob Storage dependencies see the [Azure Blob Storage Guide](AZURE_BLO | `withMaxUploadSize(Long)` | `Long.MAX_VALUE` | Maximum allowed total upload size in bytes per upload resource. | | `withMaxAppendSize(Long)` / `withMinAppendSize(Long)` / `withMinSize(Long)` | No limit | IETF RUFH only: maximum size of a single append (`PATCH`) request, minimum size of an append or creation request, and minimum total upload size (all in bytes). | | `withUploadExpirationPeriod(Long)` | `null` (disabled) | Expiration period in **milliseconds** after which incomplete/expired uploads become eligible for cleanup (run `cleanup()` periodically, see the [Integration Guide](INTEGRATION_GUIDE.md#3-cleanup)). | -| `withUploadDeduplication(Boolean)` | `false` | Enables duplicate file detection by checksum, linking new uploads (`duplicatesUploadId`) and skipping redundant storage writes. | +| `withUploadDeduplication(Boolean)` | `false` | Enables duplicate file detection by checksum, linking new uploads (`duplicatesUploadId`) and skipping redundant storage writes. See [Upload Deduplication & Migration](DEDUPLICATION.md). | ### Identity, events & serialization @@ -64,7 +64,7 @@ For Azure Blob Storage dependencies see the [Azure Blob Storage Guide](AZURE_BLO | Method | Default | Description | |---|---|---| | `withStoragePath(String)` | `${java.io.tmpdir}/tus` | Path on the filesystem or shared drive where uploaded bytes and metadata are stored when using `DiskStorageService`. Also configures `LeaseFileLockingService`. | -| `withUploadStorageService(UploadStorageService)` | `DiskStorageService` | Configures custom or cloud storage backend (`DiskStorageService`, `S3StorageService`, `AzureBlobStorageService`). Automatically copies over previous limits and configuration. | +| `withUploadStorageService(UploadStorageService)` | `DiskStorageService` | Configures custom or cloud storage backend (`DiskStorageService`, `S3StorageService`, `AzureBlobStorageService`). Automatically copies over previous limits and configuration. For guidelines on implementing custom storage backends, see [ARCHITECTURE.md](ARCHITECTURE.md#implementing-a-custom-storage-backend). | | `withUploadLockingService(UploadLockingService)` | `LeaseFileLockingService` | Configures custom or cloud locking backend (`LeaseFileLockingService`, `S3LockingService`, `AzureBlobLockingService`). | | `withLockWaitTimeout(Duration)` | `Duration.ofSeconds(60)` | Maximum duration a request waits to acquire an upload lock held by an in-flight transfer (retrying every 200ms, resulting in 300 retries). Automatically configures cloud background chunk drain timeout to 5s less than this value (default 55s). See [LOCKING.md](LOCKING.md). | | `withThreadLocalCache(Boolean)` | `false` | Enables in-memory thread-local caching of upload request data to reduce storage backend I/O load. Strongly recommended for cloud backends (S3, Azure) to eliminate redundant remote metadata lookups per request. | diff --git a/docs/DEDUPLICATION.md b/docs/DEDUPLICATION.md new file mode 100644 index 00000000..a83dfeef --- /dev/null +++ b/docs/DEDUPLICATION.md @@ -0,0 +1,185 @@ +# File Checksum Deduplication & Lifecycle Management + +This document describes the architecture, lifecycle management, and migration options for checksum-based upload deduplication in `tus-java-server`. + +--- + +## 1. Architecture Overview + +When upload deduplication is enabled (`withUploadDeduplication(true)`), `tus-java-server` detects identical files across uploads using cryptographic checksums (e.g., SHA-1, SHA-256) supplied via Tus 1.0.0 `Upload-Checksum` or IETF RUFH `Content-Digest` / `Repr-Digest` headers. + +### 1.1 Parent and Child Relationships + +- **Parent Upload ($P$)**: The first upload completed for a given checksum value. It stores the physical data payload on disk, S3, or Azure Blob storage. +- **Child Upload ($C$)**: Subsequent uploads completing with the identical checksum. The child upload's physical data payload is deleted to save storage, and its metadata (`duplicatesUploadId`) is linked to the parent upload ID. +- **Transparent Reads**: When reading or downloading child uploads (`getUploadedBytes`, `copyUploadTo`), the server dynamically streams the payload from the parent upload. +- **Isolated Writes**: Write operations (`append`, `removeLastNumberOfBytes`) never resolve duplicates recursively, guaranteeing that parent files cannot be mutated by child errors or retries. + +--- + +## 2. Storage Layout & Reference Tracking + +To safely track parent/child lifecycles and ensure thread-safe and cluster-safe cleanup without race conditions, `tus-java-server` uses the following layout: + +```text +checksums///parent # Contains the parent UploadId +checksums///parent-deleted # Tombstone marker (present only when parent is deleted) +checksums///children/ # Marker file/blob per active child upload +``` + +- ``: Canonical lowercase hexadecimal checksum normalized via `ChecksumKey` (protects against path traversal, slashes, or illegal characters across Linux, Windows, S3, and Azure Blob storage). +- `parent`: Created atomically when the first upload with this checksum completes. +- `parent-deleted`: Created if the parent upload is deleted via HTTP `DELETE` or `deleteUpload` while child uploads are still active. +- `children/`: Created when a child links to the parent; removed when the child is deleted. + +--- + +## 3. Lifecycle & Cleanup Workflows + +### 3.1 Parent Upload Deletion + +When a parent upload is deleted: +1. **If active children exist** (`hasChildren(key) == true`): + - A `parent-deleted` tombstone marker is created alongside the existing `parent` pointer file. + - Parent metadata is deleted, but **parent physical payload bytes and the `parent` pointer are preserved** so children continue downloading without disruption. + - **Tombstoned Parent Joining**: Any new duplicate upload arriving while a parent is tombstoned and still has active children will join the tombstoned parent and share its retained bytes until all children are deleted. +2. **If no children exist** (`hasChildren(key) == false`): + - The parent physical data bytes are deleted. + - The entire `checksums//` directory/prefix is removed. + +### 3.2 Child Upload Deletion + +When a child upload is deleted: +1. The child marker `children/` and child metadata are deleted. +2. **Last-Child Purge**: If the parent was previously deleted (`isParentDeleted(key) == true`) and no other children remain, the server acquires the parent lock and purges the parent data payload and the checksum directory. + +### 3.3 Self-Healing on Missing Parent Data + +If an external lifecycle policy or disk cleanup deletes a parent payload out-of-band: +- Upon the next lookup or deduplication attempt, `DeduplicationService` detects that the parent physical data is missing. +- It deletes the stale checksum directory/prefix immediately and allows the incoming upload to become the new parent, preventing data loss. + +--- + +## 4. Upgrading & Migration Guide (from version 1.x) + +In versions prior to 2.0.0, completed uploads were indexed using a flat file structure: +`/checksums//` + +Version 2.0.0 introduces directory-based reference tracking (`/parent`). When upgrading to 2.0.0 with deduplication enabled, choose one of the following two migration options: + +### Option 1: Clear the Existing Checksum Index (Recommended) + +Because the checksum index is an optimization for deduplication and does not store actual file payloads, the simplest approach is to delete the `checksums/` directory or object prefix before starting the upgraded server: + +- **Local Disk**: `rm -rf /checksums` +- **S3 Object Storage**: Remove all objects under the `checksums/` prefix (e.g. `aws s3 rm s3:///checksums/ --recursive`) +- **Azure Blob Storage**: Delete blobs with prefix `checksums/` + +New completed uploads will automatically populate the new layout. Existing completed uploads will remain accessible, and new identical files will simply register a fresh parent. + +### Option 2: Run a Manual Migration Script + +If you wish to retain existing parent deduplication references for pre-existing completed uploads, run a migration script before starting version 2.0.0: + +#### Local Disk Migration Script (Bash) + +```bash +#!/usr/bin/env bash +set -euo pipefail + +STORAGE_DIR="${1:-/tmp/tus}" +CHECKSUMS_DIR="${STORAGE_DIR}/checksums" + +if [ ! -d "${CHECKSUMS_DIR}" ]; then + echo "No checksums directory found at ${CHECKSUMS_DIR}. Nothing to migrate." + exit 0 +fi + +echo "Migrating checksum files in ${CHECKSUMS_DIR}..." + +for alg_dir in "${CHECKSUMS_DIR}"/*; do + if [ ! -d "${alg_dir}" ]; then + continue + fi + algorithm=$(basename "${alg_dir}") + + for checksum_file in "${alg_dir}"/*; do + if [ ! -f "${checksum_file}" ]; then + continue + fi + + checksum_name=$(basename "${checksum_file}") + parent_id=$(cat "${checksum_file}" | tr -d '[:space:]') + + # Create target directory: /// + target_dir="${alg_dir}/${checksum_name}_dir" + mkdir -p "${target_dir}" + + # Write parent file with upload ID + echo -n "${parent_id}" > "${target_dir}/parent" + + # Replace old flat file with the new directory + rm -f "${checksum_file}" + mv "${target_dir}" "${alg_dir}/${checksum_name}" + + echo "Migrated ${algorithm}/${checksum_name} -> parent: ${parent_id}" + done +done + +echo "Checksum migration completed successfully." +``` + +#### S3 Object Storage Migration + +For S3, move each object from `checksums//` to `checksums///parent` retaining the `UploadId` content. + +--- + +## 5. Implementing Deduplication in Custom Storage Backends + +When developing a custom storage backend (e.g., for GCP Cloud Storage, database BLOB storage, or proprietary systems), upload deduplication can be enabled by implementing the following SPI components: + +### 5.1 Critical `UploadStorageService` Methods + +| Method | Role in Deduplication | Default Behavior if Omitted | +|---|---|---| +| `deduplicate(UploadInfo, ChecksumKey)` | Orchestrates duplicate linking or CAS parent registration via `DeduplicationService`. | Returns `uploadInfo` unmodified (deduplication bypassed). | +| `dataExists(UploadId)` | Verifies that physical committed payload data exists for the upload. Required for self-healing missing parents. | Returns `false` (breaks parent reuse). | +| `deleteData(UploadId)` | Deletes physical payload bytes without deleting the `.info` metadata file. Required for child payload cleanup and last-child parent purges. | No-op (leaks child and tombstoned parent payloads). | +| `setUploadDeduplicationEnabled(boolean)` / `isUploadDeduplicationEnabled()` | Stores and exposes the deduplication feature flag. | Always returns `false`. | + +### 5.2 Read/Write Safety Rules + +- **Transparent Reads**: `getUploadedBytes(UploadId)` and `copyUploadTo(UploadInfo, OutputStream)` MUST inspect `uploadInfo.getDuplicatesUploadId()`. If non-null, data bytes must be streamed directly from the parent upload's storage location. +- **Isolated Writes**: `append(UploadInfo, InputStream)` and `removeLastNumberOfBytes(UploadInfo, long)` MUST NEVER resolve `duplicatesUploadId` to the parent; they must strictly operate on the target upload's own storage resources to prevent parent corruption. + +### 5.3 Implementing a Custom `ChecksumIndex` + +Storage backends implement the [`ChecksumIndex`](../src/main/java/me/desair/tus/server/upload/checksum/ChecksumIndex.java) interface to manage checksum index entries: + +```java +public interface ChecksumIndex { + boolean tryRegisterParent(ChecksumKey key, UploadId uploadId) throws IOException; + UploadId findParent(ChecksumKey key) throws IOException; + boolean isParentDeleted(ChecksumKey key) throws IOException; + void markParentDeleted(ChecksumKey key) throws IOException; + void addChild(ChecksumKey key, UploadId childId) throws IOException; + void removeChild(ChecksumKey key, UploadId childId) throws IOException; + boolean hasChildren(ChecksumKey key) throws IOException; + void deleteIndex(ChecksumKey key) throws IOException; + Set getAllIndexedKeys() throws IOException; +} +``` + +The storage backend instantiates a [`DeduplicationService`](../src/main/java/me/desair/tus/server/upload/checksum/DeduplicationService.java) passing its custom `ChecksumIndex`, `UploadStorageService`, and `UploadLockingService`. In `deduplicate(...)`, the storage service delegates to the deduplication service: + +```java +@Override +public UploadInfo deduplicate(UploadInfo uploadInfo, ChecksumKey checksumKey) throws IOException { + if (!isUploadDeduplicationEnabled() || uploadInfo == null || checksumKey == null) { + return uploadInfo; + } + return deduplicationService.deduplicate(uploadInfo, checksumKey); +} +``` diff --git a/docs/README.md b/docs/README.md index f1a1b1a0..b11d9266 100644 --- a/docs/README.md +++ b/docs/README.md @@ -21,6 +21,7 @@ - [Disk & network storage (NFS, EFS, SMB)](DISK_BASED_LOCKING.md) - [S3-compatible storage](S3_STORAGE.md) - [Azure Blob Storage](AZURE_BLOB_STORAGE.md) +- [Upload Deduplication & Migration](DEDUPLICATION.md) ## Contributors & maintainers - [Architecture Guide](ARCHITECTURE.md), [CONTRIBUTING](../CONTRIBUTING.md), [Testing guide](TESTING.md), [Conformity testing](CONFORMITY_TESTING.md), [Release process](RELEASE.md) diff --git a/docs/S3_STORAGE.md b/docs/S3_STORAGE.md index 8dc3222b..4e6b3a28 100644 --- a/docs/S3_STORAGE.md +++ b/docs/S3_STORAGE.md @@ -81,12 +81,14 @@ TusFileUploadService tusService = new TusFileUploadService() `S3StorageService` uses a clean, flat object key structure: ``` -/ # Final completed data object -/.info # JSON-serialized UploadInfo metadata -/.part # Incomplete sub-5MB part buffer -// # Deduplication checksum index object -/.lock # Lock lease object (JSON: holder + expiry) -/.stop # Cross-pod contention interrupt signal +/ # Final completed data object +/.info # JSON-serialized UploadInfo metadata +/.part # Incomplete sub-5MB part buffer +///parent # Deduplication parent pointer object +///parent-deleted # Tombstone marker if parent deleted +///children/ # Active duplicate child reference +/.lock # Lock lease object (JSON: holder + expiry) +/.stop # Cross-pod contention interrupt signal ``` ### Key Prefix Defaults diff --git a/scripts/rufh_conformity_test.py b/scripts/rufh_conformity_test.py index d973a588..89e53f0f 100755 --- a/scripts/rufh_conformity_test.py +++ b/scripts/rufh_conformity_test.py @@ -124,7 +124,7 @@ def http_request(method, url, headers=None, body=None, test_name=""): port = parsed.port or 80 path = parsed.path + ("?" + parsed.query if parsed.query else "") - s = socket.create_connection((host, port), timeout=5) + s = socket.create_connection((host, port), timeout=30) try: req_headers = CaseInsensitiveDict(headers) if "Connection" not in req_headers: @@ -1183,6 +1183,90 @@ def test_creation_with_invalid_content_digest(self, target_url, request): status, _, _, _ = http_request("POST", target_url, headers=headers, body=payload, test_name=request.node.name) assert status in (400, 409) + def test_content_digest_deduplication_and_parent_child_lifecycle(self, target_url, request): + """RFC 9530 & RUFH: Content-Digest deduplication multi-child lifecycle & tombstoned parent joining.""" + unique_token = str(time.time()).encode("ascii") + payload = b"RUFH RFC 9530 Multi-Child Deduplication Test Data: " + unique_token + digest_bytes = hashlib.sha256(payload).digest() + b64_digest = base64.b64encode(digest_bytes).decode("ascii") + content_digest = f"sha-256=:{b64_digest}:" + + headers = { + UPLOAD_COMPLETE: TRUE, + UPLOAD_LENGTH: str(len(payload)), + "Content-Digest": content_digest, + "Repr-Digest": content_digest, + CONTENT_TYPE: "text/plain", + } + + def make_abs(loc): + if not loc.startswith("http"): + parsed = urlparse(target_url) + return f"{parsed.scheme}://{parsed.netloc}{loc}" + return loc + + # 1. Create parent upload P + status, resp_headers, _, _ = http_request("POST", target_url, headers=headers, body=payload, test_name=request.node.name) + assert status in (200, 201) + parent_loc = make_abs(resp_headers.get(LOCATION)) + + # 2. Create child duplicate upload C1 + status2, resp2, _, _ = http_request("POST", target_url, headers=headers, body=payload, test_name=request.node.name) + assert status2 in (200, 201) + c1_loc = make_abs(resp2.get(LOCATION)) + + # 3. Terminate parent upload P (204) -> P is tombstoned while C1 remains + del_status, _, _, _ = http_request("DELETE", parent_loc, test_name=request.node.name) + assert del_status == 204 + + # Verify parent P is deleted (404) + p_head_status, _, _, _ = http_request("HEAD", parent_loc, test_name=request.node.name) + assert p_head_status in (404, 410) + + # 4. Create child duplicate upload C2 while parent P is tombstoned -> C2 joins tombstoned P + status3, resp3, _, _ = http_request("POST", target_url, headers=headers, body=payload, test_name=request.node.name) + assert status3 in (200, 201) + c2_loc = make_abs(resp3.get(LOCATION)) + + # 5. Verify GET on C1, C2 returns identical payload bytes + for loc in (c1_loc, c2_loc): + get_status, _, get_body, _ = http_request("GET", loc, test_name=request.node.name) + assert get_status == 200, f"GET {loc} must return 200, got {get_status}" + assert get_body == payload, "Child must stream full payload from shared parent data" + + # 6. Terminate child C1 (204) + del_status, _, _, _ = http_request("DELETE", c1_loc, test_name=request.node.name) + assert del_status == 204 + + # Verify C1 is deleted (404) + c1_head_status, _, _, _ = http_request("HEAD", c1_loc, test_name=request.node.name) + assert c1_head_status in (404, 410) + + # 7. Child C2 is STILL accessible via HEAD and GET + head_status, head_resp, _, _ = http_request("HEAD", c2_loc, test_name=request.node.name) + assert head_status in (200, 204) + assert head_resp.get(UPLOAD_OFFSET) == str(len(payload)) + get_status, _, get_body, _ = http_request("GET", c2_loc, test_name=request.node.name) + assert get_status == 200 + assert get_body == payload + + # 8. Terminate child C2. Last child out purges parent and checksum index. + del_status, _, _, _ = http_request("DELETE", c2_loc, test_name=request.node.name) + assert del_status == 204 + + # Verify C2 is deleted (404) + c2_head_status, _, _, _ = http_request("HEAD", c2_loc, test_name=request.node.name) + assert c2_head_status in (404, 410) + + # 9. Third upload with same content succeeds as fresh new parent + status4, resp4, _, _ = http_request("POST", target_url, headers=headers, body=payload, test_name=request.node.name) + assert status4 in (200, 201) + new_loc = make_abs(resp4.get(LOCATION)) + get_status, _, get_body, _ = http_request("GET", new_loc, test_name=request.node.name) + assert get_status == 200 + assert get_body == payload + http_request("DELETE", new_loc, test_name=request.node.name) + # CLI Entry Point & Custom Formatted Summary Reporter if __name__ == "__main__": diff --git a/scripts/tus_conformity_test.py b/scripts/tus_conformity_test.py index 5e09fa7c..f3fdb4cf 100755 --- a/scripts/tus_conformity_test.py +++ b/scripts/tus_conformity_test.py @@ -131,7 +131,7 @@ def http_request(method, url, headers=None, body=None): if not path: path = "/" - s = socket.create_connection((host, port), timeout=10) + s = socket.create_connection((host, port), timeout=30) try: req_headers = CaseInsensitiveDict(headers) if "Connection" not in req_headers: @@ -837,6 +837,164 @@ def test_patch_with_unsupported_checksum_algorithm(self, target_url): _, head_headers, _ = http_request("HEAD", upload_url, headers={TUS_RESUMABLE: TUS_API_VERSION}) assert head_headers.get(UPLOAD_OFFSET) == "0", "Upload-Offset MUST NOT be updated after invalid algorithm" + def test_creation_with_upload_valid_sha1_checksum(self, target_url): + """ + §7.4 Checksum & Creation-With-Upload: Valid SHA1 Checksum on Initial Creation. + A client creates an upload with initial body content and an Upload-Checksum header. + The server validates the payload and returns 201 Created with Upload-Offset matching length. + """ + payload = b"Creation with valid SHA1 checksum payload" + digest_bytes = hashlib.sha1(payload).digest() + b64_digest = base64.b64encode(digest_bytes).decode("ascii") + + extra_headers = { + UPLOAD_CHECKSUM: f"sha1 {b64_digest}", + } + loc, resp_headers = create_upload( + target_url, + upload_length=str(len(payload)), + content_type=APPLICATION_OFFSET_OCTET_STREAM, + extra_headers=extra_headers, + body=payload, + ) + assert resp_headers.get(UPLOAD_OFFSET) == str(len(payload)), "Upload-Offset MUST match payload length" + + # Verify via HEAD + status, head_headers, _ = http_request("HEAD", loc, headers={TUS_RESUMABLE: TUS_API_VERSION}) + assert status in (200, 204) + assert head_headers.get(UPLOAD_OFFSET) == str(len(payload)) + + def test_creation_with_upload_invalid_checksum_mismatch(self, target_url): + """ + §7.4 Checksum & Creation-With-Upload: Invalid Checksum Mismatch Returns 460. + A client provides a mismatched Upload-Checksum header on initial creation. + The server MUST reject the creation and return 460 Checksum Mismatch. + """ + payload = b"Creation with mismatching SHA1 checksum payload" + invalid_digest = base64.b64encode(b"0" * 20).decode("ascii") + + headers = { + TUS_RESUMABLE: TUS_API_VERSION, + UPLOAD_LENGTH: str(len(payload)), + CONTENT_TYPE: APPLICATION_OFFSET_OCTET_STREAM, + UPLOAD_CHECKSUM: f"sha1 {invalid_digest}", + } + status, _, _ = http_request("POST", target_url, headers=headers, body=payload) + assert status == 460, f"Checksum mismatch on creation MUST return 460, got {status}" + + def test_checksum_deduplication_and_parent_child_lifecycle(self, target_url): + """ + Checksum Deduplication Multi-Child Lifecycle & Tombstoned Parent Joining: + 1. Upload parent (P) with valid SHA1 checksum. + 2. Upload child 1 (C1) with identical content and SHA1 checksum (deduplicated against P). + 3. Delete parent P via DELETE (204) -> P is tombstoned, C1 preserved. + 4. Upload child 2 (C2) with identical content while P is tombstoned -> C2 joins tombstoned P. + 5. Verify GET on C1 and C2 returns 200 with identical uploaded bytes without duplication. + 6. Delete C1 via DELETE (204) -> C2 remains accessible via HEAD and GET. + 7. Delete C2 via DELETE (204). Last child out completely purges parent and checksum index. + 8. Upload new resource with same content/checksum succeeds as fresh new parent. + """ + unique_token = str(time.time()).encode("ascii") + payload = b"Multi-child deduplication test data payload: " + unique_token + digest_bytes = hashlib.sha1(payload).digest() + b64_digest = base64.b64encode(digest_bytes).decode("ascii") + checksum_header = f"sha1 {b64_digest}" + + # 1. Upload parent P + parent_url, parent_headers = create_upload( + target_url, + upload_length=str(len(payload)), + content_type=APPLICATION_OFFSET_OCTET_STREAM, + extra_headers={UPLOAD_CHECKSUM: checksum_header}, + body=payload, + ) + assert parent_headers.get(UPLOAD_OFFSET) == str(len(payload)) + + # 2. Upload duplicate child C1 + c1_url, c1_headers = create_upload( + target_url, + upload_length=str(len(payload)), + content_type=APPLICATION_OFFSET_OCTET_STREAM, + extra_headers={UPLOAD_CHECKSUM: checksum_header}, + body=payload, + ) + assert c1_headers.get(UPLOAD_OFFSET) == str(len(payload)) + + # 3. Delete parent P (204) -> P is tombstoned while C1 remains + del_status, _, _ = http_request("DELETE", parent_url, headers={TUS_RESUMABLE: TUS_API_VERSION}) + assert del_status == 204, f"Parent DELETE should return 204, got {del_status}" + + # Verify parent P is deleted (404) + p_head_status, _, _ = http_request("HEAD", parent_url, headers={TUS_RESUMABLE: TUS_API_VERSION}) + assert p_head_status in (404, 410) + + # Verify child C1 remains accessible via HEAD and GET + c1_head_status, c1_head_headers, _ = http_request("HEAD", c1_url, headers={TUS_RESUMABLE: TUS_API_VERSION}) + assert c1_head_status in (200, 204) + assert c1_head_headers.get(UPLOAD_OFFSET) == str(len(payload)) + get_status, _, get_body = http_request("GET", c1_url, headers={TUS_RESUMABLE: TUS_API_VERSION}) + assert get_status == 200 + assert get_body == payload + + # 4. Upload child C2 while parent P is tombstoned -> C2 joins tombstoned P + c2_url, c2_headers = create_upload( + target_url, + upload_length=str(len(payload)), + content_type=APPLICATION_OFFSET_OCTET_STREAM, + extra_headers={UPLOAD_CHECKSUM: checksum_header}, + body=payload, + ) + assert c2_headers.get(UPLOAD_OFFSET) == str(len(payload)) + + # 5. Both children C1 and C2 stream payload bytes + for child_url in (c1_url, c2_url): + head_status, head_headers, _ = http_request("HEAD", child_url, headers={TUS_RESUMABLE: TUS_API_VERSION}) + assert head_status in (200, 204) + assert head_headers.get(UPLOAD_OFFSET) == str(len(payload)) + get_status, _, get_body = http_request("GET", child_url, headers={TUS_RESUMABLE: TUS_API_VERSION}) + assert get_status == 200 + assert get_body == payload, "Child must stream full payload from shared parent data" + + # 6. Delete child C1 (204) + c1_del_status, _, _ = http_request("DELETE", c1_url, headers={TUS_RESUMABLE: TUS_API_VERSION}) + assert c1_del_status == 204 + + # Verify C1 is deleted (404) + c1_head_status, _, _ = http_request("HEAD", c1_url, headers={TUS_RESUMABLE: TUS_API_VERSION}) + assert c1_head_status in (404, 410) + + # 7. Child C2 is STILL accessible via HEAD and GET + c2_head_status, c2_head_headers, _ = http_request("HEAD", c2_url, headers={TUS_RESUMABLE: TUS_API_VERSION}) + assert c2_head_status in (200, 204) + assert c2_head_headers.get(UPLOAD_OFFSET) == str(len(payload)) + c2_get_status, _, c2_get_body = http_request("GET", c2_url, headers={TUS_RESUMABLE: TUS_API_VERSION}) + assert c2_get_status == 200 + assert c2_get_body == payload + + # 8. Delete child C2 (204). Last child out completely purges parent and checksum index. + c2_del_status, _, _ = http_request("DELETE", c2_url, headers={TUS_RESUMABLE: TUS_API_VERSION}) + assert c2_del_status == 204 + + # Verify C2 is deleted (404) + c2_head_status, _, _ = http_request("HEAD", c2_url, headers={TUS_RESUMABLE: TUS_API_VERSION}) + assert c2_head_status in (404, 410) + + # 9. Upload new resource with same content/checksum succeeds and creates clean new parent + new_url, new_headers = create_upload( + target_url, + upload_length=str(len(payload)), + content_type=APPLICATION_OFFSET_OCTET_STREAM, + extra_headers={UPLOAD_CHECKSUM: checksum_header}, + body=payload, + ) + assert new_headers.get(UPLOAD_OFFSET) == str(len(payload)) + get_status, _, get_body = http_request("GET", new_url, headers={TUS_RESUMABLE: TUS_API_VERSION}) + assert get_status == 200 + assert get_body == payload + + # Clean up + http_request("DELETE", new_url, headers={TUS_RESUMABLE: TUS_API_VERSION}) + class TestTerminationExtension: """Termination Extension compliance tests (§7.5).""" diff --git a/src/main/java/me/desair/tus/server/TusFileUploadService.java b/src/main/java/me/desair/tus/server/TusFileUploadService.java index a72ae52c..76422bc6 100644 --- a/src/main/java/me/desair/tus/server/TusFileUploadService.java +++ b/src/main/java/me/desair/tus/server/TusFileUploadService.java @@ -306,6 +306,7 @@ public TusFileUploadService withUploadStorageService(UploadStorageService upload this.uploadStorageService.isJsonSerializationEnabled()); uploadStorageService.setDrainTimeout(this.uploadStorageService.getDrainTimeout()); uploadStorageService.setIdFactory(this.idFactory); + uploadStorageService.setUploadLockingService(this.uploadLockingService); // Update the upload storage service this.uploadStorageService = uploadStorageService; prepareCacheIfEnabled(); @@ -353,8 +354,11 @@ public UploadStorageService getUploadStorageService() { public TusFileUploadService withUploadLockingService(UploadLockingService uploadLockingService) { Objects.requireNonNull(uploadLockingService, "The UploadLockingService cannot be null"); uploadLockingService.setIdFactory(this.idFactory); - // Update the upload storage service + // Update the upload locking service this.uploadLockingService = uploadLockingService; + if (this.uploadStorageService != null) { + this.uploadStorageService.setUploadLockingService(uploadLockingService); + } prepareCacheIfEnabled(); return this; } @@ -947,7 +951,6 @@ public void deleteUpload(String uploadUri, String ownerKey) throws IOException, UploadInfo uploadInfo = uploadStorageService.getUploadInfo(uploadUri, ownerKey); if (uploadInfo != null) { uploadStorageService.terminateUpload(uploadInfo); - cleanupLock(uploadInfo.getId()); } } } @@ -963,7 +966,7 @@ private UploadLock lockUploadByUri(String uploadUri) throws IOException, TusExce */ public void cleanup() throws IOException { uploadLockingService.cleanupStaleLocks(); - uploadStorageService.cleanupExpiredUploads(uploadLockingService); + uploadStorageService.cleanupExpiredUploads(); } protected UploadInfo processLockedRequest( @@ -1044,7 +1047,7 @@ protected void notifyUploadCompletionListeners(UploadInfo uploadInfo) { } private void cleanupLock(UploadId id) { - if (id != null && uploadLockingService != null) { + if (id != null) { try { uploadLockingService.cleanupLock(id); } catch (Exception e) { @@ -1176,7 +1179,7 @@ private void updateSupportedHttpMethods() { } private void prepareCacheIfEnabled() { - if (isThreadLocalCacheEnabled && uploadStorageService != null && uploadLockingService != null) { + if (isThreadLocalCacheEnabled && uploadStorageService != null) { ThreadLocalCachedStorageAndLockingService service = new ThreadLocalCachedStorageAndLockingService(uploadStorageService, uploadLockingService); service.setIdFactory(this.idFactory); @@ -1192,9 +1195,7 @@ private void prepareCacheIfEnabled() { */ @Override public void close() throws IOException { - if (uploadLockingService != null) { - uploadLockingService.close(); - } + uploadLockingService.close(); if (uploadStorageService != null) { uploadStorageService.close(); } diff --git a/src/main/java/me/desair/tus/server/checksum/ChecksumKey.java b/src/main/java/me/desair/tus/server/checksum/ChecksumKey.java new file mode 100644 index 00000000..3edfa40d --- /dev/null +++ b/src/main/java/me/desair/tus/server/checksum/ChecksumKey.java @@ -0,0 +1,111 @@ +package me.desair.tus.server.checksum; + +import java.io.Serializable; +import java.util.Locale; +import java.util.Objects; +import org.apache.commons.codec.binary.Base64; +import org.apache.commons.codec.binary.Hex; + +/** + * Immutable value object representing a normalized checksum key composed of a {@link + * ChecksumAlgorithm} and a canonical lowercase hexadecimal checksum string. + * + *

Standardizes incoming checksum strings (which may be provided as hex or Base64) to a safe, + * filesystem- and object-storage-friendly lowercase hex format, preventing directory traversal or + * invalid path characters across Linux, Windows, S3, and Azure Blob storage. + */ +public final class ChecksumKey implements Serializable { + + private static final long serialVersionUID = 1L; + + private final ChecksumAlgorithm algorithm; + private final String hexChecksum; + + private ChecksumKey(ChecksumAlgorithm algorithm, String hexChecksum) { + this.algorithm = algorithm; + this.hexChecksum = hexChecksum; + } + + /** + * Creates a normalized {@link ChecksumKey} from a raw checksum string and algorithm. + * + * @param checksum the raw checksum string (hex or Base64 encoded) + * @param algorithm the checksum algorithm used + * @return a canonical {@link ChecksumKey} + * @throws IllegalArgumentException if the checksum or algorithm is invalid, empty, or unsafe + */ + public static ChecksumKey of(String checksum, ChecksumAlgorithm algorithm) { + Objects.requireNonNull(algorithm, "Checksum algorithm must not be null"); + if (checksum == null || checksum.trim().isEmpty()) { + throw new IllegalArgumentException("Checksum string must not be null or blank"); + } + + String trimmed = checksum.trim(); + String normalizedHex; + + // Direct hexadecimal string detection + if (trimmed.matches("^[0-9a-fA-F]+$")) { + normalizedHex = trimmed.toLowerCase(Locale.ROOT); + } else if (Base64.isBase64(trimmed) && trimmed.matches("^[A-Za-z0-9+/=]+$")) { + // Decode Base64 encoded checksum values (common in HTTP Content-Digest headers) + byte[] decoded = Base64.decodeBase64(trimmed); + if (decoded != null && decoded.length > 0) { + normalizedHex = Hex.encodeHexString(decoded).toLowerCase(Locale.ROOT); + } else { + throw new IllegalArgumentException( + "The checksum value is not valid hexadecimal or Base64: " + checksum); + } + } else { + throw new IllegalArgumentException( + "The checksum value is not valid hexadecimal or Base64: " + checksum); + } + + // Guard against path traversal characters, slashes, or non-hex characters + if (!normalizedHex.matches("^[0-9a-f]+$")) { + throw new IllegalArgumentException( + "The normalized checksum contains invalid characters: " + normalizedHex); + } + + return new ChecksumKey(algorithm, normalizedHex); + } + + /** + * Returns the checksum algorithm. + * + * @return the {@link ChecksumAlgorithm} + */ + public ChecksumAlgorithm getAlgorithm() { + return algorithm; + } + + /** + * Returns the canonical lowercase hexadecimal checksum string. + * + * @return the normalized hex checksum + */ + public String getHexChecksum() { + return hexChecksum; + } + + @Override + public boolean equals(Object o) { + if (this == o) { + return true; + } + if (!(o instanceof ChecksumKey)) { + return false; + } + ChecksumKey that = (ChecksumKey) o; + return algorithm == that.algorithm && Objects.equals(hexChecksum, that.hexChecksum); + } + + @Override + public int hashCode() { + return Objects.hash(algorithm, hexChecksum); + } + + @Override + public String toString() { + return algorithm.getTusName() + "/" + hexChecksum; + } +} diff --git a/src/main/java/me/desair/tus/server/checksum/ChecksumRequestHandler.java b/src/main/java/me/desair/tus/server/checksum/ChecksumRequestHandler.java index 17ce0e99..ae823fcd 100644 --- a/src/main/java/me/desair/tus/server/checksum/ChecksumRequestHandler.java +++ b/src/main/java/me/desair/tus/server/checksum/ChecksumRequestHandler.java @@ -57,16 +57,27 @@ public void process( + calculatedValue + " with algorithm " + checksumAlgorithm); - } else if (uploadStorageService.isUploadDeduplicationEnabled()) { + } else { UploadInfo uploadInfo = uploadStorageService.getUploadInfo( Utils.getUploadUri(servletRequest, servletResponse), ownerKey); - if (uploadInfo != null - && !uploadInfo.isUploadInProgress() - && uploadInfo.getDuplicatesUploadId() == null) { + if (uploadInfo != null && !uploadInfo.isUploadInProgress()) { uploadInfo.setChecksum(expectedValue); uploadInfo.setChecksumAlgorithm(checksumAlgorithm); - uploadStorageService.update(uploadInfo); + if (uploadStorageService.isUploadDeduplicationEnabled() + && uploadInfo.getDuplicatesUploadId() == null) { + try { + ChecksumKey key = ChecksumKey.of(expectedValue, checksumAlgorithm); + UploadInfo deduped = uploadStorageService.deduplicate(uploadInfo, key); + if (deduped != null) { + uploadInfo = deduped; + } + } catch (Exception e) { + uploadStorageService.update(uploadInfo); + } + } else { + uploadStorageService.update(uploadInfo); + } } } } diff --git a/src/main/java/me/desair/tus/server/core/CorePatchRequestHandler.java b/src/main/java/me/desair/tus/server/core/CorePatchRequestHandler.java index d234c881..ee58dcff 100644 --- a/src/main/java/me/desair/tus/server/core/CorePatchRequestHandler.java +++ b/src/main/java/me/desair/tus/server/core/CorePatchRequestHandler.java @@ -45,7 +45,15 @@ public void process( UploadStorageService uploadStorageService, String ownerKey) throws IOException, TusException { - process(method, servletRequest, servletResponse, uploadStorageService, null, ownerKey, null); + UploadLockingService lockingService = Utils.getLockingServiceOrDefault(uploadStorageService); + process( + method, + servletRequest, + servletResponse, + uploadStorageService, + lockingService, + ownerKey, + null); } @Override @@ -70,29 +78,13 @@ public HttpProblemDetails process( try { InputStream stream = servletRequest.getContentInputStream(); - // If a locking service is provided, wrap the input stream in an InterruptibleInputStream - // and register it with the locking service to allow for interruption of the upload. - if (lockingService != null) { - interruptibleStream = new InterruptibleInputStream(stream); - lockingService.registerInputStream(servletRequest.getRequestURI(), interruptibleStream); - stream = interruptibleStream; - } + // Wrap the input stream in an InterruptibleInputStream and register it with the locking + // service to allow for clean interruption during lock contention or cancellation. + interruptibleStream = new InterruptibleInputStream(stream); + lockingService.registerInputStream(servletRequest.getRequestURI(), interruptibleStream); + stream = interruptibleStream; uploadInfo = uploadStorageService.append(uploadInfo, stream); - - if (uploadStorageService.isUploadDeduplicationEnabled() - && servletRequest.hasCalculatedChecksum()) { - Utils.ChecksumInfo checksumInfo = Utils.parseUploadChecksumHeader(servletRequest); - if (checksumInfo != null) { - UploadInfo duplicateInfo = - uploadStorageService.getUploadInfoByChecksum( - checksumInfo.getValue(), checksumInfo.getAlgorithm()); - if (duplicateInfo != null) { - uploadInfo.setDuplicatesUploadId(duplicateInfo.getId()); - uploadStorageService.update(uploadInfo); - } - } - } } catch (UploadNotFoundException e) { found = false; } catch (IOException e) { diff --git a/src/main/java/me/desair/tus/server/creationwithupload/CreationWithUploadPostRequestHandler.java b/src/main/java/me/desair/tus/server/creationwithupload/CreationWithUploadPostRequestHandler.java index 61b0bb39..ae253009 100644 --- a/src/main/java/me/desair/tus/server/creationwithupload/CreationWithUploadPostRequestHandler.java +++ b/src/main/java/me/desair/tus/server/creationwithupload/CreationWithUploadPostRequestHandler.java @@ -46,12 +46,9 @@ public HttpProblemDetails process( UploadInfo uploadInfo = uploadStorageService.getUploadInfo(location, ownerKey); if (uploadInfo != null && uploadInfo.isUploadInProgress()) { InputStream stream = servletRequest.getContentInputStream(); - InterruptibleInputStream interruptibleStream = null; - if (uploadLockingService != null) { - interruptibleStream = new InterruptibleInputStream(stream); - uploadLockingService.registerInputStream(location, interruptibleStream); - stream = interruptibleStream; - } + InterruptibleInputStream interruptibleStream = new InterruptibleInputStream(stream); + uploadLockingService.registerInputStream(location, interruptibleStream); + stream = interruptibleStream; try { uploadInfo = uploadStorageService.append(uploadInfo, stream); diff --git a/src/main/java/me/desair/tus/server/digest/HttpDigestsPostPutPatchRequestHandler.java b/src/main/java/me/desair/tus/server/digest/HttpDigestsPostPutPatchRequestHandler.java index 1ddc897e..0fd677f0 100644 --- a/src/main/java/me/desair/tus/server/digest/HttpDigestsPostPutPatchRequestHandler.java +++ b/src/main/java/me/desair/tus/server/digest/HttpDigestsPostPutPatchRequestHandler.java @@ -5,11 +5,11 @@ import java.security.DigestInputStream; import java.security.MessageDigest; import java.util.Map; -import java.util.Objects; import me.desair.tus.server.HttpHeader; import me.desair.tus.server.HttpMethod; import me.desair.tus.server.HttpProblemDetails; import me.desair.tus.server.checksum.ChecksumAlgorithm; +import me.desair.tus.server.checksum.ChecksumKey; import me.desair.tus.server.digest.validation.HttpDigestsValidator; import me.desair.tus.server.exception.TusException; import me.desair.tus.server.exception.UploadDigestMismatchException; @@ -78,7 +78,8 @@ public HttpProblemDetails process( captureDigestPreferences(servletRequest, uploadInfo, uploadStorageService); // 3. Verify Repr-Digest on upload completion - verifyReprDigestOnCompletion(uploadInfo, uploadUri, ownerKey, uploadStorageService); + verifyReprDigestOnCompletion( + servletRequest, uploadInfo, uploadUri, ownerKey, uploadStorageService); // 4. Provide Repr-Digest response header if requested addReprDigestResponseHeader( @@ -144,45 +145,54 @@ private void captureDigestPreferences( } private void verifyReprDigestOnCompletion( + TusServletRequest servletRequest, UploadInfo uploadInfo, String uploadUri, String ownerKey, UploadStorageService uploadStorageService) throws IOException, TusException { - if (!uploadInfo.isUploadInProgress() && uploadInfo.getRepresentationDigest() != null) { - Map expectedDigests = - ChecksumAlgorithm.parseDigestHeader(uploadInfo.getRepresentationDigest()); - - for (Map.Entry entry : expectedDigests.entrySet()) { - ChecksumAlgorithm alg = entry.getKey(); - String expectedValue = entry.getValue(); - String calculatedValue = - calculateEntireFileDigest(uploadUri, ownerKey, alg, uploadStorageService); - - if (calculatedValue != null) { - if (!Strings.CS.equals(expectedValue, calculatedValue)) { - throw new UploadDigestMismatchException( - "Repr-Digest mismatch for algorithm " - + alg.getHttpDigestNames().get(0) - + ". Expected: " - + expectedValue - + " but was: " - + calculatedValue); - } else { - // Deduplication support: If verification succeeds and deduplication is enabled - if (uploadStorageService.isUploadDeduplicationEnabled() - && uploadInfo.getDuplicatesUploadId() == null) { - UploadInfo duplicateInfo = - uploadStorageService.getUploadInfoByChecksum(expectedValue, alg); - if (duplicateInfo != null - && !Objects.equals(duplicateInfo.getId(), uploadInfo.getId())) { - uploadInfo.setDuplicatesUploadId(duplicateInfo.getId()); - uploadStorageService.update(uploadInfo); + if (!uploadInfo.isUploadInProgress()) { + String reprDigest = uploadInfo.getRepresentationDigest(); + if (reprDigest == null && servletRequest != null) { + reprDigest = servletRequest.getHeader(HttpHeader.CONTENT_DIGEST); + } + if (reprDigest != null) { + Map expectedDigests = + ChecksumAlgorithm.parseDigestHeader(reprDigest); + + for (Map.Entry entry : expectedDigests.entrySet()) { + ChecksumAlgorithm alg = entry.getKey(); + String expectedValue = entry.getValue(); + String calculatedValue = + calculateEntireFileDigest(uploadUri, ownerKey, alg, uploadStorageService); + + if (calculatedValue != null) { + if (!Strings.CS.equals(expectedValue, calculatedValue)) { + throw new UploadDigestMismatchException( + "Repr-Digest mismatch for algorithm " + + alg.getHttpDigestNames().get(0) + + ". Expected: " + + expectedValue + + " but was: " + + calculatedValue); + } else { + // Always save verified checksum on the uploadInfo + uploadInfo.setChecksum(expectedValue); + uploadInfo.setChecksumAlgorithm(alg); + + if (uploadStorageService.isUploadDeduplicationEnabled() + && uploadInfo.getDuplicatesUploadId() == null) { + try { + ChecksumKey key = ChecksumKey.of(expectedValue, alg); + UploadInfo deduped = uploadStorageService.deduplicate(uploadInfo, key); + if (deduped != null) { + uploadInfo = deduped; + } + } catch (Exception e) { + uploadStorageService.update(uploadInfo); + } } else { - // Index this file - uploadInfo.setChecksum(expectedValue); - uploadInfo.setChecksumAlgorithm(alg); uploadStorageService.update(uploadInfo); } } diff --git a/src/main/java/me/desair/tus/server/download/DownloadGetRequestHandler.java b/src/main/java/me/desair/tus/server/download/DownloadGetRequestHandler.java index 0dbff814..78f2ef3b 100644 --- a/src/main/java/me/desair/tus/server/download/DownloadGetRequestHandler.java +++ b/src/main/java/me/desair/tus/server/download/DownloadGetRequestHandler.java @@ -65,9 +65,8 @@ public void process( .replace("+", "%20"))); servletResponse.setHeader(HttpHeader.CONTENT_TYPE, info.getFileMimeType()); + servletResponse.setStatus(HttpServletResponse.SC_OK); uploadStorageService.copyUploadTo(info, servletResponse.getOutputStream()); - - servletResponse.setStatus(HttpServletResponse.SC_OK); } } diff --git a/src/main/java/me/desair/tus/server/expiration/ExpirationRequestHandler.java b/src/main/java/me/desair/tus/server/expiration/ExpirationRequestHandler.java index 84e2230a..6e8df0bc 100644 --- a/src/main/java/me/desair/tus/server/expiration/ExpirationRequestHandler.java +++ b/src/main/java/me/desair/tus/server/expiration/ExpirationRequestHandler.java @@ -58,9 +58,6 @@ public HttpProblemDetails process( if (expirationPeriod != null && expirationPeriod > 0 && uploadInfo != null) { if (uploadInfo.isExpired()) { uploadStorageService.terminateUpload(uploadInfo); - if (uploadLockingService != null) { - uploadLockingService.cleanupLock(uploadInfo.getId()); - } throw new UploadNotFoundException( "The upload for path " + uploadUri + " has expired and was removed."); } else { diff --git a/src/main/java/me/desair/tus/server/rufh/handler/RufhAppendPatchRequestHandler.java b/src/main/java/me/desair/tus/server/rufh/handler/RufhAppendPatchRequestHandler.java index 0d889b2d..f728a31e 100644 --- a/src/main/java/me/desair/tus/server/rufh/handler/RufhAppendPatchRequestHandler.java +++ b/src/main/java/me/desair/tus/server/rufh/handler/RufhAppendPatchRequestHandler.java @@ -72,11 +72,9 @@ public HttpProblemDetails process( InputStream is = servletRequest.getContentInputStream(); InterruptibleInputStream interruptibleStream = null; if (is != null) { - if (uploadLockingService != null) { - interruptibleStream = new InterruptibleInputStream(is); - uploadLockingService.registerInputStream(requestUri, interruptibleStream); - is = interruptibleStream; - } + interruptibleStream = new InterruptibleInputStream(is); + uploadLockingService.registerInputStream(requestUri, interruptibleStream); + is = interruptibleStream; try { UploadInfo appended = uploadStorageService.append(uploadInfo, is); if (appended != null) { diff --git a/src/main/java/me/desair/tus/server/rufh/handler/RufhCreationPostRequestHandler.java b/src/main/java/me/desair/tus/server/rufh/handler/RufhCreationPostRequestHandler.java index aa46f7db..df16b37f 100644 --- a/src/main/java/me/desair/tus/server/rufh/handler/RufhCreationPostRequestHandler.java +++ b/src/main/java/me/desair/tus/server/rufh/handler/RufhCreationPostRequestHandler.java @@ -115,11 +115,9 @@ public HttpProblemDetails process( InputStream is = servletRequest.getContentInputStream(); InterruptibleInputStream interruptibleStream = null; if (is != null && hasContent) { - if (uploadLockingService != null) { - interruptibleStream = new InterruptibleInputStream(is); - uploadLockingService.registerInputStream(uploadUri, interruptibleStream); - is = interruptibleStream; - } + interruptibleStream = new InterruptibleInputStream(is); + uploadLockingService.registerInputStream(uploadUri, interruptibleStream); + is = interruptibleStream; try { UploadInfo appended = uploadStorageService.append(uploadInfo, is); if (appended != null) { diff --git a/src/main/java/me/desair/tus/server/rufh/handler/RufhDeleteRequestHandler.java b/src/main/java/me/desair/tus/server/rufh/handler/RufhDeleteRequestHandler.java index f086036e..35b9f2cc 100644 --- a/src/main/java/me/desair/tus/server/rufh/handler/RufhDeleteRequestHandler.java +++ b/src/main/java/me/desair/tus/server/rufh/handler/RufhDeleteRequestHandler.java @@ -46,9 +46,6 @@ public HttpProblemDetails process( } uploadStorageService.terminateUpload(uploadInfo); - if (uploadLockingService != null) { - uploadLockingService.cleanupLock(uploadInfo.getId()); - } servletResponse.setStatus(204); return null; } diff --git a/src/main/java/me/desair/tus/server/termination/TerminationDeleteRequestHandler.java b/src/main/java/me/desair/tus/server/termination/TerminationDeleteRequestHandler.java index 25c2a7b0..0f4df363 100644 --- a/src/main/java/me/desair/tus/server/termination/TerminationDeleteRequestHandler.java +++ b/src/main/java/me/desair/tus/server/termination/TerminationDeleteRequestHandler.java @@ -41,9 +41,6 @@ public HttpProblemDetails process( if (uploadInfo != null) { uploadStorageService.terminateUpload(uploadInfo); - if (uploadLockingService != null) { - uploadLockingService.cleanupLock(uploadInfo.getId()); - } } servletResponse.setStatus(HttpServletResponse.SC_NO_CONTENT); diff --git a/src/main/java/me/desair/tus/server/upload/AbstractLeaseLockingService.java b/src/main/java/me/desair/tus/server/upload/AbstractLeaseLockingService.java index 521c3cc0..ac14ac8e 100644 --- a/src/main/java/me/desair/tus/server/upload/AbstractLeaseLockingService.java +++ b/src/main/java/me/desair/tus/server/upload/AbstractLeaseLockingService.java @@ -110,6 +110,20 @@ public UploadLock lockUploadByUri(String requestUri) throws TusException, IOExce return lock; } + @Override + public UploadLock lockUpload(UploadId uploadId) throws TusException, IOException { + if (uploadId == null) { + return null; + } + + String holderId = UUID.randomUUID().toString(); + UploadLock lock = acquireOrEvictExpiredLock(uploadId, holderId, uploadId.toString()); + if (lock == null) { + throw new UploadAlreadyLockedException("Upload with ID " + uploadId + " is currently locked"); + } + return lock; + } + @Override public boolean isLocked(UploadId id) { if (id == null) { diff --git a/src/main/java/me/desair/tus/server/upload/NoOpUploadLockingService.java b/src/main/java/me/desair/tus/server/upload/NoOpUploadLockingService.java new file mode 100644 index 00000000..6ba4cd02 --- /dev/null +++ b/src/main/java/me/desair/tus/server/upload/NoOpUploadLockingService.java @@ -0,0 +1,74 @@ +package me.desair.tus.server.upload; + +import java.io.IOException; +import java.io.InputStream; +import me.desair.tus.server.exception.TusException; + +/** + * A no-op implementation of {@link UploadLockingService} following the Null Object Pattern. + * + *

All locking operations return a no-op {@link UploadLock} handle, and lifecycle/registration + * methods execute as no-ops. Useful for unit tests or specialized embedding environments where + * concurrency control is managed externally. + */ +public class NoOpUploadLockingService implements UploadLockingService { + + private static final UploadLock NO_OP_LOCK = + new UploadLock() { + @Override + public String getUploadUri() { + return null; + } + + @Override + public void release() {} + + @Override + public void close() {} + }; + + @Override + public UploadLock lockUploadByUri(String requestURI) throws TusException, IOException { + return NO_OP_LOCK; + } + + @Override + public UploadLock lockUpload(UploadId id) throws TusException, IOException { + return NO_OP_LOCK; + } + + @Override + public void cleanupStaleLocks() throws IOException { + // No-op + } + + @Override + public boolean isLocked(UploadId id) { + return false; + } + + @Override + public void cleanupLock(UploadId id) throws IOException { + // No-op + } + + @Override + public void registerInputStream(String requestUri, InputStream inputStream) { + // No-op + } + + @Override + public void requestLockRelease(String requestUri) { + // No-op + } + + @Override + public void setIdFactory(UploadIdFactory idFactory) { + // No-op + } + + @Override + public void close() throws IOException { + // No-op + } +} diff --git a/src/main/java/me/desair/tus/server/upload/UploadLockingService.java b/src/main/java/me/desair/tus/server/upload/UploadLockingService.java index d97fb701..8929f432 100644 --- a/src/main/java/me/desair/tus/server/upload/UploadLockingService.java +++ b/src/main/java/me/desair/tus/server/upload/UploadLockingService.java @@ -1,6 +1,7 @@ package me.desair.tus.server.upload; import java.io.IOException; +import java.io.InputStream; import me.desair.tus.server.exception.TusException; /** @@ -18,6 +19,21 @@ public interface UploadLockingService { */ UploadLock lockUploadByUri(String requestUri) throws TusException, IOException; + /** + * Lock the upload resource associated with the given {@link UploadId}. + * + * @param id The upload ID to lock + * @return The lock on the upload, or null if no lock was applied + * @throws TusException If the upload is already locked + * @throws IOException If locking fails + */ + default UploadLock lockUpload(UploadId id) throws TusException, IOException { + if (id == null) { + return null; + } + return lockUploadByUri(id.toString()); + } + /** * Clean up any stale locks that are still present. * @@ -48,7 +64,7 @@ public interface UploadLockingService { * @param requestUri The request URI of the active request * @param inputStream The input stream of the active request */ - default void registerInputStream(String requestUri, java.io.InputStream inputStream) { + default void registerInputStream(String requestUri, InputStream inputStream) { // No-op by default for backwards compatibility } diff --git a/src/main/java/me/desair/tus/server/upload/UploadStorageService.java b/src/main/java/me/desair/tus/server/upload/UploadStorageService.java index 5f168ba4..88acbe4d 100644 --- a/src/main/java/me/desair/tus/server/upload/UploadStorageService.java +++ b/src/main/java/me/desair/tus/server/upload/UploadStorageService.java @@ -5,11 +5,70 @@ import java.io.OutputStream; import java.time.Duration; import me.desair.tus.server.checksum.ChecksumAlgorithm; +import me.desair.tus.server.checksum.ChecksumKey; import me.desair.tus.server.exception.TusException; import me.desair.tus.server.exception.UploadNotFoundException; import me.desair.tus.server.upload.concatenation.UploadConcatenationService; - -/** Interface to a service that is able to store the (partially) uploaded files. */ +import me.desair.tus.server.util.Utils; + +/** + * Service Provider Interface (SPI) for storing, appending, retrieving, and terminating resumable + * uploads and their associated metadata. + * + *

Implementations of this interface manage the persistence of binary payload data and {@link + * UploadInfo} metadata across local filesystems ({@code DiskStorageService}), S3-compatible object + * storage ({@code S3StorageService}), Azure Blob Storage ({@code AzureBlobStorageService}), or + * custom storage engines. + * + *

1. Upload Lifecycle

+ * + *
    + *
  • Creation: {@link #create(UploadInfo, String)} allocates initial storage resources + * and persists the initial metadata record. + *
  • Transfer: {@link #append(UploadInfo, InputStream)} streams and appends chunk bytes. + * If an append is interrupted (e.g. client disconnect, JVM shutdown, or lock contention), the + * service MUST persist all bytes received up to the interruption point, update the offset in + * {@link UploadInfo}, and re-throw the exception. + *
  • Retrieval: {@link #getUploadInfo(UploadId)} and {@link #getUploadInfo(String, + * String)} retrieve metadata. {@link #getUploadedBytes(UploadId)} and {@link + * #copyUploadTo(UploadInfo, OutputStream)} stream committed data bytes. + *
  • Termination: {@link #terminateUpload(UploadInfo)} is the authoritative method to + * free physical payload bytes, remove metadata, clean up locks, and coordinate deduplication + * index cleanup. + *
+ * + *

2. Mandatory Locking Service Integration

+ * + *

Concurrency control is mandatory across all storage backends. Every storage service receives + * an {@link UploadLockingService} instance via {@link + * #setUploadLockingService(UploadLockingService)} and exposes it via {@link + * #getUploadLockingService()}. Implementations MUST retain this reference and delegate lock cleanup + * during {@link #terminateUpload(UploadInfo)} and {@link #cleanupExpiredUploads()}. + * + *

3. Checksum Deduplication Support

+ * + *

When upload deduplication is enabled via {@link #setUploadDeduplicationEnabled(boolean)}, + * implementations MUST implement the following critical methods: + * + *

    + *
  • {@link #deduplicate(UploadInfo, ChecksumKey)}: Coordinates cluster-safe deduplication when + * an upload completes, typically delegating to {@link + * me.desair.tus.server.upload.checksum.DeduplicationService}. + *
  • {@link #dataExists(UploadId)}: Checks whether committed physical data exists in storage. + * Used by {@link me.desair.tus.server.upload.checksum.DeduplicationService} to self-heal + * stale checksum index entries when parent data has been deleted out-of-band. + *
  • {@link #deleteData(UploadId)}: Deletes physical committed data without deleting the + * upload's {@link UploadInfo} metadata record. Used to free storage for child duplicates + * linked to a parent, and to purge retained parent data when the last active child + * terminates. + *
  • Read/Write Safety Rules: Read operations ({@link #getUploadedBytes(UploadId)}, + * {@link #copyUploadTo(UploadInfo, OutputStream)}) MUST transparently resolve {@link + * UploadInfo#getDuplicatesUploadId()} to stream from the parent upload. Mutation operations + * ({@link #append(UploadInfo, InputStream)}, {@link #removeLastNumberOfBytes(UploadInfo, + * long)}) MUST NEVER resolve duplicates to the parent; they must strictly operate on + * child-owned storage. + *
+ */ public interface UploadStorageService { /** @@ -47,6 +106,10 @@ public interface UploadStorageService { * storage service MUST cleanly persist all bytes received up to the interruption point, update * the {@link UploadInfo} offset and metadata accordingly, and re-throw the exception. * + *

Deduplication Safety: When writing or appending bytes, implementations MUST strictly + * operate on the target upload's own physical storage files/objects and MUST NEVER resolve {@link + * UploadInfo#getDuplicatesUploadId()} to modify parent storage. + * * @param upload The ID of the upload * @param inputStream The input stream containing the bytes to append * @return The new {@link UploadInfo} for this upload @@ -109,9 +172,13 @@ default Long getMaxAppendSize() { /** * Get the uploaded bytes corresponding to the given upload URL as a stream. * + *

Deduplication Contract: If the upload is a child upload linked to a parent ({@link + * UploadInfo#getDuplicatesUploadId()} is non-null), implementations MUST dynamically stream data + * bytes from the parent upload's storage location. + * * @param uploadUri The URI * @param ownerKey The owner key of this upload - * @return an {@link OutputStream} containing the bytes of the upload + * @return an {@link InputStream} containing the bytes of the upload */ InputStream getUploadedBytes(String uploadUri, String ownerKey) throws IOException, UploadNotFoundException; @@ -119,33 +186,66 @@ InputStream getUploadedBytes(String uploadUri, String ownerKey) /** * Get the uploaded bytes corresponding to the given upload ID as a stream. * + *

Deduplication Contract: If the upload is a child upload linked to a parent ({@link + * UploadInfo#getDuplicatesUploadId()} is non-null), implementations MUST dynamically stream data + * bytes from the parent upload's storage location. + * * @param id the ID of the upload - * @return an {@link OutputStream} containing the bytes of the upload + * @return an {@link InputStream} containing the bytes of the upload * @throws IOException When retrieving the bytes from the storage layer fails - * @throws UploadNotFoundException When the proved id is not linked to an upload + * @throws UploadNotFoundException When the provided id is not linked to an upload */ InputStream getUploadedBytes(UploadId id) throws IOException, UploadNotFoundException; /** * Copy the uploaded bytes to the given output stream. * + *

Deduplication Contract: If the upload is a child upload linked to a parent ({@link + * UploadInfo#getDuplicatesUploadId()} is non-null), implementations MUST dynamically copy bytes + * from the parent upload's storage location. + * * @param info The upload of which we should copy the bytes * @param outputStream The output stream where we have to copy the bytes to + * @throws UploadNotFoundException When the provided upload is not found in storage + * @throws IOException When reading or writing bytes fails */ void copyUploadTo(UploadInfo info, OutputStream outputStream) throws UploadNotFoundException, IOException; + /** + * Clean up any upload data that is expired according to the configured expiration time. + * + *

Storage implementations should use the {@link UploadLockingService} provided via {@link + * #setUploadLockingService(UploadLockingService)} to check lock status and coordinate lock + * cleanup. + * + * @throws IOException When an I/O error occurs + */ + default void cleanupExpiredUploads() throws IOException { + cleanupExpiredUploads(null); + } + /** * Clean up any upload data that is expired according to the configured expiration time. * * @param uploadLockingService An {@link UploadLockingService} that can be used to check and lock * uploads + * @throws IOException When an I/O error occurs + * @deprecated As of release 2.0.0, replaced by {@link #cleanupExpiredUploads()} in conjunction + * with {@link #setUploadLockingService(UploadLockingService)}. */ - void cleanupExpiredUploads(UploadLockingService uploadLockingService) throws IOException; + @Deprecated + default void cleanupExpiredUploads(UploadLockingService uploadLockingService) throws IOException { + setUploadLockingService(Utils.getLockingServiceOrDefault(uploadLockingService)); + } /** * Remove the given last amount of bytes from the uploaded data. * + *

Deduplication Safety: When truncating bytes, implementations MUST strictly operate on + * the target upload's own physical storage files/objects and MUST NEVER resolve {@link + * UploadInfo#getDuplicatesUploadId()} to truncate parent storage. + * * @param uploadInfo Upload of which to remove the bytes * @param byteCount The number of bytes to remove at the end */ @@ -155,7 +255,21 @@ void removeLastNumberOfBytes(UploadInfo uploadInfo, long byteCount) /** * Terminate completed and unfinished uploads allowing the Server to free up used resources. * + *

Implementations must: + * + *

    + *
  • Delete physical committed data bytes and any temporary chunk buffers for this upload. + *
  • Delete the metadata record ({@code .info}) for this upload. + *
  • If upload deduplication is enabled, coordinate index removal or tombstoning via {@link + * me.desair.tus.server.upload.checksum.DeduplicationService#removeUploadFromIndex(UploadInfo)}. + *
  • Clean up any associated upload lock or contention artifacts (e.g. by delegating to {@link + * UploadLockingService#cleanupLock(UploadId)}) to ensure no orphaned lock files, stop + * signals, or lease objects remain in the locking layer. + *
+ * * @param uploadInfo The upload to terminate + * @throws UploadNotFoundException When the provided upload is not found in storage + * @throws IOException When removing data or metadata from the storage layer fails */ void terminateUpload(UploadInfo uploadInfo) throws UploadNotFoundException, IOException; @@ -198,6 +312,11 @@ void removeLastNumberOfBytes(UploadInfo uploadInfo, long byteCount) /** * Set whether duplicate file processing based on checksum is enabled. * + *

When enabled via {@code TusFileUploadService.withUploadDeduplication(true)}, completed + * uploads are deduplicated against existing parent uploads with identical checksums. Custom + * storage engines supporting deduplication must store this flag and coordinate deduplication in + * {@link #deduplicate(UploadInfo, ChecksumKey)}. + * * @param enabled True if enabled, false otherwise */ default void setUploadDeduplicationEnabled(boolean enabled) { @@ -219,12 +338,116 @@ default boolean isUploadDeduplicationEnabled() { * @param checksum The checksum value to search for * @param algorithm The checksum algorithm used * @return The UploadInfo of the existing completed upload, or null if not found + * @deprecated Use {@link #deduplicate(UploadInfo, ChecksumKey)} instead. */ + @Deprecated default UploadInfo getUploadInfoByChecksum(String checksum, ChecksumAlgorithm algorithm) throws IOException { return null; } + /** + * Performs thread-safe and cluster-safe deduplication for a completed upload using its normalized + * checksum key. + * + *

Critical for Deduplication: Implementations supporting upload deduplication MUST + * override this method (typically delegating to an instance of {@link + * me.desair.tus.server.upload.checksum.DeduplicationService} backed by a backend-specific {@link + * me.desair.tus.server.upload.checksum.ChecksumIndex}). When invoked upon upload completion, this + * method: + * + *

    + *
  • Checks if a completed parent upload with matching checksum already exists. + *
  • If a parent exists, links this upload as a child via {@link + * UploadInfo#setDuplicatesUploadId(UploadId)} and frees the child's physical data via + * {@link #deleteData(UploadId)}. + *
  • If no parent exists, atomically registers this upload as the parent in the checksum index + * using compare-and-swap (CAS) semantics. + *
+ * + *

The default implementation returns {@code uploadInfo} unchanged (no deduplication performed) + * for backward compatibility. + * + * @param uploadInfo the upload info of the completed upload + * @param checksumKey the normalized checksum key + * @return the resulting {@link UploadInfo} (either deduplicated or registered as parent) + * @throws IOException on storage I/O error + */ + default UploadInfo deduplicate(UploadInfo uploadInfo, ChecksumKey checksumKey) + throws IOException { + return uploadInfo; + } + + /** + * Sets the {@link UploadLockingService} to coordinate locking and lock cleanup. + * + *

Mandatory Locking Integration: An {@link UploadLockingService} is mandatory across + * all storage backends. Implementations MUST retain this reference and invoke {@link + * UploadLockingService#cleanupLock(UploadId)} during {@link #terminateUpload(UploadInfo)} and + * coordinate lock state during {@link #cleanupExpiredUploads()}. + * + * @param uploadLockingService the locking service instance (must not be {@code null}) + */ + default void setUploadLockingService(UploadLockingService uploadLockingService) { + java.util.Objects.requireNonNull(uploadLockingService, "uploadLockingService must not be null"); + } + + /** + * Gets the configured {@link UploadLockingService}. + * + * @return the locking service, or {@code null} if none configured + */ + default UploadLockingService getUploadLockingService() { + return null; + } + + /** + * Checks whether committed data bytes exist in storage for the specified upload ID. + * + *

Critical for Deduplication: Implementations supporting upload deduplication MUST + * override this method to accurately verify that physical data bytes exist in storage. The {@link + * me.desair.tus.server.upload.checksum.DeduplicationService} invokes this check during parent + * lookup to self-heal stale checksum index entries when parent data has been deleted out-of-band + * (e.g., via storage lifecycle policies or manual deletion). + * + *

The default implementation returns {@code false} for backward compatibility. If not + * overridden in a deduplication-enabled backend, parent uploads will be treated as missing, + * preventing duplicate linking. + * + * @param id The upload ID + * @return {@code true} if physical data exists for this upload; {@code false} otherwise + * @throws IOException on storage I/O error + */ + default boolean dataExists(UploadId id) throws IOException { + return false; + } + + /** + * Deletes the physical committed data bytes (and any partial chunk data) for the specified upload + * ID without removing the upload's {@link UploadInfo} metadata record. + * + *

Critical for Deduplication: Implementations supporting upload deduplication MUST + * override this method to free physical payload storage in two lifecycle scenarios: + * + *

    + *
  • Child Deduplication: When an upload completes and is identified as a duplicate of + * an existing parent, {@link me.desair.tus.server.upload.checksum.DeduplicationService} + * invokes this method to delete the child's physical data before linking its metadata to + * the parent upload. + *
  • Last-Child Purge: When a parent upload was previously deleted while children were + * active (tombstoned), this method is invoked upon termination of the last active child to + * permanently purge the retained parent payload bytes from storage. + *
+ * + *

The default implementation is a no-op for backward compatibility. If not overridden in a + * deduplication-enabled backend, duplicate child payloads will remain on disk/cloud storage and + * tombstoned parent payloads will be leaked. + * + * @param id The upload ID + * @throws IOException on storage I/O error + */ + default void deleteData(UploadId id) throws IOException {} + /** * Get the minimum append size limit in bytes for a single upload append or creation request. * diff --git a/src/main/java/me/desair/tus/server/upload/azure/AzureBlobChecksumIndex.java b/src/main/java/me/desair/tus/server/upload/azure/AzureBlobChecksumIndex.java new file mode 100644 index 00000000..358ef2b1 --- /dev/null +++ b/src/main/java/me/desair/tus/server/upload/azure/AzureBlobChecksumIndex.java @@ -0,0 +1,235 @@ +package me.desair.tus.server.upload.azure; + +import com.azure.core.util.BinaryData; +import com.azure.storage.blob.BlobClient; +import com.azure.storage.blob.BlobContainerClient; +import com.azure.storage.blob.models.BlobItem; +import com.azure.storage.blob.models.BlobRequestConditions; +import com.azure.storage.blob.models.BlobStorageException; +import com.azure.storage.blob.models.ListBlobsOptions; +import com.azure.storage.blob.options.BlobParallelUploadOptions; +import java.io.IOException; +import java.nio.charset.StandardCharsets; +import java.util.LinkedHashSet; +import java.util.Objects; +import java.util.Set; +import me.desair.tus.server.checksum.ChecksumAlgorithm; +import me.desair.tus.server.checksum.ChecksumKey; +import me.desair.tus.server.upload.UploadId; +import me.desair.tus.server.upload.checksum.ChecksumIndex; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +/** + * Azure Blob Storage implementation of {@link ChecksumIndex}. + * + *

Storage layout: + * + *

    + *
  • {@code ///parent} - contains parent upload ID + *
  • {@code ///parent-deleted} - tombstone marker + *
  • {@code ///children/} - child upload + * markers + *
+ */ +public class AzureBlobChecksumIndex implements ChecksumIndex { + + private static final Logger log = LoggerFactory.getLogger(AzureBlobChecksumIndex.class); + + private static final String DEFAULT_CHECKSUMS_PREFIX = "checksums/"; + private static final String PARENT_OBJECT = "parent"; + private static final String PARENT_DELETED_OBJECT = "parent-deleted"; + private static final String CHILDREN_PREFIX = "children/"; + + private final BlobContainerClient containerClient; + private final String checksumsPrefix; + + /** + * Constructs a new {@link AzureBlobChecksumIndex}. + * + * @param containerClient the Azure BlobContainerClient + */ + public AzureBlobChecksumIndex(BlobContainerClient containerClient) { + this(containerClient, DEFAULT_CHECKSUMS_PREFIX); + } + + /** + * Constructs a new {@link AzureBlobChecksumIndex} with a custom checksums prefix. + * + * @param containerClient the Azure BlobContainerClient + * @param checksumsPrefix the prefix path for checksums + */ + public AzureBlobChecksumIndex(BlobContainerClient containerClient, String checksumsPrefix) { + this.containerClient = + Objects.requireNonNull(containerClient, "containerClient must not be null"); + this.checksumsPrefix = sanitizePrefix(checksumsPrefix); + } + + @Override + public boolean tryRegisterParent(ChecksumKey key, UploadId uploadId) throws IOException { + Objects.requireNonNull(key, "key must not be null"); + Objects.requireNonNull(uploadId, "uploadId must not be null"); + + String parentKey = buildKey(key, PARENT_OBJECT); + BlobClient parentBlob = containerClient.getBlobClient(parentKey); + byte[] content = uploadId.toString().getBytes(StandardCharsets.UTF_8); + + // Atomic CAS write using Azure BlobRequestConditions.setIfNoneMatch("*") + try { + BlobRequestConditions conditions = new BlobRequestConditions().setIfNoneMatch("*"); + BlobParallelUploadOptions options = + new BlobParallelUploadOptions(BinaryData.fromBytes(content)) + .setRequestConditions(conditions); + parentBlob.uploadWithResponse(options, null, null); + return true; + } catch (BlobStorageException e) { + AzureErrorType errorType = AzureUtils.parseErrorResponse(e); + if (errorType == AzureErrorType.CONFLICT || errorType == AzureErrorType.PRECONDITION_FAILED) { + return false; + } + throw new IOException("Failed to create parent index blob in Azure: " + parentKey, e); + } + } + + @Override + public UploadId findParent(ChecksumKey key) throws IOException { + Objects.requireNonNull(key, "key must not be null"); + String parentKey = buildKey(key, PARENT_OBJECT); + BlobClient parentBlob = containerClient.getBlobClient(parentKey); + + try { + byte[] bytes = parentBlob.downloadContent().toBytes(); + String idStr = new String(bytes, StandardCharsets.UTF_8).trim(); + return idStr.isEmpty() ? null : new UploadId(idStr); + } catch (BlobStorageException e) { + if (AzureUtils.parseErrorResponse(e) == AzureErrorType.BLOB_NOT_FOUND) { + return null; + } + throw new IOException("Failed to read parent index blob in Azure: " + parentKey, e); + } + } + + @Override + public void markParentDeleted(ChecksumKey key) throws IOException { + Objects.requireNonNull(key, "key must not be null"); + String parentDeletedKey = buildKey(key, PARENT_DELETED_OBJECT); + + // Write empty tombstone marker while keeping parent pointer blob intact + // until deleteIndex(key) is invoked when the last child terminates. + BlobClient tombstoneBlob = containerClient.getBlobClient(parentDeletedKey); + tombstoneBlob.upload(BinaryData.fromBytes(new byte[0]), true); + } + + @Override + public boolean isParentDeleted(ChecksumKey key) throws IOException { + Objects.requireNonNull(key, "key must not be null"); + return blobExists(buildKey(key, PARENT_DELETED_OBJECT)); + } + + @Override + public void addChild(ChecksumKey key, UploadId childId) throws IOException { + Objects.requireNonNull(key, "key must not be null"); + Objects.requireNonNull(childId, "childId must not be null"); + String childKey = buildKey(key, CHILDREN_PREFIX + childId); + BlobClient childBlob = containerClient.getBlobClient(childKey); + childBlob.upload(BinaryData.fromBytes(new byte[0]), true); + } + + @Override + public void removeChild(ChecksumKey key, UploadId childId) throws IOException { + Objects.requireNonNull(key, "key must not be null"); + Objects.requireNonNull(childId, "childId must not be null"); + String childKey = buildKey(key, CHILDREN_PREFIX + childId); + BlobClient childBlob = containerClient.getBlobClient(childKey); + childBlob.deleteIfExists(); + } + + @Override + public boolean hasChildren(ChecksumKey key) throws IOException { + Objects.requireNonNull(key, "key must not be null"); + String prefix = buildKey(key, CHILDREN_PREFIX); + + try { + ListBlobsOptions options = new ListBlobsOptions().setPrefix(prefix).setMaxResultsPerPage(1); + return containerClient.listBlobs(options, null).iterator().hasNext(); + } catch (Exception e) { + throw new IOException("Failed to list children blobs in Azure for prefix " + prefix, e); + } + } + + @Override + public void deleteIndex(ChecksumKey key) throws IOException { + Objects.requireNonNull(key, "key must not be null"); + String prefix = buildKey(key, ""); + + try { + ListBlobsOptions options = new ListBlobsOptions().setPrefix(prefix); + for (BlobItem item : containerClient.listBlobs(options, null)) { + containerClient.getBlobClient(item.getName()).deleteIfExists(); + } + } catch (Exception e) { + throw new IOException( + "Failed to delete checksum index blobs in Azure for prefix " + prefix, e); + } + } + + @Override + public Set getAllIndexedKeys() throws IOException { + Set keys = new LinkedHashSet<>(); + try { + ListBlobsOptions options = new ListBlobsOptions().setPrefix(checksumsPrefix); + for (BlobItem item : containerClient.listBlobs(options, null)) { + if (item == null || item.getName() == null) { + continue; + } + String blobName = item.getName(); + if (blobName.endsWith("/" + PARENT_OBJECT)) { + String relative = blobName.substring(checksumsPrefix.length()); + if (relative.startsWith("/")) { + relative = relative.substring(1); + } + String[] parts = relative.split("/"); + if (parts.length == 3 && PARENT_OBJECT.equals(parts[2])) { + ChecksumAlgorithm algo = ChecksumAlgorithm.forTusName(parts[0]); + if (algo != null) { + try { + keys.add(ChecksumKey.of(parts[1], algo)); + } catch (IllegalArgumentException e) { + // Malformed checksum key, skip + } + } + } + } + } + } catch (Exception e) { + throw new IOException("Failed to list indexed checksum keys from Azure Blob Storage", e); + } + return keys; + } + + private String buildKey(ChecksumKey key, String subPath) { + return checksumsPrefix + key + (subPath.isEmpty() ? "" : "/" + subPath); + } + + private boolean blobExists(String blobKey) { + try { + return Boolean.TRUE.equals(containerClient.getBlobClient(blobKey).exists()); + } catch (Exception e) { + return false; + } + } + + private static String sanitizePrefix(String prefix) { + if (prefix == null || prefix.trim().isEmpty()) { + return ""; + } + String trimmed = prefix.trim(); + while (trimmed.startsWith("/")) { + trimmed = trimmed.substring(1); + } + if (!trimmed.isEmpty() && !trimmed.endsWith("/")) { + trimmed = trimmed + "/"; + } + return trimmed; + } +} diff --git a/src/main/java/me/desair/tus/server/upload/azure/AzureBlobLockingService.java b/src/main/java/me/desair/tus/server/upload/azure/AzureBlobLockingService.java index 059d749c..0bebcb34 100644 --- a/src/main/java/me/desair/tus/server/upload/azure/AzureBlobLockingService.java +++ b/src/main/java/me/desair/tus/server/upload/azure/AzureBlobLockingService.java @@ -126,6 +126,19 @@ public void setIdFactory(UploadIdFactory idFactory) { @Override public UploadLock lockUploadByUri(String requestUri) throws TusException, IOException { UploadId uploadId = idFactory.readUploadId(requestUri); + return lockUploadInternal(uploadId, requestUri); + } + + @Override + public UploadLock lockUpload(UploadId uploadId) throws TusException, IOException { + if (uploadId == null) { + return null; + } + return lockUploadInternal(uploadId, uploadId.toString()); + } + + private UploadLock lockUploadInternal(UploadId uploadId, String uriOrId) + throws TusException, IOException { if (uploadId == null) { return null; } @@ -158,19 +171,18 @@ public UploadLock lockUploadByUri(String requestUri) throws TusException, IOExce AzureBlobUploadLock lock = new AzureBlobUploadLock( - leaseClient, lockBlob, requestUri, idStr, activeStreams, activeLocks); + leaseClient, lockBlob, uriOrId, idStr, activeStreams, activeLocks); activeLocks.put(idStr, lock); return lock; } catch (BlobStorageException e) { AzureErrorType errorType = AzureUtils.parseErrorResponse(e); if (errorType == AzureErrorType.LEASE_ALREADY_PRESENT || errorType == AzureErrorType.CONFLICT) { - log.debug( - "Lock contention for upload URI {}: Azure blob lease is already held", requestUri); + log.debug("Lock contention for upload URI {}: Azure blob lease is already held", uriOrId); throw new UploadAlreadyLockedException( - "Upload with URI " + requestUri + " is currently locked"); + "Upload with URI " + uriOrId + " is currently locked"); } - throw new IOException("Failed to acquire Azure blob lease lock for URI " + requestUri, e); + throw new IOException("Failed to acquire Azure blob lease lock for URI " + uriOrId, e); } } diff --git a/src/main/java/me/desair/tus/server/upload/azure/AzureBlobStorageService.java b/src/main/java/me/desair/tus/server/upload/azure/AzureBlobStorageService.java index f300ac92..b5dc5de0 100644 --- a/src/main/java/me/desair/tus/server/upload/azure/AzureBlobStorageService.java +++ b/src/main/java/me/desair/tus/server/upload/azure/AzureBlobStorageService.java @@ -4,7 +4,6 @@ import com.azure.storage.blob.BlobClient; import com.azure.storage.blob.BlobContainerClient; import com.azure.storage.blob.models.BlobItem; -import com.azure.storage.blob.models.BlobProperties; import com.azure.storage.blob.models.BlobStorageException; import com.azure.storage.blob.models.Block; import com.azure.storage.blob.models.BlockList; @@ -33,6 +32,7 @@ import java.util.concurrent.TimeUnit; import java.util.concurrent.atomic.AtomicInteger; import me.desair.tus.server.checksum.ChecksumAlgorithm; +import me.desair.tus.server.checksum.ChecksumKey; import me.desair.tus.server.exception.MaxAppendSizeExceededException; import me.desair.tus.server.exception.MinAppendSizeNotMetException; import me.desair.tus.server.exception.TusException; @@ -43,6 +43,7 @@ import me.desair.tus.server.upload.UploadLockingService; import me.desair.tus.server.upload.UploadStorageService; import me.desair.tus.server.upload.UuidUploadIdFactory; +import me.desair.tus.server.upload.checksum.DeduplicationService; import me.desair.tus.server.upload.concatenation.UploadConcatenationService; import me.desair.tus.server.upload.util.AsyncChunkUploader; import me.desair.tus.server.util.InterruptibleInputStream; @@ -51,7 +52,6 @@ import org.apache.commons.io.IOUtils; import org.apache.commons.io.input.BoundedInputStream; import org.apache.commons.lang3.StringUtils; -import org.apache.commons.lang3.Strings; import org.slf4j.Logger; import org.slf4j.LoggerFactory; @@ -106,6 +106,9 @@ public class AzureBlobStorageService implements UploadStorageService { private Long minSize; private Long uploadExpirationPeriod; private boolean deduplicationEnabled = false; + private final AzureBlobChecksumIndex checksumIndex; + private DeduplicationService deduplicationService; + private UploadLockingService uploadLockingService; private UploadIdFactory idFactory = new UuidUploadIdFactory(); private UploadConcatenationService concatenationService; @@ -175,6 +178,11 @@ public AzureBlobStorageService( this.concatenationService = new AzureBlobConcatenationService(containerClient, this.uploadPrefix, this); + + this.checksumIndex = new AzureBlobChecksumIndex(this.containerClient, this.checksumsPrefix); + this.deduplicationService = new DeduplicationService(this.checksumIndex, this); + this.uploadLockingService = new AzureBlobLockingService(this.containerClient, this.locksPrefix); + this.deduplicationService.setUploadLockingService(this.uploadLockingService); } @Override @@ -468,7 +476,7 @@ public InputStream getUploadedBytes(String requestUri, String ownerKey) @SuppressWarnings("java:S2095") // InputStream is returned to caller for external management @Override - public InputStream getUploadedBytes(UploadId id) throws IOException { + public InputStream getUploadedBytes(UploadId id) throws IOException, UploadNotFoundException { UploadInfo info = getUploadInfo(id); if (info == null) { return null; @@ -486,6 +494,11 @@ public InputStream getUploadedBytes(UploadId id) throws IOException { // Data blob does not exist yet (e.g. 0-byte upload or immediately after creation) } + if (info.getOffset() != null && info.getOffset() > 0L) { + throw new UploadNotFoundException( + "Uploaded bytes blob " + targetBlobName + " not found for ID " + id); + } + return new ByteArrayInputStream(new byte[0]); } @@ -504,45 +517,44 @@ public void copyUploadTo(UploadInfo uploadInfo, OutputStream outputStream) } @Override - public void terminateUpload(UploadInfo uploadInfo) throws UploadNotFoundException, IOException { - if (uploadInfo == null || uploadInfo.getId() == null) { - return; + public void setUploadLockingService(UploadLockingService uploadLockingService) { + this.uploadLockingService = + Objects.requireNonNull(uploadLockingService, "uploadLockingService must not be null"); + if (this.deduplicationService != null) { + this.deduplicationService.setUploadLockingService(uploadLockingService); } - UploadId id = uploadInfo.getId(); + } - // 1. Delete committed data blob - containerClient.getBlobClient(uploadPrefix + id).deleteIfExists(); + @Override + public UploadLockingService getUploadLockingService() { + return this.uploadLockingService; + } - // 2. Delete metadata .info blob - containerClient.getBlobClient(metadataPrefix + id + ".info").deleteIfExists(); + @Override + public UploadInfo deduplicate(UploadInfo uploadInfo, ChecksumKey checksumKey) throws IOException { + if (!isUploadDeduplicationEnabled() || uploadInfo == null || checksumKey == null) { + return uploadInfo; + } + return deduplicationService.deduplicate(uploadInfo, checksumKey); + } - // 3. Delete checksum deduplication index blob if present - if (uploadInfo.getChecksum() != null && uploadInfo.getChecksumAlgorithm() != null) { - String checksumKey = - buildChecksumKey(uploadInfo.getChecksum(), uploadInfo.getChecksumAlgorithm()); - containerClient.getBlobClient(checksumKey).deleteIfExists(); + @Override + public void terminateUpload(UploadInfo uploadInfo) throws UploadNotFoundException, IOException { + if (uploadInfo == null || uploadInfo.getId() == null) { + return; } + UploadId id = uploadInfo.getId(); - // 4. Delete lock target and stop signal blobs - try { - containerClient.getBlobClient(locksPrefix + id + ".stop").deleteIfExists(); - } catch (Exception ignored) { + boolean deleteData = true; + if (isUploadDeduplicationEnabled() && deduplicationService != null) { + deleteData = deduplicationService.removeUploadFromIndex(uploadInfo); } - try { - BlobClient lockBlob = containerClient.getBlobClient(locksPrefix + id + ".lock"); - // Check if lock blob is actively leased by the current request before attempting deletion. - // Attempting deleteIfExists() on a leased blob without the lease ID causes Azure Blob - // Storage to reject the request with HTTP 412 (LeaseIdMissing) and logs an SDK error. - if (Boolean.TRUE.equals(lockBlob.exists())) { - BlobProperties props = lockBlob.getProperties(); - if (props.getLeaseState() != null - && !Strings.CS.equals(props.getLeaseState().toString(), "leased")) { - lockBlob.deleteIfExists(); - } - } - } catch (Exception ignored) { + if (deleteData) { + deleteData(id); } + deleteMetadata(id); + uploadLockingService.cleanupLock(id); log.debug("Terminated upload with ID {}", id); } @@ -595,42 +607,51 @@ public UploadInfo getUploadInfoByChecksum(String checksum, ChecksumAlgorithm alg if (!isUploadDeduplicationEnabled() || checksum == null || algorithm == null) { return null; } - - String checksumKey = buildChecksumKey(checksum, algorithm); - BlobClient checksumBlob = containerClient.getBlobClient(checksumKey); - + ChecksumKey key; try { - byte[] bytes = checksumBlob.downloadContent().toBytes(); - String parentIdStr = new String(bytes, StandardCharsets.UTF_8).trim(); - UploadId parentId = new UploadId(parentIdStr); - UploadInfo parentInfo = getUploadInfo(parentId); - - // Self-cleaning index: if index points to missing/deleted upload, clean up stale index - if (parentInfo == null) { - checksumBlob.deleteIfExists(); - return null; - } - - return parentInfo; - } catch (BlobStorageException e) { - if (AzureUtils.parseErrorResponse(e) == AzureErrorType.BLOB_NOT_FOUND) { - return null; - } - throw new IOException("Error reading checksum index blob " + checksumKey, e); + key = ChecksumKey.of(checksum, algorithm); } catch (Exception e) { - log.debug("Error retrieving upload info by checksum: {}", e.getMessage()); return null; } + return deduplicationService != null ? deduplicationService.getUploadInfoByChecksum(key) : null; } @Override public void update(UploadInfo uploadInfo) throws IOException { Objects.requireNonNull(uploadInfo, "UploadInfo must not be null"); saveUploadInfo(uploadInfo); + + if (isUploadDeduplicationEnabled() + && uploadInfo.getChecksum() != null + && !uploadInfo.isUploadInProgress() + && uploadInfo.getDuplicatesUploadId() == null) { + checkAndApplyDeduplication(uploadInfo); + } } @Override - public void cleanupExpiredUploads(UploadLockingService lockingService) throws IOException { + public boolean dataExists(UploadId id) { + if (id == null) { + return false; + } + return Boolean.TRUE.equals(containerClient.getBlobClient(uploadPrefix + id).exists()); + } + + @Override + public void deleteData(UploadId id) { + if (id != null) { + containerClient.getBlobClient(uploadPrefix + id).deleteIfExists(); + } + } + + private void deleteMetadata(UploadId id) { + if (id != null) { + containerClient.getBlobClient(metadataPrefix + id + ".info").deleteIfExists(); + } + } + + @Override + public void cleanupExpiredUploads() throws IOException { Utils.cleanupTempFiles(this.tempBufferDir, "tus-azure-chunk-*.tmp", 24L * 3600_000L); ListBlobsOptions options = new ListBlobsOptions().setPrefix(metadataPrefix); for (BlobItem item : containerClient.listBlobs(options, null)) { @@ -642,7 +663,7 @@ public void cleanupExpiredUploads(UploadLockingService lockingService) throws IO UploadInfo info = getUploadInfo(id); // Only clean up in-progress uploads; completed uploads must be preserved per Tus 1.0.0 spec if (info != null && info.isExpired() && info.isUploadInProgress()) { - if (lockingService != null && lockingService.isLocked(id)) { + if (this.uploadLockingService.isLocked(id)) { log.debug("Skipping cleanup of expired upload {} because it is currently locked", id); continue; } @@ -652,19 +673,20 @@ public void cleanupExpiredUploads(UploadLockingService lockingService) throws IO } catch (UploadNotFoundException ignored) { // Upload was already terminated or deleted concurrently } - if (lockingService != null) { - lockingService.cleanupLock(id); - } } } } - // Prune orphaned checksum index entries pointing to expired/deleted parent uploads - pruneOrphanedChecksumIndices(); + if (isUploadDeduplicationEnabled() && deduplicationService != null) { + deduplicationService.pruneOrphanedChecksumIndices(); + } } - public void cleanupExpiredUploads() throws IOException { - cleanupExpiredUploads(null); + @Override + @Deprecated + public void cleanupExpiredUploads(UploadLockingService lockingService) throws IOException { + setUploadLockingService(Utils.getLockingServiceOrDefault(lockingService)); + cleanupExpiredUploads(); } // --- Configuration Getters & Setters --- @@ -929,50 +951,6 @@ private void stageBlock( } } - /** - * Scans checksum deduplication index blobs under {@code checksumsPrefix} and deletes any index - * blobs whose referenced parent upload has expired or no longer exists in Azure. - */ - private void pruneOrphanedChecksumIndices() { - // 1. Skip scanning if deduplication is disabled; no checksum indices are generated or - // referenced - if (!isUploadDeduplicationEnabled()) { - return; - } - try { - // 2. Query all index blobs located under the deduplication prefix (e.g. - // checksums//) - ListBlobsOptions options = new ListBlobsOptions().setPrefix(checksumsPrefix); - for (BlobItem item : containerClient.listBlobs(options, null)) { - try { - BlobClient checksumBlob = containerClient.getBlobClient(item.getName()); - - // 3. Read the parent upload ID string stored as plain text inside the checksum index blob - String parentIdStr = checksumBlob.downloadContent().toString().trim(); - UploadId parentId = new UploadId(parentIdStr); - - // 4. Verify whether the parent upload's metadata (.info) still exists and is accessible - UploadInfo parentInfo = getUploadInfo(parentId); - - // 5. Verify whether the parent upload's final data blob exists in Azure storage - BlobClient parentDataBlob = containerClient.getBlobClient(uploadPrefix + parentId); - - // 6. If the parent upload or its data blob no longer exists (e.g. expired, deleted, or - // purged - // by Azure lifecycle policies), prune the orphaned index entry to keep storage clean - if (parentInfo == null || !Boolean.TRUE.equals(parentDataBlob.exists())) { - log.debug("Pruning orphaned Azure checksum index blob {}", item.getName()); - checksumBlob.deleteIfExists(); - } - } catch (Exception ignored) { - // Ignore individual blob read errors to allow full sweep of remaining index entries - } - } - } catch (Exception e) { - log.debug("Failed to prune orphaned Azure checksum indices: {}", e.getMessage()); - } - } - /** Commits empty block list for 0-byte upload completion. */ private void commitEmptyDataBlob(UploadId id) { BlockBlobClient blockBlobClient = @@ -995,7 +973,6 @@ String generateBlockId(int index) { return Base64.getEncoder().encodeToString(idString.getBytes(StandardCharsets.UTF_8)); } - /** Checks for deduplication match and links upload if duplicate. */ private void checkAndApplyDeduplication(UploadInfo uploadInfo) throws IOException { if (!isUploadDeduplicationEnabled() || uploadInfo == null @@ -1004,32 +981,15 @@ private void checkAndApplyDeduplication(UploadInfo uploadInfo) throws IOExceptio return; } - String checksum = uploadInfo.getChecksum(); - ChecksumAlgorithm algorithm = uploadInfo.getChecksumAlgorithm(); - - UploadInfo existingUpload = getUploadInfoByChecksum(checksum, algorithm); - if (existingUpload != null && !existingUpload.getId().equals(uploadInfo.getId())) { - log.info( - "Found duplicate upload with checksum {}/{} (parent ID {})", - algorithm, - checksum, - existingUpload.getId()); - uploadInfo.setDuplicatesUploadId(existingUpload.getId()); - containerClient.getBlobClient(uploadPrefix + uploadInfo.getId()).deleteIfExists(); - } else { - String checksumKey = buildChecksumKey(checksum, algorithm); - BlobClient checksumBlob = containerClient.getBlobClient(checksumKey); - byte[] idBytes = uploadInfo.getId().toString().getBytes(StandardCharsets.UTF_8); - checksumBlob.upload(BinaryData.fromBytes(idBytes), true); + try { + ChecksumKey key = ChecksumKey.of(uploadInfo.getChecksum(), uploadInfo.getChecksumAlgorithm()); + deduplicate(uploadInfo, key); + } catch (Exception e) { + log.warn( + "Failed to apply deduplication for upload {}: {}", uploadInfo.getId(), e.getMessage()); } } - /** Builds checksum index key path. */ - private String buildChecksumKey(String checksum, ChecksumAlgorithm algorithm) { - String algorithmName = algorithm != null ? algorithm.getTusName().toLowerCase() : "unknown"; - return checksumsPrefix + algorithmName + "/" + checksum; - } - /** Sanitizes object key prefix format. */ private String sanitizePrefix(String prefix) { if (prefix == null || prefix.isEmpty()) { diff --git a/src/main/java/me/desair/tus/server/upload/cache/ThreadLocalCachedStorageAndLockingService.java b/src/main/java/me/desair/tus/server/upload/cache/ThreadLocalCachedStorageAndLockingService.java index 9e8a1280..a428212f 100644 --- a/src/main/java/me/desair/tus/server/upload/cache/ThreadLocalCachedStorageAndLockingService.java +++ b/src/main/java/me/desair/tus/server/upload/cache/ThreadLocalCachedStorageAndLockingService.java @@ -7,6 +7,7 @@ import java.time.Duration; import java.util.Objects; import me.desair.tus.server.checksum.ChecksumAlgorithm; +import me.desair.tus.server.checksum.ChecksumKey; import me.desair.tus.server.exception.TusException; import me.desair.tus.server.exception.UploadNotFoundException; import me.desair.tus.server.upload.UploadId; @@ -166,6 +167,15 @@ public void copyUploadTo(UploadInfo info, OutputStream outputStream) } @Override + public void cleanupExpiredUploads() throws IOException { + storageServiceDelegate.cleanupExpiredUploads(); + // Since any cached uploads was potentially removed by the storage service + // we clean the cache to prevent any stale state + cleanupCache(); + } + + @Override + @Deprecated public void cleanupExpiredUploads(UploadLockingService uploadLockingService) throws IOException { storageServiceDelegate.cleanupExpiredUploads(uploadLockingService); // Since any cached uploads was potentially removed by the storage service @@ -234,6 +244,35 @@ public UploadInfo getUploadInfoByChecksum(String checksum, ChecksumAlgorithm alg return storageServiceDelegate.getUploadInfoByChecksum(checksum, algorithm); } + @Override + public UploadInfo deduplicate(UploadInfo uploadInfo, ChecksumKey checksumKey) throws IOException { + UploadInfo result = storageServiceDelegate.deduplicate(uploadInfo, checksumKey); + if (result != null) { + uploadInfoCache.set(new WeakReference<>(result)); + } + return result; + } + + @Override + public void setUploadLockingService(UploadLockingService uploadLockingService) { + storageServiceDelegate.setUploadLockingService(uploadLockingService); + } + + @Override + public UploadLockingService getUploadLockingService() { + return storageServiceDelegate.getUploadLockingService(); + } + + @Override + public boolean dataExists(UploadId id) throws IOException { + return storageServiceDelegate.dataExists(id); + } + + @Override + public void deleteData(UploadId id) throws IOException { + storageServiceDelegate.deleteData(id); + } + @Override public void setCloudUploadThreadPoolSize(int size) { storageServiceDelegate.setCloudUploadThreadPoolSize(size); @@ -260,6 +299,12 @@ public UploadLock lockUploadByUri(String requestUri) throws TusException, IOExce return new CachedLock(uploadLock); } + @Override + public UploadLock lockUpload(UploadId id) throws TusException, IOException { + UploadLock uploadLock = lockingServiceDelegate.lockUpload(id); + return new CachedLock(uploadLock); + } + @Override public void cleanupStaleLocks() throws IOException { lockingServiceDelegate.cleanupStaleLocks(); diff --git a/src/main/java/me/desair/tus/server/upload/checksum/ChecksumIndex.java b/src/main/java/me/desair/tus/server/upload/checksum/ChecksumIndex.java new file mode 100644 index 00000000..b3cf0a77 --- /dev/null +++ b/src/main/java/me/desair/tus/server/upload/checksum/ChecksumIndex.java @@ -0,0 +1,107 @@ +package me.desair.tus.server.upload.checksum; + +import java.io.IOException; +import java.util.Set; +import me.desair.tus.server.checksum.ChecksumKey; +import me.desair.tus.server.upload.UploadId; + +/** + * Service Provider Interface (SPI) for storage-backend checksum indexing and deduplication + * tracking. + * + *

Each storage backend (disk, S3, Azure Blob) implements this interface to provide atomic + * registration, parent lookup, tombstoning, and child reference tracking. + * + *

Storage layout: + * + *

    + *
  • {@code checksums///parent} - contains the parent upload ID + *
  • {@code checksums///parent-deleted} - tombstone marker indicating + * the parent was deleted + *
  • {@code checksums///children/} - child upload marker files + *
+ */ +public interface ChecksumIndex { + + /** + * Atomically registers the specified upload as the parent for the given checksum key. + * + * @param key the normalized checksum key + * @param uploadId the upload ID to register as parent + * @return {@code true} if this upload was atomically registered as the parent; {@code false} if a + * parent or tombstone already exists (CAS failure) + * @throws IOException on storage I/O error + */ + boolean tryRegisterParent(ChecksumKey key, UploadId uploadId) throws IOException; + + /** + * Looks up the registered parent upload ID for the given checksum key. + * + * @param key the normalized checksum key + * @return the parent {@link UploadId}, or {@code null} if no parent is registered + * @throws IOException on storage I/O error + */ + UploadId findParent(ChecksumKey key) throws IOException; + + /** + * Atomically marks the parent upload as deleted (tombstone) while preserving the parent marker + * file until {@link #deleteIndex(ChecksumKey)} is invoked when the last child is removed. + * + * @param key the normalized checksum key + * @throws IOException on storage I/O error + */ + void markParentDeleted(ChecksumKey key) throws IOException; + + /** + * Checks whether the parent for the given checksum key has been deleted/tombstoned. + * + * @param key the normalized checksum key + * @return {@code true} if the tombstone marker exists; {@code false} otherwise + * @throws IOException on storage I/O error + */ + boolean isParentDeleted(ChecksumKey key) throws IOException; + + /** + * Registers a child upload ID under the given checksum key. + * + * @param key the normalized checksum key + * @param childId the child upload ID to add + * @throws IOException on storage I/O error + */ + void addChild(ChecksumKey key, UploadId childId) throws IOException; + + /** + * Removes a child upload ID marker from under the given checksum key. + * + * @param key the normalized checksum key + * @param childId the child upload ID to remove + * @throws IOException on storage I/O error + */ + void removeChild(ChecksumKey key, UploadId childId) throws IOException; + + /** + * Checks whether any child upload markers currently exist for the given checksum key. + * + * @param key the normalized checksum key + * @return {@code true} if at least one child marker exists; {@code false} otherwise + * @throws IOException on storage I/O error + */ + boolean hasChildren(ChecksumKey key) throws IOException; + + /** + * Deletes the entire checksum index directory or prefix for the given checksum key, removing all + * markers and children. + * + * @param key the normalized checksum key + * @throws IOException on storage I/O error + */ + void deleteIndex(ChecksumKey key) throws IOException; + + /** + * Retrieves all currently indexed checksum keys present in this checksum index. + * + * @return a set of all indexed {@link ChecksumKey} instances + * @throws IOException on storage I/O error + */ + Set getAllIndexedKeys() throws IOException; +} diff --git a/src/main/java/me/desair/tus/server/upload/checksum/DeduplicationService.java b/src/main/java/me/desair/tus/server/upload/checksum/DeduplicationService.java new file mode 100644 index 00000000..4fb28fd5 --- /dev/null +++ b/src/main/java/me/desair/tus/server/upload/checksum/DeduplicationService.java @@ -0,0 +1,454 @@ +package me.desair.tus.server.upload.checksum; + +import java.io.IOException; +import java.util.Objects; +import java.util.Set; +import me.desair.tus.server.checksum.ChecksumKey; +import me.desair.tus.server.exception.TusException; +import me.desair.tus.server.exception.UploadNotFoundException; +import me.desair.tus.server.upload.UploadId; +import me.desair.tus.server.upload.UploadInfo; +import me.desair.tus.server.upload.UploadLock; +import me.desair.tus.server.upload.UploadLockingService; +import me.desair.tus.server.upload.UploadStorageService; +import me.desair.tus.server.util.Utils; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +/** + * Orchestrator service responsible for thread-safe and cluster-safe upload deduplication lifecycle + * management. + * + *

Handles parent registration, child deduplication linking, CAS race resolution, missing-parent + * detection, and reference-counted cleanup upon upload deletion (via DELETE request or expiration). + */ +public class DeduplicationService { + + private static final Logger log = LoggerFactory.getLogger(DeduplicationService.class); + + private final ChecksumIndex checksumIndex; + private final UploadStorageService storageService; + private UploadLockingService lockingService; + + /** + * Constructs a new {@link DeduplicationService}. + * + * @param checksumIndex the backend checksum index implementation + * @param storageService the underlying upload storage service + */ + public DeduplicationService(ChecksumIndex checksumIndex, UploadStorageService storageService) { + this(checksumIndex, storageService, Utils.getLockingServiceOrDefault(storageService)); + } + + /** + * Constructs a {@link DeduplicationService} with mandatory {@link UploadLockingService}. + * + * @param checksumIndex the checksum index + * @param storageService the storage service + * @param lockingService the locking service used for inter-upload synchronization + */ + public DeduplicationService( + ChecksumIndex checksumIndex, + UploadStorageService storageService, + UploadLockingService lockingService) { + this.checksumIndex = Objects.requireNonNull(checksumIndex, "checksumIndex must not be null"); + this.storageService = Objects.requireNonNull(storageService, "storageService must not be null"); + this.lockingService = Objects.requireNonNull(lockingService, "lockingService must not be null"); + } + + /** + * Sets the mandatory {@link UploadLockingService} used for inter-upload synchronization. + * + * @param lockingService the locking service (must not be {@code null}) + */ + public void setUploadLockingService(UploadLockingService lockingService) { + this.lockingService = Objects.requireNonNull(lockingService, "lockingService must not be null"); + } + + /** + * Attempts to deduplicate the given completed upload $U$ against an existing parent with the same + * checksum, or registers $U$ as the parent if none exists. + * + * @param uploadInfo the upload info for the completed upload + * @param key the normalized checksum key + * @return the updated {@link UploadInfo} (either linked to parent or registered as parent) + * @throws IOException on storage I/O error + */ + public UploadInfo deduplicate(UploadInfo uploadInfo, ChecksumKey key) throws IOException { + if (uploadInfo == null || key == null) { + return uploadInfo; + } + + UploadId currentId = uploadInfo.getId(); + + // 1. Check if a parent is currently registered for this checksum + UploadId parentId = checksumIndex.findParent(key); + + if (parentId == null) { + // 2. No parent registered. Attempt CAS registration of current upload as parent. + return registerAsParent(uploadInfo, key); + } + + // Guard: Do not deduplicate an upload against itself + if (Objects.equals(parentId, currentId)) { + return uploadInfo; + } + + // 3. Parent found ($P$). Acquire lock on $P$ to inspect and join. + return joinParentUnderLock(uploadInfo, key, parentId, true); + } + + /** Registers the given upload as the parent for the specified checksum key. */ + private UploadInfo registerAsParent(UploadInfo uploadInfo, ChecksumKey key) throws IOException { + UploadId currentId = uploadInfo.getId(); + boolean registered = checksumIndex.tryRegisterParent(key, currentId); + if (registered) { + log.debug("Registered upload {} as parent for checksum {}", currentId, key); + uploadInfo.setChecksum(key.getHexChecksum()); + uploadInfo.setChecksumAlgorithm(key.getAlgorithm()); + updateUploadInfo(uploadInfo); + return uploadInfo; + } + + // CAS failed (lost race to concurrent upload). Re-lookup the winning parent. + UploadId winner = checksumIndex.findParent(key); + if (winner != null && !Objects.equals(winner, currentId)) { + return joinParentUnderLock(uploadInfo, key, winner, true); + } + + uploadInfo.setChecksum(key.getHexChecksum()); + uploadInfo.setChecksumAlgorithm(key.getAlgorithm()); + updateUploadInfo(uploadInfo); + log.debug( + "Parent registration failed for upload {} under checksum {}; retaining as standalone upload", + currentId, + key); + return uploadInfo; + } + + /** + * Looks up an existing completed parent upload matching the given checksum key. + * + * @param key the normalized checksum key + * @return the parent {@link UploadInfo}, or {@code null} if no valid parent exists or its data is + * missing + * @throws IOException on storage I/O error + */ + public UploadInfo getUploadInfoByChecksum(ChecksumKey key) throws IOException { + if (key == null) { + return null; + } + UploadId parentId = checksumIndex.findParent(key); + if (parentId == null) { + return null; + } + UploadInfo parentInfo = storageService.getUploadInfo(parentId); + boolean parentTombstoned = checksumIndex.isParentDeleted(key); + boolean parentBytesExist = storageService.dataExists(parentId); + + if (parentInfo == null) { + if (parentTombstoned && parentBytesExist) { + // Parent is legitimately tombstoned and retained for active children. + // Return null as parent upload is terminated, but preserve index for children. + return null; + } + log.debug( + "Parent {} referenced by checksum {} is missing or corrupt; self-cleaning index", + parentId, + key); + checksumIndex.deleteIndex(key); + return null; + } + + if (!parentBytesExist) { + log.debug( + "Parent data for {} referenced by checksum {} is missing; self-cleaning index", + parentId, + key); + checksumIndex.deleteIndex(key); + return null; + } + + return parentInfo; + } + + /** + * Acquires the lock on parent $P$, re-verifies parent state and physical bytes, and links the + * child. + */ + private UploadInfo joinParentUnderLock( + UploadInfo uploadInfo, ChecksumKey key, UploadId expectedParentId, boolean allowRetry) + throws IOException { + try (UploadLock parentLock = lockingService.lockUpload(expectedParentId)) { + return doJoinParentUnderLock(uploadInfo, key, expectedParentId, allowRetry); + } catch (TusException e) { + // Parent is currently locked by active request. Per design, skip deduplication + // and retain current upload as an independent standalone upload. + log.debug( + "Parent {} is currently locked; skipping deduplication for upload {}", + expectedParentId, + uploadInfo.getId()); + uploadInfo.setChecksum(key.getHexChecksum()); + uploadInfo.setChecksumAlgorithm(key.getAlgorithm()); + updateUploadInfo(uploadInfo); + return uploadInfo; + } + } + + private UploadInfo doJoinParentUnderLock( + UploadInfo uploadInfo, ChecksumKey key, UploadId expectedParentId, boolean allowRetry) + throws IOException { + // Under parent lock: B2 Check - re-read current parent from index + UploadId currentParent = checksumIndex.findParent(key); + if (!Objects.equals(expectedParentId, currentParent)) { + if (allowRetry && currentParent != null) { + return joinParentUnderLock(uploadInfo, key, currentParent, false); + } + return registerAsParent(uploadInfo, key); + } + + UploadInfo parentInfo = storageService.getUploadInfo(expectedParentId); + boolean parentTombstoned = (parentInfo == null) && checksumIndex.isParentDeleted(key); + + // B3 Self-healing check: if parent is missing and not legitimately tombstoned, + // or if parent physical bytes are missing + if ((parentInfo == null && !parentTombstoned) || !storageService.dataExists(expectedParentId)) { + log.warn( + "Parent {} for checksum {} is invalid (exists={}, tombstoned={}, bytesExist={}). Purging index.", + expectedParentId, + key, + parentInfo != null, + parentTombstoned, + storageService.dataExists(expectedParentId)); + checksumIndex.deleteIndex(key); + return registerAsParent(uploadInfo, key); + } + + // Sync expiration if parent is active (not tombstoned) + if (parentInfo != null) { + syncParentExpiration(parentInfo, uploadInfo); + } + + // 4. Metadata-first child linking: + // Step 1: Register child in checksum index + checksumIndex.addChild(key, uploadInfo.getId()); + + // Step 2: Persist child metadata referencing parent + uploadInfo.setDuplicatesUploadId(expectedParentId); + uploadInfo.setChecksum(key.getHexChecksum()); + uploadInfo.setChecksumAlgorithm(key.getAlgorithm()); + updateUploadInfo(uploadInfo); + + // Step 3: Free local child data bytes (storage is now shared with parent) + storageService.deleteData(uploadInfo.getId()); + + log.debug( + "Successfully deduplicated child upload {} to parent {}", + uploadInfo.getId(), + expectedParentId); + return uploadInfo; + } + + private void syncParentExpiration(UploadInfo parentInfo, UploadInfo childInfo) + throws IOException { + Long childExpiration = childInfo.getExpirationTimestamp(); + Long parentExpiration = parentInfo.getExpirationTimestamp(); + if (childExpiration == null && parentExpiration != null) { + parentInfo.setExpirationTimestamp(null); + updateUploadInfo(parentInfo); + } else if (childExpiration != null + && parentExpiration != null + && childExpiration > parentExpiration) { + parentInfo.setExpirationTimestamp(childExpiration); + updateUploadInfo(parentInfo); + } + } + + /** + * Updates the checksum index when an upload is being terminated via DELETE or expiration sweep. + * + * @param uploadInfo the upload info being terminated + * @return {@code true} if the upload's data file can be deleted immediately by the caller; {@code + * false} if the upload is a parent with active children whose data must be preserved + * @throws IOException on storage I/O error + */ + public boolean removeUploadFromIndex(UploadInfo uploadInfo) throws IOException { + if (uploadInfo == null || uploadInfo.getId() == null) { + return true; + } + + UploadId currentId = uploadInfo.getId(); + ChecksumKey key; + try { + key = extractChecksumKey(uploadInfo); + } catch (IllegalArgumentException e) { + throw new IOException( + "The checksum contains an unsafe or invalid value: " + e.getMessage(), e); + } + + UploadId duplicatesId = uploadInfo.getDuplicatesUploadId(); + if (key == null && duplicatesId != null) { + UploadInfo parentInfo = storageService.getUploadInfo(duplicatesId); + if (parentInfo != null) { + key = extractChecksumKey(parentInfo); + } + } + + if (key == null) { + return true; + } + + if (duplicatesId != null) { + // Case 3: Terminating a child duplicate upload + return removeChildFromIndex(currentId, duplicatesId, key); + } else { + // Case 2: Terminating a parent upload + return removeParentFromIndex(currentId, key); + } + } + + /** + * Scans the checksum index and removes orphaned entries (where the parent data file/blob is + * missing, the parent metadata is missing without a tombstone, or a tombstoned parent has zero + * remaining children). + */ + public void pruneOrphanedChecksumIndices() { + Set allKeys; + try { + allKeys = checksumIndex.getAllIndexedKeys(); + } catch (Exception e) { + log.warn("Failed to retrieve indexed checksum keys for pruning: {}", e.getMessage()); + return; + } + for (ChecksumKey key : allKeys) { + try { + UploadId parentId = checksumIndex.findParent(key); + if (parentId != null) { + boolean parentDataExists = storageService.dataExists(parentId); + UploadInfo parentInfo = storageService.getUploadInfo(parentId); + boolean isTombstoned = checksumIndex.isParentDeleted(key); + boolean hasChildren = checksumIndex.hasChildren(key); + + boolean orphaned = + !parentDataExists + || (parentInfo == null && !isTombstoned) + || (isTombstoned && !hasChildren); + + if (orphaned) { + log.info( + "Pruning orphaned or abandoned checksum index for key {} (parent {})", + key, + parentId); + checksumIndex.deleteIndex(key); + if (!hasChildren) { + storageService.deleteData(parentId); + } + } + } + } catch (Exception e) { + log.warn("Error checking checksum index for key {}: {}", key, e.getMessage()); + } + } + } + + /** Handles removal of a parent upload from the index. */ + private boolean removeParentFromIndex(UploadId parentId, ChecksumKey key) throws IOException { + // B4 Check: Verify whether parentId is the registered parent in the checksum index + UploadId indexedParent = checksumIndex.findParent(key); + if (!Objects.equals(parentId, indexedParent)) { + // Unindexed standalone upload (e.g. deduplication skipped due to lock contention) + return true; + } + + if (checksumIndex.hasChildren(key)) { + // Active children exist: tombstone parent marker, preserve data bytes for children + checksumIndex.markParentDeleted(key); + log.debug("Tombstoned parent {} for checksum {} (active children remain)", parentId, key); + + // Post-tombstone race re-check: in case the last child exited while markParentDeleted was + // executing + if (!checksumIndex.hasChildren(key)) { + checksumIndex.deleteIndex(key); + log.debug("Purged parent index {} after last child exited during tombstoning", parentId); + return true; + } + return false; + } else { + // No children: delete checksum index directory + checksumIndex.deleteIndex(key); + log.debug("Fully removed unreferenced parent {} and checksum index for {}", parentId, key); + return true; + } + } + + /** Handles removal of a child duplicate upload from the index. */ + private boolean removeChildFromIndex(UploadId childId, UploadId parentId, ChecksumKey key) + throws IOException { + // Step 1: B4 Check - if parentId is the registered parent in index, remove child marker + UploadId indexedParent = checksumIndex.findParent(key); + if (Objects.equals(parentId, indexedParent)) { + checksumIndex.removeChild(key, childId); + + // Step 2: Check if parent was tombstoned and no other children remain + if (checksumIndex.isParentDeleted(key) && !checksumIndex.hasChildren(key)) { + purgeParentIfUnreferenced(key, parentId); + } + } + + return true; + } + + /** + * Under parent lock, purges parent data and checksum index if parent is tombstoned and has no + * active children. + */ + private void purgeParentIfUnreferenced(ChecksumKey key, UploadId parentId) throws IOException { + try { + try (UploadLock parentLock = lockingService.lockUpload(parentId)) { + purgeParentUnderLock(key, parentId); + } catch (TusException e) { + log.warn( + "Could not acquire parent lock {} during final child cleanup; skipping immediate purge", + parentId); + } + } finally { + cleanupLock(parentId); + } + } + + private void cleanupLock(UploadId id) { + if (id != null) { + try { + lockingService.cleanupLock(id); + } catch (Exception e) { + log.warn("Failed to clean up lock for upload {}: {}", id, e.getMessage()); + } + } + } + + private void purgeParentUnderLock(ChecksumKey key, UploadId parentId) throws IOException { + if (Objects.equals(parentId, checksumIndex.findParent(key)) + && checksumIndex.isParentDeleted(key) + && !checksumIndex.hasChildren(key)) { + storageService.deleteData(parentId); + checksumIndex.deleteIndex(key); + log.debug( + "Purged parent data {} and checksum index {} after final child deleted", parentId, key); + } + } + + private ChecksumKey extractChecksumKey(UploadInfo uploadInfo) { + if (uploadInfo.getChecksum() != null && uploadInfo.getChecksumAlgorithm() != null) { + return ChecksumKey.of(uploadInfo.getChecksum(), uploadInfo.getChecksumAlgorithm()); + } + return null; + } + + private void updateUploadInfo(UploadInfo info) throws IOException { + try { + storageService.update(info); + } catch (UploadNotFoundException e) { + throw new IOException("Failed to update upload info: " + e.getMessage(), e); + } + } +} diff --git a/src/main/java/me/desair/tus/server/upload/disk/DiskChecksumIndex.java b/src/main/java/me/desair/tus/server/upload/disk/DiskChecksumIndex.java new file mode 100644 index 00000000..2dba4002 --- /dev/null +++ b/src/main/java/me/desair/tus/server/upload/disk/DiskChecksumIndex.java @@ -0,0 +1,214 @@ +package me.desair.tus.server.upload.disk; + +import static java.nio.file.StandardOpenOption.CREATE_NEW; + +import java.io.IOException; +import java.nio.charset.StandardCharsets; +import java.nio.file.DirectoryStream; +import java.nio.file.FileAlreadyExistsException; +import java.nio.file.Files; +import java.nio.file.Path; +import java.nio.file.Paths; +import java.util.LinkedHashSet; +import java.util.Objects; +import java.util.Set; +import me.desair.tus.server.checksum.ChecksumAlgorithm; +import me.desair.tus.server.checksum.ChecksumKey; +import me.desair.tus.server.upload.UploadId; +import me.desair.tus.server.upload.checksum.ChecksumIndex; +import me.desair.tus.server.util.Utils; +import org.apache.commons.io.FileUtils; + +/** + * Disk-based implementation of {@link ChecksumIndex}. + * + *

Directory and file layout on disk: + * + *

    + *
  • {@code /checksums///parent} - contains parent upload + * ID + *
  • {@code /checksums///parent-deleted} - tombstone marker + *
  • {@code /checksums///children/} - child upload + * marker files + *
+ */ +public class DiskChecksumIndex implements ChecksumIndex { + + private static final String CHECKSUMS_SUB_DIRECTORY = "checksums"; + private static final String PARENT_FILE = "parent"; + private static final String PARENT_DELETED_FILE = "parent-deleted"; + private static final String CHILDREN_DIRECTORY = "children"; + + private final Path storagePath; + + /** + * Constructs a new {@link DiskChecksumIndex}. + * + * @param storagePath the base storage directory path + */ + public DiskChecksumIndex(String storagePath) { + Objects.requireNonNull(storagePath, "storagePath must not be null"); + this.storagePath = + Paths.get(storagePath).resolve(CHECKSUMS_SUB_DIRECTORY).normalize().toAbsolutePath(); + } + + @Override + public boolean tryRegisterParent(ChecksumKey key, UploadId uploadId) throws IOException { + Objects.requireNonNull(key, "key must not be null"); + Objects.requireNonNull(uploadId, "uploadId must not be null"); + + Path checksumDir = getChecksumDir(key); + Utils.ensureDirectoryExists(checksumDir); + + Path parentPath = checksumDir.resolve(PARENT_FILE); + // Atomic creation and write via CREATE_NEW (fails with FileAlreadyExistsException if already + // present, whether active or tombstoned) + try { + Files.write(parentPath, uploadId.toString().getBytes(StandardCharsets.UTF_8), CREATE_NEW); + return true; + } catch (FileAlreadyExistsException e) { + // CAS lost: another thread or process already created the parent marker + return false; + } + } + + @Override + public UploadId findParent(ChecksumKey key) throws IOException { + Objects.requireNonNull(key, "key must not be null"); + Path checksumDir = getChecksumDir(key); + Path parentPath = checksumDir.resolve(PARENT_FILE); + + if (Files.exists(parentPath)) { + String idStr = new String(Files.readAllBytes(parentPath), StandardCharsets.UTF_8).trim(); + return idStr.isEmpty() ? null : new UploadId(idStr); + } + return null; + } + + @Override + public void markParentDeleted(ChecksumKey key) throws IOException { + Objects.requireNonNull(key, "key must not be null"); + Path checksumDir = getChecksumDir(key); + Utils.ensureDirectoryExists(checksumDir); + + Path parentDeletedPath = checksumDir.resolve(PARENT_DELETED_FILE); + try { + // Create empty parent-deleted marker while keeping parent pointer file intact + // until deleteIndex(key) is invoked when the last child upload terminates. + Files.createFile(parentDeletedPath); + } catch (FileAlreadyExistsException ignored) { + // Tombstone already present + } + } + + @Override + public boolean isParentDeleted(ChecksumKey key) throws IOException { + Objects.requireNonNull(key, "key must not be null"); + Path checksumDir = getChecksumDir(key); + return Files.exists(checksumDir.resolve(PARENT_DELETED_FILE)); + } + + @Override + public void addChild(ChecksumKey key, UploadId childId) throws IOException { + Objects.requireNonNull(key, "key must not be null"); + Objects.requireNonNull(childId, "childId must not be null"); + + Path childrenDir = getChecksumDir(key).resolve(CHILDREN_DIRECTORY); + Utils.ensureDirectoryExists(childrenDir); + + Path childMarker = getChildMarker(childrenDir, childId); + try { + Files.createFile(childMarker); + } catch (FileAlreadyExistsException ignored) { + // Child marker already present + } + } + + @Override + public void removeChild(ChecksumKey key, UploadId childId) throws IOException { + Objects.requireNonNull(key, "key must not be null"); + Objects.requireNonNull(childId, "childId must not be null"); + + Path childrenDir = getChecksumDir(key).resolve(CHILDREN_DIRECTORY); + Path childMarker = getChildMarker(childrenDir, childId); + Files.deleteIfExists(childMarker); + } + + @Override + public boolean hasChildren(ChecksumKey key) throws IOException { + Objects.requireNonNull(key, "key must not be null"); + Path childrenDir = getChecksumDir(key).resolve(CHILDREN_DIRECTORY); + if (!Files.exists(childrenDir)) { + return false; + } + + try (DirectoryStream stream = Files.newDirectoryStream(childrenDir)) { + return stream.iterator().hasNext(); + } + } + + @Override + public void deleteIndex(ChecksumKey key) throws IOException { + Objects.requireNonNull(key, "key must not be null"); + Path checksumDir = getChecksumDir(key); + if (Files.exists(checksumDir)) { + FileUtils.deleteDirectory(checksumDir.toFile()); + } + } + + @Override + public Set getAllIndexedKeys() throws IOException { + Set keys = new LinkedHashSet<>(); + if (!Files.exists(storagePath)) { + return keys; + } + try (DirectoryStream algoDirs = Files.newDirectoryStream(storagePath)) { + for (Path algoDir : algoDirs) { + if (!Files.isDirectory(algoDir)) { + continue; + } + String algoName = algoDir.getFileName().toString(); + ChecksumAlgorithm algo = ChecksumAlgorithm.forTusName(algoName); + if (algo == null) { + continue; + } + try (DirectoryStream hashDirs = Files.newDirectoryStream(algoDir)) { + for (Path hashDir : hashDirs) { + if (!Files.isDirectory(hashDir)) { + continue; + } + Path parentFile = hashDir.resolve(PARENT_FILE); + if (Files.exists(parentFile)) { + String hash = hashDir.getFileName().toString(); + try { + keys.add(ChecksumKey.of(hash, algo)); + } catch (IllegalArgumentException e) { + // Malformed hash directory on disk, skip + } + } + } + } + } + } + return keys; + } + + private Path getChecksumDir(ChecksumKey key) { + Path path = storagePath.resolve(key.toString()); + if (!path.normalize().toAbsolutePath().startsWith(storagePath.normalize().toAbsolutePath())) { + throw new IllegalArgumentException( + "Checksum key is not valid and would result in a path traversal"); + } + return path; + } + + private Path getChildMarker(Path childrenDir, UploadId childId) { + Objects.requireNonNull(childId, "childId must not be null"); + Path path = childrenDir.resolve(childId.toString()); + if (!path.normalize().toAbsolutePath().startsWith(childrenDir.normalize().toAbsolutePath())) { + throw new IllegalArgumentException( + "Child upload ID is not valid and would result in a path traversal"); + } + return path; + } +} diff --git a/src/main/java/me/desair/tus/server/upload/disk/DiskLockingService.java b/src/main/java/me/desair/tus/server/upload/disk/DiskLockingService.java index 067fef49..72433830 100644 --- a/src/main/java/me/desair/tus/server/upload/disk/DiskLockingService.java +++ b/src/main/java/me/desair/tus/server/upload/disk/DiskLockingService.java @@ -70,20 +70,31 @@ protected void cleanupOnClose() throws IOException { } /** - * Attempts to lock the upload resource. Wraps the lock in a RegisteredLock decorator to manage - * cleanup of stop files and the active lock registry. + * Attempts to lock the upload resource by URI. Wraps the lock in a RegisteredLock decorator to + * manage cleanup of stop files and the active lock registry. */ @Override public UploadLock lockUploadByUri(String requestUri) throws TusException, IOException { - UploadId id = idFactory.readUploadId(requestUri); + return lockUploadInternal(id, requestUri); + } - UploadLock lock = null; + /** Attempts to lock the upload resource by UploadId directly without URI parsing. */ + @Override + public UploadLock lockUpload(UploadId id) throws TusException, IOException { + if (id == null) { + return null; + } + return lockUploadInternal(id, id.toString()); + } + private UploadLock lockUploadInternal(UploadId id, String uriOrId) + throws TusException, IOException { + UploadLock lock = null; Path lockPath = getLockPath(id); - // If lockPath is not null, we know this is a valid Upload URI + // If lockPath is not null, we know this is a valid Upload URI / ID if (lockPath != null) { - FileBasedLock baseLock = new FileBasedLock(requestUri, lockPath); + FileBasedLock baseLock = new FileBasedLock(uriOrId, lockPath); Path stopFilePath = baseLock.getLockPath().resolveSibling(id.toString() + ".stop"); lock = new RegisteredLock(baseLock, id.toString(), stopFilePath); } diff --git a/src/main/java/me/desair/tus/server/upload/disk/DiskStorageService.java b/src/main/java/me/desair/tus/server/upload/disk/DiskStorageService.java index 234e7dba..25be91e7 100644 --- a/src/main/java/me/desair/tus/server/upload/disk/DiskStorageService.java +++ b/src/main/java/me/desair/tus/server/upload/disk/DiskStorageService.java @@ -11,7 +11,6 @@ import java.nio.channels.FileChannel; import java.nio.channels.ReadableByteChannel; import java.nio.channels.WritableByteChannel; -import java.nio.charset.StandardCharsets; import java.nio.file.DirectoryStream; import java.nio.file.Files; import java.nio.file.Path; @@ -20,6 +19,7 @@ import java.util.List; import java.util.Objects; import me.desair.tus.server.checksum.ChecksumAlgorithm; +import me.desair.tus.server.checksum.ChecksumKey; import me.desair.tus.server.exception.InvalidUploadOffsetException; import me.desair.tus.server.exception.TusException; import me.desair.tus.server.exception.UploadNotFoundException; @@ -29,6 +29,7 @@ import me.desair.tus.server.upload.UploadLockingService; import me.desair.tus.server.upload.UploadStorageService; import me.desair.tus.server.upload.UploadType; +import me.desair.tus.server.upload.checksum.DeduplicationService; import me.desair.tus.server.upload.concatenation.UploadConcatenationService; import me.desair.tus.server.upload.concatenation.VirtualConcatenationService; import me.desair.tus.server.util.UploadInfoJsonSerializer; @@ -59,10 +60,17 @@ public class DiskStorageService extends AbstractDiskBasedService implements Uplo private UploadConcatenationService uploadConcatenationService; private boolean isUploadDeduplicationEnabled = false; private boolean jsonSerializationEnabled = false; + private final DiskChecksumIndex checksumIndex; + private DeduplicationService deduplicationService; + private UploadLockingService uploadLockingService; public DiskStorageService(String storagePath) { super(storagePath + File.separator + UPLOAD_SUB_DIRECTORY); setUploadConcatenationService(new VirtualConcatenationService(this)); + this.checksumIndex = new DiskChecksumIndex(storagePath); + this.deduplicationService = new DeduplicationService(this.checksumIndex, this); + this.uploadLockingService = new DiskLockingService(storagePath); + this.deduplicationService.setUploadLockingService(this.uploadLockingService); } public DiskStorageService(UploadIdFactory idFactory, String storagePath) { @@ -154,35 +162,46 @@ public boolean isUploadDeduplicationEnabled() { return this.isUploadDeduplicationEnabled; } + @Override + public void setUploadLockingService(UploadLockingService uploadLockingService) { + this.uploadLockingService = + Objects.requireNonNull(uploadLockingService, "uploadLockingService must not be null"); + if (this.deduplicationService != null) { + this.deduplicationService.setUploadLockingService(uploadLockingService); + } + } + + @Override + public UploadLockingService getUploadLockingService() { + return this.uploadLockingService; + } + + @Override + public UploadInfo deduplicate(UploadInfo uploadInfo, ChecksumKey checksumKey) throws IOException { + if (!isUploadDeduplicationEnabled() || uploadInfo == null || checksumKey == null) { + return uploadInfo; + } + return deduplicationService.deduplicate(uploadInfo, checksumKey); + } + @Override public UploadInfo getUploadInfoByChecksum(String checksum, ChecksumAlgorithm algorithm) throws IOException { if (checksum == null || algorithm == null) { return null; } - Path checksumFile = getChecksumPath(checksum, algorithm); - if (Files.exists(checksumFile)) { - String uploadIdStr = - new String(Files.readAllBytes(checksumFile), StandardCharsets.UTF_8).trim(); - UploadId id = new UploadId(uploadIdStr); - UploadInfo info = getUploadInfo(id); - if (info == null) { - Files.deleteIfExists(checksumFile); - return null; - } - Path bytesPath = null; - try { - bytesPath = getPathInUploadDir(id, DATA_FILE); - } catch (UploadNotFoundException e) { - // file doesn't exist - } - if (bytesPath == null || !Files.exists(bytesPath)) { - Files.deleteIfExists(checksumFile); - return null; - } - return info; + String normalized = normalizeChecksum(checksum); + if (!isSafePathComponent(normalized) || !isSafePathComponent(algorithm.toString())) { + throw new IOException("The checksum contains an unsafe value."); + } + try { + ChecksumKey key = ChecksumKey.of(normalized, algorithm); + return deduplicationService != null + ? deduplicationService.getUploadInfoByChecksum(key) + : null; + } catch (IllegalArgumentException e) { + return null; } - return null; } @Override @@ -285,6 +304,9 @@ public UploadInfo create(UploadInfo info, String ownerKey) throws IOException { @Override public void update(UploadInfo uploadInfo) throws IOException, UploadNotFoundException { if (uploadInfo != null) { + Path infoPath = getInfoPath(uploadInfo.getId()); + saveUploadInfo(uploadInfo, infoPath); + if (uploadInfo.getDuplicatesUploadId() != null) { // Delete the child's own data file if it exists try { @@ -327,16 +349,20 @@ public void update(UploadInfo uploadInfo) throws IOException, UploadNotFoundExce } else if (isUploadDeduplicationEnabled() && !uploadInfo.isUploadInProgress() && uploadInfo.getChecksum() != null - && uploadInfo.getChecksumAlgorithm() != null) { - // Index the checksum - Path checksumFile = - getChecksumPath(uploadInfo.getChecksum(), uploadInfo.getChecksumAlgorithm()); - Utils.ensureDirectoryExists(checksumFile.getParent()); - Files.write(checksumFile, uploadInfo.getId().toString().getBytes(StandardCharsets.UTF_8)); + && uploadInfo.getChecksumAlgorithm() != null + && uploadInfo.getDuplicatesUploadId() == null) { + String normalized = normalizeChecksum(uploadInfo.getChecksum()); + if (!isSafePathComponent(normalized) + || !isSafePathComponent(uploadInfo.getChecksumAlgorithm().toString())) { + throw new IOException("The checksum contains an unsafe value."); + } + try { + ChecksumKey key = ChecksumKey.of(normalized, uploadInfo.getChecksumAlgorithm()); + deduplicate(uploadInfo, key); + } catch (IllegalArgumentException e) { + throw new IOException("The checksum contains an unsafe value.", e); + } } - - Path infoPath = getInfoPath(uploadInfo.getId()); - saveUploadInfo(uploadInfo, infoPath); } } @@ -418,19 +444,85 @@ public void removeLastNumberOfBytes(UploadInfo info, long byteCount) } } + @Override + public boolean dataExists(UploadId id) { + if (id == null) { + return false; + } + try { + Path bytesPath = getBytesPath(id); + return bytesPath != null && Files.exists(bytesPath); + } catch (UploadNotFoundException e) { + return false; + } + } + + @Override + public void deleteData(UploadId id) throws IOException { + if (id != null) { + try { + Path bytesPath = getBytesPath(id); + if (bytesPath != null) { + Files.deleteIfExists(bytesPath); + } + } catch (UploadNotFoundException ignored) { + } + cleanDirectoryIfEmpty(id); + } + } + + private void deleteMetadata(UploadId id) throws IOException { + if (id != null) { + try { + Path infoPath = getInfoPath(id); + if (infoPath != null) { + Files.deleteIfExists(infoPath); + } + } catch (UploadNotFoundException ignored) { + } + cleanDirectoryIfEmpty(id); + } + } + + private void cleanDirectoryIfEmpty(UploadId id) { + Path uploadDir = getPathInStorageDirectory(id); + if (uploadDir != null && Files.exists(uploadDir)) { + try (DirectoryStream stream = Files.newDirectoryStream(uploadDir)) { + if (!stream.iterator().hasNext()) { + Files.deleteIfExists(uploadDir); + } + } catch (IOException ignored) { + } + } + } + @Override public void terminateUpload(UploadInfo info) throws UploadNotFoundException, IOException { if (info != null) { - if (info.getDuplicatesUploadId() == null - && info.getChecksum() != null - && info.getChecksumAlgorithm() != null) { - Path checksumFile = getChecksumPath(info.getChecksum(), info.getChecksumAlgorithm()); - Files.deleteIfExists(checksumFile); + if (info.getChecksum() != null && info.getChecksumAlgorithm() != null) { + String normalized = normalizeChecksum(info.getChecksum()); + if (!isSafePathComponent(normalized) + || !isSafePathComponent(info.getChecksumAlgorithm().toString())) { + throw new IOException("The checksum contains an unsafe value."); + } + } + + boolean deleteData = true; + if (isUploadDeduplicationEnabled() && deduplicationService != null) { + deleteData = deduplicationService.removeUploadFromIndex(info); } + + if (deleteData) { + deleteData(info.getId()); + } + deleteMetadata(info.getId()); + Path uploadPath = getPathInStorageDirectory(info.getId()); - if (uploadPath != null) { + if (uploadPath != null && deleteData) { FileUtils.deleteDirectory(uploadPath.toFile()); } + + uploadLockingService.cleanupLock(info.getId()); } } @@ -478,21 +570,19 @@ public InputStream getUploadedBytes(UploadId id) throws IOException, UploadNotFo throw new UploadNotFoundException("The upload with id " + id + " was not found."); } - if (uploadInfo.getDuplicatesUploadId() != null) { - return getUploadedBytes(uploadInfo.getDuplicatesUploadId()); - } - if (UploadType.CONCATENATED.equals(uploadInfo.getUploadType()) && uploadConcatenationService != null) { inputStream = uploadConcatenationService.getConcatenatedBytes(uploadInfo); } else { - Path bytesPath = getBytesPath(id); + UploadId targetDataId = + uploadInfo.getDuplicatesUploadId() != null ? uploadInfo.getDuplicatesUploadId() : id; + Path bytesPath = getBytesPath(targetDataId); // If bytesPath is not null, we know this is a valid Upload URI if (bytesPath != null) { if (!Files.exists(bytesPath)) { throw new UploadNotFoundException( - "The upload bytes for id " + id + " could not be found."); + "The upload bytes for id " + targetDataId + " could not be found."); } inputStream = Channels.newInputStream(FileChannel.open(bytesPath, READ)); } @@ -548,18 +638,38 @@ public void copyUploadTo(UploadInfo info, OutputStream outputStream) } @Override - public void cleanupExpiredUploads(UploadLockingService uploadLockingService) throws IOException { + public void cleanupExpiredUploads() throws IOException { try (DirectoryStream expiredUploadsStream = Files.newDirectoryStream( - getStoragePath(), new ExpiredUploadFilter(this, uploadLockingService))) { + getStoragePath(), new ExpiredUploadFilter(this, this.uploadLockingService))) { for (Path path : expiredUploadsStream) { - FileUtils.deleteDirectory(path.toFile()); - if (uploadLockingService != null) { - uploadLockingService.cleanupLock(new UploadId(path.getFileName().toString())); + UploadId id = new UploadId(path.getFileName().toString()); + UploadInfo info = getUploadInfo(id); + if (info != null) { + try { + terminateUpload(info); + } catch (UploadNotFoundException e) { + FileUtils.deleteDirectory(path.toFile()); + this.uploadLockingService.cleanupLock(id); + } + } else { + FileUtils.deleteDirectory(path.toFile()); + this.uploadLockingService.cleanupLock(id); } } } + + if (isUploadDeduplicationEnabled() && deduplicationService != null) { + deduplicationService.pruneOrphanedChecksumIndices(); + } + } + + @Override + @Deprecated + public void cleanupExpiredUploads(UploadLockingService uploadLockingService) throws IOException { + setUploadLockingService(Utils.getLockingServiceOrDefault(uploadLockingService)); + cleanupExpiredUploads(); } private List getUploads(UploadInfo info) throws IOException, UploadNotFoundException { @@ -584,7 +694,7 @@ private Path getInfoPath(UploadId id) throws UploadNotFoundException { return getPathInUploadDir(id, INFO_FILE); } - private Path getChecksumPath(String checksum, ChecksumAlgorithm algorithm) throws IOException { + private String normalizeChecksum(String checksum) { String filename = checksum; if (filename != null) { if (!filename.matches("^[0-9a-fA-F]+$")) { @@ -598,18 +708,7 @@ private Path getChecksumPath(String checksum, ChecksumAlgorithm algorithm) throw } } } - - if (!isSafePathComponent(filename)) { - throw new IOException("The checksum contains an unsafe value."); - } - if (!isSafePathComponent(algorithm.toString())) { - throw new IOException("The checksum algorithm contains an unsafe value."); - } - return getStoragePath() - .getParent() - .resolve("checksums") - .resolve(algorithm.toString()) - .resolve(filename); + return filename; } /** diff --git a/src/main/java/me/desair/tus/server/upload/s3/S3ChecksumIndex.java b/src/main/java/me/desair/tus/server/upload/s3/S3ChecksumIndex.java new file mode 100644 index 00000000..f47a2052 --- /dev/null +++ b/src/main/java/me/desair/tus/server/upload/s3/S3ChecksumIndex.java @@ -0,0 +1,366 @@ +package me.desair.tus.server.upload.s3; + +import io.minio.GetObjectArgs; +import io.minio.ListObjectsArgs; +import io.minio.MinioClient; +import io.minio.PutObjectArgs; +import io.minio.RemoveObjectArgs; +import io.minio.RemoveObjectsArgs; +import io.minio.Result; +import io.minio.StatObjectArgs; +import io.minio.StatObjectResponse; +import io.minio.errors.ErrorResponseException; +import io.minio.messages.DeleteRequest; +import io.minio.messages.DeleteResult; +import io.minio.messages.Item; +import java.io.ByteArrayInputStream; +import java.io.IOException; +import java.io.InputStream; +import java.nio.charset.StandardCharsets; +import java.util.ArrayList; +import java.util.Collections; +import java.util.LinkedHashSet; +import java.util.List; +import java.util.Objects; +import java.util.Set; +import java.util.concurrent.locks.ReadWriteLock; +import java.util.concurrent.locks.ReentrantReadWriteLock; +import me.desair.tus.server.checksum.ChecksumAlgorithm; +import me.desair.tus.server.checksum.ChecksumKey; +import me.desair.tus.server.upload.UploadId; +import me.desair.tus.server.upload.checksum.ChecksumIndex; +import org.apache.commons.io.IOUtils; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +/** + * S3-compatible implementation of {@link ChecksumIndex}. + * + *

Storage layout: + * + *

    + *
  • {@code ///parent} - contains parent upload ID + *
  • {@code ///parent-deleted} - tombstone marker + *
  • {@code ///children/} - child upload + * markers + *
+ */ +public class S3ChecksumIndex implements ChecksumIndex { + + private static final Logger log = LoggerFactory.getLogger(S3ChecksumIndex.class); + + private static final String DEFAULT_CHECKSUMS_PREFIX = "checksums/"; + private static final String PARENT_OBJECT = "parent"; + private static final String PARENT_DELETED_OBJECT = "parent-deleted"; + private static final String CHILDREN_PREFIX = "children/"; + + private final MinioClient minioClient; + private final String bucket; + private final String checksumsPrefix; + + private final ReadWriteLock thisObjectLock = new ReentrantReadWriteLock(); + private boolean s3ConditionalWritesSupported = true; + + /** + * Constructs a new {@link S3ChecksumIndex}. + * + * @param minioClient the MinioClient instance + * @param bucket target bucket name + */ + public S3ChecksumIndex(MinioClient minioClient, String bucket) { + this(minioClient, bucket, DEFAULT_CHECKSUMS_PREFIX); + } + + /** + * Constructs a new {@link S3ChecksumIndex} with a custom checksums prefix. + * + * @param minioClient the MinioClient instance + * @param bucket target bucket name + * @param checksumsPrefix the prefix path for checksums + */ + public S3ChecksumIndex(MinioClient minioClient, String bucket, String checksumsPrefix) { + this.minioClient = Objects.requireNonNull(minioClient, "MinioClient must not be null"); + this.bucket = Objects.requireNonNull(bucket, "Bucket must not be null"); + this.checksumsPrefix = sanitizePrefix(checksumsPrefix); + } + + /** + * Returns whether conditional writes (If-None-Match: *) are supported by the S3 endpoint. + * + * @return {@code true} if supported, {@code false} otherwise + */ + public boolean isS3ConditionalWritesSupported() { + thisObjectLock.readLock().lock(); + try { + return s3ConditionalWritesSupported; + } finally { + thisObjectLock.readLock().unlock(); + } + } + + /** + * Configures whether conditional writes (If-None-Match: *) should be attempted. + * + * @param supported {@code true} to enable, {@code false} to disable + */ + public void setS3ConditionalWritesSupported(boolean supported) { + thisObjectLock.writeLock().lock(); + try { + this.s3ConditionalWritesSupported = supported; + } finally { + thisObjectLock.writeLock().unlock(); + } + } + + @Override + public boolean tryRegisterParent(ChecksumKey key, UploadId uploadId) throws IOException { + Objects.requireNonNull(key, "key must not be null"); + Objects.requireNonNull(uploadId, "uploadId must not be null"); + + String parentKey = buildKey(key, PARENT_OBJECT); + byte[] content = uploadId.toString().getBytes(StandardCharsets.UTF_8); + + boolean conditionalSupported; + thisObjectLock.readLock().lock(); + try { + conditionalSupported = s3ConditionalWritesSupported; + } finally { + thisObjectLock.readLock().unlock(); + } + + if (conditionalSupported) { + try { + minioClient.putObject( + PutObjectArgs.builder() + .bucket(bucket) + .object(parentKey) + .extraHeaders(Collections.singletonMap("If-None-Match", "*")) + .stream(new ByteArrayInputStream(content), (long) content.length, -1L) + .build()); + return true; + } catch (ErrorResponseException e) { + S3ErrorType errorType = S3Utils.parseErrorResponse(e); + if (errorType == S3ErrorType.PRECONDITION_FAILED || errorType == S3ErrorType.CONFLICT) { + // Object already exists (lost CAS race, whether active or tombstoned) + return false; + } + if (errorType == S3ErrorType.API_NOT_IMPLEMENTED) { + log.info("S3 endpoint does not support If-None-Match on putObject; downgrading CAS"); + thisObjectLock.writeLock().lock(); + try { + s3ConditionalWritesSupported = false; + } finally { + thisObjectLock.writeLock().unlock(); + } + // Fall through to non-CAS mode + } else { + throw new IOException("Failed to put parent index object in S3: " + parentKey, e); + } + } catch (Exception e) { + throw new IOException("Failed to put parent index object in S3: " + parentKey, e); + } + } + + // Non-CAS fallback mode (fail-closed check) + if (objectExists(parentKey)) { + return false; + } + putObjectUnconditional(parentKey, content); + return true; + } + + @Override + public UploadId findParent(ChecksumKey key) throws IOException { + Objects.requireNonNull(key, "key must not be null"); + String parentKey = buildKey(key, PARENT_OBJECT); + + try (InputStream stream = + minioClient.getObject(GetObjectArgs.builder().bucket(bucket).object(parentKey).build())) { + String idStr = IOUtils.toString(stream, StandardCharsets.UTF_8).trim(); + return idStr.isEmpty() ? null : new UploadId(idStr); + } catch (ErrorResponseException e) { + if (S3Utils.parseErrorResponse(e) == S3ErrorType.NO_SUCH_KEY) { + return null; + } + throw new IOException("Failed to get parent index object from S3: " + parentKey, e); + } catch (Exception e) { + throw new IOException("Failed to get parent index object from S3: " + parentKey, e); + } + } + + @Override + public void markParentDeleted(ChecksumKey key) throws IOException { + Objects.requireNonNull(key, "key must not be null"); + String parentDeletedKey = buildKey(key, PARENT_DELETED_OBJECT); + + // Write empty tombstone marker while keeping parent pointer object intact + // until deleteIndex(key) is invoked when the last child terminates. + putObjectUnconditional(parentDeletedKey, new byte[0]); + } + + @Override + public boolean isParentDeleted(ChecksumKey key) throws IOException { + Objects.requireNonNull(key, "key must not be null"); + return objectExists(buildKey(key, PARENT_DELETED_OBJECT)); + } + + @Override + public void addChild(ChecksumKey key, UploadId childId) throws IOException { + Objects.requireNonNull(key, "key must not be null"); + Objects.requireNonNull(childId, "childId must not be null"); + String childKey = buildKey(key, CHILDREN_PREFIX + childId); + putObjectUnconditional(childKey, new byte[0]); + } + + @Override + public void removeChild(ChecksumKey key, UploadId childId) throws IOException { + Objects.requireNonNull(key, "key must not be null"); + Objects.requireNonNull(childId, "childId must not be null"); + String childKey = buildKey(key, CHILDREN_PREFIX + childId); + deleteObjectQuietly(childKey); + } + + @Override + public boolean hasChildren(ChecksumKey key) throws IOException { + Objects.requireNonNull(key, "key must not be null"); + String prefix = buildKey(key, CHILDREN_PREFIX); + + try { + Iterable> results = + minioClient.listObjects( + ListObjectsArgs.builder().bucket(bucket).prefix(prefix).maxKeys(1).build()); + for (Result result : results) { + if (result.get() != null) { + return true; + } + } + return false; + } catch (Exception e) { + throw new IOException("Failed to list children in S3 for prefix " + prefix, e); + } + } + + @Override + public void deleteIndex(ChecksumKey key) throws IOException { + Objects.requireNonNull(key, "key must not be null"); + String prefix = buildKey(key, ""); + + try { + Iterable> results = + minioClient.listObjects( + ListObjectsArgs.builder().bucket(bucket).prefix(prefix).recursive(true).build()); + + List toDelete = new ArrayList<>(); + for (Result result : results) { + Item item = result.get(); + if (item != null && item.objectName() != null) { + toDelete.add(new DeleteRequest.Object(item.objectName())); + } + } + + if (!toDelete.isEmpty()) { + Iterable> errors = + minioClient.removeObjects( + RemoveObjectsArgs.builder().bucket(bucket).objects(toDelete).build()); + for (Result errorResult : errors) { + DeleteResult.Error error = errorResult.get(); + log.warn( + "Error deleting S3 object {} in checksum index: {}", + error.objectName(), + error.message()); + } + } + } catch (Exception e) { + throw new IOException("Failed to delete checksum index in S3 for prefix " + prefix, e); + } + } + + @Override + public Set getAllIndexedKeys() throws IOException { + Set keys = new LinkedHashSet<>(); + try { + Iterable> results = + minioClient.listObjects( + ListObjectsArgs.builder() + .bucket(bucket) + .prefix(checksumsPrefix) + .recursive(true) + .build()); + for (Result result : results) { + Item item = result.get(); + if (item == null || item.objectName() == null) { + continue; + } + String objectName = item.objectName(); + if (objectName.endsWith("/" + PARENT_OBJECT)) { + String relative = objectName.substring(checksumsPrefix.length()); + if (relative.startsWith("/")) { + relative = relative.substring(1); + } + String[] parts = relative.split("/"); + if (parts.length == 3 && PARENT_OBJECT.equals(parts[2])) { + ChecksumAlgorithm algo = ChecksumAlgorithm.forTusName(parts[0]); + if (algo != null) { + try { + keys.add(ChecksumKey.of(parts[1], algo)); + } catch (IllegalArgumentException e) { + // Malformed checksum key, skip + } + } + } + } + } + } catch (Exception e) { + throw new IOException("Failed to list indexed checksum keys from S3", e); + } + return keys; + } + + private String buildKey(ChecksumKey key, String subPath) { + return checksumsPrefix + key + (subPath.isEmpty() ? "" : "/" + subPath); + } + + private boolean objectExists(String objectKey) { + try { + StatObjectResponse stat = + minioClient.statObject(StatObjectArgs.builder().bucket(bucket).object(objectKey).build()); + return stat != null; + } catch (ErrorResponseException e) { + return false; + } catch (Exception e) { + return false; + } + } + + private void putObjectUnconditional(String objectKey, byte[] data) throws IOException { + try { + minioClient.putObject( + PutObjectArgs.builder().bucket(bucket).object(objectKey).stream( + new ByteArrayInputStream(data), (long) data.length, -1L) + .build()); + } catch (Exception e) { + throw new IOException("Failed to put object in S3: " + objectKey, e); + } + } + + private void deleteObjectQuietly(String objectKey) { + try { + minioClient.removeObject(RemoveObjectArgs.builder().bucket(bucket).object(objectKey).build()); + } catch (Exception ignored) { + } + } + + private static String sanitizePrefix(String prefix) { + if (prefix == null || prefix.trim().isEmpty()) { + return ""; + } + String trimmed = prefix.trim(); + while (trimmed.startsWith("/")) { + trimmed = trimmed.substring(1); + } + if (!trimmed.isEmpty() && !trimmed.endsWith("/")) { + trimmed = trimmed + "/"; + } + return trimmed; + } +} diff --git a/src/main/java/me/desair/tus/server/upload/s3/S3LockingService.java b/src/main/java/me/desair/tus/server/upload/s3/S3LockingService.java index eddf2b45..48586424 100644 --- a/src/main/java/me/desair/tus/server/upload/s3/S3LockingService.java +++ b/src/main/java/me/desair/tus/server/upload/s3/S3LockingService.java @@ -137,6 +137,19 @@ public S3LockingService( idFactory); } + /** + * Package-private constructor accepting {@link MinioClient} with default lease and poll settings. + */ + S3LockingService(MinioClient minioClient, String bucket, String locksPrefix) { + this( + minioClient, + bucket, + locksPrefix, + DEFAULT_LEASE_DURATION_MS, + DEFAULT_POLL_INTERVAL_MS, + new UuidUploadIdFactory()); + } + /** * Package-private constructor accepting {@link MinioClient} where all parameter configuration is * concentrated. diff --git a/src/main/java/me/desair/tus/server/upload/s3/S3StorageService.java b/src/main/java/me/desair/tus/server/upload/s3/S3StorageService.java index 1d0f50e7..b062aca8 100644 --- a/src/main/java/me/desair/tus/server/upload/s3/S3StorageService.java +++ b/src/main/java/me/desair/tus/server/upload/s3/S3StorageService.java @@ -42,6 +42,7 @@ import java.util.concurrent.locks.ReadWriteLock; import java.util.concurrent.locks.ReentrantReadWriteLock; import me.desair.tus.server.checksum.ChecksumAlgorithm; +import me.desair.tus.server.checksum.ChecksumKey; import me.desair.tus.server.exception.MaxAppendSizeExceededException; import me.desair.tus.server.exception.MaxUploadLengthExceededException; import me.desair.tus.server.exception.MinAppendSizeNotMetException; @@ -55,6 +56,7 @@ import me.desair.tus.server.upload.UploadStorageService; import me.desair.tus.server.upload.UploadType; import me.desair.tus.server.upload.UuidUploadIdFactory; +import me.desair.tus.server.upload.checksum.DeduplicationService; import me.desair.tus.server.upload.concatenation.UploadConcatenationService; import me.desair.tus.server.upload.util.AsyncChunkUploader; import me.desair.tus.server.util.InterruptibleInputStream; @@ -127,6 +129,9 @@ public class S3StorageService implements UploadStorageService { private Long minSize; private Long uploadExpirationPeriod; private boolean deduplicationEnabled = false; + private final S3ChecksumIndex checksumIndex; + private DeduplicationService deduplicationService; + private UploadLockingService uploadLockingService; private UploadIdFactory idFactory = new UuidUploadIdFactory(); private UploadConcatenationService concatenationService; @@ -269,6 +274,12 @@ public S3StorageService( this.temporaryDirectory, DEFAULT_MIN_PART_SIZE, this.s3ComposeHelper); + + this.checksumIndex = new S3ChecksumIndex(this.minioClient, this.bucket, this.checksumsPrefix); + this.deduplicationService = new DeduplicationService(this.checksumIndex, this); + this.uploadLockingService = + new S3LockingService(this.minioClient, this.bucket, this.locksPrefix); + this.deduplicationService.setUploadLockingService(this.uploadLockingService); } private static MinioClient buildMinioClient( @@ -504,8 +515,7 @@ public UploadInfo append(UploadInfo upload, InputStream inputStream) return info; } - @Override - public void update(UploadInfo uploadInfo) throws IOException, UploadNotFoundException { + private void writeMetadataObject(UploadInfo uploadInfo) throws IOException { if (uploadInfo == null || uploadInfo.getId() == null) { return; } @@ -524,14 +534,24 @@ public void update(UploadInfo uploadInfo) throws IOException, UploadNotFoundExce throw new IOException( "Failed to write metadata object to S3 for ID " + uploadInfo.getId(), e); } + } + + @Override + public void update(UploadInfo uploadInfo) throws IOException, UploadNotFoundException { + writeMetadataObject(uploadInfo); // Index checksum for deduplication if upload is completed and deduplication is enabled if (isUploadDeduplicationEnabled() && uploadInfo.getChecksum() != null && !uploadInfo.isUploadInProgress() && uploadInfo.getDuplicatesUploadId() == null) { - putChecksumIndex( - uploadInfo.getChecksum(), uploadInfo.getChecksumAlgorithm(), uploadInfo.getId()); + try { + ChecksumKey key = + ChecksumKey.of(uploadInfo.getChecksum(), uploadInfo.getChecksumAlgorithm()); + deduplicate(uploadInfo, key); + } catch (Exception e) { + log.warn("Failed to index checksum for upload {}: {}", uploadInfo.getId(), e.getMessage()); + } } } @@ -552,11 +572,6 @@ public InputStream getUploadedBytes(UploadId id) throws IOException, UploadNotFo throw new UploadNotFoundException("Upload with ID " + id + " was not found"); } - // Resolve duplicate upload reference to parent upload if deduplicated - if (info.getDuplicatesUploadId() != null) { - return getUploadedBytes(info.getDuplicatesUploadId()); - } - // Handle concatenated upload resolution if applicable if (UploadType.CONCATENATED.equals(info.getUploadType()) && info.isUploadInProgress()) { if (concatenationService != null) { @@ -577,7 +592,7 @@ public void copyUploadTo(UploadInfo info, OutputStream outputStream) } @Override - public void cleanupExpiredUploads(UploadLockingService uploadLockingService) throws IOException { + public void cleanupExpiredUploads() throws IOException { // 1. Clean up orphaned local temporary chunk files Utils.cleanupTempFiles(this.temporaryDirectory, "tus-s3-chunk-*.tmp", 24L * 3600_000L); Utils.cleanupTempFiles(this.temporaryDirectory, "tus-s3-prep-*.tmp", 24L * 3600_000L); @@ -604,23 +619,27 @@ public void cleanupExpiredUploads(UploadLockingService uploadLockingService) thr if (info != null && info.isUploadInProgress() && info.isExpired() - && (uploadLockingService == null || !uploadLockingService.isLocked(id))) { + && !this.uploadLockingService.isLocked(id)) { terminateUpload(info); - if (uploadLockingService != null) { - // Clean up any remaining lock files now that the expired upload is terminated - uploadLockingService.cleanupLock(id); - } } } } - // 3. Prune orphaned checksum index entries pointing to missing or expired uploads - pruneOrphanedChecksumIndices(); + if (isUploadDeduplicationEnabled() && deduplicationService != null) { + deduplicationService.pruneOrphanedChecksumIndices(); + } } catch (Exception e) { throw new IOException("Failed to cleanup expired S3 uploads", e); } } + @Override + @Deprecated + public void cleanupExpiredUploads(UploadLockingService uploadLockingService) throws IOException { + setUploadLockingService(Utils.getLockingServiceOrDefault(uploadLockingService)); + cleanupExpiredUploads(); + } + @Override public void removeLastNumberOfBytes(UploadInfo uploadInfo, long byteCount) throws UploadNotFoundException, IOException { @@ -648,30 +667,67 @@ public void removeLastNumberOfBytes(UploadInfo uploadInfo, long byteCount) update(uploadInfo); } + @Override + public void setUploadLockingService(UploadLockingService uploadLockingService) { + this.uploadLockingService = + Objects.requireNonNull(uploadLockingService, "uploadLockingService must not be null"); + if (this.deduplicationService != null) { + this.deduplicationService.setUploadLockingService(uploadLockingService); + } + } + + @Override + public UploadLockingService getUploadLockingService() { + return this.uploadLockingService; + } + + @Override + public UploadInfo deduplicate(UploadInfo uploadInfo, ChecksumKey checksumKey) throws IOException { + if (!isUploadDeduplicationEnabled() || uploadInfo == null || checksumKey == null) { + return uploadInfo; + } + return deduplicationService.deduplicate(uploadInfo, checksumKey); + } + + @Override + public boolean dataExists(UploadId id) { + if (id == null) { + return false; + } + return objectExists(buildObjectKey(id)); + } + + @Override + public void deleteData(UploadId id) { + if (id != null) { + deleteObjectQuietly(buildObjectKey(id)); + deleteAllPartObjectsQuietly(id); + } + } + + private void deleteMetadata(UploadId id) { + if (id != null) { + deleteObjectQuietly(buildMetadataKey(id)); + } + } + @Override public void terminateUpload(UploadInfo uploadInfo) throws UploadNotFoundException, IOException { if (uploadInfo == null || uploadInfo.getId() == null) { return; } - String objectKey = getS3ObjectKey(uploadInfo); - String metadataKey = buildMetadataKey(uploadInfo.getId()); - // Delete final object and metadata object from S3 - deleteObjectQuietly(objectKey); - deleteObjectQuietly(metadataKey); - - // Delete all temporary part chunk objects (e.g. metadata/.part.00001) - deleteAllPartObjectsQuietly(uploadInfo.getId()); + boolean deleteData = true; + if (isUploadDeduplicationEnabled() && deduplicationService != null) { + deleteData = deduplicationService.removeUploadFromIndex(uploadInfo); + } - // Delete checksum deduplication index object if present - if (uploadInfo.getChecksum() != null && uploadInfo.getChecksumAlgorithm() != null) { - deleteObjectQuietly( - buildChecksumKey(uploadInfo.getChecksum(), uploadInfo.getChecksumAlgorithm())); + if (deleteData) { + deleteData(uploadInfo.getId()); } + deleteMetadata(uploadInfo.getId()); - // Delete lock target and stop signal objects - deleteObjectQuietly(buildLockKey(uploadInfo.getId())); - deleteObjectQuietly(buildStopKey(uploadInfo.getId())); + uploadLockingService.cleanupLock(uploadInfo.getId()); } @Override @@ -680,85 +736,47 @@ public UploadInfo getUploadInfoByChecksum(String checksum, ChecksumAlgorithm alg if (!isUploadDeduplicationEnabled() || checksum == null || algorithm == null) { return null; } - - String checksumKey = buildChecksumKey(checksum, algorithm); - String parentIdStr; - try (InputStream stream = - minioClient.getObject(GetObjectArgs.builder().bucket(bucket).object(checksumKey).build())) { - parentIdStr = IOUtils.toString(stream, StandardCharsets.UTF_8).trim(); - } catch (ErrorResponseException e) { - if (S3Utils.parseErrorResponse(e) == S3ErrorType.NO_SUCH_KEY) { - return null; - } - throw new IOException("Failed to read checksum index object from S3", e); + ChecksumKey key; + try { + key = ChecksumKey.of(checksum, algorithm); } catch (Exception e) { - throw new IOException("Failed to read checksum index object from S3", e); - } - - UploadId parentId = new UploadId(parentIdStr); - UploadInfo parentInfo = getUploadInfo(parentId); - // Self-cleaning: if index points to missing parent upload or object, prune stale index - if (parentInfo == null || !objectExists(buildObjectKey(parentId))) { - deleteObjectQuietly(checksumKey); return null; } + return deduplicationService != null ? deduplicationService.getUploadInfoByChecksum(key) : null; + } - return parentInfo; + // CONFIGURATION SETTERS & GETTERS + + /** + * Configures whether S3 conditional writes (If-None-Match: *) are supported by the S3 endpoint. + * + * @param supported {@code true} to enable, {@code false} to disable + * @return this storage service instance + */ + public S3StorageService withS3ConditionalWritesSupported(boolean supported) { + setS3ConditionalWritesSupported(supported); + return this; } /** - * Scans and prunes orphaned checksum deduplication index objects in S3 whose parent upload or - * final data object no longer exists (e.g. pruned by external S3 lifecycle rules). + * Sets whether S3 conditional writes (If-None-Match: *) are supported by the S3 endpoint. + * + * @param supported {@code true} to enable, {@code false} to disable */ - private void pruneOrphanedChecksumIndices() { - // 1. Skip scanning if deduplication is disabled; no checksum indices are generated or - // referenced - if (!isUploadDeduplicationEnabled()) { - return; - } - try { - // 2. Query all index objects located under the deduplication prefix (e.g. - // checksums//) - Iterable> results = - minioClient.listObjects( - ListObjectsArgs.builder() - .bucket(bucket) - .prefix(checksumsPrefix) - .recursive(true) - .build()); - for (Result result : results) { - Item item = result.get(); - if (!item.isDir()) { - try (InputStream stream = - minioClient.getObject( - GetObjectArgs.builder().bucket(bucket).object(item.objectName()).build())) { - // 3. Read the parent upload ID string stored as plain text inside the checksum index - // object - String parentIdStr = IOUtils.toString(stream, StandardCharsets.UTF_8).trim(); - UploadId parentId = new UploadId(parentIdStr); - - // 4. Verify whether the parent upload's metadata (.info) still exists and is accessible - UploadInfo parentInfo = getUploadInfo(parentId); - - // 5. Verify whether the parent upload's final data object exists in S3 storage - // 6. If the parent upload or its data object no longer exists (e.g. expired, deleted, - // or purged - // by S3 lifecycle rules), prune the orphaned index entry to keep storage clean - if (parentInfo == null || !objectExists(buildObjectKey(parentId))) { - log.debug("Pruning orphaned S3 checksum index object {}", item.objectName()); - deleteObjectQuietly(item.objectName()); - } - } catch (Exception ignored) { - // Ignore individual object read errors to allow full sweep of remaining index entries - } - } - } - } catch (Exception e) { - log.debug("Failed to prune orphaned S3 checksum indices: {}", e.getMessage()); + public void setS3ConditionalWritesSupported(boolean supported) { + if (this.checksumIndex != null) { + this.checksumIndex.setS3ConditionalWritesSupported(supported); } } - // CONFIGURATION SETTERS & GETTERS + /** + * Returns whether S3 conditional writes (If-None-Match: *) are supported by the S3 endpoint. + * + * @return {@code true} if supported, {@code false} otherwise + */ + public boolean isS3ConditionalWritesSupported() { + return this.checksumIndex != null && this.checksumIndex.isS3ConditionalWritesSupported(); + } @Override public void setMaxUploadSize(Long maxUploadSize) { @@ -1283,6 +1301,11 @@ private void uploadChunkToS3(String chunkKey, File tempChunkFile, long chunkLeng private void finalizeCompletedUploadIfFinished( UploadInfo info, String objectKey, UploadId id, long newOffset) throws IOException { + if (info.getDuplicatesUploadId() != null) { + // Child upload is deduplicated; its data is referenced from parent, no parts to compose + return; + } + if (info.getLength() != null && newOffset >= info.getLength()) { List partKeys = info.getUploadPartKeys() != null @@ -1375,7 +1398,15 @@ private void finalizeCompletedUploadIfFinished( if (isUploadDeduplicationEnabled() && info.getChecksum() != null && info.getDuplicatesUploadId() == null) { - putChecksumIndex(info.getChecksum(), info.getChecksumAlgorithm(), info.getId()); + try { + ChecksumKey key = ChecksumKey.of(info.getChecksum(), info.getChecksumAlgorithm()); + deduplicate(info, key); + } catch (Exception e) { + log.warn( + "Failed to register checksum index for completed upload {}: {}", + info.getId(), + e.getMessage()); + } } } } @@ -1687,19 +1718,6 @@ void validateRemainingPartBudget(UploadInfo upload, int currentPartCount) } } - private void putChecksumIndex(String checksum, ChecksumAlgorithm algorithm, UploadId parentId) { - String key = buildChecksumKey(checksum, algorithm); - try { - byte[] parentIdBytes = parentId.toString().getBytes(StandardCharsets.UTF_8); - minioClient.putObject( - PutObjectArgs.builder().bucket(bucket).object(key).stream( - new ByteArrayInputStream(parentIdBytes), (long) parentIdBytes.length, -1L) - .build()); - } catch (Exception e) { - log.warn("Failed to write checksum index object to S3 key {}", key, e); - } - } - private boolean objectExists(String key) { try { minioClient.statObject(StatObjectArgs.builder().bucket(bucket).object(key).build()); @@ -1748,19 +1766,6 @@ private String buildChunkPartKey(UploadId id, int partNumber) { + String.format("%05d-%s", partNumber, UUID.randomUUID().toString().substring(0, 8)); } - private String buildChecksumKey(String checksum, ChecksumAlgorithm algorithm) { - String algorithmName = algorithm != null ? algorithm.getTusName().toLowerCase() : "unknown"; - return checksumsPrefix + algorithmName + "/" + checksum; - } - - private String buildLockKey(UploadId id) { - return locksPrefix + Objects.toString(id) + ".lock"; - } - - private String buildStopKey(UploadId id) { - return locksPrefix + Objects.toString(id) + ".stop"; - } - private static class AppendResult { final long confirmedBytesAppended; final List allPartKeys; diff --git a/src/main/java/me/desair/tus/server/util/AbstractTusExtension.java b/src/main/java/me/desair/tus/server/util/AbstractTusExtension.java index c6dc66bc..b78334e4 100644 --- a/src/main/java/me/desair/tus/server/util/AbstractTusExtension.java +++ b/src/main/java/me/desair/tus/server/util/AbstractTusExtension.java @@ -69,12 +69,13 @@ public void process( UploadStorageService uploadStorageService, String ownerKey) throws IOException, TusException { + UploadLockingService lockingService = Utils.getLockingServiceOrDefault(uploadStorageService); process( method, servletRequest, servletResponse, uploadStorageService, - null, + lockingService, ownerKey, ProtocolVersion.TUS_1_0_0); } @@ -94,6 +95,9 @@ public void process( return; } + UploadLockingService lockingServiceToUse = + Utils.getLockingServiceOrDefault(uploadLockingService, uploadStorageService); + for (RequestHandler requestHandler : requestHandlers) { if (requestHandler.supports(method, version)) { requestHandler.process( @@ -101,7 +105,7 @@ public void process( servletRequest, servletResponse, uploadStorageService, - uploadLockingService, + lockingServiceToUse, ownerKey, null); } @@ -116,13 +120,14 @@ public void handleError( UploadStorageService uploadStorageService, String ownerKey) throws IOException, TusException { + UploadLockingService lockingService = Utils.getLockingServiceOrDefault(uploadStorageService); HttpProblemDetails pd = handleError( method, request, response, uploadStorageService, - null, + lockingService, ownerKey, ProtocolVersion.TUS_1_0_0, null); @@ -147,6 +152,9 @@ public HttpProblemDetails handleError( return null; } + UploadLockingService lockingServiceToUse = + Utils.getLockingServiceOrDefault(uploadLockingService, uploadStorageService); + HttpProblemDetails problemDetails = null; for (RequestHandler requestHandler : requestHandlers) { if (requestHandler.supports(method, version) && requestHandler.isErrorHandler()) { @@ -156,7 +164,7 @@ public HttpProblemDetails handleError( request, response, uploadStorageService, - uploadLockingService, + lockingServiceToUse, ownerKey, exception); if (pd != null) { diff --git a/src/main/java/me/desair/tus/server/util/Utils.java b/src/main/java/me/desair/tus/server/util/Utils.java index 906cc477..aaeb6ed4 100644 --- a/src/main/java/me/desair/tus/server/util/Utils.java +++ b/src/main/java/me/desair/tus/server/util/Utils.java @@ -32,7 +32,9 @@ import me.desair.tus.server.HttpMethod; import me.desair.tus.server.ProtocolVersion; import me.desair.tus.server.checksum.ChecksumAlgorithm; +import me.desair.tus.server.upload.NoOpUploadLockingService; import me.desair.tus.server.upload.UploadInfo; +import me.desair.tus.server.upload.UploadLockingService; import me.desair.tus.server.upload.UploadStorageService; import org.apache.commons.io.serialization.ValidatingObjectInputStream; import org.apache.commons.lang3.StringUtils; @@ -702,4 +704,49 @@ public static void cleanupTempFiles(Path dir, String globPattern, long maxAgeMil log.debug("Error scanning directory {} for stale temporary files: {}", dir, e.getMessage()); } } + + /** + * Returns the given {@link UploadLockingService} if non-null, or attempts to retrieve it from the + * provided {@link UploadStorageService}, falling back to a non-null {@link + * NoOpUploadLockingService}. + * + * @param lockingService The locking service instance (may be null) + * @param storageService The storage service instance (may be null) + * @return A non-null {@link UploadLockingService} + */ + public static UploadLockingService getLockingServiceOrDefault( + UploadLockingService lockingService, UploadStorageService storageService) { + if (lockingService != null) { + return lockingService; + } + return getLockingServiceOrDefault(storageService); + } + + /** + * Returns the {@link UploadLockingService} associated with the provided {@link + * UploadStorageService}, falling back to a non-null {@link NoOpUploadLockingService} if either is + * null or unconfigured. + * + * @param storageService The storage service instance (may be null) + * @return A non-null {@link UploadLockingService} + */ + public static UploadLockingService getLockingServiceOrDefault( + UploadStorageService storageService) { + if (storageService != null && storageService.getUploadLockingService() != null) { + return storageService.getUploadLockingService(); + } + return new NoOpUploadLockingService(); + } + + /** + * Returns the given {@link UploadLockingService} if non-null, falling back to a non-null {@link + * NoOpUploadLockingService}. + * + * @param lockingService The locking service instance (may be null) + * @return A non-null {@link UploadLockingService} + */ + public static UploadLockingService getLockingServiceOrDefault( + UploadLockingService lockingService) { + return lockingService != null ? lockingService : new NoOpUploadLockingService(); + } } diff --git a/src/test/java/me/desair/tus/server/AbstractITRufhProtocol.java b/src/test/java/me/desair/tus/server/AbstractITRufhProtocol.java index 504dd8b2..9f0aa4a7 100644 --- a/src/test/java/me/desair/tus/server/AbstractITRufhProtocol.java +++ b/src/test/java/me/desair/tus/server/AbstractITRufhProtocol.java @@ -7,14 +7,20 @@ import static org.junit.Assert.assertFalse; import static org.junit.Assert.assertNotNull; import static org.junit.Assert.assertNull; +import static org.junit.Assert.assertThrows; import static org.junit.Assert.assertTrue; import jakarta.servlet.http.HttpServletResponse; import java.io.InputStream; import java.nio.charset.StandardCharsets; +import java.util.Base64; +import java.util.UUID; import java.util.concurrent.atomic.AtomicInteger; import java.util.concurrent.atomic.AtomicReference; +import me.desair.tus.server.exception.UploadNotFoundException; +import me.desair.tus.server.upload.UploadId; import me.desair.tus.server.upload.UploadInfo; +import org.apache.commons.codec.digest.DigestUtils; import org.apache.commons.io.IOUtils; import org.apache.commons.lang3.StringUtils; import org.junit.Before; @@ -58,6 +64,16 @@ public abstract class AbstractITRufhProtocol { protected abstract TusFileUploadService createTusFileUploadService(String uploadUri) throws Exception; + /** + * Deletes the underlying physical uploaded bytes for an upload out-of-band directly on the + * storage backend (without going through Tus upload termination logic), simulating accidental + * byte loss. + * + * @param uploadId The upload identifier whose bytes should be deleted out-of-band + * @throws Exception if deletion fails + */ + protected abstract void deleteUploadedBytesOutOfBand(UploadId uploadId) throws Exception; + @Before public void setUp() throws Exception { reset(); @@ -890,6 +906,511 @@ public void testCompletingAppendPatchBypassesMinAppendSize() throws Exception { } } + @Test + public void testRufhUploadCancellationWithDeduplication() throws Exception { + tusFileUploadService.withUploadDeduplication(true); + + String uploadContent = "RUFH Deduplication Cancellation Test Content 12345"; + byte[] contentBytes = uploadContent.getBytes(StandardCharsets.UTF_8); + byte[] sha256Digest = org.apache.commons.codec.digest.DigestUtils.sha256(contentBytes); + String sha256Base64 = org.apache.commons.codec.binary.Base64.encodeBase64String(sha256Digest); + + // Step 1: Parent upload creation + servletRequest.setMethod("POST"); + servletRequest.setRequestURI(UPLOAD_URI); + servletRequest.addHeader(HttpHeader.UPLOAD_COMPLETE, "?0"); + servletRequest.addHeader(HttpHeader.UPLOAD_LENGTH, String.valueOf(contentBytes.length)); + UploadInfo parentCreated = + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + assertResponseStatus(HttpServletResponse.SC_CREATED); + String parentLocation = servletResponse.getHeader(HttpHeader.LOCATION); + + // Parent completes via PATCH with checksum + reset(); + servletRequest.setMethod("PATCH"); + servletRequest.setRequestURI(parentLocation); + servletRequest.addHeader(HttpHeader.UPLOAD_OFFSET, "0"); + servletRequest.addHeader(HttpHeader.UPLOAD_COMPLETE, "?1"); + servletRequest.addHeader(HttpHeader.CONTENT_TYPE, HttpHeader.CONTENT_TYPE_PARTIAL_UPLOAD); + servletRequest.addHeader("Content-Digest", "sha-256=:" + sha256Base64 + ":"); + servletRequest.setContent(contentBytes); + UploadInfo parentCompleted = + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + assertResponseStatus(HttpServletResponse.SC_OK); + assertNotNull(parentCompleted); + assertNull(parentCompleted.getDuplicatesUploadId()); + + // Step 2: Child duplicate upload creation + reset(); + servletRequest.setMethod("POST"); + servletRequest.setRequestURI(UPLOAD_URI); + servletRequest.addHeader(HttpHeader.UPLOAD_COMPLETE, "?0"); + servletRequest.addHeader(HttpHeader.UPLOAD_LENGTH, String.valueOf(contentBytes.length)); + UploadInfo childCreated = + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + assertResponseStatus(HttpServletResponse.SC_CREATED); + String childLocation = servletResponse.getHeader(HttpHeader.LOCATION); + + // Child completes via PATCH with same checksum -> links to parent + reset(); + servletRequest.setMethod("PATCH"); + servletRequest.setRequestURI(childLocation); + servletRequest.addHeader(HttpHeader.UPLOAD_OFFSET, "0"); + servletRequest.addHeader(HttpHeader.UPLOAD_COMPLETE, "?1"); + servletRequest.addHeader(HttpHeader.CONTENT_TYPE, HttpHeader.CONTENT_TYPE_PARTIAL_UPLOAD); + servletRequest.addHeader("Content-Digest", "sha-256=:" + sha256Base64 + ":"); + servletRequest.setContent(contentBytes); + UploadInfo childCompleted = + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + assertResponseStatus(HttpServletResponse.SC_OK); + assertNotNull(childCompleted); + assertEquals(parentCompleted.getId(), childCompleted.getDuplicatesUploadId()); + + // Step 3: Verify child can download content + try (InputStream stream = tusFileUploadService.getUploadedBytes(childCompleted)) { + assertThat(IOUtils.toString(stream, StandardCharsets.UTF_8), is(uploadContent)); + } + + // Step 4: Cancel parent upload via DELETE + reset(); + servletRequest.setMethod("DELETE"); + servletRequest.setRequestURI(parentLocation); + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + assertResponseStatus(HttpServletResponse.SC_NO_CONTENT); + + // Parent is gone (HEAD returns 404) + reset(); + servletRequest.setMethod("HEAD"); + servletRequest.setRequestURI(parentLocation); + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + assertResponseStatus(HttpServletResponse.SC_NOT_FOUND); + + // Child STILL has readable content (parent data preserved) + try (InputStream stream = tusFileUploadService.getUploadedBytes(childCompleted)) { + assertThat(IOUtils.toString(stream, StandardCharsets.UTF_8), is(uploadContent)); + } + + // Step 5: Cancel child upload via DELETE (last child out) + reset(); + servletRequest.setMethod("DELETE"); + servletRequest.setRequestURI(childLocation); + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + assertResponseStatus(HttpServletResponse.SC_NO_CONTENT); + + // Child is gone (HEAD returns 404) + reset(); + servletRequest.setMethod("HEAD"); + servletRequest.setRequestURI(childLocation); + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + assertResponseStatus(HttpServletResponse.SC_NOT_FOUND); + } + + @Test + public void testRufhDownloadFailsWhenParentBytesDeletedOutOfBandWithDeduplicationEnabled() + throws Exception { + tusFileUploadService.withUploadDeduplication(true); + + String uploadContent = "Rufh Out Of Band Byte Loss Deduplication Enabled Content"; + byte[] contentBytes = uploadContent.getBytes(StandardCharsets.UTF_8); + byte[] sha256Digest = org.apache.commons.codec.digest.DigestUtils.sha256(contentBytes); + String sha256Base64 = org.apache.commons.codec.binary.Base64.encodeBase64String(sha256Digest); + + // 1. Create and complete parent upload + servletRequest.setMethod("POST"); + servletRequest.setRequestURI(UPLOAD_URI); + servletRequest.addHeader(HttpHeader.UPLOAD_COMPLETE, "?0"); + servletRequest.addHeader(HttpHeader.UPLOAD_LENGTH, String.valueOf(contentBytes.length)); + UploadInfo parentCreated = + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + assertResponseStatus(HttpServletResponse.SC_CREATED); + String parentLocation = servletResponse.getHeader(HttpHeader.LOCATION); + + reset(); + servletRequest.setMethod("PATCH"); + servletRequest.setRequestURI(parentLocation); + servletRequest.addHeader(HttpHeader.UPLOAD_OFFSET, "0"); + servletRequest.addHeader(HttpHeader.UPLOAD_COMPLETE, "?1"); + servletRequest.addHeader(HttpHeader.CONTENT_TYPE, HttpHeader.CONTENT_TYPE_PARTIAL_UPLOAD); + servletRequest.addHeader("Content-Digest", "sha-256=:" + sha256Base64 + ":"); + servletRequest.setContent(contentBytes); + UploadInfo parentCompleted = + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + assertResponseStatus(HttpServletResponse.SC_OK); + assertNotNull(parentCompleted); + assertNull(parentCompleted.getDuplicatesUploadId()); + + // 2. Create and complete child 1 upload (deduplicated against parent) + reset(); + servletRequest.setMethod("POST"); + servletRequest.setRequestURI(UPLOAD_URI); + servletRequest.addHeader(HttpHeader.UPLOAD_COMPLETE, "?0"); + servletRequest.addHeader(HttpHeader.UPLOAD_LENGTH, String.valueOf(contentBytes.length)); + UploadInfo child1Created = + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + assertResponseStatus(HttpServletResponse.SC_CREATED); + String child1Location = servletResponse.getHeader(HttpHeader.LOCATION); + + reset(); + servletRequest.setMethod("PATCH"); + servletRequest.setRequestURI(child1Location); + servletRequest.addHeader(HttpHeader.UPLOAD_OFFSET, "0"); + servletRequest.addHeader(HttpHeader.UPLOAD_COMPLETE, "?1"); + servletRequest.addHeader(HttpHeader.CONTENT_TYPE, HttpHeader.CONTENT_TYPE_PARTIAL_UPLOAD); + servletRequest.addHeader("Content-Digest", "sha-256=:" + sha256Base64 + ":"); + servletRequest.setContent(contentBytes); + UploadInfo child1Completed = + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + assertResponseStatus(HttpServletResponse.SC_OK); + assertNotNull(child1Completed); + assertEquals(parentCompleted.getId(), child1Completed.getDuplicatesUploadId()); + + // 3. Create and complete child 2 upload (deduplicated against parent) + reset(); + servletRequest.setMethod("POST"); + servletRequest.setRequestURI(UPLOAD_URI); + servletRequest.addHeader(HttpHeader.UPLOAD_COMPLETE, "?0"); + servletRequest.addHeader(HttpHeader.UPLOAD_LENGTH, String.valueOf(contentBytes.length)); + UploadInfo child2Created = + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + assertResponseStatus(HttpServletResponse.SC_CREATED); + String child2Location = servletResponse.getHeader(HttpHeader.LOCATION); + + reset(); + servletRequest.setMethod("PATCH"); + servletRequest.setRequestURI(child2Location); + servletRequest.addHeader(HttpHeader.UPLOAD_OFFSET, "0"); + servletRequest.addHeader(HttpHeader.UPLOAD_COMPLETE, "?1"); + servletRequest.addHeader(HttpHeader.CONTENT_TYPE, HttpHeader.CONTENT_TYPE_PARTIAL_UPLOAD); + servletRequest.addHeader("Content-Digest", "sha-256=:" + sha256Base64 + ":"); + servletRequest.setContent(contentBytes); + UploadInfo child2Completed = + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + assertResponseStatus(HttpServletResponse.SC_OK); + assertNotNull(child2Completed); + assertEquals(parentCompleted.getId(), child2Completed.getDuplicatesUploadId()); + + // 4. Verify initial download succeeds for all three + try (InputStream stream = tusFileUploadService.getUploadedBytes(parentCompleted)) { + assertThat(IOUtils.toString(stream, StandardCharsets.UTF_8), is(uploadContent)); + } + try (InputStream stream = tusFileUploadService.getUploadedBytes(child1Completed)) { + assertThat(IOUtils.toString(stream, StandardCharsets.UTF_8), is(uploadContent)); + } + try (InputStream stream = tusFileUploadService.getUploadedBytes(child2Completed)) { + assertThat(IOUtils.toString(stream, StandardCharsets.UTF_8), is(uploadContent)); + } + + // 5. Delete uploaded bytes of parent OUT-OF-BAND + deleteUploadedBytesOutOfBand(parentCompleted.getId()); + + // 6. Verify programmatic download fails for parent, child 1, and child 2 + assertThrows( + UploadNotFoundException.class, + () -> tusFileUploadService.getUploadedBytes(parentCompleted)); + assertThrows( + UploadNotFoundException.class, + () -> tusFileUploadService.getUploadedBytes(child1Completed)); + assertThrows( + UploadNotFoundException.class, + () -> tusFileUploadService.getUploadedBytes(child2Completed)); + + // 7. Verify HTTP GET download fails with 404 for parent, child 1, and child 2 + reset(); + servletRequest.setMethod("GET"); + servletRequest.setRequestURI(parentLocation); + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + assertResponseStatus(HttpServletResponse.SC_NOT_FOUND); + + reset(); + servletRequest.setMethod("GET"); + servletRequest.setRequestURI(child1Location); + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + assertResponseStatus(HttpServletResponse.SC_NOT_FOUND); + + reset(); + servletRequest.setMethod("GET"); + servletRequest.setRequestURI(child2Location); + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + assertResponseStatus(HttpServletResponse.SC_NOT_FOUND); + } + + @Test + public void + testRufhDownloadSucceedsForChildrenWhenParentBytesDeletedOutOfBandWithDeduplicationDisabled() + throws Exception { + tusFileUploadService.withUploadDeduplication(false); + + String uploadContent = "Rufh Out Of Band Byte Loss Deduplication Disabled Content"; + byte[] contentBytes = uploadContent.getBytes(StandardCharsets.UTF_8); + byte[] sha256Digest = org.apache.commons.codec.digest.DigestUtils.sha256(contentBytes); + String sha256Base64 = org.apache.commons.codec.binary.Base64.encodeBase64String(sha256Digest); + + // 1. Create and complete parent upload + servletRequest.setMethod("POST"); + servletRequest.setRequestURI(UPLOAD_URI); + servletRequest.addHeader(HttpHeader.UPLOAD_COMPLETE, "?0"); + servletRequest.addHeader(HttpHeader.UPLOAD_LENGTH, String.valueOf(contentBytes.length)); + UploadInfo parentCreated = + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + assertResponseStatus(HttpServletResponse.SC_CREATED); + String parentLocation = servletResponse.getHeader(HttpHeader.LOCATION); + + reset(); + servletRequest.setMethod("PATCH"); + servletRequest.setRequestURI(parentLocation); + servletRequest.addHeader(HttpHeader.UPLOAD_OFFSET, "0"); + servletRequest.addHeader(HttpHeader.UPLOAD_COMPLETE, "?1"); + servletRequest.addHeader(HttpHeader.CONTENT_TYPE, HttpHeader.CONTENT_TYPE_PARTIAL_UPLOAD); + servletRequest.addHeader("Content-Digest", "sha-256=:" + sha256Base64 + ":"); + servletRequest.setContent(contentBytes); + UploadInfo parentCompleted = + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + assertResponseStatus(HttpServletResponse.SC_OK); + assertNotNull(parentCompleted); + assertNull(parentCompleted.getDuplicatesUploadId()); + + // 2. Create and complete child 1 upload (independent, not deduplicated) + reset(); + servletRequest.setMethod("POST"); + servletRequest.setRequestURI(UPLOAD_URI); + servletRequest.addHeader(HttpHeader.UPLOAD_COMPLETE, "?0"); + servletRequest.addHeader(HttpHeader.UPLOAD_LENGTH, String.valueOf(contentBytes.length)); + UploadInfo child1Created = + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + assertResponseStatus(HttpServletResponse.SC_CREATED); + String child1Location = servletResponse.getHeader(HttpHeader.LOCATION); + + reset(); + servletRequest.setMethod("PATCH"); + servletRequest.setRequestURI(child1Location); + servletRequest.addHeader(HttpHeader.UPLOAD_OFFSET, "0"); + servletRequest.addHeader(HttpHeader.UPLOAD_COMPLETE, "?1"); + servletRequest.addHeader(HttpHeader.CONTENT_TYPE, HttpHeader.CONTENT_TYPE_PARTIAL_UPLOAD); + servletRequest.addHeader("Content-Digest", "sha-256=:" + sha256Base64 + ":"); + servletRequest.setContent(contentBytes); + UploadInfo child1Completed = + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + assertResponseStatus(HttpServletResponse.SC_OK); + assertNotNull(child1Completed); + assertNull(child1Completed.getDuplicatesUploadId()); + + // 3. Create and complete child 2 upload (independent, not deduplicated) + reset(); + servletRequest.setMethod("POST"); + servletRequest.setRequestURI(UPLOAD_URI); + servletRequest.addHeader(HttpHeader.UPLOAD_COMPLETE, "?0"); + servletRequest.addHeader(HttpHeader.UPLOAD_LENGTH, String.valueOf(contentBytes.length)); + UploadInfo child2Created = + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + assertResponseStatus(HttpServletResponse.SC_CREATED); + String child2Location = servletResponse.getHeader(HttpHeader.LOCATION); + + reset(); + servletRequest.setMethod("PATCH"); + servletRequest.setRequestURI(child2Location); + servletRequest.addHeader(HttpHeader.UPLOAD_OFFSET, "0"); + servletRequest.addHeader(HttpHeader.UPLOAD_COMPLETE, "?1"); + servletRequest.addHeader(HttpHeader.CONTENT_TYPE, HttpHeader.CONTENT_TYPE_PARTIAL_UPLOAD); + servletRequest.addHeader("Content-Digest", "sha-256=:" + sha256Base64 + ":"); + servletRequest.setContent(contentBytes); + UploadInfo child2Completed = + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + assertResponseStatus(HttpServletResponse.SC_OK); + assertNotNull(child2Completed); + assertNull(child2Completed.getDuplicatesUploadId()); + + // 4. Delete uploaded bytes of parent OUT-OF-BAND + deleteUploadedBytesOutOfBand(parentCompleted.getId()); + + // 5. Download of parent FAILS + assertThrows( + UploadNotFoundException.class, + () -> tusFileUploadService.getUploadedBytes(parentCompleted)); + reset(); + servletRequest.setMethod("GET"); + servletRequest.setRequestURI(parentLocation); + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + assertResponseStatus(HttpServletResponse.SC_NOT_FOUND); + + // 6. Download of child 1 and child 2 SUCCEEDS + try (InputStream stream = tusFileUploadService.getUploadedBytes(child1Completed)) { + assertThat(IOUtils.toString(stream, StandardCharsets.UTF_8), is(uploadContent)); + } + try (InputStream stream = tusFileUploadService.getUploadedBytes(child2Completed)) { + assertThat(IOUtils.toString(stream, StandardCharsets.UTF_8), is(uploadContent)); + } + + reset(); + servletRequest.setMethod("GET"); + servletRequest.setRequestURI(child1Location); + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + assertResponseStatus(HttpServletResponse.SC_OK); + assertThat(servletResponse.getContentAsString(), is(uploadContent)); + + reset(); + servletRequest.setMethod("GET"); + servletRequest.setRequestURI(child2Location); + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + assertResponseStatus(HttpServletResponse.SC_OK); + assertThat(servletResponse.getContentAsString(), is(uploadContent)); + } + + @Test + public void testRufhNewChildJoinsTombstonedParentAndPurgesAfterAllChildrenDeleted() + throws Exception { + tusFileUploadService.withUploadDeduplication(true); + + String uploadContent = "RUFH content for tombstoned parent child join " + UUID.randomUUID(); + byte[] contentBytes = uploadContent.getBytes(StandardCharsets.UTF_8); + String sha256Base64 = Base64.getEncoder().encodeToString(DigestUtils.sha256(contentBytes)); + + // 1. Create and complete parent upload (P) + reset(); + servletRequest.setMethod("POST"); + servletRequest.setRequestURI(UPLOAD_URI); + servletRequest.addHeader("Upload-Draft-Interop-Version", "6"); + servletRequest.addHeader(HttpHeader.UPLOAD_COMPLETE, "?0"); + servletRequest.addHeader(HttpHeader.CONTENT_TYPE, HttpHeader.CONTENT_TYPE_PARTIAL_UPLOAD); + UploadInfo parentCreated = + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + assertResponseStatus(HttpServletResponse.SC_CREATED); + String parentLocation = servletResponse.getHeader(HttpHeader.LOCATION); + + reset(); + servletRequest.setMethod("PATCH"); + servletRequest.setRequestURI(parentLocation); + servletRequest.addHeader(HttpHeader.UPLOAD_OFFSET, "0"); + servletRequest.addHeader(HttpHeader.UPLOAD_COMPLETE, "?1"); + servletRequest.addHeader(HttpHeader.CONTENT_TYPE, HttpHeader.CONTENT_TYPE_PARTIAL_UPLOAD); + servletRequest.addHeader("Content-Digest", "sha-256=:" + sha256Base64 + ":"); + servletRequest.setContent(contentBytes); + UploadInfo parentCompleted = + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + assertResponseStatus(HttpServletResponse.SC_OK); + assertNotNull(parentCompleted); + assertNull(parentCompleted.getDuplicatesUploadId()); + + // 2. Create and complete first child upload (C1) + reset(); + servletRequest.setMethod("POST"); + servletRequest.setRequestURI(UPLOAD_URI); + servletRequest.addHeader("Upload-Draft-Interop-Version", "6"); + servletRequest.addHeader(HttpHeader.UPLOAD_COMPLETE, "?0"); + servletRequest.addHeader(HttpHeader.CONTENT_TYPE, HttpHeader.CONTENT_TYPE_PARTIAL_UPLOAD); + UploadInfo child1Created = + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + assertResponseStatus(HttpServletResponse.SC_CREATED); + String child1Location = servletResponse.getHeader(HttpHeader.LOCATION); + + reset(); + servletRequest.setMethod("PATCH"); + servletRequest.setRequestURI(child1Location); + servletRequest.addHeader(HttpHeader.UPLOAD_OFFSET, "0"); + servletRequest.addHeader(HttpHeader.UPLOAD_COMPLETE, "?1"); + servletRequest.addHeader(HttpHeader.CONTENT_TYPE, HttpHeader.CONTENT_TYPE_PARTIAL_UPLOAD); + servletRequest.addHeader("Content-Digest", "sha-256=:" + sha256Base64 + ":"); + servletRequest.setContent(contentBytes); + UploadInfo child1Completed = + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + assertResponseStatus(HttpServletResponse.SC_OK); + assertNotNull(child1Completed); + assertEquals(parentCompleted.getId(), child1Completed.getDuplicatesUploadId()); + + // 3. Delete Parent (P) via DELETE -> P is tombstoned, C1 remains + reset(); + servletRequest.setMethod("DELETE"); + servletRequest.setRequestURI(parentLocation); + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + assertResponseStatus(HttpServletResponse.SC_NO_CONTENT); + + assertNull(tusFileUploadService.getUploadInfo(parentLocation, OWNER_KEY)); + try (InputStream stream = tusFileUploadService.getUploadedBytes(child1Completed)) { + assertThat(IOUtils.toString(stream, StandardCharsets.UTF_8), is(uploadContent)); + } + + // 4. Create and complete second child (C2) while P is tombstoned + reset(); + servletRequest.setMethod("POST"); + servletRequest.setRequestURI(UPLOAD_URI); + servletRequest.addHeader("Upload-Draft-Interop-Version", "6"); + servletRequest.addHeader(HttpHeader.UPLOAD_COMPLETE, "?0"); + servletRequest.addHeader(HttpHeader.CONTENT_TYPE, HttpHeader.CONTENT_TYPE_PARTIAL_UPLOAD); + UploadInfo child2Created = + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + assertResponseStatus(HttpServletResponse.SC_CREATED); + String child2Location = servletResponse.getHeader(HttpHeader.LOCATION); + + reset(); + servletRequest.setMethod("PATCH"); + servletRequest.setRequestURI(child2Location); + servletRequest.addHeader(HttpHeader.UPLOAD_OFFSET, "0"); + servletRequest.addHeader(HttpHeader.UPLOAD_COMPLETE, "?1"); + servletRequest.addHeader(HttpHeader.CONTENT_TYPE, HttpHeader.CONTENT_TYPE_PARTIAL_UPLOAD); + servletRequest.addHeader("Content-Digest", "sha-256=:" + sha256Base64 + ":"); + servletRequest.setContent(contentBytes); + UploadInfo child2Completed = + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + assertResponseStatus(HttpServletResponse.SC_OK); + assertNotNull(child2Completed); + // C2 successfully joined tombstoned parent P! + assertEquals(parentCompleted.getId(), child2Completed.getDuplicatesUploadId()); + + // Both C1 and C2 can stream data + try (InputStream stream = tusFileUploadService.getUploadedBytes(child1Completed)) { + assertThat(IOUtils.toString(stream, StandardCharsets.UTF_8), is(uploadContent)); + } + try (InputStream stream = tusFileUploadService.getUploadedBytes(child2Completed)) { + assertThat(IOUtils.toString(stream, StandardCharsets.UTF_8), is(uploadContent)); + } + + // 5. Delete C1 via DELETE -> C2 remains + reset(); + servletRequest.setMethod("DELETE"); + servletRequest.setRequestURI(child1Location); + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + assertResponseStatus(HttpServletResponse.SC_NO_CONTENT); + + assertNull(tusFileUploadService.getUploadInfo(child1Location, OWNER_KEY)); + try (InputStream stream = tusFileUploadService.getUploadedBytes(child2Completed)) { + assertThat(IOUtils.toString(stream, StandardCharsets.UTF_8), is(uploadContent)); + } + + // 6. Delete C2 (last child) -> tombstoned parent and index purged + reset(); + servletRequest.setMethod("DELETE"); + servletRequest.setRequestURI(child2Location); + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + assertResponseStatus(HttpServletResponse.SC_NO_CONTENT); + + assertNull(tusFileUploadService.getUploadInfo(child2Location, OWNER_KEY)); + + // 7. Uploading again creates a fresh parent (P2) + reset(); + servletRequest.setMethod("POST"); + servletRequest.setRequestURI(UPLOAD_URI); + servletRequest.addHeader("Upload-Draft-Interop-Version", "6"); + servletRequest.addHeader(HttpHeader.UPLOAD_COMPLETE, "?0"); + servletRequest.addHeader(HttpHeader.CONTENT_TYPE, HttpHeader.CONTENT_TYPE_PARTIAL_UPLOAD); + UploadInfo parent2Created = + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + assertResponseStatus(HttpServletResponse.SC_CREATED); + String parent2Location = servletResponse.getHeader(HttpHeader.LOCATION); + + reset(); + servletRequest.setMethod("PATCH"); + servletRequest.setRequestURI(parent2Location); + servletRequest.addHeader(HttpHeader.UPLOAD_OFFSET, "0"); + servletRequest.addHeader(HttpHeader.UPLOAD_COMPLETE, "?1"); + servletRequest.addHeader(HttpHeader.CONTENT_TYPE, HttpHeader.CONTENT_TYPE_PARTIAL_UPLOAD); + servletRequest.addHeader("Content-Digest", "sha-256=:" + sha256Base64 + ":"); + servletRequest.setContent(contentBytes); + UploadInfo parent2Completed = + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + assertResponseStatus(HttpServletResponse.SC_OK); + assertNotNull(parent2Completed); + assertNull(parent2Completed.getDuplicatesUploadId()); + } + // =============================================================================================== // ASSERTION HELPERS // =============================================================================================== diff --git a/src/test/java/me/desair/tus/server/AbstractITTusFileUploadService.java b/src/test/java/me/desair/tus/server/AbstractITTusFileUploadService.java index 40ce876b..92909a7a 100644 --- a/src/test/java/me/desair/tus/server/AbstractITTusFileUploadService.java +++ b/src/test/java/me/desair/tus/server/AbstractITTusFileUploadService.java @@ -12,6 +12,7 @@ import static org.junit.Assert.assertFalse; import static org.junit.Assert.assertNotNull; import static org.junit.Assert.assertNull; +import static org.junit.Assert.assertThrows; import static org.junit.Assert.assertTrue; import static org.junit.Assert.fail; @@ -28,6 +29,8 @@ import java.util.concurrent.atomic.AtomicInteger; import java.util.concurrent.atomic.AtomicReference; import me.desair.tus.server.exception.TusException; +import me.desair.tus.server.exception.UploadNotFoundException; +import me.desair.tus.server.upload.UploadId; import me.desair.tus.server.upload.UploadInfo; import me.desair.tus.server.util.Utils; import org.apache.commons.io.IOUtils; @@ -45,6 +48,15 @@ public abstract class AbstractITTusFileUploadService { protected abstract TusFileUploadService createTusFileUploadService(String uploadUri) throws Exception; + /** + * Deletes the physical uploaded data bytes for the given upload ID out-of-band directly from the + * underlying storage backend. + * + * @param uploadId the upload ID whose data bytes should be deleted + * @throws Exception if deletion fails + */ + protected abstract void deleteUploadedBytesOutOfBand(UploadId uploadId) throws Exception; + protected static final String UPLOAD_URI = "/test/upload"; protected static final String OWNER_KEY = "JOHN_DOE"; @@ -2442,6 +2454,670 @@ public void testUploadCompletionListenerTusConcatenation() throws Exception { } } + @Test + public void testDeduplicationAndParentChildDeletionLifecycle() throws Exception { + tusFileUploadService.withUploadDeduplication(true); + + String content = "Hello Deduplication World 12345"; + byte[] contentBytes = content.getBytes(StandardCharsets.UTF_8); + String sha1Base64 = + org.apache.commons.codec.binary.Base64.encodeBase64String( + org.apache.commons.codec.digest.DigestUtils.sha1(contentBytes)); + + // 1. Create parent upload + reset(); + servletRequest.setMethod("POST"); + servletRequest.setRequestURI(UPLOAD_URI); + servletRequest.addHeader(HttpHeader.UPLOAD_LENGTH, contentBytes.length); + servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + assertResponseStatus(HttpServletResponse.SC_CREATED); + String parentUri = servletResponse.getHeader(HttpHeader.LOCATION); + + // Upload parent content with checksum + reset(); + servletRequest.setMethod("PATCH"); + servletRequest.setRequestURI(parentUri); + servletRequest.addHeader(HttpHeader.CONTENT_TYPE, "application/offset+octet-stream"); + servletRequest.addHeader(HttpHeader.UPLOAD_OFFSET, 0); + servletRequest.addHeader(HttpHeader.UPLOAD_CHECKSUM, "sha1 " + sha1Base64); + servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); + servletRequest.setContent(contentBytes); + UploadInfo parentInfo = + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + assertResponseStatus(HttpServletResponse.SC_NO_CONTENT); + assertNotNull(parentInfo); + assertNull(parentInfo.getDuplicatesUploadId()); + + // 2. Create child upload with identical content and checksum + reset(); + servletRequest.setMethod("POST"); + servletRequest.setRequestURI(UPLOAD_URI); + servletRequest.addHeader(HttpHeader.UPLOAD_LENGTH, contentBytes.length); + servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + assertResponseStatus(HttpServletResponse.SC_CREATED); + String childUri = servletResponse.getHeader(HttpHeader.LOCATION); + + // Upload child content with checksum + reset(); + servletRequest.setMethod("PATCH"); + servletRequest.setRequestURI(childUri); + servletRequest.addHeader(HttpHeader.CONTENT_TYPE, "application/offset+octet-stream"); + servletRequest.addHeader(HttpHeader.UPLOAD_OFFSET, 0); + servletRequest.addHeader(HttpHeader.UPLOAD_CHECKSUM, "sha1 " + sha1Base64); + servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); + servletRequest.setContent(contentBytes); + UploadInfo childInfo = tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + assertResponseStatus(HttpServletResponse.SC_NO_CONTENT); + assertNotNull(childInfo); + // Verify child upload was linked to parent + assertEquals(parentInfo.getId(), childInfo.getDuplicatesUploadId()); + + // 3. Verify child can download bytes + try (InputStream is = tusFileUploadService.getUploadedBytes(childInfo)) { + assertThat(IOUtils.toString(is, StandardCharsets.UTF_8), is(content)); + } + + // 4. Terminate parent upload via DELETE + reset(); + servletRequest.setMethod("DELETE"); + servletRequest.setRequestURI(parentUri); + servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + assertResponseStatus(HttpServletResponse.SC_NO_CONTENT); + + // Verify parent upload info is gone + UploadInfo deletedParent = tusFileUploadService.getUploadInfo(parentUri, OWNER_KEY); + assertNull(deletedParent); + + // 5. Child STILL has readable content (parent data preserved via tombstone) + try (InputStream is = tusFileUploadService.getUploadedBytes(childInfo)) { + assertThat(IOUtils.toString(is, StandardCharsets.UTF_8), is(content)); + } + + // 6. Terminate child upload via DELETE + reset(); + servletRequest.setMethod("DELETE"); + servletRequest.setRequestURI(childUri); + servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + assertResponseStatus(HttpServletResponse.SC_NO_CONTENT); + + // Verify child is also gone + UploadInfo deletedChild = tusFileUploadService.getUploadInfo(childUri, OWNER_KEY); + assertNull(deletedChild); + + // 7. Create third upload with SAME checksum - should register as new parent cleanly + reset(); + servletRequest.setMethod("POST"); + servletRequest.setRequestURI(UPLOAD_URI); + servletRequest.addHeader(HttpHeader.UPLOAD_LENGTH, contentBytes.length); + servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + assertResponseStatus(HttpServletResponse.SC_CREATED); + String newParentUri = servletResponse.getHeader(HttpHeader.LOCATION); + + reset(); + servletRequest.setMethod("PATCH"); + servletRequest.setRequestURI(newParentUri); + servletRequest.addHeader(HttpHeader.CONTENT_TYPE, "application/offset+octet-stream"); + servletRequest.addHeader(HttpHeader.UPLOAD_OFFSET, 0); + servletRequest.addHeader(HttpHeader.UPLOAD_CHECKSUM, "sha1 " + sha1Base64); + servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); + servletRequest.setContent(contentBytes); + UploadInfo newParentInfo = + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + assertResponseStatus(HttpServletResponse.SC_NO_CONTENT); + assertNotNull(newParentInfo); + assertNull(newParentInfo.getDuplicatesUploadId()); + } + + @Test + public void testMultiChildDeduplicationAndParentDeletionLifecycle() throws Exception { + tusFileUploadService.withUploadDeduplication(true); + + String content = "Multi Child Deduplication Test Payload 12345"; + byte[] contentBytes = content.getBytes(StandardCharsets.UTF_8); + String sha1Base64 = + org.apache.commons.codec.binary.Base64.encodeBase64String( + org.apache.commons.codec.digest.DigestUtils.sha1(contentBytes)); + + // 1. Create and complete parent upload + reset(); + servletRequest.setMethod("POST"); + servletRequest.setRequestURI(UPLOAD_URI); + servletRequest.addHeader(HttpHeader.UPLOAD_LENGTH, contentBytes.length); + servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + assertResponseStatus(HttpServletResponse.SC_CREATED); + String parentUri = servletResponse.getHeader(HttpHeader.LOCATION); + + reset(); + servletRequest.setMethod("PATCH"); + servletRequest.setRequestURI(parentUri); + servletRequest.addHeader(HttpHeader.CONTENT_TYPE, "application/offset+octet-stream"); + servletRequest.addHeader(HttpHeader.UPLOAD_OFFSET, 0); + servletRequest.addHeader(HttpHeader.UPLOAD_CHECKSUM, "sha1 " + sha1Base64); + servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); + servletRequest.setContent(contentBytes); + UploadInfo parentInfo = + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + assertResponseStatus(HttpServletResponse.SC_NO_CONTENT); + assertNotNull(parentInfo); + assertNull(parentInfo.getDuplicatesUploadId()); + + // 2. Create and complete child 1 upload + reset(); + servletRequest.setMethod("POST"); + servletRequest.setRequestURI(UPLOAD_URI); + servletRequest.addHeader(HttpHeader.UPLOAD_LENGTH, contentBytes.length); + servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + assertResponseStatus(HttpServletResponse.SC_CREATED); + String child1Uri = servletResponse.getHeader(HttpHeader.LOCATION); + + reset(); + servletRequest.setMethod("PATCH"); + servletRequest.setRequestURI(child1Uri); + servletRequest.addHeader(HttpHeader.CONTENT_TYPE, "application/offset+octet-stream"); + servletRequest.addHeader(HttpHeader.UPLOAD_OFFSET, 0); + servletRequest.addHeader(HttpHeader.UPLOAD_CHECKSUM, "sha1 " + sha1Base64); + servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); + servletRequest.setContent(contentBytes); + UploadInfo child1Info = + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + assertResponseStatus(HttpServletResponse.SC_NO_CONTENT); + assertNotNull(child1Info); + assertEquals(parentInfo.getId(), child1Info.getDuplicatesUploadId()); + + // 3. Create and complete child 2 upload + reset(); + servletRequest.setMethod("POST"); + servletRequest.setRequestURI(UPLOAD_URI); + servletRequest.addHeader(HttpHeader.UPLOAD_LENGTH, contentBytes.length); + servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + assertResponseStatus(HttpServletResponse.SC_CREATED); + String child2Uri = servletResponse.getHeader(HttpHeader.LOCATION); + + reset(); + servletRequest.setMethod("PATCH"); + servletRequest.setRequestURI(child2Uri); + servletRequest.addHeader(HttpHeader.CONTENT_TYPE, "application/offset+octet-stream"); + servletRequest.addHeader(HttpHeader.UPLOAD_OFFSET, 0); + servletRequest.addHeader(HttpHeader.UPLOAD_CHECKSUM, "sha1 " + sha1Base64); + servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); + servletRequest.setContent(contentBytes); + UploadInfo child2Info = + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + assertResponseStatus(HttpServletResponse.SC_NO_CONTENT); + assertNotNull(child2Info); + assertEquals(parentInfo.getId(), child2Info.getDuplicatesUploadId()); + + // 4. Terminate parent upload via DELETE + reset(); + servletRequest.setMethod("DELETE"); + servletRequest.setRequestURI(parentUri); + servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + assertResponseStatus(HttpServletResponse.SC_NO_CONTENT); + assertNull(tusFileUploadService.getUploadInfo(parentUri, OWNER_KEY)); + + // 5. Verify both children can still stream data + try (InputStream is = tusFileUploadService.getUploadedBytes(child1Info)) { + assertThat(IOUtils.toString(is, StandardCharsets.UTF_8), is(content)); + } + try (InputStream is = tusFileUploadService.getUploadedBytes(child2Info)) { + assertThat(IOUtils.toString(is, StandardCharsets.UTF_8), is(content)); + } + + // 6. Terminate child 1 via DELETE + reset(); + servletRequest.setMethod("DELETE"); + servletRequest.setRequestURI(child1Uri); + servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + assertResponseStatus(HttpServletResponse.SC_NO_CONTENT); + assertNull(tusFileUploadService.getUploadInfo(child1Uri, OWNER_KEY)); + + // 7. Child 2 must STILL have readable data + try (InputStream is = tusFileUploadService.getUploadedBytes(child2Info)) { + assertThat(IOUtils.toString(is, StandardCharsets.UTF_8), is(content)); + } + + // 8. Terminate child 2 via DELETE (last child out) + reset(); + servletRequest.setMethod("DELETE"); + servletRequest.setRequestURI(child2Uri); + servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + assertResponseStatus(HttpServletResponse.SC_NO_CONTENT); + assertNull(tusFileUploadService.getUploadInfo(child2Uri, OWNER_KEY)); + + // 9. Verify subsequent upload with same checksum registers cleanly as new parent + reset(); + servletRequest.setMethod("POST"); + servletRequest.setRequestURI(UPLOAD_URI); + servletRequest.addHeader(HttpHeader.UPLOAD_LENGTH, contentBytes.length); + servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + assertResponseStatus(HttpServletResponse.SC_CREATED); + String newParentUri = servletResponse.getHeader(HttpHeader.LOCATION); + + reset(); + servletRequest.setMethod("PATCH"); + servletRequest.setRequestURI(newParentUri); + servletRequest.addHeader(HttpHeader.CONTENT_TYPE, "application/offset+octet-stream"); + servletRequest.addHeader(HttpHeader.UPLOAD_OFFSET, 0); + servletRequest.addHeader(HttpHeader.UPLOAD_CHECKSUM, "sha1 " + sha1Base64); + servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); + servletRequest.setContent(contentBytes); + UploadInfo newParent = tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + assertResponseStatus(HttpServletResponse.SC_NO_CONTENT); + assertNotNull(newParent); + assertNull(newParent.getDuplicatesUploadId()); + } + + @Test + public void testDownloadFailsWhenParentBytesDeletedOutOfBandWithDeduplicationEnabled() + throws Exception { + tusFileUploadService.withUploadDeduplication(true); + + String content = "Test Out Of Band Deletion With Deduplication Enabled"; + byte[] contentBytes = content.getBytes(StandardCharsets.UTF_8); + String sha1Base64 = + org.apache.commons.codec.binary.Base64.encodeBase64String( + org.apache.commons.codec.digest.DigestUtils.sha1(contentBytes)); + + // 1. Create and complete parent upload + reset(); + servletRequest.setMethod("POST"); + servletRequest.setRequestURI(UPLOAD_URI); + servletRequest.addHeader(HttpHeader.UPLOAD_LENGTH, contentBytes.length); + servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + assertResponseStatus(HttpServletResponse.SC_CREATED); + String parentUri = servletResponse.getHeader(HttpHeader.LOCATION); + + reset(); + servletRequest.setMethod("PATCH"); + servletRequest.setRequestURI(parentUri); + servletRequest.addHeader(HttpHeader.CONTENT_TYPE, "application/offset+octet-stream"); + servletRequest.addHeader(HttpHeader.UPLOAD_OFFSET, 0); + servletRequest.addHeader(HttpHeader.UPLOAD_CHECKSUM, "sha1 " + sha1Base64); + servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); + servletRequest.setContent(contentBytes); + UploadInfo parentInfo = + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + assertResponseStatus(HttpServletResponse.SC_NO_CONTENT); + assertNotNull(parentInfo); + assertNull(parentInfo.getDuplicatesUploadId()); + + // 2. Create and complete child 1 upload + reset(); + servletRequest.setMethod("POST"); + servletRequest.setRequestURI(UPLOAD_URI); + servletRequest.addHeader(HttpHeader.UPLOAD_LENGTH, contentBytes.length); + servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + assertResponseStatus(HttpServletResponse.SC_CREATED); + String child1Uri = servletResponse.getHeader(HttpHeader.LOCATION); + + reset(); + servletRequest.setMethod("PATCH"); + servletRequest.setRequestURI(child1Uri); + servletRequest.addHeader(HttpHeader.CONTENT_TYPE, "application/offset+octet-stream"); + servletRequest.addHeader(HttpHeader.UPLOAD_OFFSET, 0); + servletRequest.addHeader(HttpHeader.UPLOAD_CHECKSUM, "sha1 " + sha1Base64); + servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); + servletRequest.setContent(contentBytes); + UploadInfo child1Info = + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + assertResponseStatus(HttpServletResponse.SC_NO_CONTENT); + assertNotNull(child1Info); + assertEquals(parentInfo.getId(), child1Info.getDuplicatesUploadId()); + + // 3. Create and complete child 2 upload + reset(); + servletRequest.setMethod("POST"); + servletRequest.setRequestURI(UPLOAD_URI); + servletRequest.addHeader(HttpHeader.UPLOAD_LENGTH, contentBytes.length); + servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + assertResponseStatus(HttpServletResponse.SC_CREATED); + String child2Uri = servletResponse.getHeader(HttpHeader.LOCATION); + + reset(); + servletRequest.setMethod("PATCH"); + servletRequest.setRequestURI(child2Uri); + servletRequest.addHeader(HttpHeader.CONTENT_TYPE, "application/offset+octet-stream"); + servletRequest.addHeader(HttpHeader.UPLOAD_OFFSET, 0); + servletRequest.addHeader(HttpHeader.UPLOAD_CHECKSUM, "sha1 " + sha1Base64); + servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); + servletRequest.setContent(contentBytes); + UploadInfo child2Info = + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + assertResponseStatus(HttpServletResponse.SC_NO_CONTENT); + assertNotNull(child2Info); + assertEquals(parentInfo.getId(), child2Info.getDuplicatesUploadId()); + + // 4. Verify initial download succeeds for all three + try (InputStream is = tusFileUploadService.getUploadedBytes(parentInfo)) { + assertThat(IOUtils.toString(is, StandardCharsets.UTF_8), is(content)); + } + try (InputStream is = tusFileUploadService.getUploadedBytes(child1Info)) { + assertThat(IOUtils.toString(is, StandardCharsets.UTF_8), is(content)); + } + try (InputStream is = tusFileUploadService.getUploadedBytes(child2Info)) { + assertThat(IOUtils.toString(is, StandardCharsets.UTF_8), is(content)); + } + + // 5. Delete uploaded bytes of parent OUT-OF-BAND + deleteUploadedBytesOutOfBand(parentInfo.getId()); + + // 6. Verify programmatic download fails for parent, child 1, and child 2 + assertThrows( + UploadNotFoundException.class, () -> tusFileUploadService.getUploadedBytes(parentInfo)); + assertThrows( + UploadNotFoundException.class, () -> tusFileUploadService.getUploadedBytes(child1Info)); + assertThrows( + UploadNotFoundException.class, () -> tusFileUploadService.getUploadedBytes(child2Info)); + + // 7. Verify HTTP GET download fails with 404 for parent, child 1, and child 2 + reset(); + servletRequest.setMethod("GET"); + servletRequest.setRequestURI(parentUri); + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + assertResponseStatus(HttpServletResponse.SC_NOT_FOUND); + + reset(); + servletRequest.setMethod("GET"); + servletRequest.setRequestURI(child1Uri); + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + assertResponseStatus(HttpServletResponse.SC_NOT_FOUND); + + reset(); + servletRequest.setMethod("GET"); + servletRequest.setRequestURI(child2Uri); + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + assertResponseStatus(HttpServletResponse.SC_NOT_FOUND); + } + + @Test + public void + testDownloadSucceedsForChildrenWhenParentBytesDeletedOutOfBandWithDeduplicationDisabled() + throws Exception { + tusFileUploadService.withUploadDeduplication(false); + + String content = "Test Out Of Band Deletion With Deduplication Disabled"; + byte[] contentBytes = content.getBytes(StandardCharsets.UTF_8); + String sha1Base64 = + org.apache.commons.codec.binary.Base64.encodeBase64String( + org.apache.commons.codec.digest.DigestUtils.sha1(contentBytes)); + + // 1. Create and complete parent upload + reset(); + servletRequest.setMethod("POST"); + servletRequest.setRequestURI(UPLOAD_URI); + servletRequest.addHeader(HttpHeader.UPLOAD_LENGTH, contentBytes.length); + servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + assertResponseStatus(HttpServletResponse.SC_CREATED); + String parentUri = servletResponse.getHeader(HttpHeader.LOCATION); + + reset(); + servletRequest.setMethod("PATCH"); + servletRequest.setRequestURI(parentUri); + servletRequest.addHeader(HttpHeader.CONTENT_TYPE, "application/offset+octet-stream"); + servletRequest.addHeader(HttpHeader.UPLOAD_OFFSET, 0); + servletRequest.addHeader(HttpHeader.UPLOAD_CHECKSUM, "sha1 " + sha1Base64); + servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); + servletRequest.setContent(contentBytes); + UploadInfo parentInfo = + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + assertResponseStatus(HttpServletResponse.SC_NO_CONTENT); + assertNotNull(parentInfo); + assertNull(parentInfo.getDuplicatesUploadId()); + + // 2. Create and complete child 1 upload (independent, not deduplicated) + reset(); + servletRequest.setMethod("POST"); + servletRequest.setRequestURI(UPLOAD_URI); + servletRequest.addHeader(HttpHeader.UPLOAD_LENGTH, contentBytes.length); + servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + assertResponseStatus(HttpServletResponse.SC_CREATED); + String child1Uri = servletResponse.getHeader(HttpHeader.LOCATION); + + reset(); + servletRequest.setMethod("PATCH"); + servletRequest.setRequestURI(child1Uri); + servletRequest.addHeader(HttpHeader.CONTENT_TYPE, "application/offset+octet-stream"); + servletRequest.addHeader(HttpHeader.UPLOAD_OFFSET, 0); + servletRequest.addHeader(HttpHeader.UPLOAD_CHECKSUM, "sha1 " + sha1Base64); + servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); + servletRequest.setContent(contentBytes); + UploadInfo child1Info = + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + assertResponseStatus(HttpServletResponse.SC_NO_CONTENT); + assertNotNull(child1Info); + assertNull(child1Info.getDuplicatesUploadId()); + + // 3. Create and complete child 2 upload (independent, not deduplicated) + reset(); + servletRequest.setMethod("POST"); + servletRequest.setRequestURI(UPLOAD_URI); + servletRequest.addHeader(HttpHeader.UPLOAD_LENGTH, contentBytes.length); + servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + assertResponseStatus(HttpServletResponse.SC_CREATED); + String child2Uri = servletResponse.getHeader(HttpHeader.LOCATION); + + reset(); + servletRequest.setMethod("PATCH"); + servletRequest.setRequestURI(child2Uri); + servletRequest.addHeader(HttpHeader.CONTENT_TYPE, "application/offset+octet-stream"); + servletRequest.addHeader(HttpHeader.UPLOAD_OFFSET, 0); + servletRequest.addHeader(HttpHeader.UPLOAD_CHECKSUM, "sha1 " + sha1Base64); + servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); + servletRequest.setContent(contentBytes); + UploadInfo child2Info = + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + assertResponseStatus(HttpServletResponse.SC_NO_CONTENT); + assertNotNull(child2Info); + assertNull(child2Info.getDuplicatesUploadId()); + + // 4. Delete uploaded bytes of parent OUT-OF-BAND + deleteUploadedBytesOutOfBand(parentInfo.getId()); + + // 5. Download of parent FAILS + assertThrows( + UploadNotFoundException.class, () -> tusFileUploadService.getUploadedBytes(parentInfo)); + reset(); + servletRequest.setMethod("GET"); + servletRequest.setRequestURI(parentUri); + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + assertResponseStatus(HttpServletResponse.SC_NOT_FOUND); + + // 6. Download of child 1 and child 2 SUCCEEDS + try (InputStream is = tusFileUploadService.getUploadedBytes(child1Info)) { + assertThat(IOUtils.toString(is, StandardCharsets.UTF_8), is(content)); + } + try (InputStream is = tusFileUploadService.getUploadedBytes(child2Info)) { + assertThat(IOUtils.toString(is, StandardCharsets.UTF_8), is(content)); + } + + reset(); + servletRequest.setMethod("GET"); + servletRequest.setRequestURI(child1Uri); + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + assertResponseStatus(HttpServletResponse.SC_OK); + assertThat(servletResponse.getContentAsString(), is(content)); + + reset(); + servletRequest.setMethod("GET"); + servletRequest.setRequestURI(child2Uri); + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + assertResponseStatus(HttpServletResponse.SC_OK); + assertThat(servletResponse.getContentAsString(), is(content)); + } + + @Test + public void testNewChildJoinsTombstonedParentAndPurgesAfterAllChildrenDeleted() throws Exception { + tusFileUploadService.withUploadDeduplication(true); + + String content = "Test Content For Tombstoned Parent Child Join " + UUID.randomUUID(); + byte[] contentBytes = content.getBytes(StandardCharsets.UTF_8); + String sha1Base64 = + org.apache.commons.codec.binary.Base64.encodeBase64String( + org.apache.commons.codec.digest.DigestUtils.sha1(contentBytes)); + + // 1. Create and complete parent upload (P) + reset(); + servletRequest.setMethod("POST"); + servletRequest.setRequestURI(UPLOAD_URI); + servletRequest.addHeader(HttpHeader.UPLOAD_LENGTH, contentBytes.length); + servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + assertResponseStatus(HttpServletResponse.SC_CREATED); + String parentUri = servletResponse.getHeader(HttpHeader.LOCATION); + + reset(); + servletRequest.setMethod("PATCH"); + servletRequest.setRequestURI(parentUri); + servletRequest.addHeader(HttpHeader.CONTENT_TYPE, "application/offset+octet-stream"); + servletRequest.addHeader(HttpHeader.UPLOAD_OFFSET, 0); + servletRequest.addHeader(HttpHeader.UPLOAD_CHECKSUM, "sha1 " + sha1Base64); + servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); + servletRequest.setContent(contentBytes); + UploadInfo parentInfo = + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + assertResponseStatus(HttpServletResponse.SC_NO_CONTENT); + assertNotNull(parentInfo); + assertNull(parentInfo.getDuplicatesUploadId()); + + // 2. Create and complete first child upload (C1) + reset(); + servletRequest.setMethod("POST"); + servletRequest.setRequestURI(UPLOAD_URI); + servletRequest.addHeader(HttpHeader.UPLOAD_LENGTH, contentBytes.length); + servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + assertResponseStatus(HttpServletResponse.SC_CREATED); + String child1Uri = servletResponse.getHeader(HttpHeader.LOCATION); + + reset(); + servletRequest.setMethod("PATCH"); + servletRequest.setRequestURI(child1Uri); + servletRequest.addHeader(HttpHeader.CONTENT_TYPE, "application/offset+octet-stream"); + servletRequest.addHeader(HttpHeader.UPLOAD_OFFSET, 0); + servletRequest.addHeader(HttpHeader.UPLOAD_CHECKSUM, "sha1 " + sha1Base64); + servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); + servletRequest.setContent(contentBytes); + UploadInfo child1Info = + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + assertResponseStatus(HttpServletResponse.SC_NO_CONTENT); + assertNotNull(child1Info); + assertEquals(parentInfo.getId(), child1Info.getDuplicatesUploadId()); + + // 3. Delete Parent (P) via DELETE HTTP request -> P is tombstoned while C1 remains + reset(); + servletRequest.setMethod("DELETE"); + servletRequest.setRequestURI(parentUri); + servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + assertResponseStatus(HttpServletResponse.SC_NO_CONTENT); + + // Verify P is 404 / null + assertNull(tusFileUploadService.getUploadInfo(parentUri, OWNER_KEY)); + + // Verify C1 can still download data + try (InputStream is = tusFileUploadService.getUploadedBytes(child1Info)) { + assertThat(IOUtils.toString(is, StandardCharsets.UTF_8), is(content)); + } + + // 4. Create and complete new child upload (C2) while P is tombstoned + reset(); + servletRequest.setMethod("POST"); + servletRequest.setRequestURI(UPLOAD_URI); + servletRequest.addHeader(HttpHeader.UPLOAD_LENGTH, contentBytes.length); + servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + assertResponseStatus(HttpServletResponse.SC_CREATED); + String child2Uri = servletResponse.getHeader(HttpHeader.LOCATION); + + reset(); + servletRequest.setMethod("PATCH"); + servletRequest.setRequestURI(child2Uri); + servletRequest.addHeader(HttpHeader.CONTENT_TYPE, "application/offset+octet-stream"); + servletRequest.addHeader(HttpHeader.UPLOAD_OFFSET, 0); + servletRequest.addHeader(HttpHeader.UPLOAD_CHECKSUM, "sha1 " + sha1Base64); + servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); + servletRequest.setContent(contentBytes); + UploadInfo child2Info = + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + assertResponseStatus(HttpServletResponse.SC_NO_CONTENT); + assertNotNull(child2Info); + // C2 successfully joined the tombstoned parent P! + assertEquals(parentInfo.getId(), child2Info.getDuplicatesUploadId()); + + // Both C1 and C2 stream content successfully + try (InputStream is = tusFileUploadService.getUploadedBytes(child1Info)) { + assertThat(IOUtils.toString(is, StandardCharsets.UTF_8), is(content)); + } + try (InputStream is = tusFileUploadService.getUploadedBytes(child2Info)) { + assertThat(IOUtils.toString(is, StandardCharsets.UTF_8), is(content)); + } + + // 5. Delete child 1 (C1) via DELETE -> C2 remains valid and streams content + reset(); + servletRequest.setMethod("DELETE"); + servletRequest.setRequestURI(child1Uri); + servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + assertResponseStatus(HttpServletResponse.SC_NO_CONTENT); + + assertNull(tusFileUploadService.getUploadInfo(child1Uri, OWNER_KEY)); + try (InputStream is = tusFileUploadService.getUploadedBytes(child2Info)) { + assertThat(IOUtils.toString(is, StandardCharsets.UTF_8), is(content)); + } + + // 6. Delete child 2 (C2) (last child) -> tombstoned parent data and checksum index purged + reset(); + servletRequest.setMethod("DELETE"); + servletRequest.setRequestURI(child2Uri); + servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + assertResponseStatus(HttpServletResponse.SC_NO_CONTENT); + + assertNull(tusFileUploadService.getUploadInfo(child2Uri, OWNER_KEY)); + + // 7. Uploading the same content again registers a clean new parent P2 (not child) + reset(); + servletRequest.setMethod("POST"); + servletRequest.setRequestURI(UPLOAD_URI); + servletRequest.addHeader(HttpHeader.UPLOAD_LENGTH, contentBytes.length); + servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + assertResponseStatus(HttpServletResponse.SC_CREATED); + String parent2Uri = servletResponse.getHeader(HttpHeader.LOCATION); + + reset(); + servletRequest.setMethod("PATCH"); + servletRequest.setRequestURI(parent2Uri); + servletRequest.addHeader(HttpHeader.CONTENT_TYPE, "application/offset+octet-stream"); + servletRequest.addHeader(HttpHeader.UPLOAD_OFFSET, 0); + servletRequest.addHeader(HttpHeader.UPLOAD_CHECKSUM, "sha1 " + sha1Base64); + servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); + servletRequest.setContent(contentBytes); + UploadInfo parent2Info = + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + assertResponseStatus(HttpServletResponse.SC_NO_CONTENT); + assertNotNull(parent2Info); + assertNull(parent2Info.getDuplicatesUploadId()); + } + protected void assertResponseHeader(final String header, final String value) { assertThat(servletResponse.getHeader(header), is(value)); } diff --git a/src/test/java/me/desair/tus/server/CoverageGapTest.java b/src/test/java/me/desair/tus/server/CoverageGapTest.java index 45b65b9e..db42559a 100644 --- a/src/test/java/me/desair/tus/server/CoverageGapTest.java +++ b/src/test/java/me/desair/tus/server/CoverageGapTest.java @@ -489,9 +489,16 @@ public boolean isErrorHandler() { UploadLockingService mockLocking = org.mockito.Mockito.mock(UploadLockingService.class); mockHandler.process(HttpMethod.PATCH, null, null, null, mockLocking, "owner", null); - // Test branch: uploadLockingService == null, servletRequest != null + // Test process with NoOpUploadLockingService TusServletRequest mockRequest = org.mockito.Mockito.mock(TusServletRequest.class); - mockHandler.process(HttpMethod.PATCH, mockRequest, null, null, null, "owner", null); + mockHandler.process( + HttpMethod.PATCH, + mockRequest, + null, + null, + new me.desair.tus.server.upload.NoOpUploadLockingService(), + "owner", + null); // Test branch: uploadLockingService != null, servletRequest != null mockHandler.process(HttpMethod.PATCH, mockRequest, null, null, mockLocking, "owner", null); diff --git a/src/test/java/me/desair/tus/server/ITDiskTusFileUploadService.java b/src/test/java/me/desair/tus/server/ITDiskTusFileUploadService.java index 6e70af6a..6b7efc4e 100644 --- a/src/test/java/me/desair/tus/server/ITDiskTusFileUploadService.java +++ b/src/test/java/me/desair/tus/server/ITDiskTusFileUploadService.java @@ -4,6 +4,7 @@ import java.nio.file.Files; import java.nio.file.Path; import java.nio.file.Paths; +import me.desair.tus.server.upload.UploadId; import me.desair.tus.server.upload.disk.DiskLockingService; import me.desair.tus.server.upload.disk.DiskStorageService; import org.apache.commons.io.FileUtils; @@ -45,4 +46,10 @@ protected TusFileUploadService createTusFileUploadService(String uploadUri) { .withDownloadFeature() .withChunkedTransferDecoding(true); } + + @Override + protected void deleteUploadedBytesOutOfBand(UploadId uploadId) throws Exception { + Files.deleteIfExists( + storagePath.resolve("uploads").resolve(uploadId.toString()).resolve("data")); + } } diff --git a/src/test/java/me/desair/tus/server/ITLeaseFileRufhProtocol.java b/src/test/java/me/desair/tus/server/ITLeaseFileRufhProtocol.java index 0ba2d781..a0010379 100644 --- a/src/test/java/me/desair/tus/server/ITLeaseFileRufhProtocol.java +++ b/src/test/java/me/desair/tus/server/ITLeaseFileRufhProtocol.java @@ -4,6 +4,7 @@ import java.nio.file.Files; import java.nio.file.Path; import java.nio.file.Paths; +import me.desair.tus.server.upload.UploadId; import me.desair.tus.server.upload.disk.DiskStorageService; import me.desair.tus.server.upload.disk.LeaseFileLockingService; import org.apache.commons.io.FileUtils; @@ -46,4 +47,10 @@ protected TusFileUploadService createTusFileUploadService(String uploadUri) { .withUploadExpirationPeriod(2L * 24 * 60 * 60 * 1000) .withDownloadFeature(); } + + @Override + protected void deleteUploadedBytesOutOfBand(UploadId uploadId) throws Exception { + Files.deleteIfExists( + storagePath.resolve("uploads").resolve(uploadId.toString()).resolve("data")); + } } diff --git a/src/test/java/me/desair/tus/server/ITLeaseFileTusFileUploadService.java b/src/test/java/me/desair/tus/server/ITLeaseFileTusFileUploadService.java index e473d7e6..0d3fbfbc 100644 --- a/src/test/java/me/desair/tus/server/ITLeaseFileTusFileUploadService.java +++ b/src/test/java/me/desair/tus/server/ITLeaseFileTusFileUploadService.java @@ -4,6 +4,7 @@ import java.nio.file.Files; import java.nio.file.Path; import java.nio.file.Paths; +import me.desair.tus.server.upload.UploadId; import me.desair.tus.server.upload.disk.DiskStorageService; import me.desair.tus.server.upload.disk.LeaseFileLockingService; import org.apache.commons.io.FileUtils; @@ -47,4 +48,10 @@ protected TusFileUploadService createTusFileUploadService(String uploadUri) { .withDownloadFeature() .withChunkedTransferDecoding(true); } + + @Override + protected void deleteUploadedBytesOutOfBand(UploadId uploadId) throws Exception { + Files.deleteIfExists( + storagePath.resolve("uploads").resolve(uploadId.toString()).resolve("data")); + } } diff --git a/src/test/java/me/desair/tus/server/ITRufhProtocol.java b/src/test/java/me/desair/tus/server/ITRufhProtocol.java index 9f02cba0..cc3af97f 100644 --- a/src/test/java/me/desair/tus/server/ITRufhProtocol.java +++ b/src/test/java/me/desair/tus/server/ITRufhProtocol.java @@ -4,6 +4,7 @@ import java.nio.file.Files; import java.nio.file.Path; import java.nio.file.Paths; +import me.desair.tus.server.upload.UploadId; import org.apache.commons.io.FileUtils; import org.junit.AfterClass; import org.junit.BeforeClass; @@ -40,4 +41,10 @@ protected TusFileUploadService createTusFileUploadService(String uploadUri) { .withUploadExpirationPeriod(2L * 24 * 60 * 60 * 1000) .withDownloadFeature(); } + + @Override + protected void deleteUploadedBytesOutOfBand(UploadId uploadId) throws Exception { + Files.deleteIfExists( + storagePath.resolve("uploads").resolve(uploadId.toString()).resolve("data")); + } } diff --git a/src/test/java/me/desair/tus/server/ITTusFileUploadService.java b/src/test/java/me/desair/tus/server/ITTusFileUploadService.java index f39f6112..6c06c96d 100644 --- a/src/test/java/me/desair/tus/server/ITTusFileUploadService.java +++ b/src/test/java/me/desair/tus/server/ITTusFileUploadService.java @@ -2,14 +2,17 @@ import static org.hamcrest.CoreMatchers.is; import static org.hamcrest.MatcherAssert.assertThat; +import static org.junit.Assert.assertEquals; import static org.junit.Assert.assertFalse; import static org.junit.Assert.assertTrue; import jakarta.servlet.http.HttpServletResponse; import java.io.IOException; +import java.nio.charset.StandardCharsets; import java.nio.file.Files; import java.nio.file.Path; import java.nio.file.Paths; +import me.desair.tus.server.upload.UploadId; import me.desair.tus.server.upload.UploadInfo; import org.apache.commons.io.FileUtils; import org.apache.commons.lang3.StringUtils; @@ -163,13 +166,132 @@ public void testAutomaticDeduplicationOnUploadCompletion() throws Exception { assertResponseStatus(HttpServletResponse.SC_OK); assertThat(servletResponse.getContentAsString(), is(manipulatedContent)); - // Step 10: Delete parent upload and verify child download returns 404 + // Step 10: Delete parent upload and verify child download still succeeds (parent data + // preserved) tusFileUploadService.deleteUpload(parentLocation, OWNER_KEY); + reset(); + servletRequest.setMethod("GET"); + servletRequest.setRequestURI(childLocation); + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + assertResponseStatus(HttpServletResponse.SC_OK); + assertThat(servletResponse.getContentAsString(), is(manipulatedContent)); + + // Step 11: Delete child upload and verify child download returns 404 + tusFileUploadService.deleteUpload(childLocation, OWNER_KEY); + reset(); servletRequest.setMethod("GET"); servletRequest.setRequestURI(childLocation); tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); assertResponseStatus(HttpServletResponse.SC_NOT_FOUND); } + + /** + * Disk Integration Test: Verifies that during the end-to-end Tus protocol lifecycle, the physical + * checksums directory and markers are created on disk and properly purged on final deletion. + */ + @Test + public void testPhysicalChecksumDirectoryCreationAndCleanupOnDisk() throws Exception { + tusFileUploadService.withUploadDeduplication(true); + + String content = "Physical Disk Checksum Verification Payload"; + byte[] contentBytes = content.getBytes(StandardCharsets.UTF_8); + byte[] sha1Digest = org.apache.commons.codec.digest.DigestUtils.sha1(contentBytes); + String sha1Hex = org.apache.commons.codec.binary.Hex.encodeHexString(sha1Digest); + String sha1Base64 = org.apache.commons.codec.binary.Base64.encodeBase64String(sha1Digest); + + Path indexDir = storagePath.resolve("checksums").resolve("sha1").resolve(sha1Hex); + Path parentMarker = indexDir.resolve("parent"); + Path parentDeletedMarker = indexDir.resolve("parent-deleted"); + + // 1. Create parent upload + reset(); + servletRequest.setMethod("POST"); + servletRequest.setRequestURI(UPLOAD_URI); + servletRequest.addHeader(HttpHeader.UPLOAD_LENGTH, contentBytes.length); + servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + assertResponseStatus(HttpServletResponse.SC_CREATED); + String parentUri = servletResponse.getHeader(HttpHeader.LOCATION); + + // Complete parent upload via PATCH with checksum + reset(); + servletRequest.setMethod("PATCH"); + servletRequest.setRequestURI(parentUri); + servletRequest.addHeader(HttpHeader.CONTENT_TYPE, "application/offset+octet-stream"); + servletRequest.addHeader(HttpHeader.UPLOAD_OFFSET, 0); + servletRequest.addHeader(HttpHeader.UPLOAD_CHECKSUM, "sha1 " + sha1Base64); + servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); + servletRequest.setContent(contentBytes); + UploadInfo parentInfo = + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + assertResponseStatus(HttpServletResponse.SC_NO_CONTENT); + + // Assert physical directory and parent marker exist on disk + assertTrue( + "Physical checksum directory must exist on disk after parent upload", + Files.isDirectory(indexDir)); + assertTrue("Parent pointer file must exist on disk", Files.isRegularFile(parentMarker)); + assertEquals( + parentInfo.getId().toString(), + Files.readString(parentMarker, StandardCharsets.UTF_8).trim()); + assertFalse( + "Parent-deleted tombstone must not exist initially", Files.exists(parentDeletedMarker)); + + // 2. Create duplicate child upload + reset(); + servletRequest.setMethod("POST"); + servletRequest.setRequestURI(UPLOAD_URI); + servletRequest.addHeader(HttpHeader.UPLOAD_LENGTH, contentBytes.length); + servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + assertResponseStatus(HttpServletResponse.SC_CREATED); + String childUri = servletResponse.getHeader(HttpHeader.LOCATION); + + // Complete child upload via PATCH with same checksum + reset(); + servletRequest.setMethod("PATCH"); + servletRequest.setRequestURI(childUri); + servletRequest.addHeader(HttpHeader.CONTENT_TYPE, "application/offset+octet-stream"); + servletRequest.addHeader(HttpHeader.UPLOAD_OFFSET, 0); + servletRequest.addHeader(HttpHeader.UPLOAD_CHECKSUM, "sha1 " + sha1Base64); + servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); + servletRequest.setContent(contentBytes); + UploadInfo childInfo = tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + assertResponseStatus(HttpServletResponse.SC_NO_CONTENT); + assertEquals(parentInfo.getId(), childInfo.getDuplicatesUploadId()); + + Path childMarker = indexDir.resolve("children").resolve(childInfo.getId().toString()); + assertTrue( + "Child marker file must exist on disk under children/", Files.isRegularFile(childMarker)); + + // 3. Delete parent upload via DELETE + reset(); + servletRequest.setMethod("DELETE"); + servletRequest.setRequestURI(parentUri); + servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + assertResponseStatus(HttpServletResponse.SC_NO_CONTENT); + + assertTrue( + "Parent-deleted tombstone must exist on disk", Files.isRegularFile(parentDeletedMarker)); + + // 4. Delete child upload via DELETE (last child out) + reset(); + servletRequest.setMethod("DELETE"); + servletRequest.setRequestURI(childUri); + servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + assertResponseStatus(HttpServletResponse.SC_NO_CONTENT); + + assertFalse( + "Physical checksum directory must be completely purged from disk", Files.exists(indexDir)); + } + + @Override + protected void deleteUploadedBytesOutOfBand(UploadId uploadId) throws Exception { + Files.deleteIfExists( + storagePath.resolve("uploads").resolve(uploadId.toString()).resolve("data")); + } } diff --git a/src/test/java/me/desair/tus/server/TusFileUploadServiceTest.java b/src/test/java/me/desair/tus/server/TusFileUploadServiceTest.java index b1b8ec93..25e2334c 100644 --- a/src/test/java/me/desair/tus/server/TusFileUploadServiceTest.java +++ b/src/test/java/me/desair/tus/server/TusFileUploadServiceTest.java @@ -988,7 +988,7 @@ public void testProcessCleansUpLockOnUploadCompletion() throws Exception { } @Test - public void testProcessCleansUpLockOnDeleteTermination() throws Exception { + public void testProcessTerminatesUploadOnDelete() throws Exception { UploadLockingService mockLockingService = mock(UploadLockingService.class); UploadLock mockLock = mock(UploadLock.class); when(mockLockingService.lockUploadByUri(anyString())).thenReturn(mockLock); @@ -1022,11 +1022,11 @@ public void testProcessCleansUpLockOnDeleteTermination() throws Exception { service.process(mockReq, mockResp, "owner"); - verify(mockLockingService).cleanupLock(testId); + verify(mockStorage).terminateUpload(info); } @Test - public void testDeleteUploadCleansUpLock() throws Exception { + public void testDeleteUploadTerminatesUpload() throws Exception { UploadLockingService mockLockingService = mock(UploadLockingService.class); UploadLock mockLock = mock(UploadLock.class); when(mockLockingService.lockUploadByUri(anyString())).thenReturn(mockLock); @@ -1049,6 +1049,5 @@ public void testDeleteUploadCleansUpLock() throws Exception { service.deleteUpload(uploadUri, "owner"); verify(mockStorage).terminateUpload(info); - verify(mockLockingService).cleanupLock(testId); } } diff --git a/src/test/java/me/desair/tus/server/checksum/ChecksumKeyTest.java b/src/test/java/me/desair/tus/server/checksum/ChecksumKeyTest.java new file mode 100644 index 00000000..13be06c1 --- /dev/null +++ b/src/test/java/me/desair/tus/server/checksum/ChecksumKeyTest.java @@ -0,0 +1,70 @@ +package me.desair.tus.server.checksum; + +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertNotEquals; + +import org.junit.Test; + +/** Unit tests for {@link ChecksumKey}. */ +public class ChecksumKeyTest { + + @Test + public void testHexChecksumNormalization() { + ChecksumKey key = ChecksumKey.of("ABCDEF0123456789", ChecksumAlgorithm.SHA1); + assertEquals("abcdef0123456789", key.getHexChecksum()); + assertEquals(ChecksumAlgorithm.SHA1, key.getAlgorithm()); + assertEquals("sha1/abcdef0123456789", key.toString()); + } + + @Test + public void testHexChecksumWithWhitespace() { + ChecksumKey key = ChecksumKey.of(" abcdef0123456789 ", ChecksumAlgorithm.SHA256); + assertEquals("abcdef0123456789", key.getHexChecksum()); + assertEquals(ChecksumAlgorithm.SHA256, key.getAlgorithm()); + } + + @Test + public void testBase64ChecksumDecoding() { + // "hello" in ASCII bytes is 68 65 6c 6c 6f. Base64 is "aGVsbG8=" + ChecksumKey key = ChecksumKey.of("aGVsbG8=", ChecksumAlgorithm.SHA1); + assertEquals("68656c6c6f", key.getHexChecksum()); + } + + @Test + public void testEqualsAndHashCode() { + ChecksumKey key1 = ChecksumKey.of("abcdef", ChecksumAlgorithm.SHA1); + ChecksumKey key2 = ChecksumKey.of("ABCDEF", ChecksumAlgorithm.SHA1); + ChecksumKey key3 = ChecksumKey.of("abcdef", ChecksumAlgorithm.SHA256); + + assertEquals(key1, key2); + assertEquals(key1.hashCode(), key2.hashCode()); + assertNotEquals(key1, key3); + assertNotEquals(key1, null); + assertNotEquals(key1, "not-a-checksum-key"); + } + + @Test(expected = NullPointerException.class) + public void testNullAlgorithm() { + ChecksumKey.of("abcdef", null); + } + + @Test(expected = IllegalArgumentException.class) + public void testNullChecksum() { + ChecksumKey.of(null, ChecksumAlgorithm.SHA1); + } + + @Test(expected = IllegalArgumentException.class) + public void testEmptyChecksum() { + ChecksumKey.of(" ", ChecksumAlgorithm.SHA1); + } + + @Test(expected = IllegalArgumentException.class) + public void testInvalidCharacters() { + ChecksumKey.of("invalid!@#$", ChecksumAlgorithm.SHA1); + } + + @Test(expected = IllegalArgumentException.class) + public void testPathTraversalAttempt() { + ChecksumKey.of("../../../etc/passwd", ChecksumAlgorithm.SHA1); + } +} diff --git a/src/test/java/me/desair/tus/server/core/CorePatchRequestHandlerTest.java b/src/test/java/me/desair/tus/server/core/CorePatchRequestHandlerTest.java index 1656ae53..9ee60c20 100644 --- a/src/test/java/me/desair/tus/server/core/CorePatchRequestHandlerTest.java +++ b/src/test/java/me/desair/tus/server/core/CorePatchRequestHandlerTest.java @@ -191,24 +191,7 @@ public void processWithLockingService() throws Exception { } @Test - public void testProcess7ParamsBranchCoverage() throws Exception { - UploadLockingService mockLocking = mock(UploadLockingService.class); - - // lockingService != null, servletRequest == null - try { - handler.process( - HttpMethod.PATCH, - null, - new TusServletResponse(servletResponse), - uploadStorageService, - mockLocking, - "owner", - null); - } catch (Exception e) { - // Expected, we just need to hit the branch check - } - - // lockingService == null, servletRequest != null + public void testProcessWithNoOpUploadLockingService() throws Exception { UploadInfo info = new UploadInfo(); info.setId(new UploadId(UUID.randomUUID())); info.setOffset(2L); @@ -223,9 +206,11 @@ public void testProcess7ParamsBranchCoverage() throws Exception { new TusServletRequest(servletRequest), new TusServletResponse(servletResponse), uploadStorageService, - null, + new me.desair.tus.server.upload.NoOpUploadLockingService(), "owner", null); + + verify(uploadStorageService, times(1)).append(eq(info), any(InputStream.class)); } @Test diff --git a/src/test/java/me/desair/tus/server/creationwithupload/CreationWithUploadPostRequestHandlerTest.java b/src/test/java/me/desair/tus/server/creationwithupload/CreationWithUploadPostRequestHandlerTest.java index bf7848a9..bb7b882e 100644 --- a/src/test/java/me/desair/tus/server/creationwithupload/CreationWithUploadPostRequestHandlerTest.java +++ b/src/test/java/me/desair/tus/server/creationwithupload/CreationWithUploadPostRequestHandlerTest.java @@ -92,7 +92,14 @@ public void testProcessWithoutLockingService() throws Exception { when(storageService.getUploadInfo("/files/123", "owner")).thenReturn(uploadInfo); when(storageService.append(eq(uploadInfo), any(InputStream.class))).thenReturn(updatedInfo); - handler.process(HttpMethod.POST, request, response, storageService, null, "owner", null); + handler.process( + HttpMethod.POST, + request, + response, + storageService, + new me.desair.tus.server.upload.NoOpUploadLockingService(), + "owner", + null); verify(response).setHeader(HttpHeader.UPLOAD_OFFSET, "5"); } diff --git a/src/test/java/me/desair/tus/server/expiration/ExpirationRequestHandlerTest.java b/src/test/java/me/desair/tus/server/expiration/ExpirationRequestHandlerTest.java index da6944f9..2283573a 100644 --- a/src/test/java/me/desair/tus/server/expiration/ExpirationRequestHandlerTest.java +++ b/src/test/java/me/desair/tus/server/expiration/ExpirationRequestHandlerTest.java @@ -239,7 +239,6 @@ public void testExpiredUploadTerminatesUploadAndCleansUpLock() throws Exception null); } finally { verify(uploadStorageService, times(1)).terminateUpload(info); - verify(mockLocking, times(1)).cleanupLock(info.getId()); } } diff --git a/src/test/java/me/desair/tus/server/rufh/handler/RufhAppendPatchRequestHandlerTest.java b/src/test/java/me/desair/tus/server/rufh/handler/RufhAppendPatchRequestHandlerTest.java index 27441b1e..2d6efd3c 100644 --- a/src/test/java/me/desair/tus/server/rufh/handler/RufhAppendPatchRequestHandlerTest.java +++ b/src/test/java/me/desair/tus/server/rufh/handler/RufhAppendPatchRequestHandlerTest.java @@ -14,6 +14,7 @@ import java.io.InputStream; import me.desair.tus.server.HttpHeader; import me.desair.tus.server.HttpMethod; +import me.desair.tus.server.upload.NoOpUploadLockingService; import me.desair.tus.server.upload.UploadId; import me.desair.tus.server.upload.UploadInfo; import me.desair.tus.server.upload.UploadLockingService; @@ -181,7 +182,7 @@ public void testProcessCallFiveParameterMethodAndAppendReturnsNull() throws Exce new TusServletRequest(request), new TusServletResponse(response), storageService, - null, + new NoOpUploadLockingService(), "owner", null); @@ -211,7 +212,7 @@ public void testProcessIsFinishedCombinations() throws Exception { new TusServletRequest(request), new TusServletResponse(response), storageService, - null, + new NoOpUploadLockingService(), "owner", null); diff --git a/src/test/java/me/desair/tus/server/rufh/handler/RufhCreationPostRequestHandlerTest.java b/src/test/java/me/desair/tus/server/rufh/handler/RufhCreationPostRequestHandlerTest.java index 34ad2bf4..76215ddd 100644 --- a/src/test/java/me/desair/tus/server/rufh/handler/RufhCreationPostRequestHandlerTest.java +++ b/src/test/java/me/desair/tus/server/rufh/handler/RufhCreationPostRequestHandlerTest.java @@ -15,6 +15,7 @@ import java.io.InputStream; import me.desair.tus.server.HttpHeader; import me.desair.tus.server.HttpMethod; +import me.desair.tus.server.upload.NoOpUploadLockingService; import me.desair.tus.server.upload.UploadId; import me.desair.tus.server.upload.UploadInfo; import me.desair.tus.server.upload.UploadLockingService; @@ -172,7 +173,7 @@ public void testProcessWithoutLockingServiceAndAppendReturnsNull() throws Except new TusServletRequest(request), new TusServletResponse(response), storageService, - null, + new NoOpUploadLockingService(), "owner", null); diff --git a/src/test/java/me/desair/tus/server/rufh/handler/RufhDeleteRequestHandlerTest.java b/src/test/java/me/desair/tus/server/rufh/handler/RufhDeleteRequestHandlerTest.java index e451523d..1fc28768 100644 --- a/src/test/java/me/desair/tus/server/rufh/handler/RufhDeleteRequestHandlerTest.java +++ b/src/test/java/me/desair/tus/server/rufh/handler/RufhDeleteRequestHandlerTest.java @@ -70,7 +70,6 @@ public void testProcessDeleteRequest() throws Exception { assertThat(response.getStatus(), is(204)); verify(storageService).terminateUpload(info); - verify(mockLocking).cleanupLock(info.getId()); } @Test(expected = me.desair.tus.server.exception.TusException.class) diff --git a/src/test/java/me/desair/tus/server/termination/TerminationDeleteRequestHandlerTest.java b/src/test/java/me/desair/tus/server/termination/TerminationDeleteRequestHandlerTest.java index c4753fd2..3743971b 100644 --- a/src/test/java/me/desair/tus/server/termination/TerminationDeleteRequestHandlerTest.java +++ b/src/test/java/me/desair/tus/server/termination/TerminationDeleteRequestHandlerTest.java @@ -117,7 +117,6 @@ public void testWithExistingUploadAndLockingService() throws Exception { null); verify(uploadStorageService, times(1)).terminateUpload(info); - verify(mockLocking, times(1)).cleanupLock(id); assertThat(servletResponse.getStatus(), is(HttpServletResponse.SC_NO_CONTENT)); } } diff --git a/src/test/java/me/desair/tus/server/upload/NoOpUploadLockingServiceTest.java b/src/test/java/me/desair/tus/server/upload/NoOpUploadLockingServiceTest.java new file mode 100644 index 00000000..a86f1eb6 --- /dev/null +++ b/src/test/java/me/desair/tus/server/upload/NoOpUploadLockingServiceTest.java @@ -0,0 +1,45 @@ +package me.desair.tus.server.upload; + +import static org.junit.Assert.assertFalse; +import static org.junit.Assert.assertNotNull; +import static org.junit.Assert.assertNull; + +import java.io.ByteArrayInputStream; +import org.junit.Test; + +public class NoOpUploadLockingServiceTest { + + @Test + public void testLockOperationsReturnNonNullLockThatClosesCleanly() throws Exception { + NoOpUploadLockingService service = new NoOpUploadLockingService(); + + try (UploadLock lockByUri = service.lockUploadByUri("/files/123")) { + assertNotNull(lockByUri); + assertNull(lockByUri.getUploadUri()); + lockByUri.release(); + } + + UploadId id = new UploadId("test-upload-id"); + try (UploadLock lockById = service.lockUpload(id)) { + assertNotNull(lockById); + assertNull(lockById.getUploadUri()); + lockById.release(); + } + + assertFalse(service.isLocked(id)); + } + + @Test + public void testNoOpLifecycleAndRegistrationMethodsExecuteCleanly() throws Exception { + NoOpUploadLockingService service = new NoOpUploadLockingService(); + UploadId id = new UploadId("test-id"); + + // KISS: verifying method executes cleanly without throwing an exception + service.cleanupStaleLocks(); + service.cleanupLock(id); + service.registerInputStream("/files/123", new ByteArrayInputStream(new byte[0])); + service.requestLockRelease("/files/123"); + service.setIdFactory(new UuidUploadIdFactory()); + service.close(); + } +} diff --git a/src/test/java/me/desair/tus/server/upload/UploadLockingServiceTest.java b/src/test/java/me/desair/tus/server/upload/UploadLockingServiceTest.java index f5ddb64e..c0ecfe92 100644 --- a/src/test/java/me/desair/tus/server/upload/UploadLockingServiceTest.java +++ b/src/test/java/me/desair/tus/server/upload/UploadLockingServiceTest.java @@ -38,5 +38,7 @@ public void setIdFactory(UploadIdFactory idFactory) {} dummyService.close(); assertFalse(dummyService.isLocked(new UploadId("123"))); + org.junit.Assert.assertNull(dummyService.lockUpload((UploadId) null)); + org.junit.Assert.assertNull(dummyService.lockUpload(new UploadId("123"))); } } diff --git a/src/test/java/me/desair/tus/server/upload/UploadStorageServiceTest.java b/src/test/java/me/desair/tus/server/upload/UploadStorageServiceTest.java index 151db73e..cdc1be49 100644 --- a/src/test/java/me/desair/tus/server/upload/UploadStorageServiceTest.java +++ b/src/test/java/me/desair/tus/server/upload/UploadStorageServiceTest.java @@ -9,6 +9,7 @@ import java.io.OutputStream; import java.time.Duration; import me.desair.tus.server.checksum.ChecksumAlgorithm; +import me.desair.tus.server.checksum.ChecksumKey; import me.desair.tus.server.exception.TusException; import me.desair.tus.server.exception.UploadNotFoundException; import me.desair.tus.server.upload.concatenation.UploadConcatenationService; @@ -76,10 +77,6 @@ public InputStream getUploadedBytes(UploadId id) public void copyUploadTo(UploadInfo info, OutputStream outputStream) throws UploadNotFoundException, IOException {} - @Override - public void cleanupExpiredUploads(UploadLockingService uploadLockingService) - throws IOException {} - @Override public void removeLastNumberOfBytes(UploadInfo uploadInfo, long byteCount) throws UploadNotFoundException, IOException {} @@ -152,4 +149,38 @@ public void testDefaultAppendAndSizeConfigurationMethods() throws Exception { // Default close is a no-op dummyStorageService.close(); } + + @Test + public void testDefaultDeduplicateAndLockingService() throws Exception { + UploadInfo info = new UploadInfo(); + ChecksumKey key = + ChecksumKey.of("0123456789abcdef0123456789abcdef01234567", ChecksumAlgorithm.SHA1); + assertThat(dummyStorageService.deduplicate(info, key), is(info)); + dummyStorageService.setUploadLockingService(new NoOpUploadLockingService()); + assertThat(dummyStorageService.getUploadLockingService(), is(nullValue())); + } + + @Test(expected = NullPointerException.class) + public void testSetUploadLockingServiceThrowsOnNull() { + dummyStorageService.setUploadLockingService(null); + } + + @Test + public void testDefaultDataExistsAndDeleteData() throws Exception { + UploadId id = new UploadId("test-id"); + assertThat(dummyStorageService.dataExists(id), is(false)); + + // KISS: verifying method executes cleanly without throwing an exception + dummyStorageService.deleteData(id); + } + + @Test + public void testDefaultCleanupExpiredUploads() throws Exception { + // KISS: verifying method executes cleanly without throwing an exception + dummyStorageService.cleanupExpiredUploads(); + dummyStorageService.cleanupExpiredUploads(null); + + UploadLockingService mockLocking = org.mockito.Mockito.mock(UploadLockingService.class); + dummyStorageService.cleanupExpiredUploads(mockLocking); + } } diff --git a/src/test/java/me/desair/tus/server/upload/azure/AzureBlobChecksumIndexTest.java b/src/test/java/me/desair/tus/server/upload/azure/AzureBlobChecksumIndexTest.java new file mode 100644 index 00000000..01c87d2f --- /dev/null +++ b/src/test/java/me/desair/tus/server/upload/azure/AzureBlobChecksumIndexTest.java @@ -0,0 +1,114 @@ +package me.desair.tus.server.upload.azure; + +import static org.junit.Assert.assertNotNull; + +import com.azure.storage.blob.BlobContainerClient; +import com.azure.storage.blob.BlobContainerClientBuilder; +import java.io.IOException; +import me.desair.tus.server.checksum.ChecksumAlgorithm; +import me.desair.tus.server.checksum.ChecksumKey; +import me.desair.tus.server.upload.UploadId; +import org.junit.Before; +import org.junit.Test; + +/** + * Offline unit tests for {@link AzureBlobChecksumIndex} verifying parameter validation, prefix + * normalization, and defensive guard clauses. + */ +public class AzureBlobChecksumIndexTest { + + private BlobContainerClient containerClient; + private AzureBlobChecksumIndex checksumIndex; + private ChecksumKey key; + private UploadId uploadId; + + @Before + public void setUp() { + containerClient = + new BlobContainerClientBuilder() + .endpoint("https://dummyaccount.blob.core.windows.net") + .containerName("dummy-container") + .buildClient(); + checksumIndex = new AzureBlobChecksumIndex(containerClient); + key = ChecksumKey.of("0123456789abcdef", ChecksumAlgorithm.SHA256); + uploadId = new UploadId("test-parent-id"); + } + + @Test(expected = NullPointerException.class) + public void constructorShouldThrowOnNullContainerClient() { + new AzureBlobChecksumIndex(null); + } + + @Test + public void testConstructorAndSanitizePrefix() { + AzureBlobChecksumIndex index1 = new AzureBlobChecksumIndex(containerClient); + assertNotNull(index1); + + AzureBlobChecksumIndex index2 = new AzureBlobChecksumIndex(containerClient, "custom/"); + assertNotNull(index2); + + AzureBlobChecksumIndex index3 = new AzureBlobChecksumIndex(containerClient, "/custom///"); + assertNotNull(index3); + + AzureBlobChecksumIndex index4 = new AzureBlobChecksumIndex(containerClient, ""); + assertNotNull(index4); + + AzureBlobChecksumIndex index5 = new AzureBlobChecksumIndex(containerClient, null); + assertNotNull(index5); + } + + @Test(expected = NullPointerException.class) + public void tryRegisterParentNullKeyThrows() throws IOException { + checksumIndex.tryRegisterParent(null, uploadId); + } + + @Test(expected = NullPointerException.class) + public void tryRegisterParentNullUploadIdThrows() throws IOException { + checksumIndex.tryRegisterParent(key, null); + } + + @Test(expected = NullPointerException.class) + public void findParentNullKeyThrows() throws IOException { + checksumIndex.findParent(null); + } + + @Test(expected = NullPointerException.class) + public void markParentDeletedNullKeyThrows() throws IOException { + checksumIndex.markParentDeleted(null); + } + + @Test(expected = NullPointerException.class) + public void isParentDeletedNullKeyThrows() throws IOException { + checksumIndex.isParentDeleted(null); + } + + @Test(expected = NullPointerException.class) + public void addChildNullKeyThrows() throws IOException { + checksumIndex.addChild(null, uploadId); + } + + @Test(expected = NullPointerException.class) + public void addChildNullUploadIdThrows() throws IOException { + checksumIndex.addChild(key, null); + } + + @Test(expected = NullPointerException.class) + public void removeChildNullKeyThrows() throws IOException { + checksumIndex.removeChild(null, uploadId); + } + + @Test(expected = NullPointerException.class) + public void removeChildNullUploadIdThrows() throws IOException { + checksumIndex.removeChild(key, null); + } + + @Test(expected = NullPointerException.class) + public void hasChildrenNullKeyThrows() throws IOException { + checksumIndex.hasChildren(null); + } + + @Test(expected = NullPointerException.class) + public void deleteIndexNullKeyThrows() throws IOException { + checksumIndex.deleteIndex(null); + } +} diff --git a/src/test/java/me/desair/tus/server/upload/azure/AzureBlobStorageServiceTest.java b/src/test/java/me/desair/tus/server/upload/azure/AzureBlobStorageServiceTest.java index 27dfdb53..b16bfd12 100644 --- a/src/test/java/me/desair/tus/server/upload/azure/AzureBlobStorageServiceTest.java +++ b/src/test/java/me/desair/tus/server/upload/azure/AzureBlobStorageServiceTest.java @@ -6,6 +6,7 @@ import static org.junit.Assert.assertNull; import static org.junit.Assert.assertTrue; +import com.azure.storage.blob.BlobContainerClient; import com.azure.storage.blob.BlobContainerClientBuilder; import java.io.ByteArrayInputStream; import java.io.ByteArrayOutputStream; @@ -28,7 +29,7 @@ */ public class AzureBlobStorageServiceTest { - private com.azure.storage.blob.BlobContainerClient containerClient; + private BlobContainerClient containerClient; private AzureBlobStorageService storageService; @Before @@ -369,4 +370,14 @@ public void testDeleteFileQuietly() throws Exception { Files.deleteIfExists(tempDir); } } + + @Test(expected = NullPointerException.class) + public void testSetUploadLockingServiceNullThrowsNpe() { + storageService.setUploadLockingService(null); + } + + @Test + public void testGetUploadLockingServiceReturnsConfiguredInstance() { + assertNotNull(storageService.getUploadLockingService()); + } } diff --git a/src/test/java/me/desair/tus/server/upload/azure/ITAzureBlobRufhProtocol.java b/src/test/java/me/desair/tus/server/upload/azure/ITAzureBlobRufhProtocol.java index bbf24407..c0f30932 100644 --- a/src/test/java/me/desair/tus/server/upload/azure/ITAzureBlobRufhProtocol.java +++ b/src/test/java/me/desair/tus/server/upload/azure/ITAzureBlobRufhProtocol.java @@ -4,6 +4,7 @@ import me.desair.tus.server.AbstractITRufhProtocol; import me.desair.tus.server.TestUtils; import me.desair.tus.server.TusFileUploadService; +import me.desair.tus.server.upload.UploadId; import org.junit.AfterClass; import org.junit.BeforeClass; import org.testcontainers.containers.GenericContainer; @@ -61,4 +62,9 @@ protected TusFileUploadService createTusFileUploadService(String uploadUri) { .withUploadExpirationPeriod(2L * 24 * 60 * 60 * 1000) .withDownloadFeature(); } + + @Override + protected void deleteUploadedBytesOutOfBand(UploadId uploadId) throws Exception { + containerClient.getBlobClient("uploads/" + uploadId).deleteIfExists(); + } } diff --git a/src/test/java/me/desair/tus/server/upload/azure/ITAzureBlobStorageService.java b/src/test/java/me/desair/tus/server/upload/azure/ITAzureBlobStorageService.java index 4a99704e..5d9ae13b 100644 --- a/src/test/java/me/desair/tus/server/upload/azure/ITAzureBlobStorageService.java +++ b/src/test/java/me/desair/tus/server/upload/azure/ITAzureBlobStorageService.java @@ -4,18 +4,22 @@ import static org.junit.Assert.assertFalse; import static org.junit.Assert.assertNotNull; import static org.junit.Assert.assertNull; +import static org.junit.Assert.assertThrows; import static org.junit.Assert.assertTrue; import com.azure.core.util.BinaryData; import com.azure.storage.blob.BlobClient; import com.azure.storage.blob.BlobContainerClient; +import com.azure.storage.blob.models.ListBlobsOptions; import java.io.ByteArrayInputStream; import java.io.ByteArrayOutputStream; import java.io.IOException; import java.io.InputStream; import java.nio.charset.StandardCharsets; +import java.util.Set; import me.desair.tus.server.TestUtils; import me.desair.tus.server.checksum.ChecksumAlgorithm; +import me.desair.tus.server.checksum.ChecksumKey; import me.desair.tus.server.exception.MaxAppendSizeExceededException; import me.desair.tus.server.exception.MinAppendSizeNotMetException; import me.desair.tus.server.exception.UploadNotFoundException; @@ -484,26 +488,140 @@ public void testDeduplicationOnAzurite() throws Exception { AzureBlobStorageService storage = new AzureBlobStorageService(containerClient); storage.setUploadDeduplicationEnabled(true); + String md5Hex = "5d41402abc4b2a76b9719d911017c592"; + String parentBlobName = "checksums/md5/" + md5Hex + "/parent"; + String parentDeletedBlobName = "checksums/md5/" + md5Hex + "/parent-deleted"; + + // 1. Parent upload UploadInfo parent = new UploadInfo(); parent.setLength(5L); - parent.setChecksum("5d41402abc4b2a76b9719d911017c592"); + parent.setChecksum(md5Hex); parent.setChecksumAlgorithm(ChecksumAlgorithm.MD5); parent = storage.create(parent, "owner1"); storage.append(parent, new ByteArrayInputStream("hello".getBytes())); + // Verify parent pointer blob exists in Azure container + assertTrue(containerClient.getBlobClient(parentBlobName).exists()); + + // 2. Child duplicate upload UploadInfo child = new UploadInfo(); child.setLength(5L); - child.setChecksum("5d41402abc4b2a76b9719d911017c592"); + child.setChecksum(md5Hex); child.setChecksumAlgorithm(ChecksumAlgorithm.MD5); child = storage.create(child, "owner1"); - storage.append(child, new ByteArrayInputStream("hello".getBytes())); + child = storage.append(child, new ByteArrayInputStream("hello".getBytes())); assertNotNull(child.getDuplicatesUploadId()); assertEquals(parent.getId(), child.getDuplicatesUploadId()); + String childBlobName = "checksums/md5/" + md5Hex + "/children/" + child.getId(); + assertTrue(containerClient.getBlobClient(childBlobName).exists()); + + // 3. Terminate parent upload (child still alive) storage.terminateUpload(parent); - assertNull( - storage.getUploadInfoByChecksum("5d41402abc4b2a76b9719d911017c592", ChecksumAlgorithm.MD5)); + assertTrue(containerClient.getBlobClient(parentDeletedBlobName).exists()); + assertNull(storage.getUploadInfoByChecksum(md5Hex, ChecksumAlgorithm.MD5)); + + // 4. Terminate child upload (last child out) + storage.terminateUpload(child); + + // Verify all blobs under checksum prefix are completely deleted from Azure + assertFalse(containerClient.getBlobClient(parentBlobName).exists()); + assertFalse(containerClient.getBlobClient(parentDeletedBlobName).exists()); + assertFalse( + containerClient + .listBlobs(new ListBlobsOptions().setPrefix("checksums/md5/" + md5Hex), null) + .iterator() + .hasNext()); + } + + @Test + public void testAzureBlobChecksumIndexDirectly() throws Exception { + AzureBlobChecksumIndex index = new AzureBlobChecksumIndex(containerClient, "test-checksums"); + ChecksumKey key = + ChecksumKey.of("1122334455667788990011223344556677889900", ChecksumAlgorithm.SHA1); + UploadId parentId = new UploadId("parent-upload-id"); + UploadId competingId = new UploadId("competing-upload-id"); + + // 1. Initial lookup returns null + assertNull(index.findParent(key)); + assertFalse(index.isParentDeleted(key)); + assertFalse(index.hasChildren(key)); + + // 2. First registration succeeds + assertTrue(index.tryRegisterParent(key, parentId)); + assertEquals(parentId, index.findParent(key)); + + // 3. Second registration fails due to CAS (precondition failed) + assertFalse(index.tryRegisterParent(key, competingId)); + assertEquals(parentId, index.findParent(key)); + + // 4. Add and remove child + UploadId childId = new UploadId("child-upload-id"); + index.addChild(key, childId); + assertTrue(index.hasChildren(key)); + index.removeChild(key, childId); + assertFalse(index.hasChildren(key)); + + // 5. Mark parent deleted (preserves parent file for active children and new joiners) + index.markParentDeleted(key); + assertTrue(index.isParentDeleted(key)); + assertEquals(parentId, index.findParent(key)); + + // 6. Registration after tombstone returns false + assertFalse(index.tryRegisterParent(key, competingId)); + + // 7. Empty parent blob returns null + ChecksumKey emptyKey = + ChecksumKey.of("2233445566778899001122334455667788990011", ChecksumAlgorithm.SHA1); + containerClient + .getBlobClient("test-checksums/sha1/2233445566778899001122334455667788990011/parent") + .upload(BinaryData.fromBytes(new byte[0]), true); + assertNull(index.findParent(emptyKey)); + + // Upload malformed and non-parent blobs to exercise all branches in getAllIndexedKeys + String blobTooManyParts = "test-checksums/too/many/parts/here/parent"; + String blobChild = "test-checksums/sha1/1122334455667788990011223344556677889900/child"; + String blobUnknownAlgo = + "test-checksums/unknownalgo/1122334455667788990011223344556677889900/parent"; + String blobMalformedHex = "test-checksums/sha1/invalid_hex!/parent"; + + containerClient.getBlobClient(blobTooManyParts).upload(BinaryData.fromString("test"), true); + containerClient.getBlobClient(blobChild).upload(BinaryData.fromString("test"), true); + containerClient.getBlobClient(blobUnknownAlgo).upload(BinaryData.fromString("test"), true); + containerClient.getBlobClient(blobMalformedHex).upload(BinaryData.fromString("test"), true); + + // Verify getAllIndexedKeys returns registered keys and skips malformed entries + Set allKeys = index.getAllIndexedKeys(); + assertTrue(allKeys.contains(key)); + assertTrue(allKeys.contains(emptyKey)); + + // Clean up malformed test blobs + containerClient.getBlobClient(blobTooManyParts).deleteIfExists(); + containerClient.getBlobClient(blobChild).deleteIfExists(); + containerClient.getBlobClient(blobUnknownAlgo).deleteIfExists(); + containerClient.getBlobClient(blobMalformedHex).deleteIfExists(); + + // 8. Delete entire index + index.deleteIndex(key); + assertFalse(index.isParentDeleted(key)); + index.deleteIndex(emptyKey); + assertFalse(index.getAllIndexedKeys().contains(key)); + assertFalse(index.getAllIndexedKeys().contains(emptyKey)); + + // 9. Error handling on non-existent container + BlobContainerClient nonExistentClient = + containerClient + .getServiceClient() + .getBlobContainerClient("non-existent-" + System.nanoTime()); + AzureBlobChecksumIndex badIndex = new AzureBlobChecksumIndex(nonExistentClient); + + assertThrows(IOException.class, () -> badIndex.tryRegisterParent(key, parentId)); + assertNull(badIndex.findParent(key)); + assertThrows(IOException.class, () -> badIndex.hasChildren(key)); + assertThrows(IOException.class, () -> badIndex.deleteIndex(key)); + assertThrows(IOException.class, () -> badIndex.getAllIndexedKeys()); + assertFalse(badIndex.isParentDeleted(key)); } @Test @@ -717,52 +835,6 @@ public void testAppendInterruptedMultiBlock() throws Exception { assertEquals(Long.valueOf(4L * 1024 * 1024 + 100L), fetched.getOffset()); } - @Test - public void testPruneOrphanedChecksumIndices() throws Exception { - storageService.setUploadDeduplicationEnabled(true); - - BlobClient orphanedIndex = containerClient.getBlobClient("checksums/sha1/deadbeef12345678"); - orphanedIndex.upload(BinaryData.fromString("non-existent-upload-id"), true); - assertTrue(orphanedIndex.exists()); - - storageService.cleanupExpiredUploads(); - - assertFalse(orphanedIndex.exists()); - } - - @Test - public void testPruneOrphanedChecksumIndicesMissingDataBlob() throws Exception { - storageService.setUploadDeduplicationEnabled(true); - - UploadInfo parentInfo = new UploadInfo(); - parentInfo.setLength(10L); - UploadInfo created = storageService.create(parentInfo, "owner1"); - - BlobClient orphanedIndex = containerClient.getBlobClient("checksums/sha1/missingdatablob"); - orphanedIndex.upload(BinaryData.fromString(created.getId().toString()), true); - assertTrue(orphanedIndex.exists()); - - // Data blob doesn't exist yet (upload created but no bytes uploaded) - storageService.cleanupExpiredUploads(); - - assertFalse(orphanedIndex.exists()); - } - - @Test - public void testGetUploadInfoByChecksumStaleIndexDeleted() throws Exception { - storageService.setUploadDeduplicationEnabled(true); - - BlobClient staleIndex = containerClient.getBlobClient("checksums/sha1/nonexistentparent"); - staleIndex.upload(BinaryData.fromString("missing-parent-id"), true); - assertTrue(staleIndex.exists()); - - // Looking up non-existent parent should self-clean the stale index - UploadInfo match = - storageService.getUploadInfoByChecksum("nonexistentparent", ChecksumAlgorithm.SHA1); - assertNull(match); - assertFalse(staleIndex.exists()); - } - @Test public void testRemoveLastNumberOfBytesDownToZero() throws Exception { UploadInfo info = new UploadInfo(); @@ -950,4 +1022,44 @@ public void testTerminateUploadWithUnleasedLockBlob() throws Exception { assertFalse(Boolean.TRUE.equals(lockBlob.exists())); } + + @Test + public void testPruneOrphanedChecksumIndicesMissingDataBlob() throws Exception { + storageService.setUploadDeduplicationEnabled(true); + ChecksumKey key = + ChecksumKey.of("0123456789abcdef0123456789abcdef01234567", ChecksumAlgorithm.SHA1); + UploadId dummyId = new UploadId("orphan-id"); + + // Create an orphaned index pointing to a non-existent data blob + BlobClient parentBlob = containerClient.getBlobClient("checksums/" + key + "/parent"); + parentBlob.upload(BinaryData.fromString(dummyId.toString()), true); + assertTrue(Boolean.TRUE.equals(parentBlob.exists())); + + storageService.cleanupExpiredUploads(); + + assertFalse(Boolean.TRUE.equals(parentBlob.exists())); + } + + @Test + public void testPruneOrphanedChecksumIndicesAbandonedTombstone() throws Exception { + storageService.setUploadDeduplicationEnabled(true); + ChecksumKey key = + ChecksumKey.of("fedcba9876543210fedcba9876543210fedcba98", ChecksumAlgorithm.SHA1); + UploadId dummyId = new UploadId("tombstone-orphan-id"); + + // Create parent and tombstone marker with zero children + BlobClient parentBlob = containerClient.getBlobClient("checksums/" + key + "/parent"); + parentBlob.upload(BinaryData.fromString(dummyId.toString()), true); + BlobClient tombstoneBlob = + containerClient.getBlobClient("checksums/" + key + "/parent-deleted"); + tombstoneBlob.upload(BinaryData.fromString(""), true); + + assertTrue(Boolean.TRUE.equals(parentBlob.exists())); + assertTrue(Boolean.TRUE.equals(tombstoneBlob.exists())); + + storageService.cleanupExpiredUploads(); + + assertFalse(Boolean.TRUE.equals(parentBlob.exists())); + assertFalse(Boolean.TRUE.equals(tombstoneBlob.exists())); + } } diff --git a/src/test/java/me/desair/tus/server/upload/azure/ITAzureBlobTusFileUploadService.java b/src/test/java/me/desair/tus/server/upload/azure/ITAzureBlobTusFileUploadService.java index 7be2e903..a3ca084c 100644 --- a/src/test/java/me/desair/tus/server/upload/azure/ITAzureBlobTusFileUploadService.java +++ b/src/test/java/me/desair/tus/server/upload/azure/ITAzureBlobTusFileUploadService.java @@ -2,14 +2,19 @@ import static org.junit.Assert.assertEquals; import static org.junit.Assert.assertFalse; +import static org.junit.Assert.assertTrue; import com.azure.storage.blob.BlobContainerClient; +import com.azure.storage.blob.models.ListBlobsOptions; import jakarta.servlet.http.HttpServletResponse; +import java.nio.charset.StandardCharsets; import me.desair.tus.server.AbstractITTusFileUploadService; import me.desair.tus.server.HttpHeader; import me.desair.tus.server.ProtocolVersion; import me.desair.tus.server.TestUtils; import me.desair.tus.server.TusFileUploadService; +import me.desair.tus.server.upload.UploadId; +import me.desair.tus.server.upload.UploadInfo; import org.apache.commons.lang3.StringUtils; import org.junit.AfterClass; import org.junit.BeforeClass; @@ -169,4 +174,106 @@ public void testDeleteTerminationCleansUpLockBlob() throws Exception { "Stop signal blob should not exist", containerClient.getBlobClient("locks/" + uploadIdStr + ".stop").exists()); } + + /** + * Azure Integration Test: Verifies that during the end-to-end Tus protocol lifecycle, the + * physical checksum index blobs in Azure Blob storage are created and properly purged on final + * deletion. + */ + @Test + public void testPhysicalChecksumDirectoryCreationAndCleanupInAzureBlob() throws Exception { + tusFileUploadService.withUploadDeduplication(true); + + String content = "Physical Azure Checksum Verification Payload"; + byte[] contentBytes = content.getBytes(StandardCharsets.UTF_8); + byte[] sha1Digest = org.apache.commons.codec.digest.DigestUtils.sha1(contentBytes); + String sha1Hex = org.apache.commons.codec.binary.Hex.encodeHexString(sha1Digest); + String sha1Base64 = org.apache.commons.codec.binary.Base64.encodeBase64String(sha1Digest); + + String parentBlobName = "checksums/sha1/" + sha1Hex + "/parent"; + String parentDeletedBlobName = "checksums/sha1/" + sha1Hex + "/parent-deleted"; + + // 1. Create parent upload + reset(); + servletRequest.setMethod("POST"); + servletRequest.setRequestURI(UPLOAD_URI); + servletRequest.addHeader(HttpHeader.UPLOAD_LENGTH, contentBytes.length); + servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + assertEquals(HttpServletResponse.SC_CREATED, servletResponse.getStatus()); + String parentUri = servletResponse.getHeader(HttpHeader.LOCATION); + + // Complete parent upload via PATCH with checksum + reset(); + servletRequest.setMethod("PATCH"); + servletRequest.setRequestURI(parentUri); + servletRequest.addHeader(HttpHeader.CONTENT_TYPE, "application/offset+octet-stream"); + servletRequest.addHeader(HttpHeader.UPLOAD_OFFSET, 0); + servletRequest.addHeader(HttpHeader.UPLOAD_CHECKSUM, "sha1 " + sha1Base64); + servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); + servletRequest.setContent(contentBytes); + UploadInfo parentInfo = + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + assertEquals(HttpServletResponse.SC_NO_CONTENT, servletResponse.getStatus()); + + // Assert physical parent pointer exists in Azure container + assertTrue(containerClient.getBlobClient(parentBlobName).exists()); + + // 2. Create duplicate child upload + reset(); + servletRequest.setMethod("POST"); + servletRequest.setRequestURI(UPLOAD_URI); + servletRequest.addHeader(HttpHeader.UPLOAD_LENGTH, contentBytes.length); + servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + assertEquals(HttpServletResponse.SC_CREATED, servletResponse.getStatus()); + String childUri = servletResponse.getHeader(HttpHeader.LOCATION); + + // Complete child upload via PATCH with same checksum + reset(); + servletRequest.setMethod("PATCH"); + servletRequest.setRequestURI(childUri); + servletRequest.addHeader(HttpHeader.CONTENT_TYPE, "application/offset+octet-stream"); + servletRequest.addHeader(HttpHeader.UPLOAD_OFFSET, 0); + servletRequest.addHeader(HttpHeader.UPLOAD_CHECKSUM, "sha1 " + sha1Base64); + servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); + servletRequest.setContent(contentBytes); + UploadInfo childInfo = tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + assertEquals(HttpServletResponse.SC_NO_CONTENT, servletResponse.getStatus()); + assertEquals(parentInfo.getId(), childInfo.getDuplicatesUploadId()); + + String childBlobName = "checksums/sha1/" + sha1Hex + "/children/" + childInfo.getId(); + assertTrue(containerClient.getBlobClient(childBlobName).exists()); + + // 3. Delete parent upload via DELETE + reset(); + servletRequest.setMethod("DELETE"); + servletRequest.setRequestURI(parentUri); + servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + assertEquals(HttpServletResponse.SC_NO_CONTENT, servletResponse.getStatus()); + + assertTrue(containerClient.getBlobClient(parentDeletedBlobName).exists()); + + // 4. Delete child upload via DELETE (last child out) + reset(); + servletRequest.setMethod("DELETE"); + servletRequest.setRequestURI(childUri); + servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + assertEquals(HttpServletResponse.SC_NO_CONTENT, servletResponse.getStatus()); + + assertFalse(containerClient.getBlobClient(parentBlobName).exists()); + assertFalse(containerClient.getBlobClient(parentDeletedBlobName).exists()); + assertFalse( + containerClient + .listBlobs(new ListBlobsOptions().setPrefix("checksums/sha1/" + sha1Hex), null) + .iterator() + .hasNext()); + } + + @Override + protected void deleteUploadedBytesOutOfBand(UploadId uploadId) throws Exception { + containerClient.getBlobClient("uploads/" + uploadId).deleteIfExists(); + } } diff --git a/src/test/java/me/desair/tus/server/upload/cache/ThreadLocalCachedStorageAndLockingServiceTest.java b/src/test/java/me/desair/tus/server/upload/cache/ThreadLocalCachedStorageAndLockingServiceTest.java index fc5a99d4..a1e7edf1 100644 --- a/src/test/java/me/desair/tus/server/upload/cache/ThreadLocalCachedStorageAndLockingServiceTest.java +++ b/src/test/java/me/desair/tus/server/upload/cache/ThreadLocalCachedStorageAndLockingServiceTest.java @@ -205,6 +205,9 @@ public void testDelegateMethods() throws Exception { service.copyUploadTo(info, mock(OutputStream.class)); verify(mockStorage, times(1)).copyUploadTo(any(), any()); + service.cleanupExpiredUploads(); + verify(mockStorage, times(1)).cleanupExpiredUploads(); + service.cleanupExpiredUploads(mockLocking); verify(mockStorage, times(1)).cleanupExpiredUploads(mockLocking); @@ -258,6 +261,21 @@ public void testDelegateMethods() throws Exception { service.cleanupLock("/files/1"); verify(mockLocking, times(1)).cleanupLock("/files/1"); + service.getUploadLockingService(); + verify(mockStorage, times(1)).getUploadLockingService(); + + service.dataExists(id); + verify(mockStorage, times(1)).dataExists(id); + + service.deleteData(id); + verify(mockStorage, times(1)).deleteData(id); + + service.lockUpload(id); + verify(mockLocking, times(1)).lockUpload(id); + + service.lockUploadByUri("/files/1"); + verify(mockLocking, times(1)).lockUploadByUri("/files/1"); + service.close(); verify(mockLocking, times(1)).close(); } diff --git a/src/test/java/me/desair/tus/server/upload/checksum/DeduplicationServiceTest.java b/src/test/java/me/desair/tus/server/upload/checksum/DeduplicationServiceTest.java new file mode 100644 index 00000000..ba4618f3 --- /dev/null +++ b/src/test/java/me/desair/tus/server/upload/checksum/DeduplicationServiceTest.java @@ -0,0 +1,1060 @@ +package me.desair.tus.server.upload.checksum; + +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertFalse; +import static org.junit.Assert.assertNotNull; +import static org.junit.Assert.assertNull; +import static org.junit.Assert.assertTrue; +import static org.mockito.ArgumentMatchers.any; +import static org.mockito.Mockito.mock; +import static org.mockito.Mockito.never; +import static org.mockito.Mockito.verify; +import static org.mockito.Mockito.when; + +import java.io.IOException; +import java.io.InputStream; +import java.io.OutputStream; +import java.util.HashMap; +import java.util.HashSet; +import java.util.Map; +import java.util.Set; +import me.desair.tus.server.checksum.ChecksumAlgorithm; +import me.desair.tus.server.checksum.ChecksumKey; +import me.desair.tus.server.exception.TusException; +import me.desair.tus.server.exception.UploadAlreadyLockedException; +import me.desair.tus.server.exception.UploadNotFoundException; +import me.desair.tus.server.upload.UploadId; +import me.desair.tus.server.upload.UploadIdFactory; +import me.desair.tus.server.upload.UploadInfo; +import me.desair.tus.server.upload.UploadLock; +import me.desair.tus.server.upload.UploadLockingService; +import me.desair.tus.server.upload.UploadStorageService; +import me.desair.tus.server.upload.concatenation.UploadConcatenationService; +import org.junit.Before; +import org.junit.Test; + +/** Unit tests for {@link DeduplicationService}. */ +public class DeduplicationServiceTest { + + private TestChecksumIndex index; + private TestStorageService adapter; + private UploadLockingService lockingService; + private DeduplicationService deduplicationService; + + private final ChecksumKey checksumKey = + ChecksumKey.of("0123456789abcdef0123456789abcdef01234567", ChecksumAlgorithm.SHA1); + + @Before + public void setUp() { + index = new TestChecksumIndex(); + adapter = new TestStorageService(); + lockingService = mock(UploadLockingService.class); + deduplicationService = new DeduplicationService(index, adapter); + deduplicationService.setUploadLockingService(lockingService); + adapter.setDeduplicationService(deduplicationService); + } + + @Test + public void testDeduplicateFirstUploadBecomesParent() throws Exception { + UploadId id1 = new UploadId("upload-1"); + UploadInfo info1 = createUploadInfo(id1, checksumKey); + adapter.put(id1, info1, true); + + UploadInfo result = deduplicationService.deduplicate(info1, checksumKey); + + assertEquals(id1, result.getId()); + assertNull(result.getDuplicatesUploadId()); + assertEquals(id1, index.findParent(checksumKey)); + assertFalse(index.hasChildren(checksumKey)); + } + + @Test + public void testDeduplicateSecondUploadJoinsParent() throws Exception { + UploadId parentId = new UploadId("parent-upload"); + UploadInfo parentInfo = createUploadInfo(parentId, checksumKey); + adapter.put(parentId, parentInfo, true); + index.tryRegisterParent(checksumKey, parentId); + + UploadLock mockLock = mock(UploadLock.class); + when(lockingService.lockUpload(parentId)).thenReturn(mockLock); + + UploadId childId = new UploadId("child-upload"); + UploadInfo childInfo = createUploadInfo(childId, checksumKey); + adapter.put(childId, childInfo, true); + + UploadInfo result = deduplicationService.deduplicate(childInfo, checksumKey); + + assertEquals(childId, result.getId()); + assertEquals(parentId, result.getDuplicatesUploadId()); + assertTrue(index.hasChildren(checksumKey)); + assertFalse(adapter.dataExists(childId)); // Child data deleted + assertTrue(adapter.dataExists(parentId)); // Parent data intact + verify(mockLock).close(); + } + + @Test + public void testDeduplicateSkipsWhenParentLocked() throws Exception { + UploadId parentId = new UploadId("parent-upload"); + UploadInfo parentInfo = createUploadInfo(parentId, checksumKey); + adapter.put(parentId, parentInfo, true); + index.tryRegisterParent(checksumKey, parentId); + + when(lockingService.lockUpload(parentId)) + .thenThrow(new UploadAlreadyLockedException("Parent locked")); + + UploadId childId = new UploadId("child-upload"); + UploadInfo childInfo = createUploadInfo(childId, checksumKey); + adapter.put(childId, childInfo, true); + + UploadInfo result = deduplicationService.deduplicate(childInfo, checksumKey); + + // Should skip deduplication and keep own data + assertNull(result.getDuplicatesUploadId()); + assertTrue(adapter.dataExists(childId)); + assertFalse(index.hasChildren(checksumKey)); + } + + @Test + public void testDeduplicatePurgesIndexWhenParentBytesMissing() throws Exception { + UploadId parentId = new UploadId("parent-upload"); + UploadInfo parentInfo = createUploadInfo(parentId, checksumKey); + adapter.put(parentId, parentInfo, false); // No data bytes! + index.tryRegisterParent(checksumKey, parentId); + + UploadLock mockLock = mock(UploadLock.class); + when(lockingService.lockUpload(parentId)).thenReturn(mockLock); + + UploadId childId = new UploadId("child-upload"); + UploadInfo childInfo = createUploadInfo(childId, checksumKey); + adapter.put(childId, childInfo, true); + + UploadInfo result = deduplicationService.deduplicate(childInfo, checksumKey); + + // Self-healing: stale index purged and child upload promoted to new parent + assertEquals(childId, index.findParent(checksumKey)); + assertNull(result.getDuplicatesUploadId()); + assertTrue(adapter.dataExists(childId)); + } + + @Test + public void testTerminateParentWithNoChildrenPurgesAll() throws Exception { + UploadId parentId = new UploadId("parent-upload"); + UploadInfo parentInfo = createUploadInfo(parentId, checksumKey); + adapter.put(parentId, parentInfo, true); + index.tryRegisterParent(checksumKey, parentId); + + assertTrue(deduplicationService.removeUploadFromIndex(parentInfo)); + adapter.terminateUpload(parentInfo); + + assertFalse(adapter.dataExists(parentId)); + assertNull(adapter.getUploadInfo(parentId)); + assertNull(index.findParent(checksumKey)); + } + + @Test + public void testTerminateParentWithChildrenTombstonesParent() throws Exception { + UploadId parentId = new UploadId("parent-upload"); + UploadInfo parentInfo = createUploadInfo(parentId, checksumKey); + adapter.put(parentId, parentInfo, true); + index.tryRegisterParent(checksumKey, parentId); + + UploadId childId = new UploadId("child-upload"); + index.addChild(checksumKey, childId); + + assertFalse(deduplicationService.removeUploadFromIndex(parentInfo)); + adapter.terminateUpload(parentInfo); + + // Parent data bytes MUST remain for child! Metadata deleted, tombstone created, parent pointer + // retained. + assertTrue(adapter.dataExists(parentId)); + assertNull(adapter.getUploadInfo(parentId)); + assertTrue(index.isParentDeleted(checksumKey)); + assertEquals(parentId, index.findParent(checksumKey)); + } + + @Test + public void testTerminateChildDeletesChildMetadataOnlyWhenOtherChildrenRemain() throws Exception { + UploadId parentId = new UploadId("parent-upload"); + UploadInfo parentInfo = createUploadInfo(parentId, checksumKey); + adapter.put(parentId, parentInfo, true); + index.tryRegisterParent(checksumKey, parentId); + index.markParentDeleted(checksumKey); // Parent already deleted + + UploadId child1 = new UploadId("child-1"); + UploadInfo child1Info = createUploadInfo(child1, checksumKey); + child1Info.setDuplicatesUploadId(parentId); + adapter.put(child1, child1Info, false); + index.addChild(checksumKey, child1); + + UploadId child2 = new UploadId("child-2"); + index.addChild(checksumKey, child2); + + assertTrue(deduplicationService.removeUploadFromIndex(child1Info)); + adapter.terminateUpload(child1Info); + + // child 1 metadata removed + assertNull(adapter.getUploadInfo(child1)); + // parent data bytes still preserved for child 2 + assertTrue(adapter.dataExists(parentId)); + assertTrue(index.hasChildren(checksumKey)); + } + + @Test + public void testTerminateLastChildAfterParentDeletedPurgesParentData() throws Exception { + UploadId parentId = new UploadId("parent-upload"); + UploadInfo parentInfo = createUploadInfo(parentId, checksumKey); + adapter.put(parentId, parentInfo, true); + index.tryRegisterParent(checksumKey, parentId); + index.markParentDeleted(checksumKey); + + UploadId lastChild = new UploadId("last-child"); + UploadInfo childInfo = createUploadInfo(lastChild, checksumKey); + childInfo.setDuplicatesUploadId(parentId); + adapter.put(lastChild, childInfo, false); + index.addChild(checksumKey, lastChild); + + UploadLock mockParentLock = mock(UploadLock.class); + when(lockingService.lockUpload(parentId)).thenReturn(mockParentLock); + + assertTrue(deduplicationService.removeUploadFromIndex(childInfo)); + adapter.terminateUpload(childInfo); + + // Child metadata deleted + assertNull(adapter.getUploadInfo(lastChild)); + // Parent data purged! + assertFalse(adapter.dataExists(parentId)); + assertFalse(index.hasChildren(checksumKey)); + assertFalse(index.isParentDeleted(checksumKey)); + verify(mockParentLock).close(); + } + + @Test + public void testTerminateChildWhenParentIsAliveDoesNotLockParent() throws Exception { + UploadId parentId = new UploadId("parent-upload"); + UploadInfo parentInfo = createUploadInfo(parentId, checksumKey); + adapter.put(parentId, parentInfo, true); + index.tryRegisterParent(checksumKey, parentId); + + UploadId childId = new UploadId("child-upload"); + UploadInfo childInfo = createUploadInfo(childId, checksumKey); + childInfo.setDuplicatesUploadId(parentId); + adapter.put(childId, childInfo, false); + index.addChild(checksumKey, childId); + + assertTrue(deduplicationService.removeUploadFromIndex(childInfo)); + adapter.terminateUpload(childInfo); + + // Verify parent lock is NEVER touched when parent is alive + verify(lockingService, never()).lockUpload(parentId); + verify(lockingService, never()).lockUpload(any()); + assertNull(adapter.getUploadInfo(childId)); + assertTrue(adapter.dataExists(parentId)); + assertFalse(index.hasChildren(checksumKey)); + } + + @Test + public void testTerminateChildWithNullChecksumResolvesParentChecksum() throws Exception { + UploadId parentId = new UploadId("parent-upload"); + UploadInfo parentInfo = createUploadInfo(parentId, checksumKey); + adapter.put(parentId, parentInfo, true); + index.tryRegisterParent(checksumKey, parentId); + + UploadId childId = new UploadId("child-upload"); + UploadInfo childInfo = new UploadInfo(); + childInfo.setId(childId); + childInfo.setDuplicatesUploadId(parentId); + // checksum is null on childInfo! + adapter.put(childId, childInfo, false); + index.addChild(checksumKey, childId); + + assertTrue(deduplicationService.removeUploadFromIndex(childInfo)); + adapter.terminateUpload(childInfo); + + assertNull(adapter.getUploadInfo(childId)); + assertFalse(index.hasChildren(checksumKey)); + } + + @Test + public void testDeduplicateWhenCasFailsAndWinningParentIsActive() throws Exception { + UploadId winnerParentId = new UploadId("winner-parent"); + UploadInfo winnerInfo = createUploadInfo(winnerParentId, checksumKey); + adapter.put(winnerParentId, winnerInfo, true); + + UploadLock mockLock = mock(UploadLock.class); + when(lockingService.lockUpload(winnerParentId)).thenReturn(mockLock); + + // Custom index where tryRegisterParent fails (CAS lost) but findParent returns winner + TestChecksumIndex customIndex = + new TestChecksumIndex() { + @Override + public synchronized boolean tryRegisterParent(ChecksumKey key, UploadId uploadId) { + return false; // lost race + } + + @Override + public synchronized UploadId findParent(ChecksumKey key) { + return winnerParentId; + } + }; + + DeduplicationService service = new DeduplicationService(customIndex, adapter); + service.setUploadLockingService(lockingService); + + UploadId childId = new UploadId("child-upload"); + UploadInfo childInfo = createUploadInfo(childId, checksumKey); + adapter.put(childId, childInfo, true); + + UploadInfo result = service.deduplicate(childInfo, checksumKey); + assertEquals(childId, result.getId()); + assertEquals(winnerParentId, result.getDuplicatesUploadId()); + assertFalse(adapter.dataExists(childId)); + verify(mockLock).close(); + } + + @Test + public void testDeduplicateNullInputs() throws Exception { + assertNull(deduplicationService.deduplicate(null, checksumKey)); + UploadInfo info = new UploadInfo(); + assertEquals(info, deduplicationService.deduplicate(info, null)); + } + + @Test + public void testDeduplicateWhenUploadIsAlreadyParent() throws Exception { + UploadId parentId = new UploadId("parent-upload"); + UploadInfo parentInfo = createUploadInfo(parentId, checksumKey); + adapter.put(parentId, parentInfo, true); + index.tryRegisterParent(checksumKey, parentId); + + // If deduplicate is invoked with the parent itself, it should return unmodified + UploadInfo result = deduplicationService.deduplicate(parentInfo, checksumKey); + assertEquals(parentId, result.getId()); + assertNull(result.getDuplicatesUploadId()); + } + + @Test + public void testDeduplicateWhenCasFailsAndWinningParentWasTombstoned() throws Exception { + // Simulate case where CAS fails (another upload registered), but by lookup time parent is null + TestChecksumIndex customIndex = + new TestChecksumIndex() { + private boolean firstAttempt = true; + + @Override + public synchronized boolean tryRegisterParent(ChecksumKey key, UploadId uploadId) { + return false; // lost race + } + + @Override + public synchronized UploadId findParent(ChecksumKey key) { + return null; // tombstoned concurrently + } + }; + DeduplicationService service = new DeduplicationService(customIndex, adapter); + service.setUploadLockingService(lockingService); + + UploadId childId = new UploadId("child-upload"); + UploadInfo childInfo = createUploadInfo(childId, checksumKey); + adapter.put(childId, childInfo, true); + + UploadInfo result = service.deduplicate(childInfo, checksumKey); + assertEquals(childId, result.getId()); + assertNull(result.getDuplicatesUploadId()); + } + + @Test + public void testGetUploadInfoByChecksum() throws Exception { + assertNull(deduplicationService.getUploadInfoByChecksum(null)); + + // When no parent registered + assertNull(deduplicationService.getUploadInfoByChecksum(checksumKey)); + + // When parent exists with data + UploadId parentId = new UploadId("parent-upload"); + UploadInfo parentInfo = createUploadInfo(parentId, checksumKey); + adapter.put(parentId, parentInfo, true); + index.tryRegisterParent(checksumKey, parentId); + + UploadInfo found = deduplicationService.getUploadInfoByChecksum(checksumKey); + assertEquals(parentId, found.getId()); + + // When parent data is missing + adapter.deleteData(parentId); + assertNull(deduplicationService.getUploadInfoByChecksum(checksumKey)); + } + + @Test + public void testTerminateUploadNullOrMissingId() throws Exception { + assertTrue(deduplicationService.removeUploadFromIndex(null)); + assertTrue(deduplicationService.removeUploadFromIndex(new UploadInfo())); + adapter.terminateUpload(null); + adapter.terminateUpload(new UploadInfo()); + // KISS: verifying method executes cleanly without throwing an exception + } + + @Test + public void testTerminateUploadWithoutChecksum() throws Exception { + UploadId plainId = new UploadId("plain-upload"); + UploadInfo plainInfo = new UploadInfo(); + plainInfo.setId(plainId); + adapter.put(plainId, plainInfo, true); + + assertTrue(deduplicationService.removeUploadFromIndex(plainInfo)); + adapter.terminateUpload(plainInfo); + assertFalse(adapter.dataExists(plainId)); + assertNull(adapter.getUploadInfo(plainId)); + } + + @Test(expected = java.io.IOException.class) + public void testTerminateUploadWithInvalidChecksumThrowsIOException() throws Exception { + UploadId testId = new UploadId("test-upload"); + UploadInfo testInfo = new UploadInfo(); + testInfo.setId(testId); + testInfo.setChecksum("invalid hex checksum !@#$"); + testInfo.setChecksumAlgorithm(ChecksumAlgorithm.SHA1); + + deduplicationService.removeUploadFromIndex(testInfo); + } + + @Test + public void testTerminateLastChildWhenParentLockFails() throws Exception { + UploadId parentId = new UploadId("parent-upload"); + UploadInfo parentInfo = createUploadInfo(parentId, checksumKey); + adapter.put(parentId, parentInfo, true); + index.tryRegisterParent(checksumKey, parentId); + index.markParentDeleted(checksumKey); + + UploadId lastChild = new UploadId("last-child"); + UploadInfo childInfo = createUploadInfo(lastChild, checksumKey); + childInfo.setDuplicatesUploadId(parentId); + adapter.put(lastChild, childInfo, false); + index.addChild(checksumKey, lastChild); + + when(lockingService.lockUpload(parentId)) + .thenThrow(new UploadAlreadyLockedException("Parent locked")); + + assertTrue(deduplicationService.removeUploadFromIndex(childInfo)); + adapter.terminateUpload(childInfo); + + // Child metadata deleted + assertNull(adapter.getUploadInfo(lastChild)); + // Parent data purge was skipped due to lock contention + assertTrue(adapter.dataExists(parentId)); + } + + @Test + public void testNewChildJoinsTombstonedParentWhenChildrenRemain() throws Exception { + UploadId parentId = new UploadId("parent-upload"); + UploadInfo parentInfo = createUploadInfo(parentId, checksumKey); + adapter.put(parentId, parentInfo, true); + index.tryRegisterParent(checksumKey, parentId); + + // Child 1 joins parent + UploadId child1 = new UploadId("child-1"); + UploadInfo child1Info = createUploadInfo(child1, checksumKey); + adapter.put(child1, child1Info, true); + UploadLock mockLock1 = mock(UploadLock.class); + when(lockingService.lockUpload(parentId)).thenReturn(mockLock1); + deduplicationService.deduplicate(child1Info, checksumKey); + + // Parent is deleted (tombstoned) + assertFalse(deduplicationService.removeUploadFromIndex(parentInfo)); + adapter.terminateUpload(parentInfo); + assertTrue(index.isParentDeleted(checksumKey)); + assertEquals(parentId, index.findParent(checksumKey)); + assertTrue(adapter.dataExists(parentId)); + + // Child 2 arrives while parent is tombstoned + UploadId child2 = new UploadId("child-2"); + UploadInfo child2Info = createUploadInfo(child2, checksumKey); + adapter.put(child2, child2Info, true); + UploadLock mockLock2 = mock(UploadLock.class); + when(lockingService.lockUpload(parentId)).thenReturn(mockLock2); + + UploadInfo result2 = deduplicationService.deduplicate(child2Info, checksumKey); + assertEquals(parentId, result2.getDuplicatesUploadId()); + assertFalse(adapter.dataExists(child2)); + assertTrue(adapter.dataExists(parentId)); + assertTrue(index.hasChildren(checksumKey)); + } + + @Test + public void testDeduplicateSelfHealsWhenParentInfoMissingAndNotTombstoned() throws Exception { + UploadId parentId = new UploadId("stale-parent"); + index.tryRegisterParent(checksumKey, parentId); + // parentInfo is null in adapter and isParentDeleted is false! + UploadLock mockLock = mock(UploadLock.class); + when(lockingService.lockUpload(parentId)).thenReturn(mockLock); + + UploadId newUpload = new UploadId("new-upload"); + UploadInfo newInfo = createUploadInfo(newUpload, checksumKey); + adapter.put(newUpload, newInfo, true); + + UploadInfo result = deduplicationService.deduplicate(newInfo, checksumKey); + assertEquals(newUpload, result.getId()); + assertNull(result.getDuplicatesUploadId()); + assertEquals(newUpload, index.findParent(checksumKey)); + assertTrue(adapter.dataExists(newUpload)); + } + + @Test + public void testDeduplicateB2ParentReplacementUnderLock() throws Exception { + UploadId parent1 = new UploadId("parent-1"); + UploadId parent2 = new UploadId("parent-2"); + UploadInfo parent2Info = createUploadInfo(parent2, checksumKey); + adapter.put(parent2, parent2Info, true); + index.tryRegisterParent(checksumKey, parent1); + + UploadLock mockLock1 = mock(UploadLock.class); + UploadLock mockLock2 = mock(UploadLock.class); + when(lockingService.lockUpload(parent1)) + .thenAnswer( + invocation -> { + // Concurrently, parent1 was replaced with parent2 + index.deleteIndex(checksumKey); + index.tryRegisterParent(checksumKey, parent2); + return mockLock1; + }); + when(lockingService.lockUpload(parent2)).thenReturn(mockLock2); + + UploadId child = new UploadId("child-upload"); + UploadInfo childInfo = createUploadInfo(child, checksumKey); + adapter.put(child, childInfo, true); + + UploadInfo result = deduplicationService.deduplicate(childInfo, checksumKey); + assertEquals(parent2, result.getDuplicatesUploadId()); + verify(mockLock1).close(); + verify(mockLock2).close(); + } + + @Test + public void testTerminateUnindexedStandaloneUploadWithChecksumDoesNotTouchIndex() + throws Exception { + UploadId registeredParent = new UploadId("registered-parent"); + UploadInfo parentInfo = createUploadInfo(registeredParent, checksumKey); + adapter.put(registeredParent, parentInfo, true); + index.tryRegisterParent(checksumKey, registeredParent); + + // Standalone upload that skipped deduplication (e.g. under lock contention) + UploadId standalone = new UploadId("standalone-upload"); + UploadInfo standaloneInfo = createUploadInfo(standalone, checksumKey); + adapter.put(standalone, standaloneInfo, true); + + assertTrue(deduplicationService.removeUploadFromIndex(standaloneInfo)); + adapter.terminateUpload(standaloneInfo); + + // Standalone upload data/metadata deleted + assertFalse(adapter.dataExists(standalone)); + assertNull(adapter.getUploadInfo(standalone)); + + // Registered parent remains intact in index and storage + assertEquals(registeredParent, index.findParent(checksumKey)); + assertTrue(adapter.dataExists(registeredParent)); + } + + @Test + public void testTerminateParentRaceWhenLastChildExitsDuringTombstoning() throws Exception { + UploadId parentId = new UploadId("parent-upload"); + UploadInfo parentInfo = createUploadInfo(parentId, checksumKey); + adapter.put(parentId, parentInfo, true); + index.tryRegisterParent(checksumKey, parentId); + + UploadId childId = new UploadId("child-upload"); + index.addChild(checksumKey, childId); + + // Simulate concurrent child removal right when markParentDeleted is called + TestChecksumIndex raceIndex = + new TestChecksumIndex() { + @Override + public synchronized void markParentDeleted(ChecksumKey key) { + super.markParentDeleted(key); + // Concurrent child terminates and removes its marker + removeChild(key, childId); + } + }; + raceIndex.tryRegisterParent(checksumKey, parentId); + raceIndex.addChild(checksumKey, childId); + + DeduplicationService service = new DeduplicationService(raceIndex, adapter); + service.setUploadLockingService(lockingService); + + boolean deleteData = service.removeUploadFromIndex(parentInfo); + if (deleteData) { + adapter.deleteData(parentId); + } + adapter.deleteMetadata(parentId); + + // Parent data and index purged immediately due to post-tombstone check + assertFalse(adapter.dataExists(parentId)); + assertNull(raceIndex.findParent(checksumKey)); + } + + @Test + public void testSyncParentExpirationOnChildJoin() throws Exception { + UploadId parentId = new UploadId("parent-upload"); + UploadInfo parentInfo = createUploadInfo(parentId, checksumKey); + parentInfo.setExpirationTimestamp(1000L); + adapter.put(parentId, parentInfo, true); + index.tryRegisterParent(checksumKey, parentId); + + UploadLock mockLock = mock(UploadLock.class); + when(lockingService.lockUpload(parentId)).thenReturn(mockLock); + + // Child with longer expiration extends parent expiration + UploadId child1 = new UploadId("child-1"); + UploadInfo child1Info = createUploadInfo(child1, checksumKey); + child1Info.setExpirationTimestamp(2000L); + adapter.put(child1, child1Info, true); + + deduplicationService.deduplicate(child1Info, checksumKey); + assertEquals(Long.valueOf(2000L), adapter.getUploadInfo(parentId).getExpirationTimestamp()); + + // Child with null expiration clears parent expiration (infinite) + UploadId child2 = new UploadId("child-2"); + UploadInfo child2Info = createUploadInfo(child2, checksumKey); + child2Info.setExpirationTimestamp(null); + adapter.put(child2, child2Info, true); + + deduplicationService.deduplicate(child2Info, checksumKey); + assertNull(adapter.getUploadInfo(parentId).getExpirationTimestamp()); + } + + @Test + public void + testGetUploadInfoByChecksumWhenParentTombstonedAndBytesExistReturnsNullWithoutDeletingIndex() + throws Exception { + UploadId parentId = new UploadId("tombstoned-parent"); + index.tryRegisterParent(checksumKey, parentId); + index.markParentDeleted(checksumKey); + adapter.put(parentId, null, true); // metadata deleted, physical data exists + + assertNull(deduplicationService.getUploadInfoByChecksum(checksumKey)); + // Index was NOT purged + assertEquals(parentId, index.findParent(checksumKey)); + } + + @Test + public void testDeduplicateB2ParentReplacementUnderLockWhenNewParentIsNull() throws Exception { + UploadId parent1 = new UploadId("parent-1"); + index.tryRegisterParent(checksumKey, parent1); + + UploadLock mockLock1 = mock(UploadLock.class); + when(lockingService.lockUpload(parent1)) + .thenAnswer( + invocation -> { + // Concurrently, index was cleared + index.deleteIndex(checksumKey); + return mockLock1; + }); + + UploadId newParent = new UploadId("new-parent"); + UploadInfo newParentInfo = createUploadInfo(newParent, checksumKey); + adapter.put(newParent, newParentInfo, true); + + UploadInfo result = deduplicationService.deduplicate(newParentInfo, checksumKey); + assertEquals(newParent, result.getId()); + assertNull(result.getDuplicatesUploadId()); + assertEquals(newParent, index.findParent(checksumKey)); + } + + @Test + public void testTerminateChildWithNullChecksumWhenParentInfoNull() throws Exception { + UploadId parentId = new UploadId("deleted-parent"); + UploadId childId = new UploadId("child-upload"); + UploadInfo childInfo = new UploadInfo(); + childInfo.setId(childId); + childInfo.setDuplicatesUploadId(parentId); + // checksum is null and parentInfo in adapter is null + adapter.put(childId, childInfo, true); + + assertTrue(deduplicationService.removeUploadFromIndex(childInfo)); + adapter.terminateUpload(childInfo); + assertNull(adapter.getUploadInfo(childId)); + assertFalse(adapter.dataExists(childId)); + } + + @Test + public void testRegisterAsParentCasFailsAndJoinsWinner() throws Exception { + UploadId winnerId = new UploadId("winner-parent"); + UploadInfo winnerInfo = createUploadInfo(winnerId, checksumKey); + adapter.put(winnerId, winnerInfo, true); + index.tryRegisterParent(checksumKey, winnerId); + + UploadLock mockLock = mock(UploadLock.class); + when(lockingService.lockUpload(winnerId)).thenReturn(mockLock); + + UploadId competitorId = new UploadId("competitor-upload"); + UploadInfo competitorInfo = createUploadInfo(competitorId, checksumKey); + adapter.put(competitorId, competitorInfo, true); + + // tryRegisterParent will return false because winner is already registered + UploadInfo result = deduplicationService.deduplicate(competitorInfo, checksumKey); + + assertEquals(winnerId, result.getDuplicatesUploadId()); + assertFalse(adapter.dataExists(competitorId)); + assertTrue(index.hasChildren(checksumKey)); + } + + @Test + public void testRemoveUploadFromIndexDoesNotInvokeLockCleanup() throws Exception { + UploadId parentId = new UploadId("parent-upload"); + UploadInfo parentInfo = createUploadInfo(parentId, checksumKey); + adapter.put(parentId, parentInfo, true); + index.tryRegisterParent(checksumKey, parentId); + + // Parent terminates with no children; index is removed but lockingService.cleanupLock is NOT + // invoked (delegated to storage service terminateUpload) + assertTrue(deduplicationService.removeUploadFromIndex(parentInfo)); + org.mockito.Mockito.verify(lockingService, org.mockito.Mockito.never()).cleanupLock(parentId); + } + + @Test + public void testPruneOrphanedChecksumIndicesWhenParentDataMissing() throws Exception { + UploadId parentId = new UploadId("parent-upload"); + UploadInfo parentInfo = createUploadInfo(parentId, checksumKey); + adapter.put(parentId, parentInfo, false); // metadata exists, data missing! + index.tryRegisterParent(checksumKey, parentId); + + deduplicationService.pruneOrphanedChecksumIndices(); + + assertNull(index.findParent(checksumKey)); + } + + @Test + public void testPruneOrphanedChecksumIndicesWhenParentMetadataMissingAndNotTombstoned() + throws Exception { + UploadId parentId = new UploadId("parent-upload"); + adapter.put(parentId, null, true); // metadata missing, data exists, not tombstoned! + index.tryRegisterParent(checksumKey, parentId); + + deduplicationService.pruneOrphanedChecksumIndices(); + + assertNull(index.findParent(checksumKey)); + assertFalse(adapter.dataExists(parentId)); + } + + @Test + public void testPruneOrphanedChecksumIndicesWhenTombstonedWithZeroChildren() throws Exception { + UploadId parentId = new UploadId("parent-upload"); + adapter.put(parentId, null, true); // metadata missing, data exists + index.tryRegisterParent(checksumKey, parentId); + index.markParentDeleted(checksumKey); // tombstoned, but no children! + + deduplicationService.pruneOrphanedChecksumIndices(); + + assertNull(index.findParent(checksumKey)); + assertFalse(adapter.dataExists(parentId)); + } + + @Test + public void testPruneOrphanedChecksumIndicesWhenTombstonedWithChildrenPreserved() + throws Exception { + UploadId parentId = new UploadId("parent-upload"); + adapter.put(parentId, null, true); + index.tryRegisterParent(checksumKey, parentId); + index.markParentDeleted(checksumKey); + + UploadId childId = new UploadId("child-upload"); + index.addChild(checksumKey, childId); + + deduplicationService.pruneOrphanedChecksumIndices(); + + // Tombstoned parent with active children MUST be preserved + assertEquals(parentId, index.findParent(checksumKey)); + assertTrue(adapter.dataExists(parentId)); + } + + @Test + public void testPruneOrphanedChecksumIndicesWhenValidParentPreserved() throws Exception { + UploadId parentId = new UploadId("parent-upload"); + UploadInfo parentInfo = createUploadInfo(parentId, checksumKey); + adapter.put(parentId, parentInfo, true); + index.tryRegisterParent(checksumKey, parentId); + + deduplicationService.pruneOrphanedChecksumIndices(); + + // Valid parent with metadata and data must not be pruned + assertEquals(parentId, index.findParent(checksumKey)); + assertTrue(adapter.dataExists(parentId)); + } + + @Test + public void testPruneOrphanedChecksumIndicesHandlesExceptionGracefully() throws Exception { + TestChecksumIndex errorIndex = + new TestChecksumIndex() { + @Override + public synchronized UploadId findParent(ChecksumKey key) { + throw new RuntimeException("Simulated error reading index"); + } + }; + errorIndex.tryRegisterParent(checksumKey, new UploadId("parent-id")); + + DeduplicationService service = new DeduplicationService(errorIndex, adapter); + service.pruneOrphanedChecksumIndices(); + // KISS: verify method executes cleanly without bubbling up exception + assertTrue(true); + } + + @Test + public void testPruneOrphanedChecksumIndicesHandlesGetAllIndexedKeysExceptionGracefully() + throws Exception { + TestChecksumIndex errorIndex = + new TestChecksumIndex() { + @Override + public synchronized Set getAllIndexedKeys() throws IOException { + throw new IOException("Simulated error listing keys"); + } + }; + + DeduplicationService service = new DeduplicationService(errorIndex, adapter); + service.pruneOrphanedChecksumIndices(); + // KISS: verify method executes cleanly without bubbling up exception + assertTrue(true); + } + + @Test(expected = NullPointerException.class) + public void testThreeParamConstructorThrowsOnNullLockingService() { + new DeduplicationService(index, adapter, null); + } + + @Test(expected = NullPointerException.class) + public void testSetUploadLockingServiceThrowsOnNull() { + deduplicationService.setUploadLockingService(null); + } + + @Test + public void testThreeParamConstructorWithValidLockingService() { + DeduplicationService service = new DeduplicationService(index, adapter, lockingService); + assertNotNull(service); + } + + private UploadInfo createUploadInfo(UploadId id, ChecksumKey key) { + UploadInfo info = new UploadInfo(); + info.setId(id); + info.setOffset(100L); + info.setLength(100L); + info.setChecksum(key.getHexChecksum()); + info.setChecksumAlgorithm(key.getAlgorithm()); + return info; + } + + // --- In-memory Test Doubles --- + + private static class TestChecksumIndex implements ChecksumIndex { + private final Map parents = new HashMap<>(); + private final Set tombstones = new HashSet<>(); + private final Map> children = new HashMap<>(); + + @Override + public synchronized boolean tryRegisterParent(ChecksumKey key, UploadId uploadId) { + if (parents.containsKey(key) || tombstones.contains(key)) { + return false; + } + parents.put(key, uploadId); + return true; + } + + @Override + public synchronized UploadId findParent(ChecksumKey key) { + return parents.get(key); + } + + @Override + public synchronized void markParentDeleted(ChecksumKey key) { + tombstones.add(key); + } + + @Override + public synchronized boolean isParentDeleted(ChecksumKey key) { + return tombstones.contains(key); + } + + @Override + public synchronized void addChild(ChecksumKey key, UploadId childId) { + children.computeIfAbsent(key, k -> new HashSet<>()).add(childId); + } + + @Override + public synchronized void removeChild(ChecksumKey key, UploadId childId) { + Set set = children.get(key); + if (set != null) { + set.remove(childId); + if (set.isEmpty()) { + children.remove(key); + } + } + } + + @Override + public synchronized boolean hasChildren(ChecksumKey key) { + Set set = children.get(key); + return set != null && !set.isEmpty(); + } + + @Override + public synchronized void deleteIndex(ChecksumKey key) { + parents.remove(key); + tombstones.remove(key); + children.remove(key); + } + + @Override + public synchronized Set getAllIndexedKeys() throws IOException { + return new HashSet<>(parents.keySet()); + } + } + + private static class TestStorageService implements UploadStorageService { + private final Map infoMap = new HashMap<>(); + private final Set dataFiles = new HashSet<>(); + private DeduplicationService deduplicationService; + + void setDeduplicationService(DeduplicationService deduplicationService) { + this.deduplicationService = deduplicationService; + } + + void put(UploadId id, UploadInfo info, boolean hasData) { + if (info != null) { + infoMap.put(id, info); + } else { + infoMap.remove(id); + } + if (hasData) { + dataFiles.add(id); + } else { + dataFiles.remove(id); + } + } + + @Override + public boolean dataExists(UploadId id) { + return dataFiles.contains(id); + } + + @Override + public void deleteData(UploadId id) { + dataFiles.remove(id); + } + + public void deleteMetadata(UploadId id) { + infoMap.remove(id); + } + + @Override + public UploadInfo getUploadInfo(UploadId id) { + return infoMap.get(id); + } + + @Override + public void update(UploadInfo info) { + infoMap.put(info.getId(), info); + } + + @Override + public void terminateUpload(UploadInfo uploadInfo) throws UploadNotFoundException, IOException { + if (uploadInfo == null || uploadInfo.getId() == null) { + return; + } + boolean deleteData = true; + if (deduplicationService != null) { + deleteData = deduplicationService.removeUploadFromIndex(uploadInfo); + } + if (deleteData) { + deleteData(uploadInfo.getId()); + } + deleteMetadata(uploadInfo.getId()); + } + + @Override + public UploadInfo getUploadInfo(String uploadUrl, String ownerKey) throws IOException { + return null; + } + + @Override + public String getUploadUri() { + return null; + } + + @Override + public UploadInfo append(UploadInfo upload, InputStream inputStream) + throws IOException, TusException { + return null; + } + + @Override + public void setMaxUploadSize(Long maxUploadSize) {} + + @Override + public long getMaxUploadSize() { + return 0; + } + + @Override + public UploadInfo create(UploadInfo info, String ownerKey) throws IOException { + return null; + } + + @Override + public InputStream getUploadedBytes(String uploadUri, String ownerKey) + throws IOException, UploadNotFoundException { + return null; + } + + @Override + public InputStream getUploadedBytes(UploadId id) throws IOException, UploadNotFoundException { + return null; + } + + @Override + public void copyUploadTo(UploadInfo info, OutputStream outputStream) + throws UploadNotFoundException, IOException {} + + @Override + public void cleanupExpiredUploads(UploadLockingService uploadLockingService) + throws IOException {} + + @Override + public void removeLastNumberOfBytes(UploadInfo uploadInfo, long byteCount) + throws UploadNotFoundException, IOException {} + + @Override + public Long getUploadExpirationPeriod() { + return null; + } + + @Override + public void setUploadExpirationPeriod(Long uploadExpirationPeriod) {} + + @Override + public void setUploadConcatenationService(UploadConcatenationService concatenationService) {} + + @Override + public UploadConcatenationService getUploadConcatenationService() { + return null; + } + + @Override + public void setIdFactory(UploadIdFactory idFactory) {} + + @Override + public UploadInfo deduplicate(UploadInfo uploadInfo, ChecksumKey checksumKey) + throws IOException { + return null; + } + + @Override + public void setUploadLockingService(UploadLockingService uploadLockingService) {} + + @Override + public void setUploadDeduplicationEnabled(boolean deduplicationEnabled) {} + + @Override + public boolean isUploadDeduplicationEnabled() { + return false; + } + + @Override + public UploadInfo getUploadInfoByChecksum(String checksum, ChecksumAlgorithm algorithm) + throws IOException { + return null; + } + } +} diff --git a/src/test/java/me/desair/tus/server/upload/disk/DiskChecksumIndexTest.java b/src/test/java/me/desair/tus/server/upload/disk/DiskChecksumIndexTest.java new file mode 100644 index 00000000..0b4f2eeb --- /dev/null +++ b/src/test/java/me/desair/tus/server/upload/disk/DiskChecksumIndexTest.java @@ -0,0 +1,158 @@ +package me.desair.tus.server.upload.disk; + +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertFalse; +import static org.junit.Assert.assertNull; +import static org.junit.Assert.assertTrue; + +import java.io.IOException; +import java.nio.file.Files; +import java.nio.file.Path; +import java.util.Set; +import me.desair.tus.server.checksum.ChecksumAlgorithm; +import me.desair.tus.server.checksum.ChecksumKey; +import me.desair.tus.server.upload.UploadId; +import org.apache.commons.io.FileUtils; +import org.junit.After; +import org.junit.Before; +import org.junit.Test; + +/** Unit tests for {@link DiskChecksumIndex}. */ +public class DiskChecksumIndexTest { + + private Path tempDir; + private DiskChecksumIndex index; + private final ChecksumKey checksumKey = + ChecksumKey.of("0123456789abcdef0123456789abcdef01234567", ChecksumAlgorithm.SHA1); + + @Before + public void setUp() throws Exception { + tempDir = Files.createTempDirectory("tus-disk-checksum-test-"); + index = new DiskChecksumIndex(tempDir.toString()); + } + + @After + public void tearDown() throws Exception { + FileUtils.deleteDirectory(tempDir.toFile()); + } + + @Test + public void testTryRegisterParentSuccessAndFindParent() throws IOException { + UploadId uploadId = new UploadId("parent-123"); + assertTrue(index.tryRegisterParent(checksumKey, uploadId)); + + assertEquals(uploadId, index.findParent(checksumKey)); + assertFalse(index.isParentDeleted(checksumKey)); + assertFalse(index.hasChildren(checksumKey)); + } + + @Test + public void testTryRegisterParentConflictFails() throws IOException { + UploadId uploadId1 = new UploadId("parent-1"); + UploadId uploadId2 = new UploadId("parent-2"); + + assertTrue(index.tryRegisterParent(checksumKey, uploadId1)); + assertFalse(index.tryRegisterParent(checksumKey, uploadId2)); + + assertEquals(uploadId1, index.findParent(checksumKey)); + } + + @Test + public void testMarkParentDeletedAndTombstone() throws IOException { + UploadId uploadId = new UploadId("parent-123"); + index.tryRegisterParent(checksumKey, uploadId); + + index.markParentDeleted(checksumKey); + + assertTrue(index.isParentDeleted(checksumKey)); + // Parent marker is preserved during tombstoning so children can resolve parent ID + assertEquals(uploadId, index.findParent(checksumKey)); + + // After tombstone, another upload cannot register as parent + assertFalse(index.tryRegisterParent(checksumKey, new UploadId("parent-456"))); + } + + @Test + public void testAddAndRemoveChildren() throws IOException { + UploadId child1 = new UploadId("child-1"); + UploadId child2 = new UploadId("child-2"); + + assertFalse(index.hasChildren(checksumKey)); + + index.addChild(checksumKey, child1); + assertTrue(index.hasChildren(checksumKey)); + + index.addChild(checksumKey, child2); + assertTrue(index.hasChildren(checksumKey)); + + index.removeChild(checksumKey, child1); + assertTrue(index.hasChildren(checksumKey)); + + index.removeChild(checksumKey, child2); + assertFalse(index.hasChildren(checksumKey)); + } + + @Test + public void testDeleteIndexRemovesDirectory() throws IOException { + UploadId parent = new UploadId("parent"); + UploadId child = new UploadId("child"); + + index.tryRegisterParent(checksumKey, parent); + index.addChild(checksumKey, child); + + index.deleteIndex(checksumKey); + + assertNull(index.findParent(checksumKey)); + assertFalse(index.isParentDeleted(checksumKey)); + assertFalse(index.hasChildren(checksumKey)); + } + + @Test + public void testGetAllIndexedKeysEmptyAndNonExistent() throws IOException { + assertTrue(index.getAllIndexedKeys().isEmpty()); + + DiskChecksumIndex nonExistentIndex = + new DiskChecksumIndex(tempDir.resolve("does-not-exist").toString()); + assertTrue(nonExistentIndex.getAllIndexedKeys().isEmpty()); + } + + @Test + public void testGetAllIndexedKeysWithValidAndInvalidEntries() throws IOException { + UploadId parentId = new UploadId("parent-123"); + index.tryRegisterParent(checksumKey, parentId); + + Path checksumsDir = tempDir.resolve("checksums"); + + // Create a regular file directly inside storagePath (must be skipped) + Files.createFile(checksumsDir.resolve("regular-file.tmp")); + + // Create an invalid/unknown algorithm directory (must be skipped) + Files.createDirectories(checksumsDir.resolve("unknown-algo").resolve("some-hash")); + + // Create a valid algorithm directory with a hash dir that lacks a parent file (must be skipped) + Files.createDirectories(checksumsDir.resolve("sha1").resolve("empty-hash-dir")); + + // Create a valid algorithm directory with an empty file instead of a directory (must be + // skipped) + Files.createFile(checksumsDir.resolve("sha1").resolve("not-a-dir")); + + // Create a malformed hash directory (must be caught and skipped) + Path malformedDir = checksumsDir.resolve("sha1").resolve("invalid_hex_chars_!@#$"); + Files.createDirectories(malformedDir); + Files.createFile(malformedDir.resolve("parent")); + + Set keys = index.getAllIndexedKeys(); + assertEquals(1, keys.size()); + assertTrue(keys.contains(checksumKey)); + } + + @Test(expected = IllegalArgumentException.class) + public void testChildIdPathTraversalThrowsException() throws IOException { + index.addChild(checksumKey, new UploadId("../../..%2F/etc/passwd")); + } + + @Test(expected = IllegalArgumentException.class) + public void testRemoveChildIdPathTraversalThrowsException() throws IOException { + index.removeChild(checksumKey, new UploadId("../../..%2F/etc/passwd")); + } +} diff --git a/src/test/java/me/desair/tus/server/upload/disk/DiskLockingServiceTest.java b/src/test/java/me/desair/tus/server/upload/disk/DiskLockingServiceTest.java index b9eee747..9335348f 100644 --- a/src/test/java/me/desair/tus/server/upload/disk/DiskLockingServiceTest.java +++ b/src/test/java/me/desair/tus/server/upload/disk/DiskLockingServiceTest.java @@ -5,6 +5,7 @@ import static org.hamcrest.Matchers.not; import static org.hamcrest.Matchers.nullValue; import static org.junit.Assert.assertFalse; +import static org.junit.Assert.assertNull; import static org.junit.Assert.assertTrue; import static org.mockito.ArgumentMatchers.nullable; import static org.mockito.Mockito.reset; @@ -97,6 +98,18 @@ public void lockUploadByUri() throws Exception { uploadLock.release(); } + @Test + public void lockUploadByIdDirectly() throws Exception { + UploadId uploadId = new UploadId(UUID.randomUUID()); + UploadLock uploadLock = lockingService.lockUpload(uploadId); + + assertThat(uploadLock, not(nullValue())); + assertTrue(lockingService.isLocked(uploadId)); + + uploadLock.release(); + assertNull(lockingService.lockUpload(null)); + } + @Test public void isLockedTrue() throws Exception { String uploadIdStr = UUID.randomUUID().toString(); diff --git a/src/test/java/me/desair/tus/server/upload/disk/DiskStorageServiceTest.java b/src/test/java/me/desair/tus/server/upload/disk/DiskStorageServiceTest.java index bbfc4d32..cc899fba 100644 --- a/src/test/java/me/desair/tus/server/upload/disk/DiskStorageServiceTest.java +++ b/src/test/java/me/desair/tus/server/upload/disk/DiskStorageServiceTest.java @@ -4,8 +4,11 @@ import static org.hamcrest.Matchers.is; import static org.hamcrest.Matchers.notNullValue; import static org.hamcrest.Matchers.nullValue; +import static org.junit.Assert.assertEquals; import static org.junit.Assert.assertFalse; +import static org.junit.Assert.assertNotNull; import static org.junit.Assert.assertNotSame; +import static org.junit.Assert.assertNull; import static org.junit.Assert.assertThrows; import static org.junit.Assert.assertTrue; import static org.junit.Assert.fail; @@ -30,6 +33,7 @@ import java.util.List; import java.util.UUID; import me.desair.tus.server.checksum.ChecksumAlgorithm; +import me.desair.tus.server.checksum.ChecksumKey; import me.desair.tus.server.exception.InvalidUploadOffsetException; import me.desair.tus.server.exception.UploadNotFoundException; import me.desair.tus.server.upload.UploadId; @@ -541,6 +545,28 @@ public void cleanupExpiredUploads() throws Exception { org.mockito.Mockito.verify(uploadLockingService).cleanupLock(info.getId()); } + @Test + public void cleanupExpiredUploadsNoArg() throws Exception { + when(uploadLockingService.isLocked(any(UploadId.class))).thenReturn(false); + storageService.setUploadLockingService(uploadLockingService); + + String content = "This is the content of my upload"; + + UploadInfo info = new UploadInfo(); + info.setLength((long) content.getBytes().length + 20); + info.updateExpiration(100L); + + info = storageService.create(info, null); + assertTrue(Files.exists(getUploadInfoPath(info.getId()))); + + Utils.sleep(500L); + storageService.cleanupExpiredUploads(); + + assertFalse(Files.exists(getUploadInfoPath(info.getId()))); + assertFalse(Files.exists(getStoragePath(info.getId()))); + org.mockito.Mockito.verify(uploadLockingService).cleanupLock(info.getId()); + } + @Test public void testDeduplicationEnabled() throws Exception { storageService.setUploadDeduplicationEnabled(true); @@ -561,7 +587,7 @@ public void testDeduplicationAndParentChildExpiration() throws Exception { parent.setLength(100L); parent.setOffset(100L); parent.setChecksum("parentchecksumvalue"); - parent.setChecksumAlgorithm(me.desair.tus.server.checksum.ChecksumAlgorithm.SHA256); + parent.setChecksumAlgorithm(ChecksumAlgorithm.SHA256); parent = storageService.create(parent, null); // Write dummy data so getBytesPath checks succeed Files.write(getUploadDataPath(parent.getId()), new byte[100]); @@ -571,8 +597,7 @@ public void testDeduplicationAndParentChildExpiration() throws Exception { // Verify index lookup finds it UploadInfo resolvedParent = - storageService.getUploadInfoByChecksum( - "parentchecksumvalue", me.desair.tus.server.checksum.ChecksumAlgorithm.SHA256); + storageService.getUploadInfoByChecksum("parentchecksumvalue", ChecksumAlgorithm.SHA256); assertThat(resolvedParent, is(notNullValue())); assertThat(resolvedParent.getId(), is(parent.getId())); @@ -610,11 +635,86 @@ public void testDeduplicationAndParentChildExpiration() throws Exception { // Test parent termination removes parent index storageService.terminateUpload(parent); resolvedParent = - storageService.getUploadInfoByChecksum( - "parentchecksumvalue", me.desair.tus.server.checksum.ChecksumAlgorithm.SHA256); + storageService.getUploadInfoByChecksum("parentchecksumvalue", ChecksumAlgorithm.SHA256); assertThat(resolvedParent, is(nullValue())); } + /** + * Verifies that when deduplication is enabled on DiskStorageService: 1. A completed upload with a + * checksum creates the physical directory and 'parent' marker on disk. 2. A deduplicated child + * upload creates a marker under 'children/'. 3. Terminating the parent while the child is active + * writes the 'parent-deleted' tombstone marker and preserves parent data bytes. 4. Terminating + * the child (last child) purges both parent data bytes and the checksum directory. + */ + @Test + public void testChecksumDirectoryCreationAndCleanupOnDisk() throws Exception { + storageService.setUploadDeduplicationEnabled(true); + storageService.setUploadLockingService(uploadLockingService); + + String sha1Hex = "0123456789abcdef0123456789abcdef01234567"; + Path indexDir = storagePath.resolve("checksums").resolve("sha1").resolve(sha1Hex); + Path parentMarker = indexDir.resolve("parent"); + Path parentDeletedMarker = indexDir.resolve("parent-deleted"); + + // 1. Create completed parent upload with SHA1 checksum + UploadInfo parent = new UploadInfo(); + parent.setLength(100L); + parent.setOffset(100L); + parent.setChecksum(sha1Hex); + parent.setChecksumAlgorithm(ChecksumAlgorithm.SHA1); + parent = storageService.create(parent, null); + Files.write(getUploadDataPath(parent.getId()), new byte[100]); + parent.setOffset(100L); + storageService.update(parent); + + // Assert that the checksum directory and parent pointer exist on disk + assertTrue( + "Checksum directory must exist on disk after parent upload", Files.isDirectory(indexDir)); + assertTrue("Parent pointer file must exist on disk", Files.isRegularFile(parentMarker)); + assertEquals( + parent.getId().toString(), Files.readString(parentMarker, StandardCharsets.UTF_8).trim()); + assertFalse( + "Parent-deleted marker must not exist initially", Files.exists(parentDeletedMarker)); + + // 2. Create duplicate child upload + UploadInfo child = new UploadInfo(); + child.setLength(100L); + child.setOffset(100L); + child.setChecksum(sha1Hex); + child.setChecksumAlgorithm(ChecksumAlgorithm.SHA1); + child = storageService.create(child, null); + Files.write(getUploadDataPath(child.getId()), new byte[100]); + child.setOffset(100L); + storageService.update(child); + + Path childMarker = indexDir.resolve("children").resolve(child.getId().toString()); + assertTrue( + "Child marker file must exist on disk under children/", Files.isRegularFile(childMarker)); + assertFalse( + "Child physical data file must be removed", Files.exists(getUploadDataPath(child.getId()))); + + // 3. Terminate parent upload (child still alive) + storageService.terminateUpload(parent); + + assertTrue( + "Parent-deleted tombstone marker must exist on disk", + Files.isRegularFile(parentDeletedMarker)); + assertTrue( + "Parent marker file must still exist while child is active", + Files.isRegularFile(parentMarker)); + assertTrue( + "Parent physical data file must still exist while child is active", + Files.exists(getUploadDataPath(parent.getId()))); + + // 4. Terminate child upload (last child) + storageService.terminateUpload(child); + + assertFalse( + "Parent physical data file must be deleted upon last child termination", + Files.exists(getUploadDataPath(parent.getId()))); + assertFalse("Checksum directory must be completely removed from disk", Files.exists(indexDir)); + } + @Test public void testDanglingDeduplicationIndexSelfCleaning() throws Exception { storageService.setUploadDeduplicationEnabled(true); @@ -1136,4 +1236,109 @@ public void testDefaultDrainTimeoutOnDiskStorageService() { storageService.setDrainTimeout(null); assertThat(storageService.getDrainTimeout(), is(Duration.ofSeconds(20))); } + + @Test + public void testDataExistsAndDeleteData() throws Exception { + UploadInfo info = new UploadInfo(); + info.setLength(100L); + UploadInfo created = storageService.create(info, "owner"); + assertTrue(storageService.dataExists(created.getId())); + + storageService.deleteData(created.getId()); + assertFalse(storageService.dataExists(created.getId())); + + Files.write(getUploadDataPath(created.getId()), new byte[100]); + assertTrue(storageService.dataExists(created.getId())); + + storageService.deleteData(created.getId()); + assertFalse(storageService.dataExists(created.getId())); + assertFalse(storageService.dataExists(null)); + } + + @Test + public void testPruneOrphanedChecksumIndicesOnDisk() throws Exception { + storageService.setUploadDeduplicationEnabled(true); + String sha1Hex = "0123456789abcdef0123456789abcdef01234567"; + Path indexDir = storagePath.resolve("checksums").resolve("sha1").resolve(sha1Hex); + Files.createDirectories(indexDir); + Path parentMarker = indexDir.resolve("parent"); + Files.writeString(parentMarker, "orphan-parent-id", StandardCharsets.UTF_8); + + assertTrue(Files.exists(parentMarker)); + + storageService.cleanupExpiredUploads(null); + + assertFalse(Files.exists(parentMarker)); + } + + @Test(expected = NullPointerException.class) + public void testSetUploadLockingServiceNullThrowsNpe() { + storageService.setUploadLockingService(null); + } + + @Test + public void testGetUploadLockingServiceReturnsConfiguredInstance() { + assertNotNull(storageService.getUploadLockingService()); + } + + @Test + public void testPruneOrphanedChecksumIndicesAbandonedTombstoneOnDisk() throws Exception { + storageService.setUploadDeduplicationEnabled(true); + String sha1Hex = "abcdef0123456789abcdef0123456789abcdef01"; + Path indexDir = storagePath.resolve("checksums").resolve("sha1").resolve(sha1Hex); + Files.createDirectories(indexDir); + Path parentMarker = indexDir.resolve("parent"); + Path parentDeletedMarker = indexDir.resolve("parent-deleted"); + Files.writeString(parentMarker, "tombstone-parent-id", StandardCharsets.UTF_8); + Files.writeString(parentDeletedMarker, "", StandardCharsets.UTF_8); + + assertTrue(Files.exists(parentMarker)); + assertTrue(Files.exists(parentDeletedMarker)); + + storageService.cleanupExpiredUploads(null); + + assertFalse(Files.exists(parentMarker)); + assertFalse(Files.exists(parentDeletedMarker)); + } + + @Test + public void testDeduplicateWhenDisabledOrNullParams() throws Exception { + UploadInfo info = new UploadInfo(); + info.setId(new UploadId("test-id")); + ChecksumKey key = + ChecksumKey.of("0123456789abcdef0123456789abcdef01234567", ChecksumAlgorithm.SHA1); + + storageService.setUploadDeduplicationEnabled(false); + assertEquals(info, storageService.deduplicate(info, key)); + + storageService.setUploadDeduplicationEnabled(true); + assertNull(storageService.deduplicate(null, key)); + assertEquals(info, storageService.deduplicate(info, null)); + } + + @Test + public void testGetUploadInfoByChecksumNullParams() throws Exception { + assertNull(storageService.getUploadInfoByChecksum(null, ChecksumAlgorithm.SHA1)); + assertNull( + storageService.getUploadInfoByChecksum("0123456789abcdef0123456789abcdef01234567", null)); + } + + @Test(expected = IOException.class) + public void testGetUploadInfoByChecksumUnsafeThrowsIOException() throws Exception { + storageService.getUploadInfoByChecksum(" / ", ChecksumAlgorithm.SHA1); + } + + @Test(expected = IOException.class) + public void testUpdateWithMalformedChecksumThrowsIOException() throws Exception { + storageService.setUploadDeduplicationEnabled(true); + UploadInfo info = new UploadInfo(); + info.setId(new UploadId("malformed-sum-id")); + info.setLength(10L); + info.setChecksum("!!!"); + info.setChecksumAlgorithm(ChecksumAlgorithm.SHA1); + info = storageService.create(info, null); + info.setOffset(10L); + + storageService.update(info); + } } diff --git a/src/test/java/me/desair/tus/server/upload/s3/ITS3RufhProtocol.java b/src/test/java/me/desair/tus/server/upload/s3/ITS3RufhProtocol.java index 97872e3c..4486a980 100644 --- a/src/test/java/me/desair/tus/server/upload/s3/ITS3RufhProtocol.java +++ b/src/test/java/me/desair/tus/server/upload/s3/ITS3RufhProtocol.java @@ -1,9 +1,11 @@ package me.desair.tus.server.upload.s3; import io.minio.MinioClient; +import io.minio.RemoveObjectArgs; import me.desair.tus.server.AbstractITRufhProtocol; import me.desair.tus.server.TestUtils; import me.desair.tus.server.TusFileUploadService; +import me.desair.tus.server.upload.UploadId; import org.junit.AfterClass; import org.junit.BeforeClass; @@ -64,4 +66,10 @@ protected TusFileUploadService createTusFileUploadService(String uploadUri) { .withUploadExpirationPeriod(2L * 24 * 60 * 60 * 1000) .withDownloadFeature(); } + + @Override + protected void deleteUploadedBytesOutOfBand(UploadId uploadId) throws Exception { + minioClient.removeObject( + RemoveObjectArgs.builder().bucket(BUCKET).object("uploads/" + uploadId).build()); + } } diff --git a/src/test/java/me/desair/tus/server/upload/s3/ITS3StorageService.java b/src/test/java/me/desair/tus/server/upload/s3/ITS3StorageService.java index b230dca9..2102dfe4 100644 --- a/src/test/java/me/desair/tus/server/upload/s3/ITS3StorageService.java +++ b/src/test/java/me/desair/tus/server/upload/s3/ITS3StorageService.java @@ -1,10 +1,17 @@ package me.desair.tus.server.upload.s3; import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertFalse; import static org.junit.Assert.assertNotNull; import static org.junit.Assert.assertNull; +import static org.junit.Assert.assertTrue; +import io.minio.ListObjectsArgs; import io.minio.MinioClient; +import io.minio.PutObjectArgs; +import io.minio.Result; +import io.minio.StatObjectArgs; +import io.minio.messages.Item; import java.io.ByteArrayInputStream; import java.io.InputStream; import java.nio.charset.StandardCharsets; @@ -91,10 +98,13 @@ public void testDeduplicationOnS3() throws Exception { storageService.setUploadDeduplicationEnabled(true); byte[] content = "S3 Deduplicated Content".getBytes(StandardCharsets.UTF_8); - String sha1Base64 = - org.apache.commons.codec.binary.Base64.encodeBase64String(DigestUtils.sha1(content)); + byte[] sha1Digest = DigestUtils.sha1(content); + String sha1Base64 = org.apache.commons.codec.binary.Base64.encodeBase64String(sha1Digest); + String sha1Hex = org.apache.commons.codec.binary.Hex.encodeHexString(sha1Digest); + String parentObjectKey = "checksums/sha1/" + sha1Hex + "/parent"; + String parentDeletedKey = "checksums/sha1/" + sha1Hex + "/parent-deleted"; - // Parent upload + // 1. Parent upload UploadInfo parent = new UploadInfo(); parent.setLength((long) content.length); parent.setChecksum(sha1Base64); @@ -103,10 +113,45 @@ public void testDeduplicationOnS3() throws Exception { storageService.append(parent, new ByteArrayInputStream(content)); + // Verify parent pointer exists in S3 + assertNotNull( + minioClient.statObject( + StatObjectArgs.builder().bucket(BUCKET).object(parentObjectKey).build())); + // Look up by checksum UploadInfo found = storageService.getUploadInfoByChecksum(sha1Base64, ChecksumAlgorithm.SHA1); assertNotNull(found); assertEquals(parent.getId(), found.getId()); + + // 2. Child duplicate upload + UploadInfo child = new UploadInfo(); + child.setLength((long) content.length); + child.setChecksum(sha1Base64); + child.setChecksumAlgorithm(ChecksumAlgorithm.SHA1); + child = storageService.create(child, "owner1"); + child = storageService.append(child, new ByteArrayInputStream(content)); + + assertEquals(parent.getId(), child.getDuplicatesUploadId()); + String childObjectKey = "checksums/sha1/" + sha1Hex + "/children/" + child.getId(); + assertNotNull( + minioClient.statObject( + StatObjectArgs.builder().bucket(BUCKET).object(childObjectKey).build())); + + // 3. Terminate parent upload (child still alive) + storageService.terminateUpload(parent); + + assertNotNull( + minioClient.statObject( + StatObjectArgs.builder().bucket(BUCKET).object(parentDeletedKey).build())); + + // 4. Terminate child upload (last child out) + storageService.terminateUpload(child); + + // Verify all objects under checksum prefix are completely purged from S3 + Iterable> remaining = + minioClient.listObjects( + ListObjectsArgs.builder().bucket(BUCKET).prefix("checksums/sha1/" + sha1Hex).build()); + assertFalse(remaining.iterator().hasNext()); } @Test @@ -179,4 +224,61 @@ public void testCleanupExpiredUploadsOnS3() throws Exception { // Verify upload was cleaned up from real S3 assertNull(storageService.getUploadInfo(info.getId())); } + + @Test + public void testPruneOrphanedChecksumIndicesMissingData() throws Exception { + storageService.setUploadDeduplicationEnabled(true); + String sha1Hex = "0123456789abcdef0123456789abcdef01234567"; + String parentKey = "checksums/sha1/" + sha1Hex + "/parent"; + byte[] parentContent = "orphan-parent-id".getBytes(StandardCharsets.UTF_8); + + // Create an orphaned checksum index pointing to a non-existent data object in S3 + putObject(parentKey, parentContent); + assertTrue(objectExists(parentKey)); + + // Running cleanupExpiredUploads should detect the missing parent data and prune the index + storageService.cleanupExpiredUploads(); + + // Verify the orphaned parent checksum pointer is deleted from S3 + assertFalse(objectExists(parentKey)); + } + + @Test + public void testPruneOrphanedChecksumIndicesAbandonedTombstone() throws Exception { + storageService.setUploadDeduplicationEnabled(true); + String sha1Hex = "abcdef0123456789abcdef0123456789abcdef01"; + String parentKey = "checksums/sha1/" + sha1Hex + "/parent"; + String parentDeletedKey = "checksums/sha1/" + sha1Hex + "/parent-deleted"; + byte[] parentContent = "tombstone-parent-id".getBytes(StandardCharsets.UTF_8); + + // Create parent and parent-deleted tombstone marker with zero children in S3 + putObject(parentKey, parentContent); + putObject(parentDeletedKey, new byte[0]); + + assertTrue(objectExists(parentKey)); + assertTrue(objectExists(parentDeletedKey)); + + // Running cleanupExpiredUploads should detect the abandoned tombstone and prune both markers + storageService.cleanupExpiredUploads(); + + // Verify both the parent pointer and tombstone marker are pruned from S3 + assertFalse(objectExists(parentKey)); + assertFalse(objectExists(parentDeletedKey)); + } + + private void putObject(String objectKey, byte[] data) throws Exception { + minioClient.putObject( + PutObjectArgs.builder().bucket(BUCKET).object(objectKey).stream( + new ByteArrayInputStream(data), (long) data.length, -1L) + .build()); + } + + private boolean objectExists(String objectKey) { + try { + minioClient.statObject(StatObjectArgs.builder().bucket(BUCKET).object(objectKey).build()); + return true; + } catch (Exception e) { + return false; + } + } } diff --git a/src/test/java/me/desair/tus/server/upload/s3/ITS3TusFileUploadService.java b/src/test/java/me/desair/tus/server/upload/s3/ITS3TusFileUploadService.java index 6d8da536..2d3b017f 100644 --- a/src/test/java/me/desair/tus/server/upload/s3/ITS3TusFileUploadService.java +++ b/src/test/java/me/desair/tus/server/upload/s3/ITS3TusFileUploadService.java @@ -1,12 +1,27 @@ package me.desair.tus.server.upload.s3; +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertFalse; +import static org.junit.Assert.assertNotNull; + +import io.minio.ListObjectsArgs; import io.minio.MinioClient; +import io.minio.RemoveObjectArgs; +import io.minio.Result; +import io.minio.StatObjectArgs; +import io.minio.messages.Item; +import jakarta.servlet.http.HttpServletResponse; +import java.nio.charset.StandardCharsets; import me.desair.tus.server.AbstractITTusFileUploadService; +import me.desair.tus.server.HttpHeader; import me.desair.tus.server.ProtocolVersion; import me.desair.tus.server.TestUtils; import me.desair.tus.server.TusFileUploadService; +import me.desair.tus.server.upload.UploadId; +import me.desair.tus.server.upload.UploadInfo; import org.junit.AfterClass; import org.junit.BeforeClass; +import org.junit.Test; import org.testcontainers.containers.GenericContainer; /** @@ -68,4 +83,107 @@ protected TusFileUploadService createTusFileUploadService(String uploadUri) { .withDownloadFeature() .withChunkedTransferDecoding(true); } + + /** + * S3 Integration Test: Verifies that during the end-to-end Tus protocol lifecycle, the physical + * checksum index objects in S3 are created and properly purged on final deletion. + */ + @Test + public void testPhysicalChecksumDirectoryCreationAndCleanupInS3() throws Exception { + tusFileUploadService.withUploadDeduplication(true); + + String content = "Physical S3 Checksum Verification Payload"; + byte[] contentBytes = content.getBytes(StandardCharsets.UTF_8); + byte[] sha1Digest = org.apache.commons.codec.digest.DigestUtils.sha1(contentBytes); + String sha1Hex = org.apache.commons.codec.binary.Hex.encodeHexString(sha1Digest); + String sha1Base64 = org.apache.commons.codec.binary.Base64.encodeBase64String(sha1Digest); + + String parentKey = "checksums/sha1/" + sha1Hex + "/parent"; + String parentDeletedKey = "checksums/sha1/" + sha1Hex + "/parent-deleted"; + + // 1. Create parent upload + reset(); + servletRequest.setMethod("POST"); + servletRequest.setRequestURI(UPLOAD_URI); + servletRequest.addHeader(HttpHeader.UPLOAD_LENGTH, contentBytes.length); + servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + assertResponseStatus(HttpServletResponse.SC_CREATED); + String parentUri = servletResponse.getHeader(HttpHeader.LOCATION); + + // Complete parent upload via PATCH with checksum + reset(); + servletRequest.setMethod("PATCH"); + servletRequest.setRequestURI(parentUri); + servletRequest.addHeader(HttpHeader.CONTENT_TYPE, "application/offset+octet-stream"); + servletRequest.addHeader(HttpHeader.UPLOAD_OFFSET, 0); + servletRequest.addHeader(HttpHeader.UPLOAD_CHECKSUM, "sha1 " + sha1Base64); + servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); + servletRequest.setContent(contentBytes); + UploadInfo parentInfo = + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + assertResponseStatus(HttpServletResponse.SC_NO_CONTENT); + + // Assert physical parent pointer exists in S3 + assertNotNull( + minioClient.statObject(StatObjectArgs.builder().bucket(BUCKET).object(parentKey).build())); + + // 2. Create duplicate child upload + reset(); + servletRequest.setMethod("POST"); + servletRequest.setRequestURI(UPLOAD_URI); + servletRequest.addHeader(HttpHeader.UPLOAD_LENGTH, contentBytes.length); + servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + assertResponseStatus(HttpServletResponse.SC_CREATED); + String childUri = servletResponse.getHeader(HttpHeader.LOCATION); + + // Complete child upload via PATCH with same checksum + reset(); + servletRequest.setMethod("PATCH"); + servletRequest.setRequestURI(childUri); + servletRequest.addHeader(HttpHeader.CONTENT_TYPE, "application/offset+octet-stream"); + servletRequest.addHeader(HttpHeader.UPLOAD_OFFSET, 0); + servletRequest.addHeader(HttpHeader.UPLOAD_CHECKSUM, "sha1 " + sha1Base64); + servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); + servletRequest.setContent(contentBytes); + UploadInfo childInfo = tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + assertResponseStatus(HttpServletResponse.SC_NO_CONTENT); + assertEquals(parentInfo.getId(), childInfo.getDuplicatesUploadId()); + + String childKey = "checksums/sha1/" + sha1Hex + "/children/" + childInfo.getId(); + assertNotNull( + minioClient.statObject(StatObjectArgs.builder().bucket(BUCKET).object(childKey).build())); + + // 3. Delete parent upload via DELETE + reset(); + servletRequest.setMethod("DELETE"); + servletRequest.setRequestURI(parentUri); + servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + assertResponseStatus(HttpServletResponse.SC_NO_CONTENT); + + assertNotNull( + minioClient.statObject( + StatObjectArgs.builder().bucket(BUCKET).object(parentDeletedKey).build())); + + // 4. Delete child upload via DELETE (last child out) + reset(); + servletRequest.setMethod("DELETE"); + servletRequest.setRequestURI(childUri); + servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + assertResponseStatus(HttpServletResponse.SC_NO_CONTENT); + + Iterable> remaining = + minioClient.listObjects( + ListObjectsArgs.builder().bucket(BUCKET).prefix("checksums/sha1/" + sha1Hex).build()); + assertFalse(remaining.iterator().hasNext()); + } + + @Override + protected void deleteUploadedBytesOutOfBand(UploadId uploadId) throws Exception { + minioClient.removeObject( + RemoveObjectArgs.builder().bucket(BUCKET).object("uploads/" + uploadId).build()); + } } diff --git a/src/test/java/me/desair/tus/server/upload/s3/S3ChecksumIndexTest.java b/src/test/java/me/desair/tus/server/upload/s3/S3ChecksumIndexTest.java new file mode 100644 index 00000000..ab345790 --- /dev/null +++ b/src/test/java/me/desair/tus/server/upload/s3/S3ChecksumIndexTest.java @@ -0,0 +1,328 @@ +package me.desair.tus.server.upload.s3; + +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertFalse; +import static org.junit.Assert.assertNotNull; +import static org.junit.Assert.assertNull; +import static org.junit.Assert.assertTrue; +import static org.mockito.ArgumentMatchers.any; +import static org.mockito.Mockito.mock; +import static org.mockito.Mockito.never; +import static org.mockito.Mockito.verify; +import static org.mockito.Mockito.when; + +import io.minio.GetObjectArgs; +import io.minio.GetObjectResponse; +import io.minio.ListObjectsArgs; +import io.minio.MinioClient; +import io.minio.PutObjectArgs; +import io.minio.RemoveObjectArgs; +import io.minio.RemoveObjectsArgs; +import io.minio.Result; +import io.minio.StatObjectArgs; +import io.minio.StatObjectResponse; +import io.minio.errors.ErrorResponseException; +import io.minio.messages.ErrorResponse; +import io.minio.messages.Item; +import java.io.ByteArrayInputStream; +import java.io.IOException; +import java.nio.charset.StandardCharsets; +import java.util.Arrays; +import java.util.Collections; +import java.util.Set; +import me.desair.tus.server.checksum.ChecksumAlgorithm; +import me.desair.tus.server.checksum.ChecksumKey; +import me.desair.tus.server.upload.UploadId; +import okhttp3.Headers; +import org.junit.Before; +import org.junit.Test; + +/** Offline unit tests for {@link S3ChecksumIndex}. */ +public class S3ChecksumIndexTest { + + private MinioClient minioClient; + private S3ChecksumIndex checksumIndex; + private ChecksumKey key; + private UploadId uploadId; + + @Before + public void setUp() { + minioClient = mock(MinioClient.class); + checksumIndex = new S3ChecksumIndex(minioClient, "test-bucket"); + key = ChecksumKey.of("0123456789abcdef", ChecksumAlgorithm.SHA256); + uploadId = new UploadId("test-parent-id"); + } + + @Test + public void testConstructorAndSanitizePrefix() { + S3ChecksumIndex index1 = new S3ChecksumIndex(minioClient, "bucket"); + assertNotNull(index1); + + S3ChecksumIndex index2 = new S3ChecksumIndex(minioClient, "bucket", "custom/"); + assertNotNull(index2); + + S3ChecksumIndex index3 = new S3ChecksumIndex(minioClient, "bucket", "/custom///"); + assertNotNull(index3); + + S3ChecksumIndex index4 = new S3ChecksumIndex(minioClient, "bucket", ""); + assertNotNull(index4); + } + + private ErrorResponseException createError(String code) { + return new ErrorResponseException( + new ErrorResponse(code, "Error message", null, null, null, null, null), null, null); + } + + @Test + public void testTryRegisterParentSuccess() throws Exception { + when(minioClient.statObject(any(StatObjectArgs.class))).thenThrow(createError("NoSuchKey")); + + boolean registered = checksumIndex.tryRegisterParent(key, uploadId); + assertTrue(registered); + verify(minioClient).putObject(any(PutObjectArgs.class)); + } + + @Test + public void testTryRegisterParentWhenTombstoned() throws Exception { + when(minioClient.putObject(any(PutObjectArgs.class))) + .thenThrow(createError("PreconditionFailed")); + + boolean registered = checksumIndex.tryRegisterParent(key, uploadId); + assertFalse(registered); + } + + @Test + public void testTryRegisterParentPreconditionFailedCollision() throws Exception { + when(minioClient.statObject(any(StatObjectArgs.class))).thenThrow(createError("NoSuchKey")); + + when(minioClient.putObject(any(PutObjectArgs.class))) + .thenThrow(createError("PreconditionFailed")); + + boolean registered = checksumIndex.tryRegisterParent(key, uploadId); + assertFalse(registered); + } + + @Test + public void testFindParentSuccess() throws Exception { + byte[] data = "found-parent-id\n".getBytes(StandardCharsets.UTF_8); + GetObjectResponse response = + new GetObjectResponse( + Headers.of("Content-Type", "text/plain"), + "test-bucket", + "us-east-1", + "object-name", + new ByteArrayInputStream(data)); + when(minioClient.getObject(any(GetObjectArgs.class))).thenReturn(response); + + UploadId found = checksumIndex.findParent(key); + assertNotNull(found); + assertEquals("found-parent-id", found.toString()); + } + + @Test + public void testFindParentNotFound() throws Exception { + when(minioClient.getObject(any(GetObjectArgs.class))).thenThrow(createError("NoSuchKey")); + + UploadId found = checksumIndex.findParent(key); + assertNull(found); + } + + @Test + public void testMarkParentDeletedAndIsParentDeleted() throws Exception { + checksumIndex.markParentDeleted(key); + verify(minioClient).putObject(any(PutObjectArgs.class)); + // Parent marker object must NOT be deleted during tombstoning + verify(minioClient, never()).removeObject(any(RemoveObjectArgs.class)); + + StatObjectResponse statResponse = mock(StatObjectResponse.class); + when(minioClient.statObject(any(StatObjectArgs.class))).thenReturn(statResponse); + assertTrue(checksumIndex.isParentDeleted(key)); + } + + @Test + public void testAddAndRemoveChild() throws Exception { + UploadId childId = new UploadId("child-id-1"); + checksumIndex.addChild(key, childId); + verify(minioClient).putObject(any(PutObjectArgs.class)); + + checksumIndex.removeChild(key, childId); + verify(minioClient).removeObject(any(RemoveObjectArgs.class)); + } + + @Test + public void testHasChildren() throws Exception { + Item item = mock(Item.class); + Result result = new Result<>(item); + when(minioClient.listObjects(any(ListObjectsArgs.class))) + .thenReturn(Collections.singletonList(result)); + + assertTrue(checksumIndex.hasChildren(key)); + + when(minioClient.listObjects(any(ListObjectsArgs.class))).thenReturn(Collections.emptyList()); + assertFalse(checksumIndex.hasChildren(key)); + } + + @Test + public void testDeleteIndex() throws Exception { + Item item = mock(Item.class); + when(item.objectName()).thenReturn("checksums/sha256/0123456789abcdef/parent"); + Result result = new Result<>(item); + when(minioClient.listObjects(any(ListObjectsArgs.class))) + .thenReturn(Collections.singletonList(result)); + + checksumIndex.deleteIndex(key); + verify(minioClient).removeObjects(any(RemoveObjectsArgs.class)); + } + + @Test + public void testTryRegisterParentApiNotImplementedFallback() throws Exception { + when(minioClient.statObject(any(StatObjectArgs.class))).thenThrow(createError("NoSuchKey")); + when(minioClient.putObject(any(PutObjectArgs.class))) + .thenThrow(createError("NotImplemented")) + .thenReturn(null); + + boolean registered = checksumIndex.tryRegisterParent(key, uploadId); + assertTrue(registered); + } + + @Test + public void testTryRegisterParentNonCasModeObjectExists() throws Exception { + // Disable conditional writes + checksumIndex.setS3ConditionalWritesSupported(false); + + StatObjectResponse statResponse = mock(StatObjectResponse.class); + when(minioClient.statObject(any(StatObjectArgs.class))).thenReturn(statResponse); + + boolean registered = checksumIndex.tryRegisterParent(key, uploadId); + assertFalse(registered); + } + + @Test + public void testTryRegisterParentNonCasModeSuccess() throws Exception { + checksumIndex.setS3ConditionalWritesSupported(false); + + when(minioClient.statObject(any(StatObjectArgs.class))).thenThrow(createError("NoSuchKey")); + + boolean registered = checksumIndex.tryRegisterParent(key, uploadId); + assertTrue(registered); + verify(minioClient).putObject(any(PutObjectArgs.class)); + } + + @Test(expected = java.io.IOException.class) + public void testHasChildrenThrowsOnException() throws Exception { + when(minioClient.listObjects(any(ListObjectsArgs.class))) + .thenThrow(new RuntimeException("Simulated S3 error")); + checksumIndex.hasChildren(key); + } + + @Test + public void testDeleteIndexWithErrorsLogged() throws Exception { + Item item = mock(Item.class); + when(item.objectName()).thenReturn("checksums/sha256/0123456789abcdef/parent"); + Result result = new Result<>(item); + when(minioClient.listObjects(any(ListObjectsArgs.class))) + .thenReturn(Collections.singletonList(result)); + + io.minio.messages.DeleteResult.Error s3Error = mock(io.minio.messages.DeleteResult.Error.class); + when(s3Error.objectName()).thenReturn("parent"); + when(s3Error.message()).thenReturn("Access Denied"); + Result errorResult = new Result<>(s3Error); + + when(minioClient.removeObjects(any(RemoveObjectsArgs.class))) + .thenReturn(Collections.singletonList(errorResult)); + + checksumIndex.deleteIndex(key); + verify(minioClient).removeObjects(any(RemoveObjectsArgs.class)); + } + + @Test(expected = java.io.IOException.class) + public void testDeleteIndexThrowsOnException() throws Exception { + when(minioClient.listObjects(any(ListObjectsArgs.class))) + .thenThrow(new RuntimeException("Simulated error")); + checksumIndex.deleteIndex(key); + } + + @Test + public void testGetAllIndexedKeysSuccess() throws Exception { + Item item1 = mock(Item.class); + when(item1.objectName()).thenReturn("checksums/sha256/0123456789abcdef0123456789abcdef/parent"); + Result result1 = new Result<>(item1); + + when(minioClient.listObjects(any(ListObjectsArgs.class))) + .thenReturn(Collections.singletonList(result1)); + + Set keys = checksumIndex.getAllIndexedKeys(); + assertEquals(1, keys.size()); + assertTrue( + keys.contains( + ChecksumKey.of("0123456789abcdef0123456789abcdef", ChecksumAlgorithm.SHA256))); + } + + @Test + public void testGetAllIndexedKeysWithMalformedAndNonParentObjects() throws Exception { + Item itemNull = mock(Item.class); + when(itemNull.objectName()).thenReturn(null); + + Item itemNotParent = mock(Item.class); + when(itemNotParent.objectName()).thenReturn("checksums/sha256/hash/child-1"); + + Item itemInvalidAlgo = mock(Item.class); + when(itemInvalidAlgo.objectName()).thenReturn("checksums/unknown-algo/hash/parent"); + + Item itemMalformedParts = mock(Item.class); + when(itemMalformedParts.objectName()).thenReturn("checksums/too/many/parts/parent"); + + Item itemInvalidHash = mock(Item.class); + when(itemInvalidHash.objectName()).thenReturn("checksums/sha1/invalid_hex_!@#$/parent"); + + when(minioClient.listObjects(any(ListObjectsArgs.class))) + .thenReturn( + Arrays.asList( + new Result<>(itemNull), + new Result<>(itemNotParent), + new Result<>(itemInvalidAlgo), + new Result<>(itemMalformedParts), + new Result<>(itemInvalidHash))); + + Set keys = checksumIndex.getAllIndexedKeys(); + assertTrue(keys.isEmpty()); + } + + @Test(expected = java.io.IOException.class) + public void testGetAllIndexedKeysThrowsIOExceptionOnS3Error() throws Exception { + when(minioClient.listObjects(any(ListObjectsArgs.class))) + .thenThrow(new RuntimeException("Simulated error")); + checksumIndex.getAllIndexedKeys(); + } + + @Test(expected = IOException.class) + public void testTryRegisterParentOtherErrorResponseThrowsIOException() throws Exception { + when(minioClient.putObject(any(PutObjectArgs.class))).thenThrow(createError("InternalError")); + + checksumIndex.tryRegisterParent(key, uploadId); + } + + @Test(expected = IOException.class) + public void testTryRegisterParentGenericExceptionThrowsIOException() throws Exception { + when(minioClient.putObject(any(PutObjectArgs.class))) + .thenThrow(new RuntimeException("Simulated error")); + + checksumIndex.tryRegisterParent(key, uploadId); + } + + @Test(expected = IOException.class) + public void testMarkParentDeletedExceptionThrowsIOException() throws Exception { + when(minioClient.putObject(any(PutObjectArgs.class))) + .thenThrow(new RuntimeException("Simulated error")); + + checksumIndex.markParentDeleted(key); + } + + @Test(expected = IOException.class) + public void testHasChildrenExceptionThrowsIOException() throws Exception { + when(minioClient.listObjects(any(ListObjectsArgs.class))) + .thenThrow(new RuntimeException("Simulated error")); + + checksumIndex.hasChildren(key); + } +} diff --git a/src/test/java/me/desair/tus/server/upload/s3/S3StorageServiceTest.java b/src/test/java/me/desair/tus/server/upload/s3/S3StorageServiceTest.java index b2a5b259..905dfc84 100644 --- a/src/test/java/me/desair/tus/server/upload/s3/S3StorageServiceTest.java +++ b/src/test/java/me/desair/tus/server/upload/s3/S3StorageServiceTest.java @@ -33,13 +33,17 @@ import java.io.ByteArrayOutputStream; import java.io.IOException; import java.io.InputStream; +import java.nio.charset.StandardCharsets; import java.nio.file.Paths; import java.time.Duration; import java.util.ArrayList; import java.util.Arrays; import java.util.Collections; +import java.util.HashSet; import java.util.List; +import java.util.Set; import me.desair.tus.server.checksum.ChecksumAlgorithm; +import me.desair.tus.server.checksum.ChecksumKey; import me.desair.tus.server.exception.MaxAppendSizeExceededException; import me.desair.tus.server.exception.MinUploadLengthNotReachedException; import me.desair.tus.server.exception.UploadNotFoundException; @@ -189,6 +193,15 @@ public void testGetUploadedBytesNotFound() throws Exception { storageService.getUploadedBytes(new UploadId("missing-123")); } + @Test + public void testWithS3ConditionalWritesSupported() { + assertTrue(storageService.isS3ConditionalWritesSupported()); + storageService.withS3ConditionalWritesSupported(false); + assertFalse(storageService.isS3ConditionalWritesSupported()); + storageService.setS3ConditionalWritesSupported(true); + assertTrue(storageService.isS3ConditionalWritesSupported()); + } + @Test public void testGetUploadedBytesDuplicate() throws Exception { UploadInfo child = new UploadInfo(); @@ -763,8 +776,8 @@ public void testDeduplicationChecksumLookup() throws Exception { String json = UploadInfoJsonSerializer.serialize(parentInfo); java.util.Map objectData = new java.util.HashMap<>(); - objectData.put("checksums/sha256/abc123hash", "parent-123".getBytes()); - objectData.put("uploads/checksums/sha256/abc123hash", "parent-123".getBytes()); + objectData.put("checksums/sha256/abc123hash/parent", "parent-123".getBytes()); + objectData.put("metadata/uploads/parent-123.info", json.getBytes()); objectData.put("uploads/parent-123.info", json.getBytes()); when(minioClient.getObject(any(GetObjectArgs.class))) @@ -787,6 +800,149 @@ public void testDeduplicationChecksumLookup() throws Exception { assertEquals(new UploadId("parent-123"), match.getId()); } + /** + * Verifies that when deduplication is enabled on S3StorageService: 1. A completed upload with a + * checksum creates the S3 object 'checksums///parent'. 2. A deduplicated child upload + * creates 'checksums///children/'. 3. Terminating the parent while the child + * is active writes 'parent-deleted' and preserves parent data. 4. Terminating the last child + * deletes the parent data object and all checksum index objects. + */ + @Test + public void testChecksumDirectoryCreationAndCleanupInS3() throws Exception { + storageService.setUploadDeduplicationEnabled(true); + UploadLockingService lockingService = mock(UploadLockingService.class); + storageService.setUploadLockingService(lockingService); + + String sha1Hex = "0123456789abcdef0123456789abcdef01234567"; + String parentKey = "checksums/sha1/" + sha1Hex + "/parent"; + String parentDeletedKey = "checksums/sha1/" + sha1Hex + "/parent-deleted"; + String childPrefix = "checksums/sha1/" + sha1Hex + "/children/"; + + Set s3Objects = new HashSet<>(); + + when(minioClient.putObject(any(PutObjectArgs.class))) + .thenAnswer( + invocation -> { + PutObjectArgs args = invocation.getArgument(0); + s3Objects.add(args.object()); + return null; + }); + + when(minioClient.statObject(any(StatObjectArgs.class))) + .thenAnswer( + invocation -> { + StatObjectArgs args = invocation.getArgument(0); + if (s3Objects.contains(args.object())) { + return mock(StatObjectResponse.class); + } + ErrorResponse errorResponse = mock(ErrorResponse.class); + when(errorResponse.code()).thenReturn("NoSuchKey"); + throw new ErrorResponseException(errorResponse, null, null); + }); + + when(minioClient.getObject(any(GetObjectArgs.class))) + .thenAnswer( + invocation -> { + GetObjectArgs args = invocation.getArgument(0); + if (!s3Objects.contains(args.object())) { + ErrorResponse errorResponse = mock(ErrorResponse.class); + when(errorResponse.code()).thenReturn("NoSuchKey"); + throw new ErrorResponseException(errorResponse, null, null); + } + if (args.object().equals(parentKey)) { + return mockGetObjectResponse("parent-123".getBytes(StandardCharsets.UTF_8)); + } + if (args.object().endsWith(".info")) { + UploadInfo info = new UploadInfo(); + info.setId(new UploadId("parent-123")); + info.setLength(100L); + info.setOffset(100L); + info.setChecksum(sha1Hex); + info.setChecksumAlgorithm(ChecksumAlgorithm.SHA1); + return mockGetObjectResponse( + UploadInfoJsonSerializer.serialize(info).getBytes(StandardCharsets.UTF_8)); + } + return mockGetObjectResponse(new byte[0]); + }); + + when(minioClient.listObjects(any(ListObjectsArgs.class))) + .thenAnswer( + invocation -> { + ListObjectsArgs args = invocation.getArgument(0); + List> items = new ArrayList<>(); + for (String obj : s3Objects) { + if (obj.startsWith(args.prefix())) { + Item item = mock(Item.class); + when(item.objectName()).thenReturn(obj); + items.add(new Result<>(item)); + } + } + return items; + }); + + doAnswer( + invocation -> { + RemoveObjectArgs args = invocation.getArgument(0); + s3Objects.remove(args.object()); + return null; + }) + .when(minioClient) + .removeObject(any(RemoveObjectArgs.class)); + + doAnswer( + invocation -> { + s3Objects.removeIf(key -> key.startsWith("checksums/sha1/" + sha1Hex)); + return Collections.emptyList(); + }) + .when(minioClient) + .removeObjects(any(RemoveObjectsArgs.class)); + + // 1. Parent upload completion + UploadInfo parent = new UploadInfo(); + parent.setId(new UploadId("parent-123")); + parent.setLength(100L); + parent.setOffset(100L); + parent.setChecksum(sha1Hex); + parent.setChecksumAlgorithm(ChecksumAlgorithm.SHA1); + s3Objects.add("uploads/parent-123"); + + storageService.update(parent); + + assertTrue("Parent pointer object must exist in S3", s3Objects.contains(parentKey)); + assertFalse( + "Parent-deleted marker must not exist initially", s3Objects.contains(parentDeletedKey)); + + // 2. Child duplicate upload + UploadId childId = new UploadId("child-456"); + UploadInfo child = new UploadInfo(); + child.setId(childId); + child.setLength(100L); + child.setOffset(100L); + child.setChecksum(sha1Hex); + child.setChecksumAlgorithm(ChecksumAlgorithm.SHA1); + + storageService.deduplicate(child, ChecksumKey.of(sha1Hex, ChecksumAlgorithm.SHA1)); + + String childKey = childPrefix + childId; + assertTrue( + "Child marker object must exist in S3 under children/", s3Objects.contains(childKey)); + assertEquals(parent.getId(), child.getDuplicatesUploadId()); + + // 3. Parent termination with active child + storageService.terminateUpload(parent); + + assertTrue( + "Parent-deleted tombstone marker must exist in S3", s3Objects.contains(parentDeletedKey)); + + // 4. Child termination (last child out) + storageService.terminateUpload(child); + + assertFalse("Parent pointer object must be removed from S3", s3Objects.contains(parentKey)); + assertFalse( + "Parent-deleted marker must be removed from S3", s3Objects.contains(parentDeletedKey)); + assertFalse("Child marker must be removed from S3", s3Objects.contains(childKey)); + } + @Test public void testConfigurationSettersAndGetters() { storageService.setMaxUploadSize(5000L); @@ -950,6 +1106,34 @@ public void testCleanupExpiredUploads() throws Exception { verify(mockLocking).cleanupLock(expiredId); } + @Test + public void testCleanupExpiredUploadsNoArg() throws Exception { + UploadId expiredId = new UploadId("exp-s3-no-arg"); + UploadInfo expiredInfo = new UploadInfo(); + expiredInfo.setId(expiredId); + expiredInfo.setOffset(5L); + expiredInfo.setLength(10L); + expiredInfo.setExpirationTimestamp(System.currentTimeMillis() - 10000L); + + Item item = mock(Item.class); + when(item.objectName()).thenReturn("metadata/" + expiredId + ".info"); + Result resultItem = new Result<>(item); + + when(minioClient.listObjects(any(ListObjectsArgs.class))) + .thenReturn(Collections.singletonList(resultItem)); + + String json = UploadInfoJsonSerializer.serialize(expiredInfo); + when(minioClient.getObject(any(GetObjectArgs.class))) + .thenAnswer(invocation -> mockGetObjectResponse(json.getBytes())); + + UploadLockingService mockLocking = mock(UploadLockingService.class); + when(mockLocking.isLocked(expiredId)).thenReturn(false); + storageService.setUploadLockingService(mockLocking); + + storageService.cleanupExpiredUploads(); + verify(mockLocking).cleanupLock(expiredId); + } + @Test(expected = IOException.class) public void testCleanupExpiredUploadsThrowsIOExceptionOnMinioFailure() throws Exception { when(minioClient.listObjects(any(ListObjectsArgs.class))) @@ -2689,6 +2873,135 @@ public void testPruneOrphanedChecksumIndicesListObjectsExceptionHandled() throws assertTrue(true); } + @Test + public void testPruneOrphanedChecksumIndicesMissingData() throws Exception { + storageService.setUploadDeduplicationEnabled(true); + String hex = "0123456789abcdef0123456789abcdef01234567"; + + Item item = mock(Item.class); + when(item.objectName()).thenReturn("checksums/sha1/" + hex + "/parent"); + Result result = new Result<>(item); + + when(minioClient.listObjects(any(ListObjectsArgs.class))) + .thenAnswer( + invocation -> { + ListObjectsArgs args = invocation.getArgument(0); + String prefix = args.prefix(); + if (prefix.endsWith("/children/")) { + return Collections.emptyList(); + } + if ("checksums/".equals(prefix) || prefix.startsWith("checksums/sha1/" + hex)) { + return Collections.singletonList(result); + } + return Collections.emptyList(); + }); + + when(minioClient.removeObjects(any(RemoveObjectsArgs.class))) + .thenReturn(Collections.emptyList()); + + when(minioClient.getObject(any(GetObjectArgs.class))) + .thenAnswer( + invocation -> { + GetObjectArgs args = invocation.getArgument(0); + if (("checksums/sha1/" + hex + "/parent").equals(args.object())) { + return mockGetObjectResponse("parent-123\n".getBytes(StandardCharsets.UTF_8)); + } + ErrorResponse notFound = mock(ErrorResponse.class); + when(notFound.code()).thenReturn("NoSuchKey"); + throw new ErrorResponseException(notFound, null, null); + }); + + ErrorResponse notFound = mock(ErrorResponse.class); + when(notFound.code()).thenReturn("NoSuchKey"); + ErrorResponseException notFoundEx = new ErrorResponseException(notFound, null, null); + + when(minioClient.statObject(any(StatObjectArgs.class))).thenThrow(notFoundEx); + + storageService.cleanupExpiredUploads(null); + + verify(minioClient).removeObjects(any(RemoveObjectsArgs.class)); + verify(minioClient) + .removeObject( + argThat((RemoveObjectArgs a) -> a != null && a.object().equals("uploads/parent-123"))); + } + + @Test + public void testPruneOrphanedChecksumIndicesAbandonedTombstone() throws Exception { + storageService.setUploadDeduplicationEnabled(true); + String hex = "fedcba9876543210fedcba9876543210fedcba98"; + + Item item = mock(Item.class); + when(item.objectName()).thenReturn("checksums/sha1/" + hex + "/parent"); + Result result = new Result<>(item); + + when(minioClient.listObjects(any(ListObjectsArgs.class))) + .thenAnswer( + invocation -> { + ListObjectsArgs args = invocation.getArgument(0); + String prefix = args.prefix(); + if (prefix.endsWith("/children/")) { + return Collections.emptyList(); + } + if ("checksums/".equals(prefix) || prefix.startsWith("checksums/sha1/" + hex)) { + return Collections.singletonList(result); + } + return Collections.emptyList(); + }); + + when(minioClient.removeObjects(any(RemoveObjectsArgs.class))) + .thenReturn(Collections.emptyList()); + + when(minioClient.getObject(any(GetObjectArgs.class))) + .thenAnswer( + invocation -> { + GetObjectArgs args = invocation.getArgument(0); + if (("checksums/sha1/" + hex + "/parent").equals(args.object())) { + return mockGetObjectResponse("tombstone-parent\n".getBytes(StandardCharsets.UTF_8)); + } + ErrorResponse notFound = mock(ErrorResponse.class); + when(notFound.code()).thenReturn("NoSuchKey"); + throw new ErrorResponseException(notFound, null, null); + }); + + when(minioClient.statObject(any(StatObjectArgs.class))) + .thenAnswer( + invocation -> { + StatObjectArgs args = invocation.getArgument(0); + if ("uploads/tombstone-parent".equals(args.object()) + || ("checksums/sha1/" + hex + "/parent-deleted").equals(args.object())) { + return mock(StatObjectResponse.class); + } + ErrorResponse notFound = mock(ErrorResponse.class); + when(notFound.code()).thenReturn("NoSuchKey"); + throw new ErrorResponseException(notFound, null, null); + }); + + storageService.cleanupExpiredUploads(null); + + verify(minioClient).removeObjects(any(RemoveObjectsArgs.class)); + verify(minioClient) + .removeObject( + argThat( + (RemoveObjectArgs a) -> + a != null && a.object().equals("uploads/tombstone-parent"))); + } + + @Test + public void testPruneOrphanedChecksumIndicesMalformedPathIgnored() throws Exception { + storageService.setUploadDeduplicationEnabled(true); + + Item item = mock(Item.class); + when(item.objectName()).thenReturn("checksums/invalid/parent"); + Result result = new Result<>(item); + + when(minioClient.listObjects(any(ListObjectsArgs.class))) + .thenReturn(Collections.singletonList(result)); + + storageService.cleanupExpiredUploads(null); + // KISS: malformed path is skipped without error + assertTrue(true); + } + @Test public void testIsJsonSerializationEnabled() { assertTrue(storageService.isJsonSerializationEnabled()); @@ -2940,6 +3253,16 @@ public void testAppendDrainExceptionRecordedAndThrowsIOException() throws Except } } + @Test(expected = NullPointerException.class) + public void testSetUploadLockingServiceNullThrowsNpe() { + storageService.setUploadLockingService(null); + } + + @Test + public void testGetUploadLockingServiceReturnsConfiguredInstance() { + assertNotNull(storageService.getUploadLockingService()); + } + private GetObjectResponse mockGetObjectResponse(byte[] bytes) { return new GetObjectResponse( null, "test-bucket", "eu-central-1", "object-key", new ByteArrayInputStream(bytes)); diff --git a/src/test/java/me/desair/tus/server/util/UtilsTest.java b/src/test/java/me/desair/tus/server/util/UtilsTest.java index a8d36f72..5e9c0aaf 100644 --- a/src/test/java/me/desair/tus/server/util/UtilsTest.java +++ b/src/test/java/me/desair/tus/server/util/UtilsTest.java @@ -23,6 +23,9 @@ import java.util.UUID; import me.desair.tus.server.HttpHeader; import me.desair.tus.server.checksum.ChecksumAlgorithm; +import me.desair.tus.server.upload.NoOpUploadLockingService; +import me.desair.tus.server.upload.UploadLockingService; +import me.desair.tus.server.upload.UploadStorageService; import org.apache.commons.io.FileUtils; import org.junit.AfterClass; import org.junit.BeforeClass; @@ -986,6 +989,48 @@ public void testToInterruptibleStream() throws Exception { assertThat(same, is(sameInstance(wrapped))); } + @Test + public void testGetLockingServiceOrDefault() { + UploadLockingService activeLockingService = mock(UploadLockingService.class); + UploadStorageService storageService = mock(UploadStorageService.class); + + // Single-arg: UploadLockingService + assertThat( + Utils.getLockingServiceOrDefault((UploadLockingService) null) + instanceof NoOpUploadLockingService, + is(true)); + assertThat( + Utils.getLockingServiceOrDefault(activeLockingService), + is(sameInstance(activeLockingService))); + + // Single-arg: UploadStorageService + assertThat( + Utils.getLockingServiceOrDefault((UploadStorageService) null) + instanceof NoOpUploadLockingService, + is(true)); + when(storageService.getUploadLockingService()).thenReturn(null); + assertThat( + Utils.getLockingServiceOrDefault(storageService) instanceof NoOpUploadLockingService, + is(true)); + when(storageService.getUploadLockingService()).thenReturn(activeLockingService); + assertThat( + Utils.getLockingServiceOrDefault(storageService), is(sameInstance(activeLockingService))); + + // Two-arg: UploadLockingService and UploadStorageService + assertThat( + Utils.getLockingServiceOrDefault(activeLockingService, storageService), + is(sameInstance(activeLockingService))); + assertThat( + Utils.getLockingServiceOrDefault(null, storageService), + is(sameInstance(activeLockingService))); + when(storageService.getUploadLockingService()).thenReturn(null); + assertThat( + Utils.getLockingServiceOrDefault(null, storageService) instanceof NoOpUploadLockingService, + is(true)); + assertThat( + Utils.getLockingServiceOrDefault(null, null) instanceof NoOpUploadLockingService, is(true)); + } + /** Simple serializable class for testing. */ public static class TestSerializable implements Serializable { private static final long serialVersionUID = 1L;