feat: upload deduplication, mandatory locking service, and lock cleanup consolidation

AGENTS.md

@@ -190,6 +190,11 @@ To maintain a clear separation between fast, offline unit tests and containerize
 - Use `thisObjectLock.readLock()` for fast, concurrent reads, and `thisObjectLock.writeLock()` for state mutations, lazy pattern compilation, or feature capability degradation.
 - Always acquire and release locks using `try ... finally` blocks (e.g. `thisObjectLock.readLock().lock(); try { return field; } finally { thisObjectLock.readLock().unlock(); }`).
 
+### 24. UploadLock AutoCloseable Lifecycle & Try-With-Resources
+- `UploadLock` implements `java.lang.AutoCloseable`.
+- Whenever acquiring an `UploadLock` (e.g. via `lockingService.lockUpload(uploadId)` or `lockingService.lockUploadByUri(uri)`), always manage the lock lifecycle using a `try-with-resources` block (e.g. `try (UploadLock lock = lockingService.lockUpload(id)) { ... }`).
+- This guarantees timely and reliable lock release upon normal completion or abrupt exceptions without requiring verbose and error-prone `try ... finally { if (lock != null) lock.release(); }` boilerplates.
+
 
 ## IETF Resumable Uploads for HTTP (RUFH) Spec Maintenance & Update Playbook
 

CHANGELOG.md

@@ -21,6 +21,7 @@ See [docs/UPGRADING.md](docs/UPGRADING.md) for details and code examples.
 - **Download extension (breaking)**: no longer returns `204` for uploads still in progress and no longer sets the `Tus-Resumable` header (needed for dual-protocol support). The `UploadInProgressException` class was removed.
 - **Jackson bundled**: `jackson-databind`, `jackson-annotations` and `jackson-core` are now compile-scope dependencies; no separate declaration needed.
 - **Protocol detection**: `ProtocolVersion.AUTO` is the default. Restrict with `withSupportedProtocolVersions(...)`, see [docs/PROTOCOLS.md](docs/PROTOCOLS.md).
+- **Checksum Deduplication Index**: The checksum index format has been updated to support reference tracking, parent tombstoning, and full cleanup upon upload deletion across Disk, S3, and Azure Blob storage. See [docs/DEDUPLICATION.md](docs/DEDUPLICATION.md) for migration instructions (clearing the index directory or running the migration script).
 
 ### New
 - **IETF Resumable Uploads for HTTP (RUFH)**: full draft-12 support with automatic protocol detection, RFC 9651 structured headers (`Upload-Offset`, `Upload-Complete`, `Upload-Length`, `Upload-Limit`), RFC 7807 `application/problem+json` errors, RFC 9530 HTTP digests, `104` interim responses and a spec-quoted conformity suite (`scripts/rufh_conformity_test.py`). See [docs/PROTOCOLS.md](docs/PROTOCOLS.md), [docs/MIGRATION.md](docs/MIGRATION.md), [docs/INTERIM_RESPONSES.md](docs/INTERIM_RESPONSES.md) and [docs/CONFORMITY_TESTING.md](docs/CONFORMITY_TESTING.md).
@@ -32,12 +33,17 @@ See [docs/UPGRADING.md](docs/UPGRADING.md) for details and code examples.
 - **Automatic lock cleanup** on completion, termination (`DELETE`) and expiration across all locking services (`cleanupLock(...)`), plus a fallback sweeper for abandoned locks.
 - **JSON serialization**: store `UploadInfo` as JSON with `withJsonSerialization(true)`.
 - **Absolute base URLs**: `withUploadUri(String)` accepts e.g. `https://upload.example.com/files`, returning full `Location` URLs for Tus and RUFH; relative paths keep working.
+- **Multi-Backend Upload Deduplication & Lifecycle Cleanup**: Content-based upload deduplication with `withUploadDeduplication(true)` across Disk, S3, and Azure Blob storage. Features child reference markers, parent tombstoning, and complete index directory cleanup when uploads expire or are deleted. See [docs/DEDUPLICATION.md](docs/DEDUPLICATION.md).
+- **`NoOpUploadLockingService`**: Zero-overhead Null Object implementation of `UploadLockingService` for unit tests and custom environments, making locking service usage mandatory and eliminating defensive null checks across all storage backends, request handlers, and deduplication logic.
 
 ### Fixed
 - **Content-Type matching**: media types are matched per RFC 9110 and tolerate parameters such as `;charset=UTF-8` (added by Spring `CharacterEncodingFilter`, proxies or clients) instead of returning `406`.
 - **Undertow & other containers**: `Content-Length` is cleared before `sendError(...)` ([#40](https://github.com/tomdesair/tus-java-server/issues/40)).
 - **Disk**: truncate underflow guard in `DiskStorageService`; `FileChannel` leak on lock acquisition errors fixed in `FileBasedLock`.
 
+### Deprecated
+- **`UploadStorageService.cleanupExpiredUploads(UploadLockingService)`**: Deprecated in favor of the zero-argument `cleanupExpiredUploads()`, as storage services coordinate locking through `setUploadLockingService(UploadLockingService)`.
+
 ### Security
 - **Path traversal protection**: hardened `LeaseFileLockingService` against directory traversal via crafted upload identifiers ([#141](https://github.com/tomdesair/tus-java-server/pull/141)).
 - **CORS origin validation**: prevented arbitrary `Origin` header reflection in CORS preflight responses ([#132](https://github.com/tomdesair/tus-java-server/pull/132)).

README.md

@@ -282,6 +282,7 @@ Detailed Tus and RUFH flow diagrams: [Protocols](docs/PROTOCOLS.md).
 | Tus vs. IETF RUFH, protocol version selection, extensions, HTTP digests, 104 responses | [Protocols](docs/PROTOCOLS.md), [Migrating clients to RUFH](docs/MIGRATION.md), [Interim responses](docs/INTERIM_RESPONSES.md) |
 | Multi-replica, Kubernetes, reverse proxies | [Deployment Guide](docs/DEPLOYMENT.md), [Locking](docs/LOCKING.md) |
 | Storage backends | [Disk & network (NFS/EFS/SMB)](docs/DISK_BASED_LOCKING.md), [S3-compatible](docs/S3_STORAGE.md), [Azure Blob](docs/AZURE_BLOB_STORAGE.md) |
+| Upload deduplication | [Upload Deduplication & Migration](docs/DEDUPLICATION.md) |
 | Problems | [Troubleshooting](docs/TROUBLESHOOTING.md) |
 | Upgrading from 1.x | [Upgrade Guide](docs/UPGRADING.md), [CHANGELOG](CHANGELOG.md) |
 

docs/ARCHITECTURE.md

@@ -177,13 +177,46 @@ For local filesystems and shared network volumes, `tus-java-server` provides two
 - **Upload Deduplication**: Enabled via `.withUploadDeduplication(true)`. Completed uploads are indexed by checksum. Duplicate uploads are linked via `duplicatesUploadId`. Read operations dynamically resolve to the parent upload, while write and truncation operations strictly operate on the target upload to prevent corrupting parent data.
 - **Thread-Local Request Caching (`ThreadLocalCachedStorageAndLockingService`)**: Enabled via `.withThreadLocalCache(true)`. Caches `UploadInfo` in memory for the duration of a single HTTP request to eliminate redundant network roundtrips to S3 or Azure Blob Storage.
 
+### Implementing a Custom Storage Backend
+
+Custom storage engines (e.g. Google Cloud Storage, relational database BLOB stores, or custom object systems) implement the [`UploadStorageService`](../src/main/java/me/desair/tus/server/upload/UploadStorageService.java) interface:
+
+1. **Upload Lifecycle**:
+   - `create(UploadInfo, String)`: Allocate initial storage resources and persist the upload metadata record.
+   - `append(UploadInfo, InputStream)`: Stream and append incoming payload chunks to storage. Persist partial progress if interrupted.
+   - `getUploadInfo(UploadId)` / `update(UploadInfo)`: Retrieve and update upload metadata.
+   - `getUploadedBytes(UploadId)` / `copyUploadTo(UploadInfo, OutputStream)`: Stream completed file bytes to consumers.
+2. **Resource Termination & Lock Cleanup Contract (`terminateUpload`)**:
+   `terminateUpload(UploadInfo)` is the single authoritative method responsible for freeing all resources when an upload is deleted, cancelled, or expired. Custom implementations must:
+   - Free physical committed data bytes and any temporary chunk buffers.
+   - Delete the metadata record (`.info`).
+   - Coordinate deduplication index removal or tombstoning via `deduplicationService.removeUploadFromIndex(uploadInfo)` when deduplication is active.
+   - **Delegate Lock Cleanup**: Call `uploadLockingService.cleanupLock(uploadInfo.getId())`, ensuring no orphaned lock files, stop signals, or lease objects remain in the locking layer.
+3. **Mandatory Locking Service**:
+   An `UploadLockingService` is mandatory across all storage backends. Storage services receive their `UploadLockingService` reference via `setUploadLockingService(UploadLockingService)` (enforcing non-null). For unit tests or specialized embedding environments where locking is bypassed, [`NoOpUploadLockingService`](../src/main/java/me/desair/tus/server/upload/NoOpUploadLockingService.java) provides a zero-overhead Null Object implementation.
+4. **Supporting Checksum Deduplication**:
+   When upload deduplication is enabled (`.withUploadDeduplication(true)`), storage backends must implement the following critical interface methods:
+   - `deduplicate(UploadInfo, ChecksumKey)`: Coordinate cluster-safe deduplication upon upload completion. Storage backends typically instantiate a [`DeduplicationService`](../src/main/java/me/desair/tus/server/upload/checksum/DeduplicationService.java) backed by a custom [`ChecksumIndex`](../src/main/java/me/desair/tus/server/upload/checksum/ChecksumIndex.java) implementation to check for an existing parent, link the child upload, or register a new parent.
+   - `dataExists(UploadId)`: Return `true` if physical committed data exists for the upload ID. The deduplication service invokes this method during parent lookup to detect out-of-band data deletions and automatically self-heal stale checksum index entries. *(The default interface method returns `false`, which disables parent reuse if not overridden.)*
+   - `deleteData(UploadId)`: Delete physical committed data bytes without removing the upload's `.info` metadata record. This is invoked to free physical storage when a duplicate upload is linked to an existing parent, and to purge retained parent data when the last active child terminates. *(The default interface method is a no-op, which leaks storage if not overridden.)*
+   - `setUploadDeduplicationEnabled(boolean)` / `isUploadDeduplicationEnabled()`: Store and expose whether deduplication is active.
+   - **Read/Write Safety Rules**:
+     - *Read operations* (`getUploadedBytes`, `copyUploadTo`) MUST check `uploadInfo.getDuplicatesUploadId()` and dynamically stream bytes from the parent upload if set.
+     - *Write and mutation operations* (`append`, `removeLastNumberOfBytes`) MUST NEVER resolve `duplicatesUploadId` to the parent; they must operate strictly on child storage to prevent mutating parent data.
+5. **Optional Features & Extensions**:
+   - `UploadConcatenationService`: Implement concatenation to support merging parallel chunk uploads into a single final resource.
+   - Size limits: Implement `setMinAppendSize`, `setMaxAppendSize`, `setMinSize`, and `setMaxUploadSize` to enforce upload payload constraints.
+   - Cloud streaming: Implement `setCloudUploadThreadPoolSize` and `setDrainTimeout` for asynchronous chunk uploaders.
+   - Cleanup: Implement `close()` to release underlying network clients, thread pools, or file descriptors.
+
+
 ---
 
 ## 6. Distributed Locking & Lock Contention Resolution
 
 Resumable uploads require exclusive concurrency control per upload resource to prevent race conditions (e.g., overlapping byte appends or simultaneous resume and delete requests).
 
-All locking backends implement [`UploadLockingService`](../src/main/java/me/desair/tus/server/upload/UploadLockingService.java) and return [`UploadLock`](../src/main/java/me/desair/tus/server/upload/UploadLock.java) handles.
+All locking backends implement [`UploadLockingService`](../src/main/java/me/desair/tus/server/upload/UploadLockingService.java) and return [`UploadLock`](../src/main/java/me/desair/tus/server/upload/UploadLock.java) handles. When no concurrency control is needed, [`NoOpUploadLockingService`](../src/main/java/me/desair/tus/server/upload/NoOpUploadLockingService.java) acts as the null-safe implementation.
 
 ### Building Lease-Based Lock Providers: `AbstractLeaseLockingService`
 
@@ -305,5 +338,5 @@ For detailed testing instructions, see [`docs/TESTING.md`](TESTING.md) and [`doc
 | **Add a Request Validation Rule** | Create a class implementing `RequestValidator` in the target extension's `validation` package. | Keep validation side-effect-free. Throw typed `TusException` subclasses. Register in `extension.initValidators()`. |
 | **Add a Request Processing Action** | Create a class implementing `RequestHandler` (or extending `AbstractRequestHandler`). | Wrap streaming payload inputs in `InterruptibleInputStream`. Set headers on `TusServletResponse`. Register in `extension.initRequestHandlers()`. |
 | **Add a New Protocol Extension** | Create a package under `me.desair.tus.server.<extension>` and extend `AbstractTusExtension`. | Implement `getName()`, `isApplicable(method, version)`, `initValidators()`, and `initRequestHandlers()`. Register in `TusFileUploadService.initFeatures()`. |
-| **Add a Storage Backend** | Implement `UploadStorageService` (and optionally `UploadConcatenationService`). | Support deduplication (`duplicatesUploadId`). Implement offline unit tests (`*Test.java`) and Testcontainers integration tests (`IT*`). |
+| **Add a Storage Backend** | Implement `UploadStorageService` (and optionally `UploadConcatenationService`). | Implement `terminateUpload` (free data, metadata `.info`, and delegate `cleanupLock`). If supporting deduplication, implement `deduplicate`, `dataExists`, and `deleteData`, follow read/write safety rules (`duplicatesUploadId`), and implement a `ChecksumIndex`. Implement offline unit tests (`*Test.java`) and Testcontainers integration tests (`IT*`). |
 | **Add a Distributed Locking Provider** | Extend `AbstractLeaseLockingService` and `AbstractLeaseLock`. | Implement the template hooks (`tryAcquireLock`, `isLockExpired`, `evictExpiredLock`, `writeStopSignal`, `checkStopSignals`). Inherit NTP safety margins and stream interruption automatically. |

docs/AZURE_BLOB_STORAGE.md

@@ -89,11 +89,13 @@ TusFileUploadService tusService = new TusFileUploadService()
 
 ```
 <container>/
-├── uploads/<uploadId>                # Final upload data (Block Blob)
-├── metadata/<uploadId>.info          # JSON-serialized UploadInfo
-├── checksums/<algorithm>/<hex_hash>  # Deduplication checksum index object
-├── locks/<uploadId>.lock             # Distributed lock target blob (Blob Lease)
-└── locks/<uploadId>.stop             # Cross-replica contention interrupt signal
+├── uploads/<uploadId>                                       # Final upload data (Block Blob)
+├── metadata/<uploadId>.info                                 # JSON-serialized UploadInfo
+├── checksums/<algorithm>/<hex_hash>/parent                  # Deduplication parent pointer blob
+├── checksums/<algorithm>/<hex_hash>/parent-deleted          # Tombstone marker blob if parent deleted
+├── checksums/<algorithm>/<hex_hash>/children/<childId>      # Active duplicate child reference blob
+├── locks/<uploadId>.lock                                    # Distributed lock target blob (Blob Lease)
+└── locks/<uploadId>.stop                                    # Cross-replica contention interrupt signal
 ```
 
 ### Key Prefix Defaults

docs/CONFIGURATION.md

@@ -49,7 +49,7 @@ For Azure Blob Storage dependencies see the [Azure Blob Storage Guide](AZURE_BLO
 | `withMaxUploadSize(Long)` | `Long.MAX_VALUE` | Maximum allowed total upload size in bytes per upload resource. |
 | `withMaxAppendSize(Long)` / `withMinAppendSize(Long)` / `withMinSize(Long)` | No limit | IETF RUFH only: maximum size of a single append (`PATCH`) request, minimum size of an append or creation request, and minimum total upload size (all in bytes). |
 | `withUploadExpirationPeriod(Long)` | `null` (disabled) | Expiration period in **milliseconds** after which incomplete/expired uploads become eligible for cleanup (run `cleanup()` periodically, see the [Integration Guide](INTEGRATION_GUIDE.md#3-cleanup)). |
-| `withUploadDeduplication(Boolean)` | `false` | Enables duplicate file detection by checksum, linking new uploads (`duplicatesUploadId`) and skipping redundant storage writes. |
+| `withUploadDeduplication(Boolean)` | `false` | Enables duplicate file detection by checksum, linking new uploads (`duplicatesUploadId`) and skipping redundant storage writes. See [Upload Deduplication & Migration](DEDUPLICATION.md). |
 
 ### Identity, events & serialization
 
@@ -64,7 +64,7 @@ For Azure Blob Storage dependencies see the [Azure Blob Storage Guide](AZURE_BLO
 | Method | Default | Description |
 |---|---|---|
 | `withStoragePath(String)` | `${java.io.tmpdir}/tus` | Path on the filesystem or shared drive where uploaded bytes and metadata are stored when using `DiskStorageService`. Also configures `LeaseFileLockingService`. |
-| `withUploadStorageService(UploadStorageService)` | `DiskStorageService` | Configures custom or cloud storage backend (`DiskStorageService`, `S3StorageService`, `AzureBlobStorageService`). Automatically copies over previous limits and configuration. |
+| `withUploadStorageService(UploadStorageService)` | `DiskStorageService` | Configures custom or cloud storage backend (`DiskStorageService`, `S3StorageService`, `AzureBlobStorageService`). Automatically copies over previous limits and configuration. For guidelines on implementing custom storage backends, see [ARCHITECTURE.md](ARCHITECTURE.md#implementing-a-custom-storage-backend). |
 | `withUploadLockingService(UploadLockingService)` | `LeaseFileLockingService` | Configures custom or cloud locking backend (`LeaseFileLockingService`, `S3LockingService`, `AzureBlobLockingService`). |
 | `withLockWaitTimeout(Duration)` | `Duration.ofSeconds(60)` | Maximum duration a request waits to acquire an upload lock held by an in-flight transfer (retrying every 200ms, resulting in 300 retries). Automatically configures cloud background chunk drain timeout to 5s less than this value (default 55s). See [LOCKING.md](LOCKING.md). |
 | `withThreadLocalCache(Boolean)` | `false` | Enables in-memory thread-local caching of upload request data to reduce storage backend I/O load. Strongly recommended for cloud backends (S3, Azure) to eliminate redundant remote metadata lookups per request. |