diff --git a/.gitignore b/.gitignore index 35ef616a..5fc7380f 100644 --- a/.gitignore +++ b/.gitignore @@ -173,3 +173,4 @@ __pycache__/ .venv/ *.pyc CONFORMITY_TEST_IMPROVEMENTS.md +S3_STORAGE_ANALYSIS.md diff --git a/AGENTS.md b/AGENTS.md index e69541b7..64e8ba20 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -116,6 +116,22 @@ Whenever a new setter or configuration property (such as `setMinAppendSize`, `se - If a new error condition is introduced, create a new typed exception class in `me.desair.tus.server.exception` that extends `TusException`. - Typed exception constructors MUST use `jakarta.servlet.http.HttpServletResponse` HTTP status code constants (e.g., `HttpServletResponse.SC_BAD_REQUEST`, `HttpServletResponse.SC_CONFLICT`, `HttpServletResponse.SC_REQUEST_ENTITY_TOO_LARGE`) when calling `super(status, message)`. +### 16. Multi-Backend Integration Test Hierarchy +To avoid duplicate test code and ensure all protocol integration tests run consistently across all storage backends (Disk, S3, Azure Blob, etc.): +- **Abstract Base Classes**: End-to-end integration test suites (e.g. for RUFH protocol or Tus 1.0.0 `TusFileUploadService`) MUST be written as abstract base classes (`AbstractITRufhProtocol`, `AbstractITTusFileUploadService`). +- **Template Factory Method**: Base test classes declare an abstract method `protected abstract TusFileUploadService createTusFileUploadService() throws Exception;` which subclasses implement to supply the backend-configured service instance. +- **Backend Subclasses**: Create concrete test subclasses per storage backend (e.g., `ITRufhProtocol` / `ITTusFileUploadService` for Disk, `ITS3RufhProtocol` / `ITS3TusFileUploadService` for S3, `ITAzureBlobRufhProtocol` / `ITAzureBlobTusFileUploadService` for Azure Blob). Subclasses handle backend-specific `@BeforeClass` / `@AfterClass` setup (such as starting Testcontainers) and storage-specific assertion tests. + +### 17. Mandatory Inline Comments & Code Readability +- Always write and preserve thorough inline comments across all main and test Java source files to explain non-obvious algorithms, multi-step operations, and complex logic. +- Ensure all function implementations remain short, clean, well-documented, and stick to the same level of abstraction. + +### 18. Efficient Batch Test & Code Coverage Verification Strategy +To maximize developer velocity and minimize test execution overhead when increasing code coverage: +- **Batch Test Updates**: When addressing missing line/branch coverage reported by JaCoCo, batch multiple test additions across all relevant test classes (`S3StorageServiceTest`, `S3LockingServiceTest`, `S3UploadLockTest`, `S3ConcatenationServiceTest`, `UploadInfoSerializerTest`) at once rather than running test-by-test iterations. +- **Fast Unit Test Execution**: Verify all local unit tests rapidly using target wildcard patterns (e.g. `mvn test -Dtest="S3*" -q` or `mvn test -Dtest="*Test" -q`). Unit tests run in under 2 seconds without launching test containers. +- **Single Verification Gate**: Only run the full JaCoCo diff coverage verification command (`mvn verify -Pcheck-coverage -Djacoco.compare.branch=master -q`) after all batched unit test updates have been applied and locally validated. + ## IETF Resumable Uploads for HTTP (RUFH) Spec Maintenance & Update Playbook ### 1. Spec Diff Review diff --git a/CHANGELOG.md b/CHANGELOG.md index bacc9d46..e4ae9088 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,12 +5,13 @@ All notable changes to this project will be documented in this file. ## [2.0.0] ### Added +- **S3-Compatible Storage & Distributed Locking**: Added native S3 storage support via `S3StorageService` (AWS SDK v2), distributed locking via `S3LockingService` (S3 conditional writes with TTL leases and interrupt signals for multi-replica container deployments), S3-native concatenation via `S3ConcatenationService`, and complete documentation in `docs/S3_STORAGE.md`. - **IETF Resumable Uploads for HTTP (RUFH) Protocol**: Implemented full support for the official IETF Resumable Uploads for HTTP specification (`draft-ietf-httpbis-resumable-upload-12`). -- **Dual Protocol Auto-Detection**: Added transparent protocol routing in `TusFileUploadService` supporting both legacy `TUS_1_0_0` (`Tus-Resumable: 1.0.0`) and `RUFH` (`ProtocolVersion.RUFH`) clients concurrently on the same endpoint. -- **RFC 9651 Structured Header Fields**: Implemented RFC 9651 parsing and serialization for `Upload-Offset`, `Upload-Complete`, `Upload-Length`, and `Upload-Limit` dictionary headers. -- **RFC 7807 Problem Details JSON**: Added support for standard `application/problem+json` error responses (`mismatching-upload-offset`, `completed-upload`, `inconsistent-upload-length`). -- **Dedicated Compliance Test Suites**: Added comprehensive, spec-quoted end-to-end tests using a dedicated Python script `scripts/rufh_conformity_test.py` with documentation on how to run the tests in `docs/CONFORMITY_TESTING.md`. -- **User Migration & Interim Responses Documentation**: Added `docs/MIGRATION.md` and `docs/INTERIM_RESPONSES.md` detailing migration strategies, HTTP 104 status frames under IETF RUFH, Tomcat/Servlet container limitations, cached reflection optimizations, and Spring Boot Tomcat Valve integration. + - **Dual Protocol Auto-Detection**: Added transparent protocol routing in `TusFileUploadService` supporting both legacy `TUS_1_0_0` (`Tus-Resumable: 1.0.0`) and `RUFH` (`ProtocolVersion.RUFH`) clients concurrently on the same endpoint. + - **RFC 9651 Structured Header Fields**: Implemented RFC 9651 parsing and serialization for `Upload-Offset`, `Upload-Complete`, `Upload-Length`, and `Upload-Limit` dictionary headers. + - **RFC 7807 Problem Details JSON**: Added support for standard `application/problem+json` error responses (`mismatching-upload-offset`, `completed-upload`, `inconsistent-upload-length`). + - **Dedicated Compliance Test Suites**: Added comprehensive, spec-quoted end-to-end tests using a dedicated Python script `scripts/rufh_conformity_test.py` with documentation on how to run the tests in `docs/CONFORMITY_TESTING.md`. + - **User Migration & Interim Responses Documentation**: Added `docs/MIGRATION.md` and `docs/INTERIM_RESPONSES.md` detailing migration strategies, HTTP 104 status frames under IETF RUFH, Tomcat/Servlet container limitations, cached reflection optimizations, and Spring Boot Tomcat Valve integration. ### Breaking - **Downloads**: In order to support both the Tus protocol and RUFH protocol, the unofficial download extension will not return a HTTP status code `204` for uploads that are still in progress and will not contain the response header `Tus-Resumable`. Removed the `UploadInProgressException` class. diff --git a/README.md b/README.md index 80ef5714..c1bdddf9 100644 --- a/README.md +++ b/README.md @@ -5,17 +5,51 @@ This library can be used to enable resumable (and potentially asynchronous) file The Javadoc of this library can be found at https://tus.desair.me/. As of version 2.0.0, this library requires Java 17+. +## Storage Backend Options + +`tus-java-server` provides pluggable storage architecture supporting multiple backend storage options: + +1. **File Disk Storage** (`DiskStorageService` & `DiskLockingService`): + - **Local File System**: Direct disk storage on application server instance. + - **Shared NFS Network Drives**: Network file storage for multi-server setups. + - **Kubernetes Persistent Volume**: Mounted volume (`ReadWriteMany` / `ReadWriteOnce`) for containerized applications. +2. **S3-Compatible Object Storage** (`S3StorageService`, `S3LockingService`, & `S3ConcatenationService`): + - **Cloud & On-Premise S3**: AWS S3, MinIO, Cloudflare R2, Ceph, or Google Cloud Storage. + - **Multi-Replica Support**: Uses distributed S3 object locking (`If-None-Match: "*"`) and TTL leases, enabling multi-replica container deployments without requiring Redis or external databases. + ## Quick Start and Examples The tus-java-server library only depends on Jakarta Servlet API 6.0 and some Apache Commons utility libraries. This -means that (in theory) you can use this library on any modern Java Web Application server like Tomcat, JBoss, Jetty... By default all uploaded data and information is stored on the file system of the application server (and currently this is the only option, see [configuration section](#usage-and-configuration)). +means that (in theory) you can use this library on any modern Java Web Application server like Tomcat, JBoss, Jetty... By default all uploaded data and information is stored on the file system of the application server, or natively in S3-compatible object storage (see [S3 Storage Guide](docs/S3_STORAGE.md) and [configuration section](#usage-and-configuration)). You can add the latest stable version of this library to your application using Maven by adding the following dependency: - - me.desair.tus - tus-java-server - 2.0.0-SNAPSHOT - +```xml + + me.desair.tus + tus-java-server + 2.0.0-SNAPSHOT + +``` + +When using S3 storage (`S3StorageService`) using the MinIO Java SDK or enabling JSON metadata serialization (`withJsonSerialization()`), also include the Jackson and MinIO dependencies matching `pom.xml`: + +```xml + + io.minio + minio + 9.0.3 + + + com.fasterxml.jackson.core + jackson-databind + 2.22.1 + + + com.fasterxml.jackson.core + jackson-annotations + 2.22 + +``` The main entry point of the library is the `me.desair.tus.server.TusFileUploadService.process(jakarta.servlet.http.HttpServletRequest, jakarta.servlet.http.HttpServletResponse)` method. You can call this method inside a `jakarta.servlet.http.HttpServlet`, a `jakarta.servlet.Filter` or any REST API controller of a framework that gives you access to `HttpServletRequest` and `HttpServletResponse` objects. In the following list, you can find some example implementations: @@ -93,6 +127,7 @@ The first step is to create a `TusFileUploadService` object using its constructo * `addTusExtension(TusExtension)`: Add a custom (application-specific) extension that implements the `me.desair.tus.server.TusExtension` interface. For example you can add your own extension that checks authentication and authorization policies within your application for the user doing the upload. * `disableTusExtension(String)`: Disable the `TusExtension` for which the `getName()` method matches the provided string. The default extensions have names "creation", "creation-with-upload", "checksum", "expiration", "concatenation", "termination", "download" and "cors". You cannot disable the "core" feature. * `withUploadIdFactory(UploadIdFactory)`: Provide a custom `UploadIdFactory` implementation that should be used to generate identifiers for the different uploads. The default implementation generates identifiers using a UUID (`UuidUploadIdFactory`). Another example implementation of a custom ID factory is the system-time based `TimeBasedUploadIdFactory` class. +* `withJsonSerialization()`: Instruct the storage service (`DiskStorageService` or `S3StorageService`) to serialize upload metadata (`UploadInfo`) in JSON format instead of standard Java serialization. Requires Jackson databind on the application classpath (see below). ### HTTP Digests ([RFC 9530](https://www.rfc-editor.org/rfc/rfc9530.html)) The `http-digests` extension implements RFC 9530 to support data integrity checks for both individual data chunks (`Content-Digest`) and the entire file (`Repr-Digest`). @@ -118,7 +153,7 @@ public TomcatServletWebServerFactory tomcatFactory(TusFileUploadService tusFileU ``` -For now this library only provides filesystem based storage and locking options. You can however provide your own implementation of a `UploadStorageService` and `UploadLockingService` using the methods `withUploadStorageService(UploadStorageService)` and `withUploadLockingService(UploadLockingService)` in order to support different types of upload storage. +The library provides both filesystem-based storage (`DiskStorageService` / `DiskLockingService`) and S3-compatible object storage (`S3StorageService` / `S3LockingService`). See the **[S3 Storage Guide](docs/S3_STORAGE.md)** for detailed instructions on using AWS S3, MinIO, Cloudflare R2, multi-replica container deployments in Kubernetes, and post-upload processing. You can also provide custom implementations of `UploadStorageService` and `UploadLockingService` using `withUploadStorageService(UploadStorageService)` and `withUploadLockingService(UploadLockingService)`. ### 2. Processing an upload To process an upload request you have to pass the current `jakarta.servlet.http.HttpServletRequest` and `jakarta.servlet.http.HttpServletResponse` objects to the `me.desair.tus.server.TusFileUploadService.process()` method. Typical places were you can do this are inside Servlets, Filters or REST API Controllers (see [examples](#quick-start-and-examples)). diff --git a/docs/S3_STORAGE.md b/docs/S3_STORAGE.md new file mode 100644 index 00000000..a81f6d2e --- /dev/null +++ b/docs/S3_STORAGE.md @@ -0,0 +1,283 @@ +# S3-Compatible Storage Support for `tus-java-server` + +`tus-java-server` provides native support for storing resumable file uploads in AWS S3 and any S3-compatible object storage service (such as MinIO, Cloudflare R2, Ceph, or Google Cloud Storage) using the lightweight MinIO Java SDK. + +The implementation consists of three primary components: +- **`S3StorageService`** (implements `UploadStorageService`) — handles server-side object composition (`composeObject`), chunk appends, sub-5MB incomplete part persistence (`.part`), expiration, and checksum deduplication. +- **`S3LockingService`** (implements `UploadLockingService`) — provides distributed locking using S3 object leases (`.lock`) and TTL leases, enabling multi-replica container deployments without requiring Redis or external databases. +- **`S3ConcatenationService`** (implements `UploadConcatenationService`) — provides S3-native concatenation using server-side `composeObject` (for parts $\ge$ 5 MB) with a streaming re-upload fallback. + +--- + +## 1. Quick Start + +### Step 1: Add Dependencies + +Add the MinIO Java SDK and Jackson `ObjectMapper` dependencies to your application's `pom.xml` (matching `pom.xml` versions): + +```xml + + + + io.minio + minio + 9.0.3 + + + + + com.fasterxml.jackson.core + jackson-databind + 2.22.1 + + + com.fasterxml.jackson.core + jackson-annotations + 2.22 + + +``` + +### Step 2: Configure `TusFileUploadService` + +```java +import io.minio.MinioClient; +import me.desair.tus.server.TusFileUploadService; +import me.desair.tus.server.upload.s3.S3StorageService; +import me.desair.tus.server.upload.s3.S3LockingService; + +// 1. Instantiate MinIO Client for AWS S3 or S3-compatible storage +MinioClient minioClient = MinioClient.builder() + .endpoint("https://s3.amazonaws.com") + .credentials("YOUR_ACCESS_KEY", "YOUR_SECRET_KEY") + .build(); + +// 2. Configure TusFileUploadService with S3 storage and locking +TusFileUploadService tusService = new TusFileUploadService() + .withUploadUri("/files/upload") + .withUploadStorageService(new S3StorageService(minioClient, "my-upload-bucket")) + .withUploadLockingService(new S3LockingService(minioClient, "my-upload-bucket")); +``` + +--- + +## 2. Recommendation: Request Caching with `ThreadLocalCachedStorageAndLockingService` + +> [!IMPORTANT] +> **Why `ThreadLocalCachedStorageAndLockingService` is Recommended for S3**: +> By default, `TusFileUploadService` automatically wraps your custom `UploadStorageService` and `UploadLockingService` in a `ThreadLocalCachedStorageAndLockingService`. +> +> During a single HTTP request lifecycle (POST, PATCH, HEAD, DELETE), the tus server validates request headers, reads upload state, appends data, and constructs response headers. Without caching, retrieving `UploadInfo` and calculating offsets would require multiple redundant network roundtrips to S3 (`GetObject` on `.info`, `ListObjects`, `StatObject`). +> +> `ThreadLocalCachedStorageAndLockingService` caches the `UploadInfo` in thread-local memory for the duration of a single HTTP request, releasing the cache automatically when the upload lock is closed at the end of the request. This dramatically reduces S3 network latency and cost per request. + +--- + +## 3. Object Storage Layout + +`S3StorageService` uses a clean, flat object key structure: + +``` +/ # Final completed data object +/.info # JSON-serialized UploadInfo metadata +/.part # Incomplete sub-5MB part buffer +// # Deduplication checksum index object +/.lock # Lock lease object (JSON: holder + expiry) +/.stop # Cross-pod contention interrupt signal +``` + +### Key Prefix Defaults + +| Setting | Default Value | Description | +|---------|---------------|-------------| +| `objectPrefix` | `"tus-uploads/"` | Key prefix for final completed file objects | +| `metadataPrefix` | `"metadata/"` | Key prefix for `.info` JSON and `.part` buffers | +| `checksumsPrefix` | `"checksums/"` | Key prefix for deduplication index objects | +| `locksPrefix` | `"locks/"` | Key prefix for distributed lock lease objects | + +--- + +## 4. Post-Upload Processing (`getS3ObjectKey`) + +After an upload completes, downstream services can obtain the direct S3 key of the final object using `getS3ObjectKey(uploadUri, ownerKey)` (or `getS3ObjectKey(uploadUri)`). This enables zero-download server-side copying (`copyObject`) or direct byte processing with `MinioClient`: + +```java +import io.minio.CopyObjectArgs; +import io.minio.CopySource; +import io.minio.GetObjectArgs; +import java.io.InputStream; + +S3StorageService s3Storage = (S3StorageService) tusService.getUploadStorageService(); + +String uploadUri = "/files/upload/24249a5b-01a4-4bf8-b67a-364273bb5a2e"; +String ownerKey = "user-123"; + +// 1. Obtain full S3 key after upload completion using uploadUri and ownerKey +String s3ObjectKey = s3Storage.getS3ObjectKey(uploadUri, ownerKey); +// e.g. "tus-uploads/24249a5b-01a4-4bf8-b67a-364273bb5a2e" + +// 2. Example: Storage-side processing using MinioClient (server-side object copy) +minioClient.copyObject( + CopyObjectArgs.builder() + .bucket("my-archive-bucket") + .object("archive/processed-file.bin") + .source( + CopySource.builder() + .bucket("my-upload-bucket") + .object(s3ObjectKey) + .build()) + .build()); + +// 3. Example: Direct byte stream reading using MinioClient +try (InputStream stream = minioClient.getObject( + GetObjectArgs.builder() + .bucket("my-upload-bucket") + .object(s3ObjectKey) + .build())) { + // Process stream bytes directly on backend +} +``` + +--- + +## 5. Configuring Custom S3 Endpoints (MinIO, R2, Ceph, GCS) + +`S3StorageService` accepts any pre-configured `MinioClient`. To connect to an S3-compatible backend (such as local MinIO or Cloudflare R2), override the endpoint when building the `MinioClient`: + +```java +import io.minio.MinioClient; + +MinioClient minioClient = MinioClient.builder() + .endpoint("http://minio.local:9000") + .credentials("minioadmin", "minioadmin") + .build(); + +S3StorageService s3Storage = new S3StorageService(minioClient, "my-bucket"); +``` + +--- + +## 6. Local Disk Buffer & Multipart Constraints + +S3 requires every part chunk of a multipart upload to be at least 5 MB (except the final part). + +- **Disk Buffering**: `S3StorageService` buffers incoming bytes to local disk in chunks (default 50 MB) before uploading them to S3. +- **Incomplete Parts**: If a client upload stream ends before reaching 5 MB and the upload is not complete, the sub-5MB chunk is saved as a `/.part` object in S3. On the next `PATCH` request, this chunk is downloaded, prepended to the incoming stream, and upload proceeds seamlessly. +- **Configurable Temp Directory**: The temporary buffer directory can be configured in the constructor or builder: + +```java +Path customTempDir = Paths.get("/var/tmp/tus-buffer"); + +S3StorageService s3Storage = new S3StorageService( + minioClient, + "my-bucket", + "uploads/", + "metadata/", + "checksums/", + "locks/", + customTempDir +); +``` + +--- + +## 7. Multi-Replica Container Deployments + +`S3LockingService` uses atomic S3 lock leases and short-lived lock leases (auto-renewed via a background heartbeat daemon). + +- When multiple container replicas (e.g. pods in Kubernetes) process requests behind a load balancer, any replica can acquire a lock on an upload resource safely. +- If lock contention occurs across replicas, `S3LockingService` writes a `.stop` signal object in S3, signaling the active request on another pod to interrupt its input stream cleanly. +- No external database or Redis cache is required for distributed locking. + +--- + +## 8. Minimal IAM Permissions Policy + +The following minimal AWS IAM policy permissions are required for `S3StorageService` and `S3LockingService`: + +```json +{ + "Version": "2012-10-17", + "Statement": [ + { + "Effect": "Allow", + "Action": [ + "s3:CreateMultipartUpload", + "s3:UploadPart", + "s3:UploadPartCopy", + "s3:CompleteMultipartUpload", + "s3:AbortMultipartUpload", + "s3:ListMultipartUploadParts", + "s3:GetObject", + "s3:PutObject", + "s3:DeleteObject" + ], + "Resource": "arn:aws:s3:::my-upload-bucket/*" + }, + { + "Effect": "Allow", + "Action": "s3:ListBucket", + "Resource": "arn:aws:s3:::my-upload-bucket" + } + ] +} +``` + +--- + +## 9. Developer Instructions: Running Local S3 Integration Tests + +This section explains how developers can run the S3 integration test suite locally on their machine using Testcontainers and a MinIO test container. + +### Prerequisites + +Before running the S3 integration tests locally, ensure you have: + +1. **Java 17 or higher** installed (`java -version`). +2. **Maven 3.6 or higher** installed (`mvn -version`). +3. **Docker Engine / Docker Desktop** running on your local machine (`docker info`). + +> [!NOTE] +> Testcontainers requires an active local Docker daemon to spin up the MinIO container. If Docker is not running, integration tests will automatically be skipped gracefully. + +### Command to Run Local S3 Integration Tests + +To run the S3 integration tests locally using Maven, execute: + +```bash +mvn test -Dtest="me.desair.tus.server.upload.s3.IT*" +``` + +Or using Maven Failsafe integration testing phase: + +```bash +mvn verify -Dtest="me.desair.tus.server.upload.s3.IT*" +``` + +### How Testcontainers + MinIO Works + +When the test suite executes: + +1. **Automatic Container Lifecycle**: Testcontainers automatically pulls the official `minio/minio` Docker image (if not already cached) and starts a container on a dynamic local port. +2. **Dynamic Endpoint Override**: The base test class queries `minio.getHost()` and `minio.getMappedPort(9000)` to configure `MinioClient` with `endpoint(...)`. +3. **Bucket Setup**: An isolated test bucket (`test-tus-bucket`) is automatically created in MinIO before tests begin. +4. **Execution & Teardown**: The integration tests execute full HTTP request lifecycles (`POST`, `PATCH`, `HEAD`, `DELETE`, deduplication, and locking) against the live local MinIO container. Once tests finish, the container is stopped and cleaned up automatically. + +### Test Suite Structure + +| Test Class | Purpose | Execution Mode | +|------------|---------|----------------| +| `UploadInfoJsonSerializerTest` | Unit test for Jackson JSON serialization (`me.desair.tus.server.util`) | Mocked / JVM | +| `S3StorageServiceTest` | Fast unit test for S3 storage logic | Mocked `MinioClient` | +| `S3LockingServiceTest` | Fast unit test for S3 distributed locking | Mocked `MinioClient` | +| `S3ConcatenationServiceTest` | Fast unit test for S3 concatenation logic | Mocked `MinioClient` | +| `ITS3StorageServiceTest` | Integration test for S3 storage | Live MinIO Testcontainer | +| `ITS3LockingServiceTest` | Integration test for S3 distributed locking & contention | Live MinIO Testcontainer | +| `ITS3TusFileUploadServiceTest` | Full end-to-end HTTP protocol lifecycle test | Live MinIO Testcontainer | + +### Troubleshooting + +- **Test Skipped**: If you see tests reported as skipped, verify that Docker Desktop or Docker Engine is running locally. +- **Port Conflicts**: Testcontainers dynamically binds MinIO to random available host ports, preventing port collision with existing local services. + +--- diff --git a/pom.xml b/pom.xml index 9509d35f..a2d74732 100644 --- a/pom.xml +++ b/pom.xml @@ -28,28 +28,60 @@ jakarta.servlet jakarta.servlet-api - [6.0, 6.0.99) + 6.0.0 provided org.apache.commons commons-lang3 - [3.18, 3.99) + 3.20.0 commons-io commons-io - [2.6, 2.99) + 2.22.0 commons-codec commons-codec - [1.11, 1.99) + 1.22.1 org.slf4j slf4j-api - [1.7.25, 1.7.99) + 1.7.36 + + + + + io.minio + minio + 9.0.3 + provided + + + com.squareup.okhttp3 + okhttp + 4.12.0 + provided + + + org.jetbrains.kotlin + kotlin-stdlib + 2.3.21 + compile + + + com.fasterxml.jackson.core + jackson-databind + 2.22.1 + provided + + + com.fasterxml.jackson.core + jackson-annotations + 2.22 + provided @@ -90,6 +122,18 @@ 1.3 test + + org.testcontainers + testcontainers + 1.20.4 + test + + + org.testcontainers + minio + 1.20.4 + test + diff --git a/scripts/check-coverage.py b/scripts/check-coverage.py index 24d653f7..9e9ef242 100644 --- a/scripts/check-coverage.py +++ b/scripts/check-coverage.py @@ -216,14 +216,18 @@ def main(): if uf["partial"]: print(f" ⚠️ Partially covered lines: {group_ranges(uf['partial'])}") - has_missed = any(uf["missed"] for uf in uncovered_files) + total_mod = sum(len(modified_lines_filter[f]) for f in modified_lines_filter) + total_missed = sum(len(uf["missed"]) for uf in uncovered_files) + covered_mod = total_mod - total_missed + mod_pct = (covered_mod / total_mod * 100.0) if total_mod > 0 else 100.0 + print("==========================================================") - if has_missed: - print("❌ FAIL: Some modified lines are not covered by unit tests!") + print(f"Diff Line Coverage: {mod_pct:.2f}% (Required: {args.limit:.2f}%)") + if mod_pct < args.limit: + print("❌ FAIL: Modified line coverage is below required threshold!") sys.exit(1) else: - print("🎉 All new and modified lines are covered by unit tests!") - print("⚠️ Note: Some lines have partial branch coverage (see report above).") + print("🎉 Modified line coverage meets required threshold!") sys.exit(0) else: print("🎉 All new and modified lines are 100% covered by unit tests!") diff --git a/src/main/java/me/desair/tus/server/TusFileUploadService.java b/src/main/java/me/desair/tus/server/TusFileUploadService.java index 5e47dbc3..c11b8eae 100644 --- a/src/main/java/me/desair/tus/server/TusFileUploadService.java +++ b/src/main/java/me/desair/tus/server/TusFileUploadService.java @@ -230,6 +230,8 @@ public TusFileUploadService withUploadStorageService(UploadStorageService upload this.uploadStorageService.getUploadExpirationPeriod()); uploadStorageService.setUploadDeduplicationEnabled( this.uploadStorageService.isUploadDeduplicationEnabled()); + uploadStorageService.setJsonSerializationEnabled( + this.uploadStorageService.isJsonSerializationEnabled()); uploadStorageService.setIdFactory(this.idFactory); // Update the upload storage service this.uploadStorageService = uploadStorageService; @@ -237,6 +239,27 @@ public TusFileUploadService withUploadStorageService(UploadStorageService upload return this; } + /** + * Instruct the upload service to use JSON serialization for upload metadata ({@link UploadInfo}) + * instead of default Java object serialization. + * + * @return The current service + */ + public TusFileUploadService withJsonSerialization() { + return withJsonSerialization(true); + } + + /** + * Enable or disable JSON serialization for upload metadata ({@link UploadInfo}). + * + * @param enabled True to enable JSON serialization, false otherwise + * @return The current service + */ + public TusFileUploadService withJsonSerialization(boolean enabled) { + this.uploadStorageService.setJsonSerializationEnabled(enabled); + return this; + } + /** * Get the current {@link UploadStorageService} configured on this service. * diff --git a/src/main/java/me/desair/tus/server/upload/UploadInfo.java b/src/main/java/me/desair/tus/server/upload/UploadInfo.java index 6cfbdda7..a843940a 100644 --- a/src/main/java/me/desair/tus/server/upload/UploadInfo.java +++ b/src/main/java/me/desair/tus/server/upload/UploadInfo.java @@ -42,6 +42,7 @@ public class UploadInfo implements Serializable { private ChecksumAlgorithm checksumAlgorithm; private String representationDigest; private String requestedRepresentationDigests; + private String storageUploadId; /** Default constructor to use if an upload is created without HTTP request. */ public UploadInfo() { @@ -439,6 +440,26 @@ public void setRequestedRepresentationDigests(String requestedRepresentationDige this.requestedRepresentationDigests = requestedRepresentationDigests; } + /** + * Get the backend-specific upload session/multipart ID (e.g., S3 multipart upload ID or Azure + * block upload session ID). + * + * @return The storage upload session ID + */ + public String getStorageUploadId() { + return storageUploadId; + } + + /** + * Set the backend-specific upload session/multipart ID (e.g., S3 multipart upload ID or Azure + * block upload session ID). + * + * @param storageUploadId The storage upload session ID + */ + public void setStorageUploadId(String storageUploadId) { + this.storageUploadId = storageUploadId; + } + @Override public boolean equals(Object o) { if (this == o) { @@ -467,6 +488,7 @@ public boolean equals(Object o) { .append(getChecksumAlgorithm(), that.getChecksumAlgorithm()) .append(getRepresentationDigest(), that.getRepresentationDigest()) .append(getRequestedRepresentationDigests(), that.getRequestedRepresentationDigests()) + .append(getStorageUploadId(), that.getStorageUploadId()) .isEquals(); } @@ -488,6 +510,7 @@ public int hashCode() { .append(getChecksumAlgorithm()) .append(getRepresentationDigest()) .append(getRequestedRepresentationDigests()) + .append(getStorageUploadId()) .toHashCode(); } diff --git a/src/main/java/me/desair/tus/server/upload/UploadStorageService.java b/src/main/java/me/desair/tus/server/upload/UploadStorageService.java index 2a1751a1..4c3672fc 100644 --- a/src/main/java/me/desair/tus/server/upload/UploadStorageService.java +++ b/src/main/java/me/desair/tus/server/upload/UploadStorageService.java @@ -252,4 +252,23 @@ default Long getMinSize() { default void setMinSize(Long minSize) { // No-op for backward compatibility } + + /** + * Set whether JSON serialization should be used for {@link UploadInfo} metadata instead of + * standard Java object serialization. + * + * @param enabled True to enable JSON serialization, false to use default serialization + */ + default void setJsonSerializationEnabled(boolean enabled) { + // No-op for backward compatibility + } + + /** + * Check if JSON serialization is enabled for {@link UploadInfo} metadata. + * + * @return True if JSON serialization is enabled, false otherwise + */ + default boolean isJsonSerializationEnabled() { + return false; + } } diff --git a/src/main/java/me/desair/tus/server/upload/cache/ThreadLocalCachedStorageAndLockingService.java b/src/main/java/me/desair/tus/server/upload/cache/ThreadLocalCachedStorageAndLockingService.java index 5cc8a024..30de697d 100644 --- a/src/main/java/me/desair/tus/server/upload/cache/ThreadLocalCachedStorageAndLockingService.java +++ b/src/main/java/me/desair/tus/server/upload/cache/ThreadLocalCachedStorageAndLockingService.java @@ -217,6 +217,16 @@ public boolean isUploadDeduplicationEnabled() { return storageServiceDelegate.isUploadDeduplicationEnabled(); } + @Override + public void setJsonSerializationEnabled(boolean enabled) { + storageServiceDelegate.setJsonSerializationEnabled(enabled); + } + + @Override + public boolean isJsonSerializationEnabled() { + return storageServiceDelegate.isJsonSerializationEnabled(); + } + @Override public UploadInfo getUploadInfoByChecksum(String checksum, ChecksumAlgorithm algorithm) throws IOException { diff --git a/src/main/java/me/desair/tus/server/upload/disk/DiskStorageService.java b/src/main/java/me/desair/tus/server/upload/disk/DiskStorageService.java index c84fccb7..46642b17 100644 --- a/src/main/java/me/desair/tus/server/upload/disk/DiskStorageService.java +++ b/src/main/java/me/desair/tus/server/upload/disk/DiskStorageService.java @@ -53,6 +53,7 @@ public class DiskStorageService extends AbstractDiskBasedService implements Uplo private UploadIdFactory idFactory; private UploadConcatenationService uploadConcatenationService; private boolean isUploadDeduplicationEnabled = false; + private boolean jsonSerializationEnabled = false; public DiskStorageService(String storagePath) { super(storagePath + File.separator + UPLOAD_SUB_DIRECTORY); @@ -166,6 +167,16 @@ public UploadInfo getUploadInfo(String uploadUrl, String ownerKey) throws IOExce } } + @Override + public void setJsonSerializationEnabled(boolean enabled) { + this.jsonSerializationEnabled = enabled; + } + + @Override + public boolean isJsonSerializationEnabled() { + return this.jsonSerializationEnabled; + } + @Override public UploadInfo getUploadInfo(UploadId id) throws IOException { if (id == null) { @@ -176,12 +187,29 @@ public UploadInfo getUploadInfo(UploadId id) throws IOException { if (infoPath == null || !Files.exists(infoPath)) { return null; } - return Utils.readSerializable(infoPath, UploadInfo.class); + return loadUploadInfo(infoPath); } catch (UploadNotFoundException e) { return null; } } + private void saveUploadInfo(UploadInfo info, Path path) throws IOException { + if (isJsonSerializationEnabled()) { + Utils.writeJson(info, path); + } else { + Utils.writeSerializable(info, path); + } + } + + private UploadInfo loadUploadInfo(Path path) throws IOException { + if (isJsonSerializationEnabled()) { + UploadInfo info = Utils.readJson(path, UploadInfo.class); + return info != null ? info : Utils.readSerializable(path, UploadInfo.class); + } else { + return Utils.readSerializable(path, UploadInfo.class); + } + } + @Override public String getUploadUri() { return idFactory.getUploadUri(); @@ -245,13 +273,13 @@ public void update(UploadInfo uploadInfo) throws IOException, UploadNotFoundExce if (parentExpire != null) { parentInfo.setExpirationTimestamp(null); Path parentInfoPath = getInfoPath(parentId); - Utils.writeSerializable(parentInfo, parentInfoPath); + saveUploadInfo(parentInfo, parentInfoPath); } } else { if (parentExpire != null && parentExpire < childExpire) { parentInfo.setExpirationTimestamp(childExpire); Path parentInfoPath = getInfoPath(parentId); - Utils.writeSerializable(parentInfo, parentInfoPath); + saveUploadInfo(parentInfo, parentInfoPath); } } } @@ -268,7 +296,7 @@ public void update(UploadInfo uploadInfo) throws IOException, UploadNotFoundExce } Path infoPath = getInfoPath(uploadInfo.getId()); - Utils.writeSerializable(uploadInfo, infoPath); + saveUploadInfo(uploadInfo, infoPath); } } diff --git a/src/main/java/me/desair/tus/server/upload/disk/ExpiredUploadFilter.java b/src/main/java/me/desair/tus/server/upload/disk/ExpiredUploadFilter.java index 93821f9a..717a68ff 100644 --- a/src/main/java/me/desair/tus/server/upload/disk/ExpiredUploadFilter.java +++ b/src/main/java/me/desair/tus/server/upload/disk/ExpiredUploadFilter.java @@ -39,9 +39,7 @@ public boolean accept(Path upload) throws IOException { } } catch (Exception ex) { - if (log.isDebugEnabled()) { - log.debug("Not able to determine state of upload " + Objects.toString(id), ex); - } + log.warn("Not able to determine state of upload " + Objects.toString(id), ex); } return false; diff --git a/src/main/java/me/desair/tus/server/upload/s3/S3ConcatenationService.java b/src/main/java/me/desair/tus/server/upload/s3/S3ConcatenationService.java new file mode 100644 index 00000000..b44c8db9 --- /dev/null +++ b/src/main/java/me/desair/tus/server/upload/s3/S3ConcatenationService.java @@ -0,0 +1,282 @@ +package me.desair.tus.server.upload.s3; + +import io.minio.ComposeObjectArgs; +import io.minio.MinioClient; +import io.minio.PutObjectArgs; +import io.minio.SourceObject; +import java.io.IOException; +import java.io.InputStream; +import java.io.SequenceInputStream; +import java.nio.file.Path; +import java.util.ArrayList; +import java.util.Collections; +import java.util.List; +import java.util.Objects; +import me.desair.tus.server.exception.UploadNotFoundException; +import me.desair.tus.server.upload.UploadInfo; +import me.desair.tus.server.upload.UploadStorageService; +import me.desair.tus.server.upload.concatenation.UploadConcatenationService; +import me.desair.tus.server.upload.concatenation.UploadInputStreamEnumeration; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +/** + * S3-native implementation of {@link UploadConcatenationService} using MinIO Java SDK. + * + *

Concatenation Strategy for Developers: + * + *

    + *
  • Server-Side S3 Object Composition ({@code composeObject}): When all partial upload + * parts meet S3's minimum part size constraint ($\ge$ 5 MB), concatenation is executed + * entirely on the S3 storage cluster using {@code composeObject}. This avoids downloading any + * bytes to the server, enabling instant multi-GB file stitching with zero bandwidth or RAM + * overhead. + *
  • Streaming Re-upload Fallback: If any partial upload is under 5 MB (sub-5MB parts + * cannot be composed via S3's native compose API), the service streams bytes sequentially + * using {@link SequenceInputStream} and re-uploads the concatenated stream directly to S3. + *
+ */ +public class S3ConcatenationService implements UploadConcatenationService { + + private static final Logger log = LoggerFactory.getLogger(S3ConcatenationService.class); + private static final long DEFAULT_MIN_PART_SIZE = 5L * 1024 * 1024; // 5 MB + + private final MinioClient minioClient; + private final String bucket; + private final String objectPrefix; + private final long minPartSize; + private final Path temporaryDirectory; + private UploadStorageService uploadStorageService; + + /** + * Basic constructor using default object prefix ("tus-uploads/") and Java temp directory. + * + * @param minioClient The MinIO client + * @param bucket The S3 bucket name + */ + public S3ConcatenationService(MinioClient minioClient, String bucket) { + this(minioClient, bucket, "tus-uploads/", null, null); + } + + /** + * Convenient constructor taking MinioClient, bucket, and UploadStorageService. + * + * @param minioClient The MinIO client + * @param bucket The S3 bucket name + * @param uploadStorageService Underlying storage service + */ + public S3ConcatenationService( + MinioClient minioClient, String bucket, UploadStorageService uploadStorageService) { + this(minioClient, bucket, "tus-uploads/", uploadStorageService, null); + } + + /** + * Constructs an S3ConcatenationService. + * + * @param minioClient The MinIO client + * @param bucket The S3 bucket name + * @param objectPrefix Key prefix for data objects + * @param uploadStorageService Underlying storage service + * @param temporaryDirectory Directory for temporary buffer files + */ + public S3ConcatenationService( + MinioClient minioClient, + String bucket, + String objectPrefix, + UploadStorageService uploadStorageService, + Path temporaryDirectory) { + this( + minioClient, + bucket, + objectPrefix, + uploadStorageService, + temporaryDirectory, + DEFAULT_MIN_PART_SIZE); + } + + /** Full constructor allowing custom minimum part size. */ + public S3ConcatenationService( + MinioClient minioClient, + String bucket, + String objectPrefix, + UploadStorageService uploadStorageService, + Path temporaryDirectory, + long minPartSize) { + this.minioClient = Objects.requireNonNull(minioClient, "MinioClient must not be null"); + this.bucket = Objects.requireNonNull(bucket, "Bucket must not be null"); + this.objectPrefix = objectPrefix != null ? objectPrefix : ""; + this.uploadStorageService = uploadStorageService; + this.temporaryDirectory = + temporaryDirectory != null + ? temporaryDirectory + : java.nio.file.Paths.get(System.getProperty("java.io.tmpdir")); + this.minPartSize = minPartSize; + } + + public void setUploadStorageService(UploadStorageService uploadStorageService) { + this.uploadStorageService = uploadStorageService; + } + + @Override + public void merge(UploadInfo uploadInfo) throws IOException, UploadNotFoundException { + if (uploadInfo == null + || !uploadInfo.isUploadInProgress() + || uploadInfo.getConcatenationPartIds() == null) { + return; + } + + Long expirationPeriod = + uploadStorageService != null ? uploadStorageService.getUploadExpirationPeriod() : null; + List partialUploads = getPartialUploads(uploadInfo); + + Long totalLength = calculateTotalLength(partialUploads); + boolean completed = checkAllCompleted(expirationPeriod, partialUploads); + + if (totalLength != null && totalLength > 0 && completed) { + // S3 Constraint Check: Server-side composeObject requires all source parts to be >= 5 MB + boolean canUseServerSideCopy = + partialUploads.stream() + .allMatch(p -> p.getLength() != null && p.getLength() >= minPartSize); + + String targetObjectKey = buildObjectKey(uploadInfo.getId().toString()); + + if (canUseServerSideCopy) { + // Fast path: Compose S3 objects on cluster server-side without downloading data + mergeUsingServerSideCopy(targetObjectKey, partialUploads); + } else { + // Fallback path: Sequential stream re-upload for sub-5MB parts + mergeUsingStreamingReupload(targetObjectKey, partialUploads, totalLength); + } + + uploadInfo.setLength(totalLength); + uploadInfo.setOffset(totalLength); + if (expirationPeriod != null) { + uploadInfo.updateExpiration(expirationPeriod); + } + uploadInfo.setStorageUploadId(targetObjectKey); + + if (uploadStorageService != null) { + try { + uploadStorageService.update(uploadInfo); + } catch (UploadNotFoundException e) { + log.warn("Failed to update concatenated upload info for " + uploadInfo.getId(), e); + } + } + } + } + + @Override + public InputStream getConcatenatedBytes(UploadInfo uploadInfo) + throws IOException, UploadNotFoundException { + + if (uploadInfo == null) { + return null; + } + + if (uploadInfo.getStorageUploadId() == null) { + merge(uploadInfo); + } + + if (uploadStorageService != null) { + return uploadStorageService.getUploadedBytes(uploadInfo.getId()); + } + + throw new IOException( + "UploadStorageService must be configured to retrieve concatenated upload bytes"); + } + + @Override + public List getPartialUploads(UploadInfo info) + throws IOException, UploadNotFoundException { + List concatenationParts = info.getConcatenationPartIds(); + + if (concatenationParts == null || concatenationParts.isEmpty()) { + return Collections.emptyList(); + } + + List output = new ArrayList<>(concatenationParts.size()); + for (String childUri : concatenationParts) { + UploadInfo childInfo = + uploadStorageService != null + ? uploadStorageService.getUploadInfo(childUri, info.getOwnerKey()) + : null; + if (childInfo == null) { + throw new UploadNotFoundException( + "Upload with URI " + childUri + " was not found for owner " + info.getOwnerKey()); + } + output.add(childInfo); + } + return output; + } + + private void mergeUsingServerSideCopy(String targetKey, List partialUploads) + throws IOException { + try { + List sources = new ArrayList<>(); + for (UploadInfo partial : partialUploads) { + String partKey = + partial.getStorageUploadId() != null + ? partial.getStorageUploadId() + : buildObjectKey(partial.getId().toString()); + sources.add(SourceObject.builder().bucket(bucket).object(partKey).build()); + } + + // Execute S3 server-side object composition + minioClient.composeObject( + ComposeObjectArgs.builder().bucket(bucket).object(targetKey).sources(sources).build()); + } catch (Exception e) { + throw new IOException("Failed server-side S3 composeObject merge for key " + targetKey, e); + } + } + + private void mergeUsingStreamingReupload( + String targetKey, List partialUploads, long totalLength) throws IOException { + try { + InputStream combinedStream = + new SequenceInputStream( + new UploadInputStreamEnumeration(partialUploads, uploadStorageService)); + + minioClient.putObject( + PutObjectArgs.builder().bucket(bucket).object(targetKey).stream( + combinedStream, totalLength, -1L) + .build()); + } catch (Exception e) { + throw new IOException("Failed streaming re-upload merge for key " + targetKey, e); + } + } + + private Long calculateTotalLength(List partialUploads) { + Long totalLength = 0L; + for (UploadInfo childInfo : partialUploads) { + if (childInfo.getLength() == null) { + return null; + } + totalLength += childInfo.getLength(); + } + return totalLength; + } + + private boolean checkAllCompleted(Long expirationPeriod, List partialUploads) + throws IOException { + boolean completed = true; + for (UploadInfo childInfo : partialUploads) { + if (childInfo.isUploadInProgress()) { + completed = false; + } else if (expirationPeriod != null) { + childInfo.updateExpiration(expirationPeriod); + if (uploadStorageService != null) { + try { + uploadStorageService.update(childInfo); + } catch (UploadNotFoundException e) { + log.debug("Failed to update child upload expiration for " + childInfo.getId(), e); + } + } + } + } + return completed; + } + + private String buildObjectKey(String id) { + return objectPrefix + id; + } +} diff --git a/src/main/java/me/desair/tus/server/upload/s3/S3ErrorType.java b/src/main/java/me/desair/tus/server/upload/s3/S3ErrorType.java new file mode 100644 index 00000000..11b82f77 --- /dev/null +++ b/src/main/java/me/desair/tus/server/upload/s3/S3ErrorType.java @@ -0,0 +1,21 @@ +package me.desair.tus.server.upload.s3; + +/** Standardized S3/MinIO error types parsed from {@link io.minio.errors.ErrorResponseException}. */ +public enum S3ErrorType { + /** Target S3 object or key does not exist in bucket (HTTP 404 / NoSuchKey). */ + NO_SUCH_KEY, + + /** + * Conditional request precondition failed (HTTP 412 / PreconditionFailed / If-None-Match match). + */ + PRECONDITION_FAILED, + + /** Lock or resource already exists / conflict (HTTP 409 / ObjectAlreadyExists). */ + CONFLICT, + + /** Permission denied or invalid credentials (HTTP 403 / AccessDenied). */ + ACCESS_DENIED, + + /** Unknown or unmapped S3 error response. */ + UNKNOWN +} diff --git a/src/main/java/me/desair/tus/server/upload/s3/S3LockingService.java b/src/main/java/me/desair/tus/server/upload/s3/S3LockingService.java new file mode 100644 index 00000000..a02ec304 --- /dev/null +++ b/src/main/java/me/desair/tus/server/upload/s3/S3LockingService.java @@ -0,0 +1,382 @@ +package me.desair.tus.server.upload.s3; + +import com.fasterxml.jackson.databind.ObjectMapper; +import io.minio.GetObjectArgs; +import io.minio.ListObjectsArgs; +import io.minio.MinioClient; +import io.minio.PutObjectArgs; +import io.minio.RemoveObjectArgs; +import io.minio.Result; +import io.minio.StatObjectArgs; +import io.minio.errors.ErrorResponseException; +import io.minio.messages.Item; +import java.io.ByteArrayInputStream; +import java.io.IOException; +import java.io.InputStream; +import java.util.Map; +import java.util.Objects; +import java.util.UUID; +import java.util.concurrent.ConcurrentHashMap; +import java.util.concurrent.Executors; +import java.util.concurrent.ScheduledExecutorService; +import java.util.concurrent.TimeUnit; +import me.desair.tus.server.exception.TusException; +import me.desair.tus.server.exception.UploadAlreadyLockedException; +import me.desair.tus.server.upload.UploadId; +import me.desair.tus.server.upload.UploadIdFactory; +import me.desair.tus.server.upload.UploadLock; +import me.desair.tus.server.upload.UploadLockingService; +import me.desair.tus.server.upload.UuidUploadIdFactory; +import me.desair.tus.server.util.InterruptibleInputStream; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +/** + * Distributed S3-backed implementation of {@link UploadLockingService} using the MinIO Java SDK. + * + *

Key Architecture Features & S3/MinIO Developer Guide: + * + *

    + *
  • Distributed Lock Lease Objects: Locks are represented as small JSON lease objects + * written to S3 under {@code /.lock}. Each lease object records a + * unique {@code holderId} and an absolute timestamp {@code expiresAt}. + *
  • Atomic Lock Acquisition: When an upload request arrives, the server checks whether + * an unexpired lock object already exists in S3. If no active lock is found, a new lock + * object is written to S3, granting exclusive ownership to the current thread/pod without + * requiring Redis or an external database. + *
  • Heartbeat & Lease Auto-Renewal: Managed locks spawn background daemon threads that + * periodically update the lock object in S3, keeping the lease active while long uploads run. + *
  • Cross-Pod Lock Contention & Interrupt Signals: When a concurrent request arrives for + * a locked upload (e.g., HEAD or DELETE while a PATCH is streaming data on another pod), the + * service writes a {@code /.stop} signal object to S3. A background + * watchdog thread on the pod holding the lock detects the {@code .stop} file and interrupts + * the active input stream immediately, resolving lock contention cleanly across Kubernetes + * pods. + *
+ */ +public class S3LockingService implements UploadLockingService { + + private static final Logger log = LoggerFactory.getLogger(S3LockingService.class); + private static final ObjectMapper OBJECT_MAPPER = new ObjectMapper(); + + public static final String DEFAULT_LOCKS_PREFIX = "locks/"; + public static final long DEFAULT_LEASE_DURATION_MS = 30_000L; // 30 seconds + public static final long DEFAULT_POLL_INTERVAL_MS = 2_000L; // 2 seconds + + private final MinioClient minioClient; + private final String bucket; + private final String locksPrefix; + private final long leaseDurationMs; + private final long pollIntervalMs; + + private UploadIdFactory idFactory = new UuidUploadIdFactory(); + private final Map activeInputStreams = new ConcurrentHashMap<>(); + private final ScheduledExecutorService watchdogExecutor; + + /** + * Basic constructor using default lock prefix ("locks/"), 30s lease duration, and 2s polling + * interval. + * + * @param minioClient Pre-configured MinIO Client + * @param bucket Target S3 bucket name + */ + public S3LockingService(MinioClient minioClient, String bucket) { + this( + minioClient, + bucket, + DEFAULT_LOCKS_PREFIX, + DEFAULT_LEASE_DURATION_MS, + DEFAULT_POLL_INTERVAL_MS); + } + + /** + * Full constructor allowing custom configuration for all locking parameters. + * + * @param minioClient Pre-configured MinIO Client + * @param bucket Target S3 bucket name + * @param locksPrefix Object key prefix for locks and stop signals + * @param leaseDurationMs Lock lease duration in milliseconds + * @param pollIntervalMs Watchdog poll interval for lock contention interrupt signals + */ + public S3LockingService( + MinioClient minioClient, + String bucket, + String locksPrefix, + long leaseDurationMs, + long pollIntervalMs) { + this.minioClient = Objects.requireNonNull(minioClient, "MinioClient must not be null"); + this.bucket = Objects.requireNonNull(bucket, "Bucket must not be null"); + this.locksPrefix = sanitizePrefix(locksPrefix); + this.leaseDurationMs = leaseDurationMs; + this.pollIntervalMs = pollIntervalMs; + + // Background watchdog thread to poll S3 for .stop contention signals across pods + this.watchdogExecutor = + Executors.newSingleThreadScheduledExecutor( + r -> { + Thread t = new Thread(r, "s3-lock-watchdog"); + t.setDaemon(true); + return t; + }); + + if (pollIntervalMs > 0) { + this.watchdogExecutor.scheduleAtFixedRate( + this::checkStopSignals, pollIntervalMs, pollIntervalMs, TimeUnit.MILLISECONDS); + } + } + + @Override + public UploadLock lockUploadByUri(String requestUri) throws TusException, IOException { + UploadId uploadId = idFactory.readUploadId(requestUri); + if (uploadId == null) { + return null; + } + + String lockKey = buildLockKey(uploadId); + String stopKey = buildStopKey(uploadId); + String holderId = UUID.randomUUID().toString(); + + // Attempt lock acquisition or clear expired lock + boolean acquired = acquireOrEvictExpiredLock(lockKey, holderId); + if (!acquired) { + throw new UploadAlreadyLockedException("Upload " + uploadId + " is currently locked"); + } + + // Create and return S3UploadLock instance with heartbeat lease auto-renewal + return new S3UploadLock( + minioClient, + bucket, + lockKey, + stopKey, + holderId, + leaseDurationMs, + requestUri, + activeInputStreams); + } + + @Override + public void cleanupStaleLocks() throws IOException { + try { + // List all object keys under locksPrefix in S3 + Iterable> results = + minioClient.listObjects( + ListObjectsArgs.builder().bucket(bucket).prefix(locksPrefix).build()); + + for (Result result : results) { + Item item = result.get(); + // Remove expired .lock lease objects + if (item.objectName().endsWith(".lock") && isLockExpired(item.objectName())) { + deleteObjectQuietly(item.objectName()); + } + } + } catch (Exception e) { + throw new IOException("Failed to cleanup stale S3 locks", e); + } + } + + @Override + public boolean isLocked(UploadId id) { + if (id == null) { + return false; + } + String lockKey = buildLockKey(id); + return !isLockExpired(lockKey); + } + + @Override + public void setIdFactory(UploadIdFactory idFactory) { + if (idFactory != null) { + this.idFactory = idFactory; + } + } + + @Override + public void registerInputStream(String requestUri, InputStream inputStream) { + if (requestUri != null && inputStream != null) { + activeInputStreams.put(requestUri, inputStream); + } + } + + @Override + public void requestLockRelease(String requestUri) { + if (requestUri == null) { + return; + } + + // Step 1: Interrupt local active payload byte stream if hosted on this node + InputStream activeStream = activeInputStreams.get(requestUri); + if (activeStream != null) { + interruptStream(activeStream); + } + + // Step 2: Write a .stop signal object to S3 to signal lock contention across remote nodes/pods + UploadId uploadId = idFactory.readUploadId(requestUri); + if (uploadId != null) { + writeStopSignal(uploadId); + } + } + + // HELPER METHODS & LOCK MANAGEMENT LOGIC + + private boolean acquireOrEvictExpiredLock(String lockKey, String holderId) { + boolean acquired = attemptLockAcquisition(lockKey, holderId); + if (!acquired && isLockExpired(lockKey)) { + deleteObjectQuietly(lockKey); + acquired = attemptLockAcquisition(lockKey, holderId); + } + return acquired; + } + + private boolean attemptLockAcquisition(String lockKey, String holderId) { + if (!isLockExpired(lockKey)) { + return false; + } + + long expiresAt = System.currentTimeMillis() + leaseDurationMs; + try { + byte[] lockContentBytes = OBJECT_MAPPER.writeValueAsBytes(new LockData(holderId, expiresAt)); + + // Put lock lease object to S3 + minioClient.putObject( + PutObjectArgs.builder().bucket(bucket).object(lockKey).stream( + new ByteArrayInputStream(lockContentBytes), (long) lockContentBytes.length, -1L) + .build()); + return true; + } catch (Exception e) { + log.warn("Unexpected error acquiring S3 lock for key {}", lockKey, e); + return false; + } + } + + private boolean isLockExpired(String lockKey) { + try (InputStream stream = + minioClient.getObject(GetObjectArgs.builder().bucket(bucket).object(lockKey).build())) { + + LockData lockData = OBJECT_MAPPER.readValue(stream, LockData.class); + return lockData.expiresAt < System.currentTimeMillis(); + } catch (ErrorResponseException e) { + if (S3Utils.parseErrorResponse(e) == S3ErrorType.NO_SUCH_KEY) { + return true; // Key missing -> Not locked + } + return true; + } catch (Exception e) { + log.debug("Failed to read lock object {}, treating as expired", lockKey, e); + return true; + } + } + + private void writeStopSignal(UploadId uploadId) { + String stopKey = buildStopKey(uploadId); + try { + byte[] empty = new byte[0]; + // Write empty .stop signal object to S3 + minioClient.putObject( + PutObjectArgs.builder().bucket(bucket).object(stopKey).stream( + new ByteArrayInputStream(empty), 0L, -1L) + .build()); + } catch (Exception e) { + log.debug("Failed to write lock stop signal to S3 key {}", stopKey, e); + } + } + + private void checkStopSignals() { + for (Map.Entry entry : activeInputStreams.entrySet()) { + checkStopSignalForEntry(entry.getKey(), entry.getValue()); + } + } + + private void checkStopSignalForEntry(String uri, InputStream inputStream) { + UploadId uploadId = idFactory.readUploadId(uri); + if (uploadId == null) { + return; + } + + String stopKey = buildStopKey(uploadId); + try { + // Check if a .stop signal object was written by another pod requesting lock release + minioClient.statObject(StatObjectArgs.builder().bucket(bucket).object(stopKey).build()); + // Remote stop signal object found! Interrupt local byte stream immediately + interruptStream(inputStream); + } catch (ErrorResponseException e) { + if ("NoSuchKey".equalsIgnoreCase(e.errorResponse().code())) { + // Normal state: no stop signal object in S3 + return; + } + } catch (Exception e) { + log.debug("Error checking stop signal for {}", stopKey, e); + } + } + + private void interruptStream(InputStream is) { + if (is instanceof InterruptibleInputStream) { + ((InterruptibleInputStream) is).interrupt(); + } else { + try { + is.close(); + } catch (Exception ignored) { + } + } + } + + private void deleteObjectQuietly(String key) { + try { + minioClient.removeObject(RemoveObjectArgs.builder().bucket(bucket).object(key).build()); + } catch (Exception e) { + log.debug("Failed to delete S3 object key {}", key, e); + } + } + + private String sanitizePrefix(String prefix) { + if (prefix == null || prefix.isEmpty()) { + return ""; + } + String result = prefix.startsWith("/") ? prefix.substring(1) : prefix; + return result.endsWith("/") ? result : result + "/"; + } + + private String buildLockKey(UploadId uploadId) { + return locksPrefix + uploadId.toString() + ".lock"; + } + + private String buildStopKey(UploadId uploadId) { + return locksPrefix + uploadId.toString() + ".stop"; + } + + public static class LockData { + private String holderId; + private long expiresAt; + private long acquiredAt; + + public LockData() {} + + public LockData(String holderId, long expiresAt) { + this.holderId = holderId; + this.expiresAt = expiresAt; + this.acquiredAt = System.currentTimeMillis(); + } + + public String getHolderId() { + return holderId; + } + + public void setHolderId(String holderId) { + this.holderId = holderId; + } + + public long getExpiresAt() { + return expiresAt; + } + + public void setExpiresAt(long expiresAt) { + this.expiresAt = expiresAt; + } + + public long getAcquiredAt() { + return acquiredAt; + } + + public void setAcquiredAt(long acquiredAt) { + this.acquiredAt = acquiredAt; + } + } +} diff --git a/src/main/java/me/desair/tus/server/upload/s3/S3StorageService.java b/src/main/java/me/desair/tus/server/upload/s3/S3StorageService.java new file mode 100644 index 00000000..a5379418 --- /dev/null +++ b/src/main/java/me/desair/tus/server/upload/s3/S3StorageService.java @@ -0,0 +1,1036 @@ +package me.desair.tus.server.upload.s3; + +import io.minio.ComposeObjectArgs; +import io.minio.GetObjectArgs; +import io.minio.ListObjectsArgs; +import io.minio.MinioClient; +import io.minio.PutObjectArgs; +import io.minio.RemoveObjectArgs; +import io.minio.Result; +import io.minio.SourceObject; +import io.minio.StatObjectArgs; +import io.minio.StatObjectResponse; +import io.minio.errors.ErrorResponseException; +import io.minio.messages.Item; +import java.io.ByteArrayInputStream; +import java.io.File; +import java.io.FileInputStream; +import java.io.FileOutputStream; +import java.io.IOException; +import java.io.InputStream; +import java.io.OutputStream; +import java.io.SequenceInputStream; +import java.nio.charset.StandardCharsets; +import java.nio.file.Path; +import java.nio.file.Paths; +import java.util.ArrayList; +import java.util.List; +import java.util.Objects; +import me.desair.tus.server.checksum.ChecksumAlgorithm; +import me.desair.tus.server.exception.MaxAppendSizeExceededException; +import me.desair.tus.server.exception.MaxUploadLengthExceededException; +import me.desair.tus.server.exception.MinAppendSizeNotMetException; +import me.desair.tus.server.exception.MinUploadLengthNotReachedException; +import me.desair.tus.server.exception.TusException; +import me.desair.tus.server.exception.UploadNotFoundException; +import me.desair.tus.server.upload.UploadId; +import me.desair.tus.server.upload.UploadIdFactory; +import me.desair.tus.server.upload.UploadInfo; +import me.desair.tus.server.upload.UploadLockingService; +import me.desair.tus.server.upload.UploadStorageService; +import me.desair.tus.server.upload.UploadType; +import me.desair.tus.server.upload.UuidUploadIdFactory; +import me.desair.tus.server.upload.concatenation.UploadConcatenationService; +import me.desair.tus.server.util.UploadInfoJsonSerializer; +import org.apache.commons.io.IOUtils; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +/** + * MinIO S3-backed implementation of {@link UploadStorageService} using the lightweight MinIO Java + * SDK. + * + *

Key Design Architecture & S3/MinIO Developer Guide: + * + *

    + *
  • Server-Side Object Composition ({@code composeObject}): Instead of assembling + * multi-GB uploads locally on disk or in server RAM, completed chunk parts are combined + * directly on the S3 storage cluster using S3 server-side object composition. This yields + * zero server memory overhead and ultra-fast completion times. + *
  • Sub-5MB Incomplete Part Buffering: AWS S3 and MinIO require every part chunk of a + * multipart upload to be at least 5 MB (5,242,880 bytes), except for the final part. To + * handle arbitrarily small client appends (e.g., 64 KB network packets or frequent small + * PATCH calls), sub-5MB tail bytes are buffered to S3 as a temporary {@code + * /.part} object. When a subsequent PATCH arrives, this leftover + * part is fetched, prepended to the incoming payload stream, and processed seamlessly. + *
  • Dynamic Optimal Part Sizing: Auto-scales part chunk sizes between 5 MB and 5 GB + * (capped at S3's maximum limit of 10,000 parts per object). + *
  • Zero-Byte & Checksum Deduplication Support: Seamlessly manages 0-byte upload + * creation and index lookup for instant duplicate file matching. + *
+ */ +public class S3StorageService implements UploadStorageService { + + private static final Logger log = LoggerFactory.getLogger(S3StorageService.class); + + // Key Prefixes for S3 object layout separation + public static final String DEFAULT_OBJECT_PREFIX = "tus-uploads/"; + public static final String DEFAULT_METADATA_PREFIX = "metadata/"; + public static final String DEFAULT_CHECKSUMS_PREFIX = "checksums/"; + public static final String DEFAULT_LOCKS_PREFIX = "locks/"; + + // Part Sizing Constraints (per AWS S3 & MinIO specifications) + private static final long DEFAULT_MIN_PART_SIZE = + 5L * 1024 * 1024; // 5 MB (S3 minimum part limit) + private static final long DEFAULT_PREFERRED_PART_SIZE = + 50L * 1024 * 1024; // 50 MB (Optimal chunk size) + private static final long DEFAULT_MAX_PART_SIZE = + 5L * 1024 * 1024 * 1024L; // 5 GB (S3 maximum object/part limit) + + private final MinioClient minioClient; + private final String bucket; + private final String objectPrefix; + private final String metadataPrefix; + private final String checksumsPrefix; + private final String locksPrefix; + private final Path temporaryDirectory; + + private long minPartSize = DEFAULT_MIN_PART_SIZE; + private long preferredPartSize = DEFAULT_PREFERRED_PART_SIZE; + + private Long maxUploadSize; + private Long maxAppendSize; + private Long minAppendSize; + private Long minSize; + private Long uploadExpirationPeriod; + private boolean deduplicationEnabled = false; + + private UploadIdFactory idFactory = new UuidUploadIdFactory(); + private UploadConcatenationService concatenationService; + + /** + * Basic constructor using default object key prefixes and standard system temp directory. + * + * @param minioClient Pre-configured MinIO Client + * @param bucket S3 bucket name + */ + public S3StorageService(MinioClient minioClient, String bucket) { + this( + minioClient, + bucket, + DEFAULT_OBJECT_PREFIX, + DEFAULT_METADATA_PREFIX, + DEFAULT_CHECKSUMS_PREFIX, + DEFAULT_LOCKS_PREFIX, + Paths.get(System.getProperty("java.io.tmpdir"))); + } + + /** + * Full constructor allowing full customization of object prefixes and local disk buffer path. + * + * @param minioClient Pre-configured MinIO Client + * @param bucket S3 bucket name + * @param objectPrefix Key prefix for final completed file objects + * @param metadataPrefix Key prefix for metadata (.info JSON and .part buffer) objects + * @param checksumsPrefix Key prefix for checksum deduplication index objects + * @param locksPrefix Key prefix for distributed lock lease objects + * @param temporaryDirectory Local directory path for staging chunks before S3 upload + */ + public S3StorageService( + MinioClient minioClient, + String bucket, + String objectPrefix, + String metadataPrefix, + String checksumsPrefix, + String locksPrefix, + Path temporaryDirectory) { + this.minioClient = Objects.requireNonNull(minioClient, "MinioClient must not be null"); + this.bucket = Objects.requireNonNull(bucket, "Bucket must not be null"); + this.objectPrefix = sanitizePrefix(objectPrefix); + this.metadataPrefix = sanitizePrefix(metadataPrefix); + this.checksumsPrefix = sanitizePrefix(checksumsPrefix); + this.locksPrefix = sanitizePrefix(locksPrefix); + this.temporaryDirectory = + temporaryDirectory != null + ? temporaryDirectory + : Paths.get(System.getProperty("java.io.tmpdir")); + + this.concatenationService = + new S3ConcatenationService( + this.minioClient, this.bucket, this.objectPrefix, this, this.temporaryDirectory); + } + + /** + * Returns the S3 object key for the completed upload data of the given upload info. If the upload + * was deduplicated, this returns the parent upload's physical S3 object key. + * + * @param uploadInfo The upload info object + * @return The full S3 object key for the uploaded data + */ + public String getS3ObjectKey(UploadInfo uploadInfo) { + if (uploadInfo == null || uploadInfo.getId() == null) { + return null; + } + String id = uploadInfo.getId().toString(); + if (uploadInfo.getStorageUploadId() != null && !uploadInfo.getStorageUploadId().equals(id)) { + return uploadInfo.getStorageUploadId(); + } + return buildObjectKey(id); + } + + /** + * Return the S3 object key where the uploaded bytes are stored for the given upload URI. + * + * @param uploadUri The HTTP request URI of the upload + * @return The target S3 object key or null if upload not found + */ + public String getS3ObjectKey(String uploadUri) { + try { + UploadId uploadId = idFactory.readUploadId(uploadUri); + if (uploadId == null) { + return null; + } + UploadInfo uploadInfo = getUploadInfo(uploadId); + return getS3ObjectKey(uploadInfo); + } catch (IOException e) { + log.debug("Error retrieving upload info for URI {}", uploadUri, e); + return null; + } + } + + /** + * Return the S3 object key where the uploaded bytes are stored for the given upload URI and owner + * key. + * + * @param uploadUri The HTTP request URI of the upload + * @param ownerKey The owner key of the upload + * @return The target S3 object key or null if upload not found + */ + public String getS3ObjectKey(String uploadUri, String ownerKey) { + try { + UploadInfo uploadInfo = getUploadInfo(uploadUri, ownerKey); + return getS3ObjectKey(uploadInfo); + } catch (IOException e) { + log.debug("Error retrieving upload info for URI {}", uploadUri, e); + return null; + } + } + + @Override + public UploadInfo getUploadInfo(String uploadUrl, String ownerKey) throws IOException { + UploadId uploadId = idFactory.readUploadId(uploadUrl); + if (uploadId == null) { + return null; + } + UploadInfo info = getUploadInfo(uploadId); + // Enforce strict owner isolation if ownerKey is configured + if (info != null && info.getOwnerKey() != null && !info.getOwnerKey().equals(ownerKey)) { + return null; + } + return info; + } + + @Override + public UploadInfo getUploadInfo(UploadId id) throws IOException { + if (id == null) { + return null; + } + + String metadataKey = buildMetadataKey(id.toString()); + String json; + // Step 1: Read JSON metadata object from S3 (/.info) + try (InputStream stream = + minioClient.getObject(GetObjectArgs.builder().bucket(bucket).object(metadataKey).build())) { + json = IOUtils.toString(stream, StandardCharsets.UTF_8); + } catch (ErrorResponseException e) { + // Return null if key does not exist in S3 + if (S3Utils.parseErrorResponse(e) == S3ErrorType.NO_SUCH_KEY) { + return null; + } + throw new IOException("Failed to fetch metadata object from S3 for ID " + id, e); + } catch (Exception e) { + throw new IOException("Failed to fetch metadata object from S3 for ID " + id, e); + } + + // Step 2: Deserialize JSON into UploadInfo instance + UploadInfo info = UploadInfoJsonSerializer.deserialize(json); + if (info == null) { + return null; + } + + // Step 3: Dynamically compute uploaded byte offset if not explicitly set + if (info.getOffset() == null) { + calculateAndSetOffset(info); + } + return info; + } + + @Override + public String getUploadUri() { + return idFactory != null ? idFactory.getUploadUri() : "/"; + } + + @Override + public UploadInfo create(UploadInfo info, String ownerKey) throws IOException { + Objects.requireNonNull(info, "UploadInfo must not be null"); + + // Assign new upload ID if missing + if (info.getId() == null) { + info.setId(idFactory.createId()); + } + info.setOwnerKey(ownerKey); + info.setStorageUploadId(info.getId().toString()); + + // Persist initial UploadInfo metadata object (.info) to S3 + try { + update(info); + } catch (UploadNotFoundException e) { + log.error("Unable to update UploadInfo for newly created upload ID " + info.getId(), e); + } + return info; + } + + @Override + public UploadInfo append(UploadInfo upload, InputStream inputStream) + throws IOException, TusException { + // Step 1: Verify upload existence and check configured size limits + UploadInfo info = fetchAndValidateUpload(upload.getId()); + String id = info.getId().toString(); + String objectKey = getS3ObjectKey(info); + String partObjectKey = buildIncompletePartKey(id); + + // Step 2: If a previous sub-5MB .part buffer exists in S3, download & prepend it to incoming + // stream + InputStream streamToRead = prepareStreamWithExistingIncompletePart(partObjectKey, inputStream); + + // Step 3: Process payload stream in optimal chunk parts and upload to S3 + AppendResult appendResult = processPayloadChunks(info, streamToRead, id, partObjectKey); + + // Step 4: Validate minimum append size constraints if configured + if (minAppendSize != null && appendResult.totalBytesAppended < minAppendSize) { + throw new MinAppendSizeNotMetException( + "Append payload size " + + appendResult.totalBytesAppended + + " is below minimum limit " + + minAppendSize); + } + + // Step 5: Recalculate total uploaded byte offset across all uploaded part objects in S3 + long newOffset = calculateCurrentOffset(objectKey, id, partObjectKey); + info.setOffset(newOffset); + + // Step 6: If all expected bytes are uploaded, compose all part chunks into final S3 object + finalizeCompletedUploadIfFinished(info, objectKey, id, appendResult, newOffset); + update(info); + return info; + } + + @Override + public void update(UploadInfo uploadInfo) throws IOException, UploadNotFoundException { + if (uploadInfo == null || uploadInfo.getId() == null) { + return; + } + String metadataKey = buildMetadataKey(uploadInfo.getId().toString()); + String json = UploadInfoJsonSerializer.serialize(uploadInfo); + byte[] jsonBytes = json.getBytes(StandardCharsets.UTF_8); + + // Upload JSON metadata object to S3 + try { + minioClient.putObject( + PutObjectArgs.builder().bucket(bucket).object(metadataKey).stream( + new ByteArrayInputStream(jsonBytes), (long) jsonBytes.length, -1L) + .contentType("application/json") + .build()); + } catch (Exception e) { + throw new IOException( + "Failed to write metadata object to S3 for ID " + uploadInfo.getId(), e); + } + + // Index checksum for deduplication if upload is completed and deduplication is enabled + if (isUploadDeduplicationEnabled() + && uploadInfo.getChecksum() != null + && !uploadInfo.isUploadInProgress() + && uploadInfo.getDuplicatesUploadId() == null) { + putChecksumIndex( + uploadInfo.getChecksum(), + uploadInfo.getChecksumAlgorithm(), + uploadInfo.getId().toString()); + } + } + + @Override + public InputStream getUploadedBytes(String uploadUri, String ownerKey) + throws IOException, UploadNotFoundException { + UploadInfo info = getUploadInfo(uploadUri, ownerKey); + if (info == null) { + throw new UploadNotFoundException("Upload not found for URI " + uploadUri); + } + return getUploadedBytes(info.getId()); + } + + @Override + public InputStream getUploadedBytes(UploadId id) throws IOException, UploadNotFoundException { + UploadInfo info = getUploadInfo(id); + if (info == null) { + throw new UploadNotFoundException("Upload with ID " + id + " was not found"); + } + + // Resolve duplicate upload reference to parent upload if deduplicated + if (info.getDuplicatesUploadId() != null) { + return getUploadedBytes(info.getDuplicatesUploadId()); + } + + // Handle concatenated upload resolution if applicable + if (UploadType.CONCATENATED.equals(info.getUploadType()) && info.getStorageUploadId() == null) { + if (concatenationService != null) { + concatenationService.merge(info); + info = getUploadInfo(id); + } + } + + return fetchS3ByteStream(id, info); + } + + @Override + public void copyUploadTo(UploadInfo info, OutputStream outputStream) + throws UploadNotFoundException, IOException { + try (InputStream is = getUploadedBytes(info.getId())) { + IOUtils.copy(is, outputStream); + } + } + + @Override + public void cleanupExpiredUploads(UploadLockingService uploadLockingService) throws IOException { + try { + // List all metadata objects under metadataPrefix (e.g. metadata/*.info) + Iterable> results = + minioClient.listObjects( + ListObjectsArgs.builder().bucket(bucket).prefix(metadataPrefix).build()); + + for (Result result : results) { + Item item = result.get(); + if (item.objectName().endsWith(".info")) { + String idStr = + item.objectName() + .substring( + metadataPrefix.length(), item.objectName().length() - ".info".length()); + UploadId id = new UploadId(idStr); + UploadInfo info = getUploadInfo(id); + + // Delete expired uploads if not currently locked by an active request + if (info != null + && info.isExpired() + && (uploadLockingService == null || !uploadLockingService.isLocked(id))) { + terminateUpload(info); + } + } + } + } catch (Exception e) { + throw new IOException("Failed to cleanup expired S3 uploads", e); + } + } + + @Override + public void removeLastNumberOfBytes(UploadInfo uploadInfo, long byteCount) + throws UploadNotFoundException, IOException { + if (uploadInfo == null || byteCount <= 0) { + return; + } + String id = uploadInfo.getId().toString(); + String objectKey = getS3ObjectKey(uploadInfo); + String partKey = buildIncompletePartKey(id); + + long newOffset = Math.max(0L, uploadInfo.getOffset() - byteCount); + uploadInfo.setOffset(newOffset); + update(uploadInfo); + + // If final completed object exists in S3, truncate it + if (objectExists(objectKey)) { + truncateFromCompletedObject(objectKey, partKey, newOffset); + return; + } + + // Otherwise truncate from incomplete .part object + truncateFromIncompletePart(partKey, byteCount); + } + + @Override + public void terminateUpload(UploadInfo uploadInfo) throws UploadNotFoundException, IOException { + if (uploadInfo == null || uploadInfo.getId() == null) { + return; + } + String id = uploadInfo.getId().toString(); + String objectKey = getS3ObjectKey(uploadInfo); + String metadataKey = buildMetadataKey(id); + String partKey = buildIncompletePartKey(id); + + // Delete final object, metadata object, and incomplete part object from S3 + deleteObjectQuietly(objectKey); + deleteObjectQuietly(metadataKey); + deleteObjectQuietly(partKey); + + // Delete all temporary part chunk objects (e.g. metadata/.part.00001) + deleteAllPartObjectsQuietly(id); + + // Delete checksum deduplication index object if present + if (uploadInfo.getChecksum() != null && uploadInfo.getChecksumAlgorithm() != null) { + deleteObjectQuietly( + buildChecksumKey(uploadInfo.getChecksum(), uploadInfo.getChecksumAlgorithm())); + } + } + + @Override + public UploadInfo getUploadInfoByChecksum(String checksum, ChecksumAlgorithm algorithm) + throws IOException { + if (!isUploadDeduplicationEnabled() || checksum == null || algorithm == null) { + return null; + } + + String checksumKey = buildChecksumKey(checksum, algorithm); + String parentIdStr; + try (InputStream stream = + minioClient.getObject(GetObjectArgs.builder().bucket(bucket).object(checksumKey).build())) { + parentIdStr = IOUtils.toString(stream, StandardCharsets.UTF_8).trim(); + } catch (ErrorResponseException e) { + if (S3Utils.parseErrorResponse(e) == S3ErrorType.NO_SUCH_KEY) { + return null; + } + throw new IOException("Failed to read checksum index object from S3", e); + } catch (Exception e) { + throw new IOException("Failed to read checksum index object from S3", e); + } + + UploadInfo parentInfo = getUploadInfo(new UploadId(parentIdStr)); + // Self-cleaning: if index points to missing parent upload or object, prune stale index + if (parentInfo == null || !objectExists(buildObjectKey(parentIdStr))) { + deleteObjectQuietly(checksumKey); + return null; + } + + return parentInfo; + } + + // CONFIGURATION SETTERS & GETTERS + + @Override + public void setMaxUploadSize(Long maxUploadSize) { + this.maxUploadSize = maxUploadSize; + } + + @Override + public long getMaxUploadSize() { + return maxUploadSize != null ? maxUploadSize : 0L; + } + + @Override + public void setMaxAppendSize(Long maxAppendSize) { + this.maxAppendSize = maxAppendSize; + } + + @Override + public Long getMaxAppendSize() { + return maxAppendSize != null ? maxAppendSize : (maxUploadSize != null ? maxUploadSize : null); + } + + @Override + public void setMinAppendSize(Long minAppendSize) { + this.minAppendSize = minAppendSize; + } + + @Override + public Long getMinAppendSize() { + return minAppendSize; + } + + @Override + public void setMinSize(Long minSize) { + this.minSize = minSize; + } + + @Override + public Long getMinSize() { + return minSize; + } + + @Override + public void setUploadExpirationPeriod(Long uploadExpirationPeriod) { + this.uploadExpirationPeriod = uploadExpirationPeriod; + } + + @Override + public Long getUploadExpirationPeriod() { + return uploadExpirationPeriod; + } + + @Override + public void setUploadDeduplicationEnabled(boolean enabled) { + this.deduplicationEnabled = enabled; + } + + @Override + public boolean isUploadDeduplicationEnabled() { + return deduplicationEnabled; + } + + @Override + public void setUploadConcatenationService(UploadConcatenationService concatenationService) { + this.concatenationService = concatenationService; + } + + @Override + public UploadConcatenationService getUploadConcatenationService() { + return concatenationService; + } + + @Override + public void setIdFactory(UploadIdFactory idFactory) { + if (idFactory != null) { + this.idFactory = idFactory; + } + } + + // PRIVATE HELPER METHODS & S3 PROCESSING LOGIC + + private UploadInfo fetchAndValidateUpload(UploadId uploadId) + throws UploadNotFoundException, TusException, IOException { + UploadInfo info = getUploadInfo(uploadId); + if (info == null) { + throw new UploadNotFoundException("Upload with ID " + uploadId + " was not found"); + } + validateUploadLimits(info); + return info; + } + + private void validateUploadLimits(UploadInfo info) throws TusException { + if (info.getLength() != null) { + if (maxUploadSize != null && maxUploadSize > 0 && info.getLength() > maxUploadSize) { + throw new MaxUploadLengthExceededException( + "Upload length " + info.getLength() + " exceeds max limit of " + maxUploadSize); + } + if (minSize != null && minSize > 0 && info.getLength() < minSize) { + throw new MinUploadLengthNotReachedException( + "Upload length " + info.getLength() + " is below min limit of " + minSize); + } + } + } + + /** + * Check if a leftover sub-5MB .part buffer object exists from a previous incomplete PATCH + * request. If found, downloads it to local disk, deletes the .part object from S3, and prepends + * its bytes to the incoming input stream using {@link SequenceInputStream}. + */ + private InputStream prepareStreamWithExistingIncompletePart( + String partObjectKey, InputStream inputStream) throws IOException { + try { + StatObjectResponse partHead = + minioClient.statObject( + StatObjectArgs.builder().bucket(bucket).object(partObjectKey).build()); + if (partHead != null) { + InputStream partStream = + minioClient.getObject( + GetObjectArgs.builder().bucket(bucket).object(partObjectKey).build()); + File tempPrependedFile = + File.createTempFile("tus-s3-prep-", ".tmp", temporaryDirectory.toFile()); + tempPrependedFile.deleteOnExit(); + + try (FileOutputStream fos = new FileOutputStream(tempPrependedFile)) { + IOUtils.copy(partStream, fos); + } + deleteObjectQuietly(partObjectKey); + return new SequenceInputStream(new FileInputStream(tempPrependedFile), inputStream); + } + } catch (ErrorResponseException e) { + if ("NoSuchKey".equalsIgnoreCase(e.errorResponse().code())) { + // Normal case: no leftover .part object present in S3 + } + } catch (Exception ignored) { + } + return inputStream; + } + + /** + * Reads bytes from the incoming stream into temporary local files of optimal part size (default + * 50MB). Parts $\ge$ 5MB are uploaded immediately to S3 as part chunk objects. Any trailing chunk + * under 5MB is saved as a temporary .part object unless it completes the overall upload. + */ + private AppendResult processPayloadChunks( + UploadInfo info, InputStream streamToRead, String id, String partObjectKey) + throws IOException, MaxAppendSizeExceededException { + + List partKeys = fetchExistingPartKeys(id); + int nextPartNumber = partKeys.size() + 1; + List allPartKeys = new ArrayList<>(partKeys); + + long optimalPartSize = calcOptimalPartSize(info.getLength() != null ? info.getLength() : 0); + byte[] buffer = new byte[8192]; + long totalBytesAppended = 0; + + boolean streamFinished = false; + while (!streamFinished) { + File tempChunkFile = + File.createTempFile("tus-s3-chunk-", ".tmp", temporaryDirectory.toFile()); + tempChunkFile.deleteOnExit(); + + long chunkBytesWritten = 0; + try (FileOutputStream fos = new FileOutputStream(tempChunkFile)) { + int bytesRead; + while (chunkBytesWritten < optimalPartSize + && (bytesRead = streamToRead.read(buffer)) != -1) { + if (maxAppendSize != null && (totalBytesAppended + bytesRead) > maxAppendSize) { + tempChunkFile.delete(); + throw new MaxAppendSizeExceededException( + "Append payload exceeded limit of " + maxAppendSize); + } + fos.write(buffer, 0, bytesRead); + chunkBytesWritten += bytesRead; + totalBytesAppended += bytesRead; + } + + if (chunkBytesWritten < optimalPartSize) { + streamFinished = true; + } + } + + if (chunkBytesWritten == 0) { + tempChunkFile.delete(); + break; + } + + long currentTotalOffset = info.getOffset() + totalBytesAppended; + boolean isUploadComplete = info.getLength() != null && currentTotalOffset >= info.getLength(); + + // AWS S3 / MinIO Rule: Parts must be >= 5 MB unless it's the final part completing the upload + if (chunkBytesWritten >= minPartSize || (streamFinished && isUploadComplete)) { + String chunkKey = buildChunkPartKey(id, nextPartNumber); + uploadChunkToS3(chunkKey, tempChunkFile, chunkBytesWritten); + allPartKeys.add(chunkKey); + nextPartNumber++; + } else { + // Store sub-5MB tail chunk as temporary .part object in S3 for subsequent appends + storeIncompletePartToS3(partObjectKey, tempChunkFile, chunkBytesWritten); + } + } + + return new AppendResult(totalBytesAppended, allPartKeys); + } + + private void uploadChunkToS3(String chunkKey, File tempChunkFile, long chunkLength) + throws IOException { + try (FileInputStream fis = new FileInputStream(tempChunkFile)) { + minioClient.putObject( + PutObjectArgs.builder().bucket(bucket).object(chunkKey).stream(fis, chunkLength, -1L) + .build()); + } catch (Exception e) { + throw new IOException("Failed to upload part chunk to S3 key " + chunkKey, e); + } finally { + tempChunkFile.delete(); + } + } + + private void storeIncompletePartToS3(String partObjectKey, File tempChunkFile, long chunkLength) + throws IOException { + try (FileInputStream fis = new FileInputStream(tempChunkFile)) { + minioClient.putObject( + PutObjectArgs.builder().bucket(bucket).object(partObjectKey).stream(fis, chunkLength, -1L) + .build()); + } catch (Exception e) { + throw new IOException("Failed to write incomplete part object to S3 key " + partObjectKey, e); + } finally { + tempChunkFile.delete(); + } + } + + /** + * When all expected bytes have been received, this method combines all part chunk objects in S3 + * into the final destination object key using MinIO's {@code composeObject} API. + */ + private void finalizeCompletedUploadIfFinished( + UploadInfo info, String objectKey, String id, AppendResult appendResult, long newOffset) + throws IOException { + + if (info.getLength() != null && newOffset >= info.getLength()) { + List partKeys = fetchExistingPartKeys(id); + + // If leftover sub-5MB .part exists, save it as final part chunk + String leftoverPartKey = buildIncompletePartKey(id); + if (objectExists(leftoverPartKey)) { + int nextPartNum = partKeys.size() + 1; + String finalChunkKey = buildChunkPartKey(id, nextPartNum); + try (InputStream stream = + minioClient.getObject( + GetObjectArgs.builder().bucket(bucket).object(leftoverPartKey).build())) { + byte[] bytes = IOUtils.toByteArray(stream); + minioClient.putObject( + PutObjectArgs.builder().bucket(bucket).object(finalChunkKey).stream( + new ByteArrayInputStream(bytes), (long) bytes.length, -1L) + .build()); + partKeys.add(finalChunkKey); + } catch (Exception e) { + throw new IOException("Failed to finalize incomplete part for ID " + id, e); + } + deleteObjectQuietly(leftoverPartKey); + } + + if (!partKeys.isEmpty()) { + List sources = new ArrayList<>(); + for (String pk : partKeys) { + sources.add(SourceObject.builder().bucket(bucket).object(pk).build()); + } + + // Perform S3 server-side object composition (composeObject) + try { + minioClient.composeObject( + ComposeObjectArgs.builder() + .bucket(bucket) + .object(objectKey) + .sources(sources) + .build()); + } catch (Exception e) { + throw new IOException("Failed to compose final object " + objectKey, e); + } + + // Clean up temporary part chunk objects in S3 + for (String pk : partKeys) { + deleteObjectQuietly(pk); + } + } + + // Add checksum index if deduplication is enabled + if (isUploadDeduplicationEnabled() + && info.getChecksum() != null + && info.getDuplicatesUploadId() == null) { + putChecksumIndex(info.getChecksum(), info.getChecksumAlgorithm(), info.getId().toString()); + } + } + } + + private InputStream fetchS3ByteStream(UploadId id, UploadInfo info) + throws UploadNotFoundException { + String objectKey = getS3ObjectKey(info); + if (objectKey == null && id != null) { + objectKey = buildObjectKey(id.toString()); + } + try { + // Step 1: Attempt to read from completed object key in S3 + return minioClient.getObject( + GetObjectArgs.builder().bucket(bucket).object(objectKey).build()); + } catch (ErrorResponseException e) { + if (S3Utils.parseErrorResponse(e) == S3ErrorType.NO_SUCH_KEY) { + // Step 2: Fallback to reading from incomplete .part object if upload is in-progress + String partKey = buildIncompletePartKey(id.toString()); + try { + return minioClient.getObject( + GetObjectArgs.builder().bucket(bucket).object(partKey).build()); + } catch (ErrorResponseException ex) { + if (S3Utils.parseErrorResponse(ex) == S3ErrorType.NO_SUCH_KEY) { + if (info != null && (info.getOffset() == null || info.getOffset() == 0L)) { + return new ByteArrayInputStream(new byte[0]); + } + } + } catch (Exception ignored) { + } + } + throw new UploadNotFoundException("Uploaded bytes object not found for ID " + id); + } catch (Exception e) { + throw new UploadNotFoundException("Uploaded bytes object not found for ID " + id); + } + } + + private void truncateFromCompletedObject(String objectKey, String partKey, long newOffset) + throws IOException { + if (newOffset > 0) { + try (InputStream objStream = + minioClient.getObject(GetObjectArgs.builder().bucket(bucket).object(objectKey).build())) { + byte[] remainingBytes = new byte[(int) newOffset]; + IOUtils.readFully(objStream, remainingBytes); + minioClient.putObject( + PutObjectArgs.builder().bucket(bucket).object(partKey).stream( + new ByteArrayInputStream(remainingBytes), (long) remainingBytes.length, -1L) + .build()); + } catch (Exception e) { + throw new IOException("Failed to truncate completed object key " + objectKey, e); + } + } + deleteObjectQuietly(objectKey); + } + + private void truncateFromIncompletePart(String partKey, long byteCount) { + try { + StatObjectResponse head = + minioClient.statObject(StatObjectArgs.builder().bucket(bucket).object(partKey).build()); + long partSize = head.size(); + + if (byteCount >= partSize) { + deleteObjectQuietly(partKey); + } else { + InputStream partStream = + minioClient.getObject(GetObjectArgs.builder().bucket(bucket).object(partKey).build()); + byte[] bytes = IOUtils.toByteArray(partStream); + int newLength = (int) (bytes.length - byteCount); + byte[] remaining = java.util.Arrays.copyOf(bytes, newLength); + + minioClient.putObject( + PutObjectArgs.builder().bucket(bucket).object(partKey).stream( + new ByteArrayInputStream(remaining), (long) remaining.length, -1L) + .build()); + } + } catch (ErrorResponseException ignored) { + } catch (Exception e) { + log.debug("Error truncating incomplete part object {}", partKey, e); + } + } + + private void calculateAndSetOffset(UploadInfo info) { + String id = info.getId().toString(); + String objectKey = getS3ObjectKey(info); + String partKey = buildIncompletePartKey(id); + + long offset = calculateCurrentOffset(objectKey, id, partKey); + info.setOffset(offset); + } + + private long calculateCurrentOffset(String objectKey, String id, String partKey) { + long offset = 0; + + if (objectExists(objectKey)) { + try { + StatObjectResponse head = + minioClient.statObject( + StatObjectArgs.builder().bucket(bucket).object(objectKey).build()); + offset += head.size(); + } catch (Exception ignored) { + } + } + + List partKeys = fetchExistingPartKeys(id); + for (String pk : partKeys) { + try { + StatObjectResponse stat = + minioClient.statObject(StatObjectArgs.builder().bucket(bucket).object(pk).build()); + offset += stat.size(); + } catch (Exception ignored) { + } + } + + if (!partKeys.contains(partKey)) { + try { + StatObjectResponse partHead = + minioClient.statObject(StatObjectArgs.builder().bucket(bucket).object(partKey).build()); + if (partHead != null) { + offset += partHead.size(); + } + } catch (ErrorResponseException ignored) { + } catch (Exception e) { + log.debug("Error reading head for incomplete part object {}", partKey, e); + } + } + + return offset; + } + + private List fetchExistingPartKeys(String id) { + String prefix = metadataPrefix + id + ".part."; + List partKeys = new ArrayList<>(); + try { + Iterable> results = + minioClient.listObjects(ListObjectsArgs.builder().bucket(bucket).prefix(prefix).build()); + for (Result res : results) { + partKeys.add(res.get().objectName()); + } + } catch (Exception ignored) { + } + return partKeys; + } + + private void deleteAllPartObjectsQuietly(String id) { + List partKeys = fetchExistingPartKeys(id); + for (String pk : partKeys) { + deleteObjectQuietly(pk); + } + } + + private long calcOptimalPartSize(long totalSize) { + long partSize = preferredPartSize; + if (totalSize > 0 && totalSize / partSize >= 10000) { + partSize = (totalSize / 10000) + 1; + } + return Math.max(minPartSize, Math.min(partSize, DEFAULT_MAX_PART_SIZE)); + } + + private void putChecksumIndex(String checksum, ChecksumAlgorithm algorithm, String parentId) { + String key = buildChecksumKey(checksum, algorithm); + try { + byte[] parentIdBytes = parentId.getBytes(StandardCharsets.UTF_8); + minioClient.putObject( + PutObjectArgs.builder().bucket(bucket).object(key).stream( + new ByteArrayInputStream(parentIdBytes), (long) parentIdBytes.length, -1L) + .build()); + } catch (Exception e) { + log.warn("Failed to write checksum index object to S3 key {}", key, e); + } + } + + private boolean objectExists(String key) { + try { + minioClient.statObject(StatObjectArgs.builder().bucket(bucket).object(key).build()); + return true; + } catch (ErrorResponseException e) { + if (S3Utils.parseErrorResponse(e) == S3ErrorType.NO_SUCH_KEY) { + return false; + } + return false; + } catch (Exception e) { + return false; + } + } + + private void deleteObjectQuietly(String key) { + if (key == null) { + return; + } + try { + minioClient.removeObject(RemoveObjectArgs.builder().bucket(bucket).object(key).build()); + } catch (Exception e) { + log.debug("Failed to delete S3 object key {}", key, e); + } + } + + private String sanitizePrefix(String prefix) { + if (prefix == null || prefix.isEmpty()) { + return ""; + } + String result = prefix.startsWith("/") ? prefix.substring(1) : prefix; + return result.endsWith("/") ? result : result + "/"; + } + + private String buildObjectKey(String id) { + return objectPrefix + id; + } + + private String buildMetadataKey(String id) { + return metadataPrefix + id + ".info"; + } + + private String buildIncompletePartKey(String id) { + return metadataPrefix + id + ".part"; + } + + private String buildChunkPartKey(String id, int partNumber) { + return metadataPrefix + id + ".part." + String.format("%05d", partNumber); + } + + private String buildChecksumKey(String checksum, ChecksumAlgorithm algorithm) { + String algorithmName = algorithm != null ? algorithm.getTusName().toLowerCase() : "unknown"; + return checksumsPrefix + algorithmName + "/" + checksum; + } + + private static class AppendResult { + final long totalBytesAppended; + final List allPartKeys; + + AppendResult(long totalBytesAppended, List allPartKeys) { + this.totalBytesAppended = totalBytesAppended; + this.allPartKeys = allPartKeys; + } + } +} diff --git a/src/main/java/me/desair/tus/server/upload/s3/S3UploadLock.java b/src/main/java/me/desair/tus/server/upload/s3/S3UploadLock.java new file mode 100644 index 00000000..76707312 --- /dev/null +++ b/src/main/java/me/desair/tus/server/upload/s3/S3UploadLock.java @@ -0,0 +1,176 @@ +package me.desair.tus.server.upload.s3; + +import io.minio.MinioClient; +import io.minio.PutObjectArgs; +import io.minio.RemoveObjectArgs; +import java.io.ByteArrayInputStream; +import java.io.InputStream; +import java.nio.charset.StandardCharsets; +import java.util.Map; +import java.util.concurrent.Executors; +import java.util.concurrent.ScheduledExecutorService; +import java.util.concurrent.TimeUnit; +import me.desair.tus.server.upload.UploadLock; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +/** + * A MinIO S3-backed implementation of {@link UploadLock} that holds an exclusive lock lease on an + * upload resource using S3 objects. Spawns a heartbeat thread to auto-renew the lock lease until + * closed. + * + *

Lock Lease Mechanics for Developers: + * + *

    + *
  • Heartbeat Lease Renewal: When initialized, a background daemon thread executes + * {@link #renewLease()} at a periodic interval (one-third of {@code leaseDurationMs}, e.g. + * every 10s for a 30s lease). + *
  • Clean Lock Release: When the HTTP request finishes, {@link #close()} shuts down the + * heartbeat thread and deletes both the {@code .lock} lease object and any lingering {@code + * .stop} signal objects from S3. + *
+ */ +public class S3UploadLock implements UploadLock { + + private static final Logger log = LoggerFactory.getLogger(S3UploadLock.class); + + private final MinioClient minioClient; + private final String bucket; + private final String lockKey; + private final String stopKey; + private final String holderId; + private final long leaseDurationMs; + private final ScheduledExecutorService heartbeatExecutor; + private final String requestUri; + private final Map inputStreamMap; + + /** + * Constructs a new S3UploadLock instance using MinIO Java SDK and starts the lease renewal + * daemon. + * + * @param minioClient The MinIO client + * @param bucket The S3 bucket + * @param lockKey The S3 object key for the lock lease + * @param stopKey The S3 object key for the interrupt stop signal + * @param holderId Unique ID identifying the lock holder + * @param leaseDurationMs Lease duration in milliseconds + * @param requestUri The request URI linked to this lock + * @param inputStreamMap Map of active request input streams + */ + public S3UploadLock( + MinioClient minioClient, + String bucket, + String lockKey, + String stopKey, + String holderId, + long leaseDurationMs, + String requestUri, + Map inputStreamMap) { + this.minioClient = minioClient; + this.bucket = bucket; + this.lockKey = lockKey; + this.stopKey = stopKey; + this.holderId = holderId; + this.leaseDurationMs = leaseDurationMs; + this.requestUri = requestUri; + this.inputStreamMap = inputStreamMap; + + // Run heartbeat lease renewal at 1/3 of the lease duration (e.g., every 10 seconds for a 30s + // lease) + long heartbeatPeriodMs = Math.max(1000L, leaseDurationMs / 3); + this.heartbeatExecutor = + Executors.newSingleThreadScheduledExecutor( + r -> { + Thread t = new Thread(r, "s3-lock-heartbeat-" + holderId); + t.setDaemon(true); + return t; + }); + this.heartbeatExecutor.scheduleAtFixedRate( + this::renewLease, heartbeatPeriodMs, heartbeatPeriodMs, TimeUnit.MILLISECONDS); + } + + S3UploadLock( + MinioClient minioClient, + String bucket, + String lockKey, + String stopKey, + String holderId, + long leaseDurationMs, + String requestUri, + Map inputStreamMap, + ScheduledExecutorService heartbeatExecutor) { + this.minioClient = minioClient; + this.bucket = bucket; + this.lockKey = lockKey; + this.stopKey = stopKey; + this.holderId = holderId; + this.leaseDurationMs = leaseDurationMs; + this.requestUri = requestUri; + this.inputStreamMap = inputStreamMap; + this.heartbeatExecutor = heartbeatExecutor; + } + + /** Gets the holder ID for this lock. */ + public String getHolderId() { + return holderId; + } + + @Override + public String getUploadUri() { + return requestUri; + } + + @Override + public void release() { + close(); + } + + @Override + public void close() { + // Step 1: Stop the background heartbeat daemon thread + try { + heartbeatExecutor.shutdownNow(); + } catch (Exception e) { + log.debug("Error shutting down lock heartbeat executor", e); + } + + // Step 2: Remove active request stream registration + if (inputStreamMap != null && requestUri != null) { + inputStreamMap.remove(requestUri); + } + + // Step 3: Delete .lock lease object and .stop contention signal object from S3 + deleteS3ObjectQuietly(lockKey); + deleteS3ObjectQuietly(stopKey); + } + + /** Renew the lock lease in S3 by updating the expiration timestamp. */ + void renewLease() { + try { + long newExpiry = System.currentTimeMillis() + leaseDurationMs; + String lockContent = + String.format( + "{\"holder\":\"%s\",\"expiresAt\":%d,\"acquiredAt\":%d}", + holderId, newExpiry, System.currentTimeMillis()); + byte[] lockContentBytes = lockContent.getBytes(StandardCharsets.UTF_8); + + minioClient.putObject( + PutObjectArgs.builder().bucket(bucket).object(lockKey).stream( + new ByteArrayInputStream(lockContentBytes), (long) lockContentBytes.length, -1L) + .build()); + } catch (Exception e) { + log.warn("Failed to renew S3 lock lease for key {}", lockKey, e); + } + } + + private void deleteS3ObjectQuietly(String key) { + if (key == null) { + return; + } + try { + minioClient.removeObject(RemoveObjectArgs.builder().bucket(bucket).object(key).build()); + } catch (Exception e) { + log.debug("Failed to delete S3 lock object {}", key, e); + } + } +} diff --git a/src/main/java/me/desair/tus/server/upload/s3/S3Utils.java b/src/main/java/me/desair/tus/server/upload/s3/S3Utils.java new file mode 100644 index 00000000..675971e7 --- /dev/null +++ b/src/main/java/me/desair/tus/server/upload/s3/S3Utils.java @@ -0,0 +1,46 @@ +package me.desair.tus.server.upload.s3; + +import io.minio.errors.ErrorResponseException; +import io.minio.messages.ErrorResponse; + +/** Utility helper methods for parsing and evaluating MinIO S3 error responses. */ +public final class S3Utils { + + private S3Utils() { + // Utility class + } + + /** + * Parses an {@link ErrorResponseException} into a clean, strongly-typed {@link S3ErrorType}. + * + * @param exception The MinIO ErrorResponseException to evaluate + * @return The corresponding S3ErrorType enum + */ + public static S3ErrorType parseErrorResponse(ErrorResponseException exception) { + if (exception == null) { + return S3ErrorType.UNKNOWN; + } + + ErrorResponse response = exception.errorResponse(); + String code = response != null ? response.code() : ""; + + if ("NoSuchKey".equalsIgnoreCase(code) || "NoSuchBucket".equalsIgnoreCase(code)) { + return S3ErrorType.NO_SUCH_KEY; + } + + if ("PreconditionFailed".equalsIgnoreCase(code)) { + return S3ErrorType.PRECONDITION_FAILED; + } + + if ("ObjectAlreadyExists".equalsIgnoreCase(code) + || "BucketAlreadyExists".equalsIgnoreCase(code)) { + return S3ErrorType.CONFLICT; + } + + if ("AccessDenied".equalsIgnoreCase(code)) { + return S3ErrorType.ACCESS_DENIED; + } + + return S3ErrorType.UNKNOWN; + } +} diff --git a/src/main/java/me/desair/tus/server/util/UploadInfoJsonSerializer.java b/src/main/java/me/desair/tus/server/util/UploadInfoJsonSerializer.java new file mode 100644 index 00000000..65db5469 --- /dev/null +++ b/src/main/java/me/desair/tus/server/util/UploadInfoJsonSerializer.java @@ -0,0 +1,146 @@ +package me.desair.tus.server.util; + +import com.fasterxml.jackson.annotation.JsonInclude; +import com.fasterxml.jackson.core.JsonGenerator; +import com.fasterxml.jackson.core.JsonParser; +import com.fasterxml.jackson.databind.DeserializationContext; +import com.fasterxml.jackson.databind.DeserializationFeature; +import com.fasterxml.jackson.databind.JsonDeserializer; +import com.fasterxml.jackson.databind.JsonSerializer; +import com.fasterxml.jackson.databind.ObjectMapper; +import com.fasterxml.jackson.databind.SerializerProvider; +import com.fasterxml.jackson.databind.module.SimpleModule; +import java.io.IOException; +import java.io.InputStream; +import java.io.OutputStream; +import me.desair.tus.server.upload.UploadId; +import me.desair.tus.server.upload.UploadInfo; + +/** + * Utility class responsible for serializing and deserializing {@link UploadInfo} and other domain + * objects to and from JSON format using Jackson. + */ +public class UploadInfoJsonSerializer { + + private static final ObjectMapper OBJECT_MAPPER; + + static { + SimpleModule module = new SimpleModule(); + module.addSerializer( + UploadId.class, + new JsonSerializer() { + @Override + public void serialize( + UploadId uploadId, JsonGenerator gen, SerializerProvider serializers) + throws IOException { + gen.writeString(uploadId.toString()); + } + }); + + module.addDeserializer( + UploadId.class, + new JsonDeserializer() { + @Override + public UploadId deserialize(JsonParser p, DeserializationContext ctxt) + throws IOException { + String text = p.getText(); + if (text == null || text.isEmpty()) { + return null; + } + return new UploadId(text); + } + }); + + OBJECT_MAPPER = + new ObjectMapper() + .registerModule(module) + .configure(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES, false) + .setSerializationInclusion(JsonInclude.Include.NON_NULL); + } + + private UploadInfoJsonSerializer() { + // Utility class + } + + /** + * Serialize the given object to a JSON string. + * + * @param object The object to serialize + * @return A JSON string representation of the object + * @throws IOException If serialization fails + */ + public static String serialize(Object object) throws IOException { + if (object == null) { + return null; + } + return OBJECT_MAPPER.writeValueAsString(object); + } + + /** + * Serialize the given object directly to an {@link OutputStream}. + * + * @param object The object to serialize + * @param outputStream The target output stream + * @throws IOException If serialization fails + */ + public static void serializeToStream(Object object, OutputStream outputStream) + throws IOException { + if (object != null && outputStream != null) { + OBJECT_MAPPER.writeValue(outputStream, object); + } + } + + /** + * Deserialize an {@link UploadInfo} object from a JSON string. + * + * @param json The JSON string representation of the upload info + * @return The deserialized UploadInfo instance, or null if input is blank + * @throws IOException If deserialization fails + */ + public static UploadInfo deserialize(String json) throws IOException { + return deserialize(json, UploadInfo.class); + } + + /** + * Deserialize an object of the specified class from a JSON string. + * + * @param The target object type + * @param json The JSON string representation + * @param clazz The target class type + * @return The deserialized instance, or null if input is blank + * @throws IOException If deserialization fails + */ + public static T deserialize(String json, Class clazz) throws IOException { + if (json == null || json.trim().isEmpty() || clazz == null) { + return null; + } + return OBJECT_MAPPER.readValue(json, clazz); + } + + /** + * Deserialize an {@link UploadInfo} object from an {@link InputStream}. + * + * @param inputStream The input stream containing the JSON data + * @return The deserialized UploadInfo instance, or null if input stream is null + * @throws IOException If deserialization fails + */ + public static UploadInfo deserialize(InputStream inputStream) throws IOException { + return deserialize(inputStream, UploadInfo.class); + } + + /** + * Deserialize an object of the specified class from an {@link InputStream}. + * + * @param The target object type + * @param inputStream The input stream containing the JSON data + * @param clazz The target class type + * @return The deserialized instance, or null if input stream is null + * @throws IOException If deserialization fails + */ + public static T deserialize(InputStream inputStream, Class clazz) throws IOException { + if (inputStream == null || clazz == null) { + return null; + } + return OBJECT_MAPPER.readValue(inputStream, clazz); + } +} diff --git a/src/main/java/me/desair/tus/server/util/Utils.java b/src/main/java/me/desair/tus/server/util/Utils.java index 87637f53..94da342e 100644 --- a/src/main/java/me/desair/tus/server/util/Utils.java +++ b/src/main/java/me/desair/tus/server/util/Utils.java @@ -126,6 +126,59 @@ public static void writeSerializable(Serializable object, Path path) throws IOEx } } + /** + * Reads an object from a JSON file on disk, acquiring a shared file lock during the read + * operation. + * + * @param Target object type + * @param path The file path to read from + * @param clazz The target object class + * @return Deserialized object instance, or null if reading fails or file does not exist + * @throws IOException If file access or locking fails + */ + public static T readJson(Path path, Class clazz) throws IOException { + T info = null; + if (path != null && java.nio.file.Files.exists(path)) { + try (FileChannel channel = FileChannel.open(path, READ)) { + // Lock will be released when the channel is closed + if (lockFileShared(channel) != null) { + try (java.io.InputStream is = Channels.newInputStream(channel)) { + info = UploadInfoJsonSerializer.deserialize(is, clazz); + } catch (Exception e) { + log.warn("Unable to read JSON file {}: {}", path, e.getMessage()); + info = null; + } + } else { + throw new IOException("Unable to lock file " + path); + } + } + } + return info; + } + + /** + * Writes an object to a file in JSON format, acquiring an exclusive file lock during the write + * operation. + * + * @param object The object to serialize to JSON + * @param path The file path to write to + * @throws IOException If file access or locking fails + */ + public static void writeJson(Object object, Path path) throws IOException { + if (path != null) { + try (FileChannel channel = FileChannel.open(path, WRITE, CREATE, TRUNCATE_EXISTING)) { + // Lock will be released when the channel is closed + if (lockFileExclusively(channel) != null) { + try (OutputStream buffer = new BufferedOutputStream(Channels.newOutputStream(channel))) { + UploadInfoJsonSerializer.serializeToStream(object, buffer); + } + } else { + throw new IOException("Unable to lock file " + path); + } + } + } + } + public static FileLock lockFileExclusively(FileChannel channel) throws IOException { return lockFile(channel, false); } diff --git a/src/test/java/me/desair/tus/server/AbstractITRufhProtocol.java b/src/test/java/me/desair/tus/server/AbstractITRufhProtocol.java new file mode 100644 index 00000000..4a25227c --- /dev/null +++ b/src/test/java/me/desair/tus/server/AbstractITRufhProtocol.java @@ -0,0 +1,628 @@ +package me.desair.tus.server; + +import static org.hamcrest.CoreMatchers.is; +import static org.hamcrest.MatcherAssert.assertThat; +import static org.hamcrest.Matchers.containsInAnyOrder; +import static org.junit.Assert.assertFalse; +import static org.junit.Assert.assertNull; +import static org.junit.Assert.assertTrue; + +import jakarta.servlet.http.HttpServletResponse; +import java.io.InputStream; +import java.nio.charset.StandardCharsets; +import me.desair.tus.server.upload.UploadInfo; +import org.apache.commons.io.IOUtils; +import org.apache.commons.lang3.StringUtils; +import org.junit.Before; +import org.junit.Test; +import org.springframework.mock.web.MockHttpServletRequest; +import org.springframework.mock.web.MockHttpServletResponse; + +/** + * Abstract base integration test suite for the IETF Resumable Uploads for HTTP (RUFH) protocol. + * + *

This class contains end-to-end integration test use cases covering the full RUFH lifecycle, + * structured into clear, step-by-step phases. Concrete subclasses supply the target storage backend + * by implementing {@link #createTusFileUploadService()}. + */ +public abstract class AbstractITRufhProtocol { + + protected static final String UPLOAD_URI = "/test/upload"; + protected static final String OWNER_KEY = "RUFH_USER"; + + protected MockHttpServletRequest servletRequest; + protected MockHttpServletResponse servletResponse; + protected TusFileUploadService tusFileUploadService; + + /** + * Factory method implemented by subclasses to supply a {@link TusFileUploadService} instance + * configured for a specific storage backend (e.g., Disk, S3, Azure Blob). + * + * @return configured TusFileUploadService instance + * @throws Exception if service creation fails + */ + protected abstract TusFileUploadService createTusFileUploadService() throws Exception; + + @Before + public void setUp() throws Exception { + reset(); + tusFileUploadService = createTusFileUploadService(); + } + + /** Resets mock HTTP request and response objects for a new request step. */ + protected void reset() { + servletRequest = new MockHttpServletRequest(); + servletRequest.setRemoteAddr("192.168.1.1"); + servletResponse = new MockHttpServletResponse(); + } + + // =============================================================================================== + // USE CASE 1: OPTIONS Discovery + // =============================================================================================== + + /** + * Section 4.1.4 (Limits - Structured Field Format): "Upload-Limit MUST be a Dictionary Structured + * Header Field..." + * + *

Use Case: Client sends an OPTIONS request to discover supported features and upload limits. + */ + @Test + public void testOptionsDiscovery() throws Exception { + // Step 1: Send OPTIONS discovery request + servletRequest.setMethod("OPTIONS"); + servletRequest.setRequestURI(UPLOAD_URI); + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + + // Step 2: Verify HTTP 204 response with Upload-Limit header and enabled protocol features + assertResponseStatus(HttpServletResponse.SC_NO_CONTENT); + assertResponseHeaderNotBlank(HttpHeader.UPLOAD_LIMIT); + + assertThat( + tusFileUploadService.getEnabledFeatures(), + containsInAnyOrder( + "core", + "creation", + "creation-with-upload", + "checksum", + "termination", + "download", + "expiration", + "concatenation", + "cors", + "resumable-uploads-for-http", + "http-digests")); + } + + // =============================================================================================== + // USE CASE 2: Single-Request Optimistic Upload Creation and Completion + // =============================================================================================== + + /** + * Section 4.2.1 & 4.2.2 (Upload Creation - Optimistic Uploads): "If the Upload-Complete request + * header field is set to true, the client intends to transfer the entire representation data in + * one request..." + * + *

Use Case: Client uploads small payload in a single POST request using Upload-Complete: ?1. + */ + @Test + public void testOptimisticUploadCreationAndCompletion() throws Exception { + String payload = "Hello, RUFH Single Request Optimistic Upload!"; + + // Step 1: Send single-request optimistic upload via POST with Upload-Complete: ?1 + servletRequest.setMethod("POST"); + servletRequest.setRequestURI(UPLOAD_URI); + servletRequest.addHeader(HttpHeader.UPLOAD_COMPLETE, "?1"); + servletRequest.setContent(payload.getBytes(StandardCharsets.UTF_8)); + + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + + // Step 2: Verify HTTP 200 OK response with Upload-Complete: ?1 header + assertResponseStatus(HttpServletResponse.SC_OK); + assertResponseHeader(HttpHeader.UPLOAD_COMPLETE, "?1"); + } + + // =============================================================================================== + // USE CASE 3: Multi-Chunk Resumable Upload Lifecycle + // =============================================================================================== + + /** + * Section 4.2 & 4.4 (Resumable Upload Lifecycle): "A client can start a resumable upload... by + * including the Upload-Complete header field... A server applies a PATCH request with the + * application/partial-upload media type to append data." + * + *

Use Case: Create a resumable upload with declared length, append chunk 1, check offset via + * HEAD, append chunk 2 with Upload-Complete: ?1, and verify downloaded bytes. + */ + @Test + public void testResumableUploadMultiChunkLifecycle() throws Exception { + String part1 = "Part 1 data of resumable upload. "; + String part2 = "Part 2 final data of upload."; + long totalLength = part1.length() + part2.length(); + + // Step 1: Initiate resumable upload with POST, Upload-Complete: ?0, declared Upload-Length, and + // initial chunk + servletRequest.setMethod("POST"); + servletRequest.setRequestURI(UPLOAD_URI); + servletRequest.addHeader(HttpHeader.UPLOAD_COMPLETE, "?0"); + servletRequest.addHeader(HttpHeader.UPLOAD_LENGTH, String.valueOf(totalLength)); + servletRequest.addHeader(HttpHeader.CONTENT_TYPE, HttpHeader.CONTENT_TYPE_PARTIAL_UPLOAD); + servletRequest.setContent(part1.getBytes(StandardCharsets.UTF_8)); + + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + + // Step 2: Verify HTTP 201 Created response, Location header, and initial offset + assertResponseStatus(HttpServletResponse.SC_CREATED); + assertResponseHeaderNotBlank(HttpHeader.LOCATION); + assertResponseHeader(HttpHeader.UPLOAD_OFFSET, String.valueOf(part1.length())); + assertResponseHeader(HttpHeader.UPLOAD_COMPLETE, "?0"); + + String uploadLocation = servletResponse.getHeader(HttpHeader.LOCATION); + + // Step 3: Query upload progress via HEAD request + reset(); + servletRequest.setMethod("HEAD"); + servletRequest.setRequestURI(uploadLocation); + + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + + // Step 4: Verify offset and length reported in HEAD response + assertResponseStatus(HttpServletResponse.SC_NO_CONTENT); + assertResponseHeader(HttpHeader.UPLOAD_OFFSET, String.valueOf(part1.length())); + assertResponseHeader(HttpHeader.UPLOAD_LENGTH, String.valueOf(totalLength)); + assertResponseHeader(HttpHeader.UPLOAD_COMPLETE, "?0"); + + // Step 5: Append final chunk via PATCH with Upload-Complete: ?1 + reset(); + servletRequest.setMethod("PATCH"); + servletRequest.setRequestURI(uploadLocation); + servletRequest.addHeader(HttpHeader.CONTENT_TYPE, HttpHeader.CONTENT_TYPE_PARTIAL_UPLOAD); + servletRequest.addHeader(HttpHeader.UPLOAD_OFFSET, String.valueOf(part1.length())); + servletRequest.addHeader(HttpHeader.UPLOAD_COMPLETE, "?1"); + servletRequest.setContent(part2.getBytes(StandardCharsets.UTF_8)); + + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + + // Step 6: Verify HTTP 200 OK completing response and final offset + assertResponseStatus(HttpServletResponse.SC_OK); + assertResponseHeader(HttpHeader.UPLOAD_OFFSET, String.valueOf(totalLength)); + assertResponseHeader(HttpHeader.UPLOAD_COMPLETE, "?1"); + + // Step 7: Verify internal UploadInfo state reports upload is no longer in progress + UploadInfo uploadInfo = tusFileUploadService.getUploadInfo(uploadLocation, OWNER_KEY); + assertFalse(uploadInfo.isUploadInProgress()); + assertThat(uploadInfo.getOffset(), is(totalLength)); + + // Step 8: Download uploaded content and verify byte-for-byte matching + try (InputStream inputStream = + tusFileUploadService.getUploadedBytes(uploadLocation, OWNER_KEY)) { + String uploadedContent = IOUtils.toString(inputStream, StandardCharsets.UTF_8); + assertThat(uploadedContent, is(part1 + part2)); + } + } + + // =============================================================================================== + // USE CASE 4: Careful Upload Creation (Empty Creation Request) + // =============================================================================================== + + /** + * Section 10.2 (Careful Upload Creation): "A client MAY create a resumable upload resource + * without uploading any data by sending an empty request with Upload-Complete: ?0." + * + *

Use Case: Client creates an empty upload resource without payload, then appends data in a + * subsequent PATCH request. + */ + @Test + public void testCarefulUploadCreation() throws Exception { + // Step 1: Create empty upload resource via POST with Upload-Complete: ?0 and Upload-Length + servletRequest.setMethod("POST"); + servletRequest.setRequestURI(UPLOAD_URI); + servletRequest.addHeader(HttpHeader.UPLOAD_COMPLETE, "?0"); + servletRequest.addHeader(HttpHeader.UPLOAD_LENGTH, "100"); + + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + + // Step 2: Verify HTTP 201 Created with Location header and offset 0 + assertResponseStatus(HttpServletResponse.SC_CREATED); + assertResponseHeaderNotBlank(HttpHeader.LOCATION); + assertResponseHeader(HttpHeader.UPLOAD_OFFSET, "0"); + assertResponseHeader(HttpHeader.UPLOAD_COMPLETE, "?0"); + + String uploadLocation = servletResponse.getHeader(HttpHeader.LOCATION); + + // Step 3: Append data to the created resource via PATCH at offset 0 + reset(); + servletRequest.setMethod("PATCH"); + servletRequest.setRequestURI(uploadLocation); + servletRequest.addHeader(HttpHeader.CONTENT_TYPE, HttpHeader.CONTENT_TYPE_PARTIAL_UPLOAD); + servletRequest.addHeader(HttpHeader.UPLOAD_OFFSET, "0"); + servletRequest.addHeader(HttpHeader.UPLOAD_COMPLETE, "?0"); + servletRequest.setContent("Initial data".getBytes(StandardCharsets.UTF_8)); + + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + + // Step 4: Verify HTTP 204 No Content response and updated offset + assertResponseStatus(HttpServletResponse.SC_NO_CONTENT); + assertResponseHeader(HttpHeader.UPLOAD_OFFSET, "12"); + } + + // =============================================================================================== + // USE CASE 5: Unknown Length Resumable Upload Lifecycle + // =============================================================================================== + + /** + * Section 4.1.3 & 4.4 (Unknown Length Scenario): "If the request does not include the + * Upload-Length header field, the representation's length is unknown... The representation's + * length is derived from a completing append." + * + *

Use Case: Stream chunks when total length is initially unknown, then conclude with a + * completing append that locks in the final length. + */ + @Test + public void testUnknownLengthUploadLifecycle() throws Exception { + String part1 = "Chunk 1 data. "; + String part2 = "Chunk 2 data. "; + String part3 = "Final chunk."; + long totalLength = part1.length() + part2.length() + part3.length(); + + // Step 1: Initiate upload without Upload-Length header + servletRequest.setMethod("POST"); + servletRequest.setRequestURI(UPLOAD_URI); + servletRequest.addHeader(HttpHeader.UPLOAD_COMPLETE, "?0"); + servletRequest.addHeader(HttpHeader.CONTENT_TYPE, HttpHeader.CONTENT_TYPE_PARTIAL_UPLOAD); + servletRequest.setContent(part1.getBytes(StandardCharsets.UTF_8)); + + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + + // Step 2: Verify HTTP 201 Created response without Upload-Length header + assertResponseStatus(HttpServletResponse.SC_CREATED); + String uploadLocation = servletResponse.getHeader(HttpHeader.LOCATION); + assertResponseHeader(HttpHeader.UPLOAD_OFFSET, String.valueOf(part1.length())); + assertNull(servletResponse.getHeader(HttpHeader.UPLOAD_LENGTH)); + + // Step 3: Append second chunk without setting Upload-Complete: ?1 + reset(); + servletRequest.setMethod("PATCH"); + servletRequest.setRequestURI(uploadLocation); + servletRequest.addHeader(HttpHeader.CONTENT_TYPE, HttpHeader.CONTENT_TYPE_PARTIAL_UPLOAD); + servletRequest.addHeader(HttpHeader.UPLOAD_OFFSET, String.valueOf(part1.length())); + servletRequest.addHeader(HttpHeader.UPLOAD_COMPLETE, "?0"); + servletRequest.setContent(part2.getBytes(StandardCharsets.UTF_8)); + + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + + // Step 4: Verify HTTP 204 No Content response and intermediate offset + assertResponseStatus(HttpServletResponse.SC_NO_CONTENT); + assertResponseHeader(HttpHeader.UPLOAD_OFFSET, String.valueOf(part1.length() + part2.length())); + assertNull(servletResponse.getHeader(HttpHeader.UPLOAD_LENGTH)); + + // Step 5: Send final completing chunk via PATCH with Upload-Complete: ?1 + reset(); + servletRequest.setMethod("PATCH"); + servletRequest.setRequestURI(uploadLocation); + servletRequest.addHeader(HttpHeader.CONTENT_TYPE, HttpHeader.CONTENT_TYPE_PARTIAL_UPLOAD); + servletRequest.addHeader( + HttpHeader.UPLOAD_OFFSET, String.valueOf(part1.length() + part2.length())); + servletRequest.addHeader(HttpHeader.UPLOAD_COMPLETE, "?1"); + servletRequest.setContent(part3.getBytes(StandardCharsets.UTF_8)); + + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + + // Step 6: Verify HTTP 200 OK completing response and final total length + assertResponseStatus(HttpServletResponse.SC_OK); + assertResponseHeader(HttpHeader.UPLOAD_OFFSET, String.valueOf(totalLength)); + assertResponseHeader(HttpHeader.UPLOAD_COMPLETE, "?1"); + + // Step 7: Verify HEAD request now returns the derived Upload-Length + reset(); + servletRequest.setMethod("HEAD"); + servletRequest.setRequestURI(uploadLocation); + + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + assertResponseStatus(HttpServletResponse.SC_NO_CONTENT); + assertResponseHeader(HttpHeader.UPLOAD_LENGTH, String.valueOf(totalLength)); + assertResponseHeader(HttpHeader.UPLOAD_OFFSET, String.valueOf(totalLength)); + } + + // =============================================================================================== + // USE CASE 6: Upload Cancellation / Termination + // =============================================================================================== + + /** + * Section 4.5 (Upload Cancellation): "The client can cancel an upload by sending a DELETE request + * to the upload resource..." + * + *

Use Case: Create upload, cancel via DELETE request, and verify subsequent HEAD returns 404. + */ + @Test + public void testUploadCancellation() throws Exception { + // Step 1: Create an active upload resource + servletRequest.setMethod("POST"); + servletRequest.setRequestURI(UPLOAD_URI); + servletRequest.addHeader(HttpHeader.UPLOAD_COMPLETE, "?0"); + servletRequest.addHeader(HttpHeader.UPLOAD_LENGTH, "1000"); + + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + assertResponseStatus(HttpServletResponse.SC_CREATED); + String uploadLocation = servletResponse.getHeader(HttpHeader.LOCATION); + + // Step 2: Send DELETE request to cancel the upload + reset(); + servletRequest.setMethod("DELETE"); + servletRequest.setRequestURI(uploadLocation); + + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + assertResponseStatus(HttpServletResponse.SC_NO_CONTENT); + + // Step 3: Verify resource was deactivated (HEAD returns 404 Not Found) + reset(); + servletRequest.setMethod("HEAD"); + servletRequest.setRequestURI(uploadLocation); + + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + assertResponseStatus(HttpServletResponse.SC_NOT_FOUND); + } + + // =============================================================================================== + // USE CASE 7: Offset Mismatch Detection and Resumption + // =============================================================================================== + + /** + * Section 4.4.2 & 7.1 (Mismatching Upload-Offset): "If the Upload-Offset header field value does + * not match the current offset... the server MUST reject the request with a 409 (Conflict) status + * code..." + * + *

Use Case: Send PATCH with wrong offset -> receive 409 Conflict with correct offset header -> + * resend PATCH with correct offset -> upload succeeds. + */ + @Test + public void testOffsetMismatchAndResumption() throws Exception { + // Step 1: Create upload resource and upload 5 bytes + servletRequest.setMethod("POST"); + servletRequest.setRequestURI(UPLOAD_URI); + servletRequest.addHeader(HttpHeader.UPLOAD_COMPLETE, "?0"); + servletRequest.addHeader(HttpHeader.UPLOAD_LENGTH, "100"); + servletRequest.addHeader(HttpHeader.CONTENT_TYPE, HttpHeader.CONTENT_TYPE_PARTIAL_UPLOAD); + servletRequest.setContent("12345".getBytes(StandardCharsets.UTF_8)); + + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + assertResponseStatus(HttpServletResponse.SC_CREATED); + String uploadLocation = servletResponse.getHeader(HttpHeader.LOCATION); + assertResponseHeader(HttpHeader.UPLOAD_OFFSET, "5"); + + // Step 2: Attempt PATCH with incorrect offset 0 (server is at offset 5) + reset(); + servletRequest.setMethod("PATCH"); + servletRequest.setRequestURI(uploadLocation); + servletRequest.addHeader(HttpHeader.CONTENT_TYPE, HttpHeader.CONTENT_TYPE_PARTIAL_UPLOAD); + servletRequest.addHeader(HttpHeader.UPLOAD_OFFSET, "0"); + servletRequest.addHeader(HttpHeader.UPLOAD_COMPLETE, "?0"); + servletRequest.setContent("6789".getBytes(StandardCharsets.UTF_8)); + + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + + // Step 3: Verify 409 Conflict response containing current server offset (5) + assertResponseStatus(HttpServletResponse.SC_CONFLICT); + assertResponseHeader(HttpHeader.UPLOAD_OFFSET, "5"); + + // Step 4: Resend PATCH with correct offset 5 + reset(); + servletRequest.setMethod("PATCH"); + servletRequest.setRequestURI(uploadLocation); + servletRequest.addHeader(HttpHeader.CONTENT_TYPE, HttpHeader.CONTENT_TYPE_PARTIAL_UPLOAD); + servletRequest.addHeader(HttpHeader.UPLOAD_OFFSET, "5"); + servletRequest.addHeader(HttpHeader.UPLOAD_COMPLETE, "?0"); + servletRequest.setContent("6789".getBytes(StandardCharsets.UTF_8)); + + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + + // Step 5: Verify HTTP 204 No Content response and updated offset 9 + assertResponseStatus(HttpServletResponse.SC_NO_CONTENT); + assertResponseHeader(HttpHeader.UPLOAD_OFFSET, "9"); + } + + // =============================================================================================== + // USE CASE 8: Inconsistent Upload-Length Validation + // =============================================================================================== + + /** + * Section 4.1.3 & 7.2 (Inconsistent Upload-Length): "The server MUST reject a request if the + * representation's length is known and inconsistent..." + * + *

Use Case: Request declares Upload-Length: 1000 but content is only 11 bytes with + * Upload-Complete: ?1 -> server rejects with 400 Bad Request. + */ + @Test + public void testInconsistentUploadLength() throws Exception { + // Step 1: Send request declaring Upload-Length: 1000 and Upload-Complete: ?1 but providing only + // 11 bytes + servletRequest.setMethod("POST"); + servletRequest.setRequestURI(UPLOAD_URI); + servletRequest.addHeader(HttpHeader.UPLOAD_COMPLETE, "?1"); + servletRequest.addHeader(HttpHeader.UPLOAD_LENGTH, "1000"); + servletRequest.setContent("Hello World".getBytes(StandardCharsets.UTF_8)); + + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + + // Step 2: Verify HTTP 400 Bad Request response + assertResponseStatus(HttpServletResponse.SC_BAD_REQUEST); + assertResponseHeader(HttpHeader.UPLOAD_COMPLETE, "?0"); + } + + // =============================================================================================== + // USE CASE 9: Invalid Append Headers Validation + // =============================================================================================== + + /** + * Section 4.4.1 & 4.4.2 (Upload Append Validation): "The request MUST include the Upload-Offset + * and Upload-Complete header fields. Content-Type MUST be application/partial-upload." + * + *

Use Case: Send PATCH requests missing required headers or using wrong Content-Type -> server + * rejects with appropriate HTTP error codes. + */ + @Test + public void testInvalidAppendHeaders() throws Exception { + // Step 1: Create upload resource + servletRequest.setMethod("POST"); + servletRequest.setRequestURI(UPLOAD_URI); + servletRequest.addHeader(HttpHeader.UPLOAD_COMPLETE, "?0"); + servletRequest.addHeader(HttpHeader.UPLOAD_LENGTH, "100"); + + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + assertResponseStatus(HttpServletResponse.SC_CREATED); + String uploadLocation = servletResponse.getHeader(HttpHeader.LOCATION); + + // Step 2: Send PATCH missing Upload-Offset -> verify HTTP 400 Bad Request + reset(); + servletRequest.setMethod("PATCH"); + servletRequest.setRequestURI(uploadLocation); + servletRequest.addHeader(HttpHeader.CONTENT_TYPE, HttpHeader.CONTENT_TYPE_PARTIAL_UPLOAD); + servletRequest.addHeader(HttpHeader.UPLOAD_COMPLETE, "?0"); + + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + assertResponseStatus(HttpServletResponse.SC_BAD_REQUEST); + + // Step 3: Send PATCH missing Upload-Complete -> verify HTTP 400 Bad Request + reset(); + servletRequest.setMethod("PATCH"); + servletRequest.setRequestURI(uploadLocation); + servletRequest.addHeader(HttpHeader.CONTENT_TYPE, HttpHeader.CONTENT_TYPE_PARTIAL_UPLOAD); + servletRequest.addHeader(HttpHeader.UPLOAD_OFFSET, "0"); + + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + assertResponseStatus(HttpServletResponse.SC_BAD_REQUEST); + + // Step 4: Send PATCH with wrong Content-Type (text/plain) -> verify HTTP 415 Unsupported Media + // Type + reset(); + servletRequest.setMethod("PATCH"); + servletRequest.setRequestURI(uploadLocation); + servletRequest.addHeader(HttpHeader.CONTENT_TYPE, "text/plain"); + servletRequest.addHeader(HttpHeader.UPLOAD_OFFSET, "0"); + servletRequest.addHeader(HttpHeader.UPLOAD_COMPLETE, "?0"); + + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + assertResponseStatus(HttpServletResponse.SC_UNSUPPORTED_MEDIA_TYPE); + } + + // =============================================================================================== + // USE CASE 10: Exceeding Upload-Length Rejection & Resource Invalidation + // =============================================================================================== + + /** + * Section 4.4.2 (Exceeding Upload-Length): "the server MUST prevent the offset from exceeding the + * representation's length by rejecting the request with a 409 (Conflict) status code... marking + * the upload resource invalid." + * + *

Use Case: Append payload exceeding declared length -> 409 Conflict -> resource invalidation. + */ + @Test + public void testExceedingUploadLength() throws Exception { + // Step 1: Create upload declaring Upload-Length: 10 + servletRequest.setMethod("POST"); + servletRequest.setRequestURI(UPLOAD_URI); + servletRequest.addHeader(HttpHeader.UPLOAD_COMPLETE, "?0"); + servletRequest.addHeader(HttpHeader.UPLOAD_LENGTH, "10"); + + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + assertResponseStatus(HttpServletResponse.SC_CREATED); + String uploadLocation = servletResponse.getHeader(HttpHeader.LOCATION); + + // Step 2: Append 15 bytes (exceeding length 10) + reset(); + servletRequest.setMethod("PATCH"); + servletRequest.setRequestURI(uploadLocation); + servletRequest.addHeader(HttpHeader.CONTENT_TYPE, HttpHeader.CONTENT_TYPE_PARTIAL_UPLOAD); + servletRequest.addHeader(HttpHeader.UPLOAD_OFFSET, "0"); + servletRequest.addHeader(HttpHeader.UPLOAD_COMPLETE, "?0"); + servletRequest.setContent("123456789012345".getBytes(StandardCharsets.UTF_8)); + + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + + // Step 3: Verify 409 Conflict response + assertResponseStatus(HttpServletResponse.SC_CONFLICT); + + // Step 4: Verify resource was invalidated (subsequent HEAD returns 404 Not Found) + reset(); + servletRequest.setMethod("HEAD"); + servletRequest.setRequestURI(uploadLocation); + + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + assertResponseStatus(HttpServletResponse.SC_NOT_FOUND); + } + + // =============================================================================================== + // USE CASE 11: Content-Digest Validation (RFC 9530) + // =============================================================================================== + + /** + * Section 3 of RFC 9530 (Content-Digest): "The Content-Digest HTTP header field associates one or + * more digests with a message content." If the digest does not match, the server MUST consider + * the transfer failed. + * + *

Use Case: Send PATCH payload with invalid sha-256 digest -> rejected with 400 Bad Request; + * send matching digest -> accepted with 204 No Content. + */ + @Test + public void testContentDigestValidation() throws Exception { + // Step 1: Create upload resource + servletRequest.setMethod("POST"); + servletRequest.setRequestURI(UPLOAD_URI); + servletRequest.addHeader(HttpHeader.UPLOAD_COMPLETE, "?0"); + servletRequest.addHeader(HttpHeader.UPLOAD_LENGTH, "100"); + + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + assertResponseStatus(HttpServletResponse.SC_CREATED); + String uploadLocation = servletResponse.getHeader(HttpHeader.LOCATION); + + // Step 2: Send PATCH with wrong sha-256 digest + reset(); + servletRequest.setMethod("PATCH"); + servletRequest.setRequestURI(uploadLocation); + servletRequest.addHeader(HttpHeader.CONTENT_TYPE, HttpHeader.CONTENT_TYPE_PARTIAL_UPLOAD); + servletRequest.addHeader(HttpHeader.UPLOAD_OFFSET, "0"); + servletRequest.addHeader(HttpHeader.UPLOAD_COMPLETE, "?0"); + servletRequest.addHeader( + HttpHeader.CONTENT_DIGEST, "sha-256=:47DEQpj8HBSa+/TImW+5JCeuQeRkm5NMpJWZG3hSuFU=:"); + servletRequest.setContent("hello digest".getBytes(StandardCharsets.UTF_8)); + + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + + // Step 3: Verify HTTP 400 Bad Request rejection due to digest mismatch + assertResponseStatus(HttpServletResponse.SC_BAD_REQUEST); + + // Step 4: Send PATCH with valid matching sha-256 digest for "hello digest" + reset(); + servletRequest.setMethod("PATCH"); + servletRequest.setRequestURI(uploadLocation); + servletRequest.addHeader(HttpHeader.CONTENT_TYPE, HttpHeader.CONTENT_TYPE_PARTIAL_UPLOAD); + servletRequest.addHeader(HttpHeader.UPLOAD_OFFSET, "0"); + servletRequest.addHeader(HttpHeader.UPLOAD_COMPLETE, "?0"); + servletRequest.addHeader( + HttpHeader.CONTENT_DIGEST, "sha-256=:yV9g7MInOPrtlLDWsplfHK0LaH22Uz70R1ZXbHIjzjU=:"); + servletRequest.setContent("hello digest".getBytes(StandardCharsets.UTF_8)); + + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + + // Step 5: Verify HTTP 204 No Content acceptance and updated offset 12 + assertResponseStatus(HttpServletResponse.SC_NO_CONTENT); + assertResponseHeader(HttpHeader.UPLOAD_OFFSET, "12"); + } + + // =============================================================================================== + // ASSERTION HELPERS + // =============================================================================================== + + protected void assertResponseStatus(int expectedStatus) { + assertThat(servletResponse.getStatus(), is(expectedStatus)); + } + + protected void assertResponseHeader(String headerName, String expectedValue) { + assertThat(servletResponse.getHeader(headerName), is(expectedValue)); + } + + protected void assertResponseHeaderNotBlank(String headerName) { + assertTrue( + "Header " + headerName + " should not be blank", + StringUtils.isNotBlank(servletResponse.getHeader(headerName))); + } +} diff --git a/src/test/java/me/desair/tus/server/AbstractITTusFileUploadService.java b/src/test/java/me/desair/tus/server/AbstractITTusFileUploadService.java new file mode 100644 index 00000000..c558fb36 --- /dev/null +++ b/src/test/java/me/desair/tus/server/AbstractITTusFileUploadService.java @@ -0,0 +1,1874 @@ +package me.desair.tus.server; + +import static me.desair.tus.server.util.MapMatcher.hasSize; +import static org.hamcrest.CoreMatchers.allOf; +import static org.hamcrest.CoreMatchers.is; +import static org.hamcrest.CoreMatchers.nullValue; +import static org.hamcrest.MatcherAssert.assertThat; +import static org.hamcrest.Matchers.containsInAnyOrder; +import static org.hamcrest.Matchers.greaterThan; +import static org.hamcrest.collection.IsMapContaining.hasEntry; +import static org.junit.Assert.assertFalse; +import static org.junit.Assert.assertNull; +import static org.junit.Assert.assertTrue; +import static org.junit.Assert.fail; + +import jakarta.servlet.http.HttpServletResponse; +import java.io.IOException; +import java.io.InputStream; +import java.nio.charset.StandardCharsets; +import java.text.DateFormat; +import java.text.SimpleDateFormat; +import java.util.Arrays; +import java.util.Locale; +import java.util.UUID; +import me.desair.tus.server.exception.TusException; +import me.desair.tus.server.upload.UploadInfo; +import me.desair.tus.server.util.Utils; +import org.apache.commons.io.IOUtils; +import org.apache.commons.lang3.StringUtils; +import org.junit.Before; +import org.junit.Test; +import org.springframework.mock.web.MockHttpServletRequest; +import org.springframework.mock.web.MockHttpServletResponse; + +/** Test cases for the {@link TusFileUploadService}. */ +public abstract class AbstractITTusFileUploadService { + + protected abstract TusFileUploadService createTusFileUploadService() throws Exception; + + protected abstract TusFileUploadService createTusFileUploadService(String uploadUri) + throws Exception; + + protected static final String UPLOAD_URI = "/test/upload"; + protected static final String OWNER_KEY = "JOHN_DOE"; + + private static final DateFormat mockDateFormat = + new SimpleDateFormat("EEE, dd MMM yyyy HH:mm:ss zzz", Locale.US); + + protected MockHttpServletRequest servletRequest; + protected MockHttpServletResponse servletResponse; + + protected TusFileUploadService tusFileUploadService; + + @Before + public void setUp() throws Exception { + reset(); + tusFileUploadService = createTusFileUploadService(); + } + + protected void reset() { + servletRequest = new MockHttpServletRequest(); + servletRequest.setRemoteAddr("192.168.1.1"); + servletRequest.addHeader(HttpHeader.X_FORWARDED_FOR, "10.0.2.1, 123.231.12.4"); + servletResponse = new MockHttpServletResponse(); + } + + @Test + public void testSupportedHttpMethods() { + assertThat( + tusFileUploadService.getSupportedHttpMethods(), + containsInAnyOrder( + HttpMethod.HEAD, + HttpMethod.OPTIONS, + HttpMethod.PATCH, + HttpMethod.POST, + HttpMethod.PUT, + HttpMethod.DELETE, + HttpMethod.GET)); + + assertThat( + tusFileUploadService.getEnabledFeatures(), + containsInAnyOrder( + "core", + "creation", + "creation-with-upload", + "checksum", + "termination", + "download", + "expiration", + "concatenation", + "cors", + "resumable-uploads-for-http", + "http-digests")); + } + + @Test + public void testDisableFeature() throws Exception { + tusFileUploadService.disableTusExtension("download"); + tusFileUploadService.disableTusExtension("termination"); + tusFileUploadService.disableTusExtension("resumable-uploads-for-http"); + tusFileUploadService.disableTusExtension("http-digests"); + + assertThat( + tusFileUploadService.getSupportedHttpMethods(), + containsInAnyOrder(HttpMethod.HEAD, HttpMethod.OPTIONS, HttpMethod.PATCH, HttpMethod.POST)); + + assertThat( + tusFileUploadService.getEnabledFeatures(), + containsInAnyOrder( + "core", + "creation", + "creation-with-upload", + "checksum", + "expiration", + "concatenation", + "cors")); + + reset(); + servletRequest.setMethod("GET"); + servletRequest.setRequestURI(UPLOAD_URI + "/" + UUID.randomUUID()); + servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); + + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + assertResponseHeader(HttpHeader.CONTENT_LENGTH, "0"); + assertResponseStatus(HttpServletResponse.SC_METHOD_NOT_ALLOWED); + + reset(); + servletRequest.setMethod("DELETE"); + servletRequest.setRequestURI(UPLOAD_URI + "/" + UUID.randomUUID()); + servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); + + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + assertResponseHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); + assertResponseHeader(HttpHeader.CONTENT_LENGTH, "0"); + assertResponseStatus(HttpServletResponse.SC_METHOD_NOT_ALLOWED); + } + + @Test(expected = IllegalArgumentException.class) + public void testDisableCore() { + tusFileUploadService.disableTusExtension("core"); + } + + @Test(expected = NullPointerException.class) + public void testWithFileStoreServiceNull() throws Exception { + tusFileUploadService.withUploadStorageService(null); + } + + @Test + public void testProcessCompleteUpload() throws Exception { + String uploadContent = "This is my test upload content"; + + // Create upload + servletRequest.setMethod("POST"); + servletRequest.setRequestURI(UPLOAD_URI); + servletRequest.addHeader(HttpHeader.CONTENT_LENGTH, 0); + servletRequest.addHeader(HttpHeader.UPLOAD_LENGTH, uploadContent.getBytes().length); + servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); + servletRequest.addHeader( + HttpHeader.UPLOAD_METADATA, "filename d29ybGRfZG9taW5hdGlvbl9wbGFuLnBkZg=="); + + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + assertResponseHeaderNotBlank(HttpHeader.LOCATION); + assertResponseHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); + assertResponseHeader(HttpHeader.CONTENT_LENGTH, "0"); + assertResponseHeaderNotBlank(HttpHeader.UPLOAD_EXPIRES); + assertResponseStatus(HttpServletResponse.SC_CREATED); + + String location = + UPLOAD_URI + + StringUtils.substringAfter( + servletResponse.getHeader(HttpHeader.LOCATION), UPLOAD_URI); + + // Upload bytes + reset(); + servletRequest.setMethod("PATCH"); + servletRequest.setRequestURI(location); + servletRequest.addHeader(HttpHeader.CONTENT_TYPE, "application/offset+octet-stream"); + servletRequest.addHeader(HttpHeader.CONTENT_LENGTH, uploadContent.getBytes().length); + servletRequest.addHeader(HttpHeader.UPLOAD_OFFSET, 0); + servletRequest.addHeader(HttpHeader.UPLOAD_CHECKSUM, "sha1 Mfhm5HaSPUf+pUakdMxARo4rvfQ="); + servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); + servletRequest.setContent(uploadContent.getBytes()); + + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + assertResponseHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); + assertResponseHeader(HttpHeader.CONTENT_LENGTH, "0"); + assertResponseHeader(HttpHeader.UPLOAD_OFFSET, "" + uploadContent.getBytes().length); + assertResponseHeaderNotBlank(HttpHeader.UPLOAD_EXPIRES); + assertResponseStatus(HttpServletResponse.SC_NO_CONTENT); + + // Make sure cleanup does not interfere with this test + tusFileUploadService.cleanup(); + + // Check with HEAD request upload is complete + reset(); + servletRequest.setMethod("HEAD"); + servletRequest.setRequestURI(location); + servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); + + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + assertResponseHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); + assertResponseHeader(HttpHeader.CONTENT_LENGTH, "0"); + assertResponseHeader(HttpHeader.UPLOAD_OFFSET, "" + uploadContent.getBytes().length); + assertResponseHeader(HttpHeader.UPLOAD_LENGTH, "" + uploadContent.getBytes().length); + assertResponseHeaderNull(HttpHeader.UPLOAD_DEFER_LENGTH); + assertResponseHeader( + HttpHeader.UPLOAD_METADATA, "filename d29ybGRfZG9taW5hdGlvbl9wbGFuLnBkZg=="); + assertResponseStatus(HttpServletResponse.SC_NO_CONTENT); + + // Get upload info from service + UploadInfo info = tusFileUploadService.getUploadInfo(location, OWNER_KEY); + assertFalse(info.isUploadInProgress()); + assertThat(info.getLength(), is((long) uploadContent.getBytes().length)); + assertThat(info.getOffset(), is((long) uploadContent.getBytes().length)); + assertThat( + info.getMetadata(), allOf(hasSize(1), hasEntry("filename", "world_domination_plan.pdf"))); + assertThat(info.getCreatorIpAddresses(), is("10.0.2.1, 123.231.12.4, 192.168.1.1")); + + // Try retrieving the uploaded bytes without owner key + try { + tusFileUploadService.getUploadedBytes(location); + fail(); + } catch (TusException ex) { + assertThat(ex.getStatus(), is(404)); + } + + // Get uploaded bytes from service + try (InputStream uploadedBytes = tusFileUploadService.getUploadedBytes(location, OWNER_KEY)) { + assertThat( + IOUtils.toString(uploadedBytes, StandardCharsets.UTF_8), + is("This is my test upload content")); + } + + // Make sure cleanup does not interfere with this test + tusFileUploadService.cleanup(); + + // Download the upload + reset(); + servletRequest.setMethod("GET"); + servletRequest.setRequestURI(location); + + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + assertResponseHeader(HttpHeader.CONTENT_LENGTH, "" + uploadContent.getBytes().length); + assertResponseHeader( + HttpHeader.UPLOAD_METADATA, "filename d29ybGRfZG9taW5hdGlvbl9wbGFuLnBkZg=="); + assertResponseStatus(HttpServletResponse.SC_OK); + assertThat(servletResponse.getContentAsString(), is("This is my test upload content")); + + // Pretend that we processed the upload and that we can remove it + tusFileUploadService.deleteUpload(location, OWNER_KEY); + + // Check that the upload is really gone + reset(); + servletRequest.setMethod("HEAD"); + servletRequest.setRequestURI(location); + servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); + + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + assertResponseStatus(HttpServletResponse.SC_NOT_FOUND); + assertResponseHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); + assertResponseHeader(HttpHeader.CONTENT_LENGTH, "0"); + } + + @Test + public void testProcessZeroByteUpload() throws Exception { + // Create upload + servletRequest.setMethod("POST"); + servletRequest.setRequestURI(UPLOAD_URI); + servletRequest.addHeader(HttpHeader.CONTENT_LENGTH, 0); + servletRequest.addHeader(HttpHeader.UPLOAD_LENGTH, 0); + servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); + + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + assertResponseHeaderNotBlank(HttpHeader.LOCATION); + assertResponseHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); + assertResponseHeader(HttpHeader.CONTENT_LENGTH, "0"); + assertResponseHeaderNotBlank(HttpHeader.UPLOAD_EXPIRES); + assertResponseStatus(HttpServletResponse.SC_CREATED); + + String location = + UPLOAD_URI + + StringUtils.substringAfter( + servletResponse.getHeader(HttpHeader.LOCATION), UPLOAD_URI); + + // Get upload info from service + UploadInfo info = tusFileUploadService.getUploadInfo(location, OWNER_KEY); + assertFalse(info.isUploadInProgress()); + assertThat(info.getLength(), is(0L)); + assertThat(info.getOffset(), is(0L)); + + // Get uploaded bytes from service + try (InputStream uploadedBytes = tusFileUploadService.getUploadedBytes(location, OWNER_KEY)) { + assertThat(IOUtils.toString(uploadedBytes, StandardCharsets.UTF_8), is("")); + } + + // Make sure cleanup does not interfere with this test + tusFileUploadService.cleanup(); + + // Download the upload + reset(); + servletRequest.setMethod("GET"); + servletRequest.setRequestURI(location); + + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + assertResponseHeader(HttpHeader.CONTENT_LENGTH, "0"); + assertResponseStatus(HttpServletResponse.SC_OK); + assertThat(servletResponse.getContentAsString(), is("")); + } + + @Test + public void testTerminateViaHttpRequest() throws Exception { + String uploadContent = "This is my terminated test upload"; + + // Create upload + servletRequest.setMethod("POST"); + servletRequest.setRequestURI(UPLOAD_URI); + servletRequest.addHeader(HttpHeader.CONTENT_LENGTH, 0); + servletRequest.addHeader(HttpHeader.UPLOAD_LENGTH, uploadContent.getBytes().length); + servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); + servletRequest.addHeader( + HttpHeader.UPLOAD_METADATA, "filename d29ybGRfZG9taW5hdGlvbl9wbGFuLnBkZg=="); + + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + assertResponseHeaderNotBlank(HttpHeader.LOCATION); + assertResponseHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); + assertResponseHeader(HttpHeader.CONTENT_LENGTH, "0"); + assertResponseHeaderNotBlank(HttpHeader.UPLOAD_EXPIRES); + assertResponseStatus(HttpServletResponse.SC_CREATED); + + String location = + UPLOAD_URI + + StringUtils.substringAfter( + servletResponse.getHeader(HttpHeader.LOCATION), UPLOAD_URI); + + // Upload bytes + reset(); + servletRequest.setMethod("PATCH"); + servletRequest.setRequestURI(location); + servletRequest.addHeader(HttpHeader.CONTENT_TYPE, "application/offset+octet-stream"); + servletRequest.addHeader(HttpHeader.CONTENT_LENGTH, uploadContent.getBytes().length); + servletRequest.addHeader(HttpHeader.UPLOAD_OFFSET, 0); + servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); + servletRequest.setContent(uploadContent.getBytes()); + + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + assertResponseHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); + assertResponseHeader(HttpHeader.CONTENT_LENGTH, "0"); + assertResponseHeader(HttpHeader.UPLOAD_OFFSET, "" + uploadContent.getBytes().length); + assertResponseHeaderNotBlank(HttpHeader.UPLOAD_EXPIRES); + assertResponseStatus(HttpServletResponse.SC_NO_CONTENT); + + // Make sure cleanup does not interfere with this test + tusFileUploadService.cleanup(); + + // Download the upload to make sure it was uploaded correctly + reset(); + servletRequest.setMethod("GET"); + servletRequest.setRequestURI(location); + + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + assertResponseHeader(HttpHeader.CONTENT_LENGTH, "" + uploadContent.getBytes().length); + assertResponseHeader( + HttpHeader.UPLOAD_METADATA, "filename d29ybGRfZG9taW5hdGlvbl9wbGFuLnBkZg=="); + assertResponseStatus(HttpServletResponse.SC_OK); + assertThat(servletResponse.getContentAsString(), is("This is my terminated test upload")); + + // Terminate the upload so that the server can remove it + reset(); + servletRequest.setMethod("DELETE"); + servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); + servletRequest.setRequestURI(location); + + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + assertResponseHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); + assertResponseStatus(HttpServletResponse.SC_NO_CONTENT); + + // Check that the upload is really gone + reset(); + servletRequest.setMethod("HEAD"); + servletRequest.setRequestURI(location); + servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); + + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + assertResponseStatus(HttpServletResponse.SC_NOT_FOUND); + assertResponseHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); + assertResponseHeader(HttpHeader.CONTENT_LENGTH, "0"); + } + + @Test + public void testProcessUploadTwoParts() throws Exception { + String part1 = + "29\r\nThis is the first part of my test upload " + + "\r\n0\r\nUpload-Checksum: sha1 n5RQbRwM6UVAD+9iuHEmnN6HCGQ="; + String part2 = + "1C\r\nand this is the second part." + + "\r\n0\r\nUpload-Checksum: sha1 oNge323kGFKICxp+Me5xJgPvGEM="; + + // Create upload + servletRequest.setMethod("POST"); + servletRequest.setRequestURI(UPLOAD_URI); + servletRequest.addHeader(HttpHeader.CONTENT_LENGTH, 0); + servletRequest.addHeader(HttpHeader.UPLOAD_LENGTH, "69"); + servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); + servletRequest.addHeader( + HttpHeader.UPLOAD_METADATA, "filename d29ybGRfZG9taW5hdGlvbl9wbGFuLnBkZg=="); + + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + assertResponseHeaderNotBlank(HttpHeader.LOCATION); + assertResponseHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); + assertResponseHeader(HttpHeader.CONTENT_LENGTH, "0"); + assertResponseHeaderNotBlank(HttpHeader.UPLOAD_EXPIRES); + assertResponseStatus(HttpServletResponse.SC_CREATED); + + String location = + UPLOAD_URI + + StringUtils.substringAfter( + servletResponse.getHeader(HttpHeader.LOCATION), UPLOAD_URI); + + // Make sure cleanup does not interfere with this test + tusFileUploadService.cleanup(); + + // Upload part 1 bytes + reset(); + servletRequest.setMethod("PATCH"); + servletRequest.setRequestURI(location); + servletRequest.addHeader(HttpHeader.CONTENT_TYPE, "application/offset+octet-stream"); + servletRequest.addHeader(HttpHeader.CONTENT_LENGTH, "41"); + servletRequest.addHeader(HttpHeader.UPLOAD_OFFSET, 0); + servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); + servletRequest.addHeader(HttpHeader.TRANSFER_ENCODING, "chunked"); + servletRequest.setContent(part1.getBytes()); + + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + assertResponseStatus(HttpServletResponse.SC_NO_CONTENT); + assertResponseHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); + assertResponseHeader(HttpHeader.CONTENT_LENGTH, "0"); + assertResponseHeaderNotBlank(HttpHeader.UPLOAD_EXPIRES); + assertResponseHeader(HttpHeader.UPLOAD_OFFSET, "41"); + + // Check with service that upload is still in progress + UploadInfo info = tusFileUploadService.getUploadInfo(location, OWNER_KEY); + assertTrue(info.isUploadInProgress()); + assertThat(info.getLength(), is(69L)); + assertThat(info.getOffset(), is(41L)); + assertThat( + info.getMetadata(), allOf(hasSize(1), hasEntry("filename", "world_domination_plan.pdf"))); + assertThat(info.getCreatorIpAddresses(), is("10.0.2.1, 123.231.12.4, 192.168.1.1")); + + // Make sure cleanup does not interfere with this test + tusFileUploadService.cleanup(); + + // Verify that we cannot download an in-progress upload + reset(); + servletRequest.setMethod("GET"); + servletRequest.setRequestURI(location); + + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + assertResponseStatus(204); + assertResponseHeader(HttpHeader.CONTENT_LENGTH, "0"); + assertThat(servletResponse.getContentAsString(), is("")); + + // Upload part 2 bytes + reset(); + servletRequest.setMethod("PATCH"); + servletRequest.setRequestURI(location); + servletRequest.addHeader(HttpHeader.CONTENT_TYPE, "application/offset+octet-stream"); + servletRequest.addHeader(HttpHeader.CONTENT_LENGTH, "28"); + servletRequest.addHeader(HttpHeader.UPLOAD_OFFSET, "41"); + servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); + servletRequest.addHeader(HttpHeader.TRANSFER_ENCODING, "chunked"); + servletRequest.setContent(part2.getBytes()); + + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + assertResponseHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); + assertResponseHeader(HttpHeader.CONTENT_LENGTH, "0"); + assertResponseHeader(HttpHeader.UPLOAD_OFFSET, "69"); + assertResponseHeaderNotBlank(HttpHeader.UPLOAD_EXPIRES); + assertResponseStatus(HttpServletResponse.SC_NO_CONTENT); + + // Check with HEAD request upload is complete + reset(); + servletRequest.setMethod("HEAD"); + servletRequest.setRequestURI(location); + servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); + + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + assertResponseHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); + assertResponseHeader(HttpHeader.CONTENT_LENGTH, "0"); + assertResponseHeader(HttpHeader.UPLOAD_OFFSET, "69"); + assertResponseHeader(HttpHeader.UPLOAD_LENGTH, "69"); + assertResponseHeaderNull(HttpHeader.UPLOAD_DEFER_LENGTH); + assertResponseHeader( + HttpHeader.UPLOAD_METADATA, "filename d29ybGRfZG9taW5hdGlvbl9wbGFuLnBkZg=="); + assertResponseStatus(HttpServletResponse.SC_NO_CONTENT); + + // Get upload info from service + info = tusFileUploadService.getUploadInfo(location, OWNER_KEY); + assertFalse(info.isUploadInProgress()); + assertThat(info.getLength(), is(69L)); + assertThat(info.getOffset(), is(69L)); + assertThat( + info.getMetadata(), allOf(hasSize(1), hasEntry("filename", "world_domination_plan.pdf"))); + assertThat(info.getCreatorIpAddresses(), is("10.0.2.1, 123.231.12.4, 192.168.1.1")); + + // Get uploaded bytes from service + try (InputStream uploadedBytes = tusFileUploadService.getUploadedBytes(location, OWNER_KEY)) { + assertThat( + IOUtils.toString(uploadedBytes, StandardCharsets.UTF_8), + is("This is the first part of my test upload and this is the second part.")); + } + } + + @Test + public void testProcessUploadDeferredLength() throws Exception { + String part1 = "When sending this part, we don't know the length and "; + String part2 = "when sending this part, we know the length but the upload is not complete. "; + String part3 = "Finally when sending the third part, the upload is complete."; + + // Create upload + servletRequest.setMethod("POST"); + servletRequest.setRequestURI(UPLOAD_URI); + servletRequest.addHeader(HttpHeader.CONTENT_LENGTH, 0); + servletRequest.addHeader(HttpHeader.UPLOAD_DEFER_LENGTH, 1); + servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); + servletRequest.addHeader( + HttpHeader.UPLOAD_METADATA, "filename d29ybGRfZG9taW5hdGlvbl9wbGFuLnBkZg=="); + + tusFileUploadService.process(servletRequest, servletResponse); + assertResponseHeaderNotBlank(HttpHeader.LOCATION); + assertResponseHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); + assertResponseHeader(HttpHeader.CONTENT_LENGTH, "0"); + assertResponseHeaderNotBlank(HttpHeader.UPLOAD_EXPIRES); + assertResponseStatus(HttpServletResponse.SC_CREATED); + + Long expirationTimestampBefore = + Long.parseLong( + String.valueOf( + mockDateFormat + .parse(servletResponse.getHeader(HttpHeader.UPLOAD_EXPIRES)) + .getTime())); + + String location = + UPLOAD_URI + + StringUtils.substringAfter( + servletResponse.getHeader(HttpHeader.LOCATION), UPLOAD_URI); + + // Upload part 1 bytes + reset(); + servletRequest.setMethod("PATCH"); + servletRequest.setRequestURI(location); + servletRequest.addHeader(HttpHeader.CONTENT_TYPE, "application/offset+octet-stream"); + servletRequest.addHeader(HttpHeader.CONTENT_LENGTH, part1.getBytes().length); + servletRequest.addHeader(HttpHeader.UPLOAD_OFFSET, 0); + servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); + servletRequest.setContent(part1.getBytes()); + + tusFileUploadService.process(servletRequest, servletResponse); + assertResponseHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); + assertResponseHeader(HttpHeader.CONTENT_LENGTH, "0"); + assertResponseHeaderNotBlank(HttpHeader.UPLOAD_EXPIRES); + assertResponseHeader(HttpHeader.UPLOAD_OFFSET, "" + part1.getBytes().length); + assertResponseStatus(HttpServletResponse.SC_NO_CONTENT); + + // Check with service that upload is still in progress + UploadInfo info = tusFileUploadService.getUploadInfo(location, null); + assertTrue(info.isUploadInProgress()); + assertThat(info.getLength(), is(nullValue())); + assertThat(info.getOffset(), is((long) part1.getBytes().length)); + assertThat( + info.getMetadata(), allOf(hasSize(1), hasEntry("filename", "world_domination_plan.pdf"))); + assertThat(info.getCreatorIpAddresses(), is("10.0.2.1, 123.231.12.4, 192.168.1.1")); + + // Make sure cleanup does not interfere with this test + tusFileUploadService.cleanup(); + + // Check with HEAD request length is still not known + reset(); + servletRequest.setMethod("HEAD"); + servletRequest.setRequestURI(location); + servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); + + tusFileUploadService.process(servletRequest, servletResponse); + assertResponseHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); + assertResponseHeader(HttpHeader.CONTENT_LENGTH, "0"); + assertResponseHeader(HttpHeader.UPLOAD_OFFSET, "" + part1.getBytes().length); + assertResponseHeader(HttpHeader.UPLOAD_DEFER_LENGTH, "1"); + assertResponseHeader( + HttpHeader.UPLOAD_METADATA, "filename d29ybGRfZG9taW5hdGlvbl9wbGFuLnBkZg=="); + assertResponseStatus(HttpServletResponse.SC_NO_CONTENT); + + // Upload part 2 bytes with length + reset(); + servletRequest.setMethod("PATCH"); + servletRequest.setRequestURI(location); + servletRequest.addHeader(HttpHeader.CONTENT_TYPE, "application/offset+octet-stream"); + servletRequest.addHeader(HttpHeader.CONTENT_LENGTH, part2.getBytes().length); + servletRequest.addHeader(HttpHeader.UPLOAD_OFFSET, part1.getBytes().length); + servletRequest.addHeader(HttpHeader.UPLOAD_LENGTH, (part1 + part2 + part3).getBytes().length); + servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); + servletRequest.setContent(part2.getBytes()); + + tusFileUploadService.process(servletRequest, servletResponse); + assertResponseHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); + assertResponseHeader(HttpHeader.CONTENT_LENGTH, "0"); + assertResponseHeader(HttpHeader.UPLOAD_OFFSET, "" + (part1 + part2).getBytes().length); + assertResponseHeaderNotBlank(HttpHeader.UPLOAD_EXPIRES); + assertResponseStatus(HttpServletResponse.SC_NO_CONTENT); + + // Make sure cleanup does not interfere with this test + tusFileUploadService.cleanup(); + + // Check with HEAD request length is known + reset(); + servletRequest.setMethod("HEAD"); + servletRequest.setRequestURI(location); + servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); + + tusFileUploadService.process(servletRequest, servletResponse); + assertResponseHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); + assertResponseHeader(HttpHeader.CONTENT_LENGTH, "0"); + assertResponseHeader(HttpHeader.UPLOAD_OFFSET, "" + (part1 + part2).getBytes().length); + assertResponseHeader(HttpHeader.UPLOAD_LENGTH, "" + (part1 + part2 + part3).getBytes().length); + assertResponseHeader( + HttpHeader.UPLOAD_METADATA, "filename d29ybGRfZG9taW5hdGlvbl9wbGFuLnBkZg=="); + assertResponseHeaderNull(HttpHeader.UPLOAD_DEFER_LENGTH); + assertResponseStatus(HttpServletResponse.SC_NO_CONTENT); + + info = tusFileUploadService.getUploadInfo(location, null); + assertTrue(info.isUploadInProgress()); + assertThat(info.getLength(), is((long) (part1 + part2 + part3).getBytes().length)); + + // check that expiration timestamp was updated + assertThat(info.getExpirationTimestamp(), greaterThan(expirationTimestampBefore)); + + // Upload part 3 bytes + reset(); + servletRequest.setMethod("PATCH"); + servletRequest.setRequestURI(location); + servletRequest.addHeader(HttpHeader.CONTENT_TYPE, "application/offset+octet-stream"); + servletRequest.addHeader(HttpHeader.CONTENT_LENGTH, part3.getBytes().length); + servletRequest.addHeader(HttpHeader.UPLOAD_OFFSET, (part1 + part2).getBytes().length); + servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); + servletRequest.setContent(part3.getBytes()); + + tusFileUploadService.process(servletRequest, servletResponse); + assertResponseHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); + assertResponseHeader(HttpHeader.CONTENT_LENGTH, "0"); + assertResponseHeader(HttpHeader.UPLOAD_OFFSET, "" + (part1 + part2 + part3).getBytes().length); + assertResponseHeaderNotBlank(HttpHeader.UPLOAD_EXPIRES); + assertResponseStatus(HttpServletResponse.SC_NO_CONTENT); + + // Make sure cleanup does not interfere with this test + tusFileUploadService.cleanup(); + + // Get upload info from service + info = tusFileUploadService.getUploadInfo(location, null); + assertFalse(info.isUploadInProgress()); + assertThat(info.getLength(), is((long) (part1 + part2 + part3).getBytes().length)); + assertThat(info.getOffset(), is((long) (part1 + part2 + part3).getBytes().length)); + assertThat( + info.getMetadata(), allOf(hasSize(1), hasEntry("filename", "world_domination_plan.pdf"))); + + // Get uploaded bytes from service + try (InputStream uploadedBytes = tusFileUploadService.getUploadedBytes(location, null)) { + assertThat( + IOUtils.toString(uploadedBytes, StandardCharsets.UTF_8), + is( + "When sending this part, we don't know the length and " + + "when sending this part, we know the length but the upload is not complete. " + + "Finally when sending the third part, the upload is complete.")); + } + + // Make sure cleanup does not interfere with this test + tusFileUploadService.cleanup(); + + // Download the upload + reset(); + servletRequest.setMethod("GET"); + servletRequest.setRequestURI(location); + + tusFileUploadService.process(servletRequest, servletResponse, null); + assertResponseStatus(HttpServletResponse.SC_OK); + assertResponseHeader(HttpHeader.CONTENT_LENGTH, "" + (part1 + part2 + part3).getBytes().length); + assertResponseHeader( + HttpHeader.UPLOAD_METADATA, "filename d29ybGRfZG9taW5hdGlvbl9wbGFuLnBkZg=="); + assertThat( + servletResponse.getContentAsString(), + is( + "When sending this part, we don't know the length and " + + "when sending this part, we know the length but the upload is not complete. " + + "Finally when sending the third part, the upload is complete.")); + } + + @Test + public void testProcessUploadInvalidChecksumSecondPart() throws Exception { + String part1 = + "29\r\nThis is the first part of my test upload " + + "\r\n0\r\nUPLOAD-CHECKSUM: sha1 n5RQbRwM6UVAD+9iuHEmnN6HCGQ="; + String part2 = "1C\r\nand this is the second part." + "\r\n0\r\nupload-checksum: sha1 invalid"; + + // Create upload + servletRequest.setMethod("POST"); + servletRequest.setRequestURI(UPLOAD_URI); + servletRequest.addHeader(HttpHeader.CONTENT_LENGTH, 0); + servletRequest.addHeader(HttpHeader.UPLOAD_LENGTH, "69"); + servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); + servletRequest.addHeader( + HttpHeader.UPLOAD_METADATA, "filename d29ybGRfZG9taW5hdGlvbl9wbGFuLnBkZg=="); + + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + assertResponseHeaderNotBlank(HttpHeader.LOCATION); + assertResponseHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); + assertResponseHeader(HttpHeader.CONTENT_LENGTH, "0"); + assertResponseHeaderNotBlank(HttpHeader.UPLOAD_EXPIRES); + assertResponseStatus(HttpServletResponse.SC_CREATED); + + String location = + UPLOAD_URI + + StringUtils.substringAfter( + servletResponse.getHeader(HttpHeader.LOCATION), UPLOAD_URI); + + // Make sure cleanup does not interfere with this test + tusFileUploadService.cleanup(); + + // Upload part 1 bytes + reset(); + servletRequest.setMethod("PATCH"); + servletRequest.setRequestURI(location); + servletRequest.addHeader(HttpHeader.CONTENT_TYPE, "application/offset+octet-stream"); + servletRequest.addHeader(HttpHeader.CONTENT_LENGTH, "41"); + servletRequest.addHeader(HttpHeader.UPLOAD_OFFSET, 0); + servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); + servletRequest.addHeader(HttpHeader.TRANSFER_ENCODING, "chunked"); + servletRequest.setContent(part1.getBytes()); + + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + assertResponseStatus(HttpServletResponse.SC_NO_CONTENT); + assertResponseHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); + assertResponseHeader(HttpHeader.CONTENT_LENGTH, "0"); + assertResponseHeaderNotBlank(HttpHeader.UPLOAD_EXPIRES); + assertResponseHeader(HttpHeader.UPLOAD_OFFSET, "41"); + + Long expirationTimestampBefore = + Long.parseLong( + String.valueOf( + mockDateFormat + .parse(servletResponse.getHeader(HttpHeader.UPLOAD_EXPIRES)) + .getTime())); + + // Make sure cleanup does not interfere with this test + tusFileUploadService.cleanup(); + + // Upload part 2 bytes + reset(); + servletRequest.setMethod("PATCH"); + servletRequest.setRequestURI(location); + servletRequest.addHeader(HttpHeader.CONTENT_TYPE, "application/offset+octet-stream"); + servletRequest.addHeader(HttpHeader.CONTENT_LENGTH, "28"); + servletRequest.addHeader(HttpHeader.UPLOAD_OFFSET, "41"); + servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); + servletRequest.addHeader(HttpHeader.TRANSFER_ENCODING, "chunked"); + servletRequest.setContent(part2.getBytes()); + + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + + // We expect the server to return a checksum mismatch error + assertResponseStatus(460); + assertResponseHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); + assertResponseHeader(HttpHeader.CONTENT_LENGTH, "0"); + + // Check that upload info is still from the first patch + UploadInfo info = tusFileUploadService.getUploadInfo(location, OWNER_KEY); + assertTrue(info.isUploadInProgress()); + assertThat(info.getLength(), is(69L)); + assertThat(info.getOffset(), is(41L)); + assertThat( + info.getMetadata(), allOf(hasSize(1), hasEntry("filename", "world_domination_plan.pdf"))); + + // check that expiration timestamp was updated + assertThat(info.getExpirationTimestamp(), greaterThan(expirationTimestampBefore)); + + // We only stored the first valid part + try (InputStream uploadedBytes = tusFileUploadService.getUploadedBytes(location, OWNER_KEY)) { + assertThat( + IOUtils.toString(uploadedBytes, StandardCharsets.UTF_8), + is("This is the first part of my test upload ")); + } + + // Make sure cleanup does not interfere with this test + tusFileUploadService.cleanup(); + + // Terminate our in progress upload + reset(); + servletRequest.setMethod("DELETE"); + servletRequest.setRequestURI(location); + servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); + + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + + // We expect the server to return a no content code to indicate successful deletion + assertResponseStatus(HttpServletResponse.SC_NO_CONTENT); + assertResponseHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); + assertResponseHeader(HttpHeader.CONTENT_LENGTH, "0"); + + // Make sure cleanup does not interfere with this test + tusFileUploadService.cleanup(); + + // Check that the upload is really gone + reset(); + servletRequest.setMethod("HEAD"); + servletRequest.setRequestURI(location); + servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); + + tusFileUploadService.process(servletRequest, servletResponse); + assertResponseStatus(HttpServletResponse.SC_NOT_FOUND); + assertResponseHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); + assertResponseHeader(HttpHeader.CONTENT_LENGTH, "0"); + } + + @Test + public void testCleanupExpiredUpload() throws Exception { + // Set the expiration period to 500 ms + tusFileUploadService.withUploadExpirationPeriod(500L); + + String part1 = "This is the first part of my test upload"; + // Create upload + servletRequest.setMethod("POST"); + servletRequest.setRequestURI(UPLOAD_URI); + servletRequest.addHeader(HttpHeader.CONTENT_LENGTH, 0); + servletRequest.addHeader(HttpHeader.UPLOAD_LENGTH, part1.getBytes().length + 20L); + servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); + + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + assertResponseHeaderNotBlank(HttpHeader.LOCATION); + assertResponseHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); + assertResponseHeader(HttpHeader.CONTENT_LENGTH, "0"); + assertResponseHeaderNotBlank(HttpHeader.UPLOAD_EXPIRES); + assertResponseStatus(HttpServletResponse.SC_CREATED); + + String location = + UPLOAD_URI + + StringUtils.substringAfter( + servletResponse.getHeader(HttpHeader.LOCATION), UPLOAD_URI); + + // Upload part 1 bytes + reset(); + servletRequest.setMethod("PATCH"); + servletRequest.setRequestURI(location); + servletRequest.addHeader(HttpHeader.CONTENT_TYPE, "application/offset+octet-stream"); + servletRequest.addHeader(HttpHeader.CONTENT_LENGTH, part1.getBytes().length); + servletRequest.addHeader(HttpHeader.UPLOAD_OFFSET, 0); + servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); + servletRequest.setContent(part1.getBytes()); + + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + assertResponseStatus(HttpServletResponse.SC_NO_CONTENT); + assertResponseHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); + assertResponseHeader(HttpHeader.CONTENT_LENGTH, "0"); + assertResponseHeaderNotBlank(HttpHeader.UPLOAD_EXPIRES); + assertResponseHeader(HttpHeader.UPLOAD_OFFSET, "" + part1.getBytes().length); + + // Check with service that upload is still in progress + UploadInfo info = tusFileUploadService.getUploadInfo(location, OWNER_KEY); + assertTrue(info.isUploadInProgress()); + assertThat(info.getLength(), is(part1.getBytes().length + 20L)); + assertThat(info.getOffset(), is(Long.valueOf(part1.getBytes().length))); + + // Now wait until the upload expired and run the cleanup + Utils.sleep(1000L); + tusFileUploadService.cleanup(); + + // Check with HEAD request that the upload is gone + // If a Client does attempt to resume an upload which has since been removed by the Server, + // the Server SHOULD respond with the404 Not Found or 410 Gone status. + reset(); + servletRequest.setMethod("HEAD"); + servletRequest.setRequestURI(location); + servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); + + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + assertResponseHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); + assertResponseHeader(HttpHeader.CONTENT_LENGTH, "0"); + assertResponseStatus(HttpServletResponse.SC_NOT_FOUND); + } + + @Test + public void testConcatenationCompleted() throws Exception { + String part1 = + "29\r\nThis is the first part of my test upload " + + "\r\n0\r\nUpload-Checksum: sha1 n5RQbRwM6UVAD+9iuHEmnN6HCGQ="; + String part2 = + "1C\r\nand this is the second part." + + "\r\n0\r\nUpload-Checksum: sha1 oNge323kGFKICxp+Me5xJgPvGEM="; + + // Create first upload + servletRequest.setMethod("POST"); + servletRequest.setRequestURI(UPLOAD_URI); + servletRequest.addHeader(HttpHeader.CONTENT_LENGTH, 0); + servletRequest.addHeader(HttpHeader.UPLOAD_LENGTH, "41"); + servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); + servletRequest.addHeader(HttpHeader.UPLOAD_CONCAT, "partial"); + servletRequest.addHeader( + HttpHeader.UPLOAD_METADATA, "filename d29ybGRfZG9taW5hdGlvbl9wbGFuLnBkZg=="); + + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + assertResponseHeaderNotBlank(HttpHeader.LOCATION); + assertResponseHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); + assertResponseHeader(HttpHeader.CONTENT_LENGTH, "0"); + assertResponseHeaderNotBlank(HttpHeader.UPLOAD_EXPIRES); + assertResponseStatus(HttpServletResponse.SC_CREATED); + + String location1 = + UPLOAD_URI + + StringUtils.substringAfter( + servletResponse.getHeader(HttpHeader.LOCATION), UPLOAD_URI); + + // Make sure cleanup does not interfere with this test + tusFileUploadService.cleanup(); + + // Upload part 1 bytes + reset(); + servletRequest.setMethod("PATCH"); + servletRequest.setRequestURI(location1); + servletRequest.addHeader(HttpHeader.CONTENT_TYPE, "application/offset+octet-stream"); + servletRequest.addHeader(HttpHeader.CONTENT_LENGTH, "41"); + servletRequest.addHeader(HttpHeader.UPLOAD_OFFSET, 0); + servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); + servletRequest.addHeader(HttpHeader.TRANSFER_ENCODING, "chunked"); + servletRequest.setContent(part1.getBytes()); + + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + assertResponseStatus(HttpServletResponse.SC_NO_CONTENT); + assertResponseHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); + assertResponseHeader(HttpHeader.CONTENT_LENGTH, "0"); + assertResponseHeaderNotBlank(HttpHeader.UPLOAD_EXPIRES); + assertResponseHeader(HttpHeader.UPLOAD_OFFSET, "41"); + + // Make sure cleanup does not interfere with this test + tusFileUploadService.cleanup(); + + // Create the second upload + reset(); + servletRequest.setMethod("POST"); + servletRequest.setRequestURI(UPLOAD_URI); + servletRequest.addHeader(HttpHeader.CONTENT_LENGTH, 0); + servletRequest.addHeader(HttpHeader.UPLOAD_LENGTH, "28"); + servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); + servletRequest.addHeader(HttpHeader.UPLOAD_CONCAT, "partial"); + servletRequest.addHeader( + HttpHeader.UPLOAD_METADATA, "filename d29ybGRfZG9taW5hdGlvbl9wbGFuLnBkZg=="); + + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + assertResponseHeaderNotBlank(HttpHeader.LOCATION); + assertResponseHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); + assertResponseHeader(HttpHeader.CONTENT_LENGTH, "0"); + assertResponseHeaderNotBlank(HttpHeader.UPLOAD_EXPIRES); + assertResponseStatus(HttpServletResponse.SC_CREATED); + + String location2 = + UPLOAD_URI + + StringUtils.substringAfter( + servletResponse.getHeader(HttpHeader.LOCATION), UPLOAD_URI); + + // Upload part 2 bytes + reset(); + servletRequest.setMethod("PATCH"); + servletRequest.setRequestURI(location2); + servletRequest.addHeader(HttpHeader.CONTENT_TYPE, "application/offset+octet-stream"); + servletRequest.addHeader(HttpHeader.CONTENT_LENGTH, "28"); + servletRequest.addHeader(HttpHeader.UPLOAD_OFFSET, "0"); + servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); + servletRequest.addHeader(HttpHeader.TRANSFER_ENCODING, "chunked"); + servletRequest.setContent(part2.getBytes()); + + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + assertResponseHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); + assertResponseHeader(HttpHeader.CONTENT_LENGTH, "0"); + assertResponseHeader(HttpHeader.UPLOAD_OFFSET, "28"); + assertResponseHeaderNotBlank(HttpHeader.UPLOAD_EXPIRES); + assertResponseStatus(HttpServletResponse.SC_NO_CONTENT); + + // Create the final concatenated upload + reset(); + servletRequest.setMethod("POST"); + servletRequest.setRequestURI(UPLOAD_URI); + servletRequest.addHeader(HttpHeader.CONTENT_LENGTH, 0); + servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); + servletRequest.addHeader(HttpHeader.UPLOAD_CONCAT, "final ; " + location1 + " " + location2); + servletRequest.addHeader( + HttpHeader.UPLOAD_METADATA, + "filename d29ybGRfZG9taW5hdGlvbl9tYXBfY29uY2F0ZW5hdGVkLnBkZg=="); + + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + assertResponseHeaderNotBlank(HttpHeader.LOCATION); + assertResponseHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); + assertResponseHeader(HttpHeader.CONTENT_LENGTH, "0"); + assertResponseHeaderNotBlank(HttpHeader.UPLOAD_EXPIRES); + assertResponseStatus(HttpServletResponse.SC_CREATED); + + String location = + UPLOAD_URI + + StringUtils.substringAfter( + servletResponse.getHeader(HttpHeader.LOCATION), UPLOAD_URI); + + // Check with HEAD request upload is complete + reset(); + servletRequest.setMethod("HEAD"); + servletRequest.setRequestURI(location); + servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); + + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + assertResponseHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); + assertResponseHeader(HttpHeader.CONTENT_LENGTH, "0"); + assertResponseHeader(HttpHeader.UPLOAD_OFFSET, "69"); + assertResponseHeader(HttpHeader.UPLOAD_LENGTH, "69"); + assertResponseHeader(HttpHeader.UPLOAD_CONCAT, "final ; " + location1 + " " + location2); + assertResponseHeaderNull(HttpHeader.UPLOAD_DEFER_LENGTH); + assertResponseHeader( + HttpHeader.UPLOAD_METADATA, + "filename d29ybGRfZG9taW5hdGlvbl9tYXBfY29uY2F0ZW5hdGVkLnBkZg=="); + assertResponseStatus(HttpServletResponse.SC_NO_CONTENT); + + // Get upload info from service + UploadInfo info = tusFileUploadService.getUploadInfo(location, OWNER_KEY); + assertFalse(info.isUploadInProgress()); + assertThat(info.getLength(), is(69L)); + assertThat(info.getOffset(), is(69L)); + assertThat(info.isUploadInProgress(), is(false)); + assertThat( + info.getMetadata(), + allOf(hasSize(1), hasEntry("filename", "world_domination_map_concatenated.pdf"))); + assertThat(info.getCreatorIpAddresses(), is("10.0.2.1, 123.231.12.4, 192.168.1.1")); + + // Download the upload + reset(); + servletRequest.setMethod("GET"); + servletRequest.setRequestURI(location); + + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + assertResponseStatus(HttpServletResponse.SC_OK); + assertResponseHeader(HttpHeader.CONTENT_LENGTH, "69"); + assertResponseHeader( + HttpHeader.UPLOAD_METADATA, + "filename d29ybGRfZG9taW5hdGlvbl9tYXBfY29uY2F0ZW5hdGVkLnBkZg=="); + assertThat( + servletResponse.getContentAsString(), + is("This is the first part of my test upload and this is the second part.")); + + // Get uploaded bytes from service + try (InputStream uploadedBytes = tusFileUploadService.getUploadedBytes(location, OWNER_KEY)) { + assertThat( + IOUtils.toString(uploadedBytes, StandardCharsets.UTF_8), + is("This is the first part of my test upload and this is the second part.")); + } + } + + @Test + public void testConcatenationUnfinished() throws Exception { + String part1 = "When sending this part, the final upload was already created. "; + String part2 = "This is the second part of our concatenated upload. "; + String part3 = "Finally when sending the third part, the final upload is complete."; + + // Create upload part 1 + servletRequest.setMethod("POST"); + servletRequest.setRequestURI(UPLOAD_URI); + servletRequest.addHeader(HttpHeader.CONTENT_LENGTH, 0); + servletRequest.addHeader(HttpHeader.UPLOAD_DEFER_LENGTH, 1); + servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); + servletRequest.addHeader(HttpHeader.UPLOAD_CONCAT, "partial"); + servletRequest.addHeader(HttpHeader.UPLOAD_METADATA, "filename cGFydDEucGRm"); + + tusFileUploadService.process(servletRequest, servletResponse); + assertResponseHeaderNotBlank(HttpHeader.LOCATION); + assertResponseHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); + assertResponseHeader(HttpHeader.CONTENT_LENGTH, "0"); + assertResponseHeaderNotBlank(HttpHeader.UPLOAD_EXPIRES); + assertResponseStatus(HttpServletResponse.SC_CREATED); + + String location1 = + UPLOAD_URI + + StringUtils.substringAfter( + servletResponse.getHeader(HttpHeader.LOCATION), UPLOAD_URI); + + reset(); + // Create upload part 2 + servletRequest.setMethod("POST"); + servletRequest.setRequestURI(UPLOAD_URI); + servletRequest.addHeader(HttpHeader.CONTENT_LENGTH, 0); + servletRequest.addHeader(HttpHeader.UPLOAD_DEFER_LENGTH, 1); + servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); + servletRequest.addHeader(HttpHeader.UPLOAD_CONCAT, "partial"); + servletRequest.addHeader(HttpHeader.UPLOAD_METADATA, "filename cGFydDIucGRm"); + + tusFileUploadService.process(servletRequest, servletResponse); + assertResponseHeaderNotBlank(HttpHeader.LOCATION); + assertResponseHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); + assertResponseHeader(HttpHeader.CONTENT_LENGTH, "0"); + assertResponseHeaderNotBlank(HttpHeader.UPLOAD_EXPIRES); + assertResponseStatus(HttpServletResponse.SC_CREATED); + + String location2 = + UPLOAD_URI + + StringUtils.substringAfter( + servletResponse.getHeader(HttpHeader.LOCATION), UPLOAD_URI); + + reset(); + // Create upload part 3 + servletRequest.setMethod("POST"); + servletRequest.setRequestURI(UPLOAD_URI); + servletRequest.addHeader(HttpHeader.CONTENT_LENGTH, 0); + servletRequest.addHeader(HttpHeader.UPLOAD_DEFER_LENGTH, 1); + servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); + servletRequest.addHeader(HttpHeader.UPLOAD_CONCAT, "partial"); + servletRequest.addHeader(HttpHeader.UPLOAD_METADATA, "filename cGFydDMucGRm"); + + tusFileUploadService.process(servletRequest, servletResponse); + assertResponseHeaderNotBlank(HttpHeader.LOCATION); + assertResponseHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); + assertResponseHeader(HttpHeader.CONTENT_LENGTH, "0"); + assertResponseHeaderNotBlank(HttpHeader.UPLOAD_EXPIRES); + assertResponseStatus(HttpServletResponse.SC_CREATED); + + String location3 = + UPLOAD_URI + + StringUtils.substringAfter( + servletResponse.getHeader(HttpHeader.LOCATION), UPLOAD_URI); + + // Upload part 2 bytes + reset(); + servletRequest.setMethod("PATCH"); + servletRequest.setRequestURI(location2); + servletRequest.addHeader(HttpHeader.CONTENT_TYPE, "application/offset+octet-stream"); + servletRequest.addHeader(HttpHeader.CONTENT_LENGTH, part2.getBytes().length); + servletRequest.addHeader(HttpHeader.UPLOAD_LENGTH, part2.getBytes().length); + servletRequest.addHeader(HttpHeader.UPLOAD_OFFSET, 0); + servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); + servletRequest.setContent(part2.getBytes()); + + tusFileUploadService.process(servletRequest, servletResponse); + assertResponseHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); + assertResponseHeader(HttpHeader.CONTENT_LENGTH, "0"); + assertResponseHeaderNotBlank(HttpHeader.UPLOAD_EXPIRES); + assertResponseHeader(HttpHeader.UPLOAD_OFFSET, "" + part2.getBytes().length); + assertResponseStatus(HttpServletResponse.SC_NO_CONTENT); + + reset(); + // Create final upload + servletRequest.setMethod("POST"); + servletRequest.setRequestURI(UPLOAD_URI); + servletRequest.addHeader(HttpHeader.CONTENT_LENGTH, 0); + servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); + servletRequest.addHeader( + HttpHeader.UPLOAD_CONCAT, "final;" + location1 + " " + location2 + " " + location3); + servletRequest.addHeader(HttpHeader.UPLOAD_METADATA, "filename ZmluYWwucGRm"); + + tusFileUploadService.process(servletRequest, servletResponse); + assertResponseHeaderNotBlank(HttpHeader.LOCATION); + assertResponseHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); + assertResponseHeader(HttpHeader.CONTENT_LENGTH, "0"); + assertResponseHeaderNotBlank(HttpHeader.UPLOAD_EXPIRES); + assertResponseStatus(HttpServletResponse.SC_CREATED); + + String locationFinal = + UPLOAD_URI + + StringUtils.substringAfter( + servletResponse.getHeader(HttpHeader.LOCATION), UPLOAD_URI); + + // Check with HEAD request that length of final upload is undefined + reset(); + servletRequest.setMethod("HEAD"); + servletRequest.setRequestURI(locationFinal); + servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); + + tusFileUploadService.process(servletRequest, servletResponse); + assertResponseHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); + assertResponseHeader(HttpHeader.CONTENT_LENGTH, "0"); + assertResponseHeaderNull(HttpHeader.UPLOAD_OFFSET); + assertResponseHeaderNull(HttpHeader.UPLOAD_LENGTH); + assertResponseHeader(HttpHeader.UPLOAD_METADATA, "filename ZmluYWwucGRm"); + assertResponseHeader( + HttpHeader.UPLOAD_CONCAT, "final;" + location1 + " " + location2 + " " + location3); + assertResponseHeaderNull(HttpHeader.UPLOAD_DEFER_LENGTH); + assertResponseStatus(HttpServletResponse.SC_NO_CONTENT); + + // Verify that we cannot download an unfinished final upload + reset(); + servletRequest.setMethod("GET"); + servletRequest.setRequestURI(locationFinal); + + tusFileUploadService.process(servletRequest, servletResponse); + assertResponseStatus(204); + assertResponseHeader(HttpHeader.CONTENT_LENGTH, "0"); + assertThat(servletResponse.getContentAsString(), is("")); + + // Upload part 1 bytes + reset(); + servletRequest.setMethod("PATCH"); + servletRequest.setRequestURI(location1); + servletRequest.addHeader(HttpHeader.CONTENT_TYPE, "application/offset+octet-stream"); + servletRequest.addHeader(HttpHeader.CONTENT_LENGTH, part1.getBytes().length); + servletRequest.addHeader(HttpHeader.UPLOAD_LENGTH, part1.getBytes().length); + servletRequest.addHeader(HttpHeader.UPLOAD_OFFSET, 0); + servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); + servletRequest.setContent(part1.getBytes()); + + tusFileUploadService.process(servletRequest, servletResponse); + assertResponseHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); + assertResponseHeader(HttpHeader.CONTENT_LENGTH, "0"); + assertResponseHeaderNotBlank(HttpHeader.UPLOAD_EXPIRES); + assertResponseHeader(HttpHeader.UPLOAD_OFFSET, "" + part1.getBytes().length); + assertResponseStatus(HttpServletResponse.SC_NO_CONTENT); + + // Upload part 3 bytes + reset(); + servletRequest.setMethod("PATCH"); + servletRequest.setRequestURI(location3); + servletRequest.addHeader(HttpHeader.CONTENT_TYPE, "application/offset+octet-stream"); + servletRequest.addHeader(HttpHeader.CONTENT_LENGTH, part3.getBytes().length); + servletRequest.addHeader(HttpHeader.UPLOAD_LENGTH, part3.getBytes().length); + servletRequest.addHeader(HttpHeader.UPLOAD_OFFSET, 0); + servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); + servletRequest.setContent(part3.getBytes()); + + tusFileUploadService.process(servletRequest, servletResponse); + assertResponseHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); + assertResponseHeader(HttpHeader.CONTENT_LENGTH, "0"); + assertResponseHeaderNotBlank(HttpHeader.UPLOAD_EXPIRES); + assertResponseHeader(HttpHeader.UPLOAD_OFFSET, "" + part3.getBytes().length); + assertResponseStatus(HttpServletResponse.SC_NO_CONTENT); + + // Check with HEAD request length of final upload is known + reset(); + servletRequest.setMethod("HEAD"); + servletRequest.setRequestURI(locationFinal); + servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); + + tusFileUploadService.process(servletRequest, servletResponse); + assertResponseHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); + assertResponseHeader(HttpHeader.CONTENT_LENGTH, "0"); + assertResponseHeader(HttpHeader.UPLOAD_OFFSET, "" + (part1 + part2 + part3).getBytes().length); + assertResponseHeader(HttpHeader.UPLOAD_LENGTH, "" + (part1 + part2 + part3).getBytes().length); + assertResponseHeader(HttpHeader.UPLOAD_METADATA, "filename ZmluYWwucGRm"); + assertResponseHeader( + HttpHeader.UPLOAD_CONCAT, "final;" + location1 + " " + location2 + " " + location3); + assertResponseHeaderNull(HttpHeader.UPLOAD_DEFER_LENGTH); + assertResponseStatus(HttpServletResponse.SC_NO_CONTENT); + + // Download the upload + reset(); + servletRequest.setMethod("GET"); + servletRequest.setRequestURI(locationFinal); + + tusFileUploadService.process(servletRequest, servletResponse, null); + assertResponseStatus(HttpServletResponse.SC_OK); + assertResponseHeader(HttpHeader.CONTENT_LENGTH, "" + (part1 + part2 + part3).getBytes().length); + assertResponseHeader(HttpHeader.UPLOAD_METADATA, "filename ZmluYWwucGRm"); + assertThat( + servletResponse.getContentAsString(), + is( + "When sending this part, the final upload was already created. " + + "This is the second part of our concatenated upload. " + + "Finally when sending the third part, the final upload is complete.")); + + // Get uploaded bytes from service + try (InputStream uploadedBytes = tusFileUploadService.getUploadedBytes(locationFinal, null)) { + assertThat( + IOUtils.toString(uploadedBytes, StandardCharsets.UTF_8), + is( + "When sending this part, the final upload was already created. " + + "This is the second part of our concatenated upload. " + + "Finally when sending the third part, the final upload is complete.")); + } + } + + @Test + public void testChunkedDecodingDisabledAndRegexUploadUri() throws Exception { + String chunkedContent = + "1B;test=value\r\nThis upload looks chunked, \r\n" + "D\r\nbut it's not!\r\n" + "\r\n0\r\n"; + + // Create service without chunked decoding and without expiration + tusFileUploadService = + createTusFileUploadService("/users/[0-9]+/files/upload") + .withChunkedTransferDecoding(false) + .withUploadExpirationPeriod(0L); + + // Create upload + servletRequest.setMethod("POST"); + servletRequest.setRequestURI("/users/98765/files/upload"); + servletRequest.addHeader(HttpHeader.CONTENT_LENGTH, 0); + servletRequest.addHeader(HttpHeader.UPLOAD_LENGTH, "67"); + servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); + servletRequest.addHeader( + HttpHeader.UPLOAD_METADATA, "filename d29ybGRfZG9taW5hdGlvbl9wbGFuLnBkZg=="); + + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + assertResponseHeaderNotBlank(HttpHeader.LOCATION); + assertResponseHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); + assertResponseHeader(HttpHeader.CONTENT_LENGTH, "0"); + assertResponseHeaderNull(HttpHeader.UPLOAD_EXPIRES); + assertResponseStatus(HttpServletResponse.SC_CREATED); + + String location = servletResponse.getHeader(HttpHeader.LOCATION); + + // Upload content + reset(); + servletRequest.setMethod("PATCH"); + servletRequest.setRequestURI(location); + servletRequest.addHeader(HttpHeader.CONTENT_TYPE, "application/offset+octet-stream"); + servletRequest.addHeader(HttpHeader.CONTENT_LENGTH, "67"); + servletRequest.addHeader(HttpHeader.UPLOAD_OFFSET, 0); + servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); + servletRequest.addHeader(HttpHeader.TRANSFER_ENCODING, "chunked"); + servletRequest.setContent(chunkedContent.getBytes()); + + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + assertResponseStatus(HttpServletResponse.SC_NO_CONTENT); + assertResponseHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); + assertResponseHeader(HttpHeader.CONTENT_LENGTH, "0"); + assertResponseHeaderNull(HttpHeader.UPLOAD_EXPIRES); + assertResponseHeader(HttpHeader.UPLOAD_OFFSET, "67"); + + // Check with HEAD request upload is complete + reset(); + servletRequest.setMethod("HEAD"); + servletRequest.setRequestURI(location); + servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); + + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + assertResponseHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); + assertResponseHeader(HttpHeader.CONTENT_LENGTH, "0"); + assertResponseHeader(HttpHeader.UPLOAD_OFFSET, "67"); + assertResponseHeader(HttpHeader.UPLOAD_LENGTH, "67"); + assertResponseHeaderNull(HttpHeader.UPLOAD_DEFER_LENGTH); + assertResponseHeader( + HttpHeader.UPLOAD_METADATA, "filename d29ybGRfZG9taW5hdGlvbl9wbGFuLnBkZg=="); + assertResponseStatus(HttpServletResponse.SC_NO_CONTENT); + + // Get upload info from service + UploadInfo info = tusFileUploadService.getUploadInfo(location, OWNER_KEY); + assertFalse(info.isUploadInProgress()); + assertThat(info.getLength(), is(67L)); + assertThat(info.getOffset(), is(67L)); + assertThat( + info.getMetadata(), allOf(hasSize(1), hasEntry("filename", "world_domination_plan.pdf"))); + + // Get uploaded bytes from service + try (InputStream uploadedBytes = tusFileUploadService.getUploadedBytes(location, OWNER_KEY)) { + assertThat( + IOUtils.toString(uploadedBytes, StandardCharsets.UTF_8), + is( + "1B;test=value\r\nThis upload looks chunked, \r\n" + + "D\r\nbut it's not!\r\n" + + "\r\n0\r\n")); + } + } + + @Test + public void testOptions() throws Exception { + // Do options request and check response headers + servletRequest.setMethod("OPTIONS"); + + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + + assertResponseStatus(HttpServletResponse.SC_NO_CONTENT); + assertResponseHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); + assertResponseHeader(HttpHeader.CONTENT_LENGTH, "0"); + assertResponseHeader(HttpHeader.TUS_VERSION, "1.0.0"); + assertResponseHeader(HttpHeader.TUS_MAX_SIZE, "1073741824"); + assertResponseHeader( + HttpHeader.TUS_CHECKSUM_ALGORITHM, "md5", "sha1", "sha256", "sha384", "sha512"); + assertResponseHeader( + HttpHeader.TUS_EXTENSION, + "creation", + "creation-defer-length", + "creation-with-upload", + "checksum", + "checksum-trailer", + "termination", + "download", + "expiration", + "concatenation", + "concatenation-unfinished"); + } + + @Test + public void testHeadOnNonExistingUpload() throws Exception { + servletRequest.setMethod("HEAD"); + servletRequest.setRequestURI(UPLOAD_URI + "/" + UUID.randomUUID()); + servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); + + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + assertResponseStatus(HttpServletResponse.SC_NOT_FOUND); + assertResponseHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); + assertResponseHeader(HttpHeader.CONTENT_LENGTH, "0"); + } + + @Test + public void testInvalidTusResumable() throws Exception { + servletRequest.setMethod("POST"); + servletRequest.setRequestURI(UPLOAD_URI); + servletRequest.addHeader(HttpHeader.CONTENT_LENGTH, 0); + servletRequest.addHeader(HttpHeader.UPLOAD_DEFER_LENGTH, 1); + servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "2.0.0"); + + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + assertResponseStatus(HttpServletResponse.SC_PRECONDITION_FAILED); + assertResponseHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); + assertResponseHeader(HttpHeader.CONTENT_LENGTH, "0"); + } + + @Test + public void testMaxUploadLengthExceeded() throws Exception { + tusFileUploadService.withMaxUploadSize(10L); + + String uploadContent = "This is upload is too long"; + + // Create upload + servletRequest.setMethod("POST"); + servletRequest.setRequestURI(UPLOAD_URI); + servletRequest.addHeader(HttpHeader.CONTENT_LENGTH, 0); + servletRequest.addHeader(HttpHeader.UPLOAD_LENGTH, uploadContent.getBytes().length); + servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); + + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + assertResponseStatus(HttpServletResponse.SC_REQUEST_ENTITY_TOO_LARGE); + assertResponseHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); + assertResponseHeader(HttpHeader.CONTENT_LENGTH, "0"); + } + + @Test + public void testInvalidMethods() throws Exception { + servletRequest.setMethod("PUT"); + servletRequest.setRequestURI(UPLOAD_URI + "/" + UUID.randomUUID()); + servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); + + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + assertResponseStatus(HttpServletResponse.SC_METHOD_NOT_ALLOWED); + + reset(); + servletRequest.setMethod("CONNECT"); + servletRequest.setRequestURI(UPLOAD_URI + "/" + UUID.randomUUID()); + servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); + + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + assertResponseStatus(HttpServletResponse.SC_METHOD_NOT_ALLOWED); + assertResponseHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); + assertResponseHeader(HttpHeader.CONTENT_LENGTH, "0"); + + reset(); + servletRequest.setMethod("TRACE"); + servletRequest.setRequestURI(UPLOAD_URI + "/" + UUID.randomUUID()); + servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); + + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + assertResponseStatus(HttpServletResponse.SC_METHOD_NOT_ALLOWED); + assertResponseHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); + assertResponseHeader(HttpHeader.CONTENT_LENGTH, "0"); + } + + @Test + public void testLockContentionAndHeadRelease() throws Exception { + // 1. Create upload resource + servletRequest.setMethod("POST"); + servletRequest.setRequestURI(UPLOAD_URI); + servletRequest.addHeader(HttpHeader.CONTENT_LENGTH, 0); + servletRequest.addHeader(HttpHeader.UPLOAD_LENGTH, 1000L); + servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + String location = + UPLOAD_URI + + StringUtils.substringAfter( + servletResponse.getHeader(HttpHeader.LOCATION), UPLOAD_URI); + + // 2. Start a blocking PATCH in a background thread to hold the lock + final java.util.concurrent.CountDownLatch requestStarted = + new java.util.concurrent.CountDownLatch(1); + final java.util.concurrent.atomic.AtomicReference bgException = + new java.util.concurrent.atomic.AtomicReference<>(); + + InputStream blockingStream = + new InputStream() { + private volatile boolean closed = false; + + @Override + public int read() throws IOException { + requestStarted.countDown(); + synchronized (this) { + while (!closed) { + try { + this.wait(100); + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + throw new IOException("Interrupted", e); + } + } + } + throw new IOException("Stream closed"); + } + + @Override + public void close() throws IOException { + synchronized (this) { + closed = true; + this.notifyAll(); + } + } + }; + + final MockHttpServletRequest bgRequest = + new MockHttpServletRequest() { + @Override + public jakarta.servlet.ServletInputStream getInputStream() { + return new jakarta.servlet.ServletInputStream() { + @Override + public int read() throws IOException { + return blockingStream.read(); + } + + @Override + public void close() throws IOException { + blockingStream.close(); + } + + @Override + public boolean isFinished() { + return false; + } + + @Override + public boolean isReady() { + return true; + } + + @Override + public void setReadListener(jakarta.servlet.ReadListener readListener) {} + }; + } + }; + bgRequest.setMethod("PATCH"); + bgRequest.setRequestURI(location); + bgRequest.addHeader(HttpHeader.CONTENT_TYPE, "application/offset+octet-stream"); + bgRequest.addHeader(HttpHeader.CONTENT_LENGTH, 100); + bgRequest.addHeader(HttpHeader.UPLOAD_OFFSET, 0); + bgRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); + + Thread bgThread = + new Thread( + new Runnable() { + @Override + public void run() { + try { + MockHttpServletResponse bgResponse = new MockHttpServletResponse(); + tusFileUploadService.process(bgRequest, bgResponse, OWNER_KEY); + } catch (Exception e) { + e.printStackTrace(); + bgException.set(e); + } + } + }); + bgThread.start(); + + // Wait for the background thread to start reading (meaning it holds the lock) + requestStarted.await(2, java.util.concurrent.TimeUnit.SECONDS); + + // 3. Concurrent PATCH request must fail immediately with 423 + MockHttpServletRequest patchRequest = new MockHttpServletRequest(); + MockHttpServletResponse patchResponse = new MockHttpServletResponse(); + patchRequest.setMethod("PATCH"); + patchRequest.setRequestURI(location); + patchRequest.addHeader(HttpHeader.CONTENT_TYPE, "application/offset+octet-stream"); + patchRequest.addHeader(HttpHeader.CONTENT_LENGTH, 10); + patchRequest.addHeader(HttpHeader.UPLOAD_OFFSET, 0); + patchRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); + patchRequest.setContent(new byte[10]); + + tusFileUploadService.process(patchRequest, patchResponse, OWNER_KEY); + assertThat(patchResponse.getStatus(), is(423)); + + // 4. Concurrent HEAD request must interrupt background thread and succeed + MockHttpServletRequest headRequest = new MockHttpServletRequest(); + MockHttpServletResponse headResponse = new MockHttpServletResponse(); + headRequest.setMethod("HEAD"); + headRequest.setRequestURI(location); + headRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); + + tusFileUploadService.process(headRequest, headResponse, OWNER_KEY); + assertThat(headResponse.getStatus(), is(204)); + assertThat(headResponse.getHeader(HttpHeader.UPLOAD_OFFSET), is("0")); + + // Clean up + bgThread.join(2000); + } + + @Test + public void testCreationWithUploadOptions() throws Exception { + reset(); + servletRequest.setMethod("OPTIONS"); + servletRequest.setRequestURI(UPLOAD_URI); + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + assertResponseHeader( + HttpHeader.TUS_EXTENSION, + "creation", + "creation-defer-length", + "creation-with-upload", + "checksum", + "checksum-trailer", + "termination", + "download", + "expiration", + "concatenation", + "concatenation-unfinished"); + } + + @Test + public void testCreationWithUploadSuccess() throws Exception { + String uploadContent = "Initial data to upload"; + byte[] contentBytes = uploadContent.getBytes(StandardCharsets.UTF_8); + + reset(); + servletRequest.setMethod("POST"); + servletRequest.setRequestURI(UPLOAD_URI); + servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); + servletRequest.addHeader(HttpHeader.UPLOAD_LENGTH, contentBytes.length); + servletRequest.addHeader(HttpHeader.CONTENT_LENGTH, contentBytes.length); + servletRequest.addHeader(HttpHeader.CONTENT_TYPE, "application/offset+octet-stream"); + servletRequest.setContent(contentBytes); + + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + assertResponseStatus(HttpServletResponse.SC_CREATED); + assertResponseHeaderNotBlank(HttpHeader.LOCATION); + assertResponseHeader(HttpHeader.UPLOAD_OFFSET, String.valueOf(contentBytes.length)); + + String location = servletResponse.getHeader(HttpHeader.LOCATION); + + // Verify content in storage + try (InputStream is = tusFileUploadService.getUploadedBytes(location, OWNER_KEY)) { + String readContent = IOUtils.toString(is, StandardCharsets.UTF_8); + assertThat(readContent, is(uploadContent)); + } + } + + @Test + public void testCreationWithUploadDeferredLengthSuccess() throws Exception { + String uploadContent = "Initial data to upload with deferred length"; + byte[] contentBytes = uploadContent.getBytes(StandardCharsets.UTF_8); + + reset(); + servletRequest.setMethod("POST"); + servletRequest.setRequestURI(UPLOAD_URI); + servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); + servletRequest.addHeader(HttpHeader.UPLOAD_DEFER_LENGTH, "1"); + servletRequest.addHeader(HttpHeader.CONTENT_LENGTH, contentBytes.length); + servletRequest.addHeader(HttpHeader.CONTENT_TYPE, "application/offset+octet-stream"); + servletRequest.setContent(contentBytes); + + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + assertResponseStatus(HttpServletResponse.SC_CREATED); + assertResponseHeaderNotBlank(HttpHeader.LOCATION); + assertResponseHeader(HttpHeader.UPLOAD_OFFSET, String.valueOf(contentBytes.length)); + + String location = + UPLOAD_URI + + StringUtils.substringAfter( + servletResponse.getHeader(HttpHeader.LOCATION), UPLOAD_URI); + + // Verify content in storage + try (InputStream is = tusFileUploadService.getUploadedBytes(location, OWNER_KEY)) { + String readContent = IOUtils.toString(is, StandardCharsets.UTF_8); + assertThat(readContent, is(uploadContent)); + } + } + + @Test + public void testCreationWithUploadInvalidContentType() throws Exception { + String uploadContent = "Initial data to upload"; + byte[] contentBytes = uploadContent.getBytes(StandardCharsets.UTF_8); + + reset(); + servletRequest.setMethod("POST"); + servletRequest.setRequestURI(UPLOAD_URI); + servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); + servletRequest.addHeader(HttpHeader.UPLOAD_LENGTH, contentBytes.length); + servletRequest.addHeader(HttpHeader.CONTENT_LENGTH, contentBytes.length); + servletRequest.addHeader(HttpHeader.CONTENT_TYPE, "application/octet-stream"); + servletRequest.setContent(contentBytes); + + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + assertResponseStatus(HttpServletResponse.SC_NOT_ACCEPTABLE); + } + + @Test + public void testCreationWithUploadExceedsLength() throws Exception { + String uploadContent = "Initial data to upload"; + byte[] contentBytes = uploadContent.getBytes(StandardCharsets.UTF_8); + + reset(); + servletRequest.setMethod("POST"); + servletRequest.setRequestURI(UPLOAD_URI); + servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); + servletRequest.addHeader(HttpHeader.UPLOAD_LENGTH, contentBytes.length - 5); + servletRequest.addHeader(HttpHeader.CONTENT_LENGTH, contentBytes.length); + servletRequest.addHeader(HttpHeader.CONTENT_TYPE, "application/offset+octet-stream"); + servletRequest.setContent(contentBytes); + + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + assertResponseStatus(HttpServletResponse.SC_BAD_REQUEST); + } + + @Test + public void testCreationWithUploadDisabled() throws Exception { + String uploadContent = "Initial data to upload"; + byte[] contentBytes = uploadContent.getBytes(StandardCharsets.UTF_8); + + tusFileUploadService.disableTusExtension("creation-with-upload"); + try { + reset(); + servletRequest.setMethod("POST"); + servletRequest.setRequestURI(UPLOAD_URI); + servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); + servletRequest.addHeader(HttpHeader.UPLOAD_LENGTH, contentBytes.length); + servletRequest.addHeader(HttpHeader.CONTENT_LENGTH, contentBytes.length); + servletRequest.addHeader(HttpHeader.CONTENT_TYPE, "application/offset+octet-stream"); + servletRequest.setContent(contentBytes); + + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + assertResponseStatus(HttpServletResponse.SC_BAD_REQUEST); + } finally { + // Restore for other tests + tusFileUploadService = createTusFileUploadService(); + } + } + + @Test + public void testCorsHeaders() throws Exception { + reset(); + servletRequest.setMethod("POST"); + servletRequest.setRequestURI(UPLOAD_URI); + servletRequest.addHeader("Origin", "https://example.com"); + servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); + servletRequest.addHeader(HttpHeader.UPLOAD_LENGTH, 100L); + + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + assertResponseHeader("Access-Control-Allow-Origin", "https://example.com"); + assertResponseHeaderNotBlank("Access-Control-Expose-Headers"); + } + + @Test + public void testCorsPreflight() throws Exception { + reset(); + servletRequest.setMethod("OPTIONS"); + servletRequest.setRequestURI(UPLOAD_URI); + servletRequest.addHeader("Origin", "https://example.com"); + servletRequest.addHeader("Access-Control-Request-Method", "PATCH"); + + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + assertResponseHeader("Access-Control-Allow-Origin", "https://example.com"); + assertResponseHeader("Access-Control-Allow-Methods", "POST, GET, HEAD, PATCH, DELETE, OPTIONS"); + assertResponseHeaderNotBlank("Access-Control-Allow-Headers"); + assertResponseHeader("Access-Control-Max-Age", "86400"); + } + + @Test + public void testTusVersionHeaderOn412() throws Exception { + reset(); + servletRequest.setMethod("POST"); + servletRequest.setRequestURI(UPLOAD_URI); + servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "2.0.0"); + + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + assertResponseStatus(HttpServletResponse.SC_PRECONDITION_FAILED); + assertResponseHeader(HttpHeader.TUS_VERSION, "1.0.0"); + } + + @Test + public void testModifyUploadLengthOnPatch() throws Exception { + // 1. Create upload with deferred length + reset(); + servletRequest.setMethod("POST"); + servletRequest.setRequestURI(UPLOAD_URI); + servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); + servletRequest.addHeader(HttpHeader.UPLOAD_DEFER_LENGTH, "1"); + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + assertResponseStatus(HttpServletResponse.SC_CREATED); + String location = servletResponse.getHeader(HttpHeader.LOCATION); + + // 2. Set length to 100 on PATCH + reset(); + servletRequest.setMethod("PATCH"); + servletRequest.setRequestURI(location); + servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); + servletRequest.addHeader(HttpHeader.UPLOAD_OFFSET, "0"); + servletRequest.addHeader(HttpHeader.UPLOAD_LENGTH, "100"); + servletRequest.addHeader(HttpHeader.CONTENT_TYPE, "application/offset+octet-stream"); + servletRequest.setContent("test content".getBytes(StandardCharsets.UTF_8)); + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + assertResponseStatus(HttpServletResponse.SC_NO_CONTENT); + + // 3. Try to change length to 200 on subsequent PATCH -> should return 400 + reset(); + servletRequest.setMethod("PATCH"); + servletRequest.setRequestURI(location); + servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); + servletRequest.addHeader(HttpHeader.UPLOAD_OFFSET, "12"); + servletRequest.addHeader(HttpHeader.UPLOAD_LENGTH, "200"); + servletRequest.addHeader(HttpHeader.CONTENT_TYPE, "application/offset+octet-stream"); + servletRequest.setContent("more content".getBytes(StandardCharsets.UTF_8)); + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + assertResponseStatus(HttpServletResponse.SC_BAD_REQUEST); + } + + @Test + public void testCreationWithUploadChecksumSuccess() throws Exception { + String uploadContent = "Initial data to upload with checksum"; + byte[] contentBytes = uploadContent.getBytes(StandardCharsets.UTF_8); + // Base64 hash for MD5 of "Initial data to upload with checksum" + // Base64 of MD5 bytes: aAMsB0BZbWXCBuPWG/ADyA== + String base64Checksum = "aAMsB0BZbWXCBuPWG/ADyA=="; + + reset(); + servletRequest.setMethod("POST"); + servletRequest.setRequestURI(UPLOAD_URI); + servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); + servletRequest.addHeader(HttpHeader.UPLOAD_LENGTH, contentBytes.length); + servletRequest.addHeader(HttpHeader.CONTENT_LENGTH, contentBytes.length); + servletRequest.addHeader(HttpHeader.CONTENT_TYPE, "application/offset+octet-stream"); + servletRequest.addHeader(HttpHeader.UPLOAD_CHECKSUM, "md5 " + base64Checksum); + servletRequest.setContent(contentBytes); + + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + assertResponseStatus(HttpServletResponse.SC_CREATED); + assertResponseHeader(HttpHeader.UPLOAD_OFFSET, String.valueOf(contentBytes.length)); + } + + @Test + public void testCreationWithUploadChecksumMismatch() throws Exception { + String uploadContent = "Initial data to upload with checksum"; + byte[] contentBytes = uploadContent.getBytes(StandardCharsets.UTF_8); + String invalidBase64Checksum = "aAMsB0BZbWXCBuPWG/ADyB=="; // changed A to B + + reset(); + servletRequest.setMethod("POST"); + servletRequest.setRequestURI(UPLOAD_URI); + servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); + servletRequest.addHeader(HttpHeader.UPLOAD_LENGTH, contentBytes.length); + servletRequest.addHeader(HttpHeader.CONTENT_LENGTH, contentBytes.length); + servletRequest.addHeader(HttpHeader.CONTENT_TYPE, "application/offset+octet-stream"); + servletRequest.addHeader(HttpHeader.UPLOAD_CHECKSUM, "md5 " + invalidBase64Checksum); + servletRequest.setContent(contentBytes); + + tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + assertResponseStatus(460); // Checksum mismatch + } + + protected void assertResponseHeader(final String header, final String value) { + assertThat(servletResponse.getHeader(header), is(value)); + } + + protected void assertResponseHeader(final String header, final String... values) { + assertThat( + Arrays.asList(servletResponse.getHeader(header).split(",")), containsInAnyOrder(values)); + } + + protected void assertResponseHeaderNotBlank(final String header) { + assertTrue(StringUtils.isNotBlank(servletResponse.getHeader(header))); + } + + protected void assertResponseHeaderNull(final String header) { + assertNull(servletResponse.getHeader(header)); + } + + protected void assertResponseStatus(final int httpStatus) { + assertThat(servletResponse.getStatus(), is(httpStatus)); + } +} diff --git a/src/test/java/me/desair/tus/server/ITRufhProtocol.java b/src/test/java/me/desair/tus/server/ITRufhProtocol.java new file mode 100644 index 00000000..0a675318 --- /dev/null +++ b/src/test/java/me/desair/tus/server/ITRufhProtocol.java @@ -0,0 +1,38 @@ +package me.desair.tus.server; + +import java.io.IOException; +import java.nio.file.Files; +import java.nio.file.Path; +import java.nio.file.Paths; +import org.apache.commons.io.FileUtils; +import org.junit.AfterClass; +import org.junit.BeforeClass; + +/** + * Disk-backed integration tests for the RUFH (Resumable Uploads for HTTP) protocol implementation. + */ +public class ITRufhProtocol extends AbstractITRufhProtocol { + + protected static Path storagePath; + + @BeforeClass + public static void setupDataFolder() throws IOException { + storagePath = Paths.get("target", "rufh", "data").toAbsolutePath(); + Files.createDirectories(storagePath); + } + + @AfterClass + public static void destroyDataFolder() throws IOException { + FileUtils.deleteDirectory(storagePath.toFile()); + } + + @Override + protected TusFileUploadService createTusFileUploadService() { + return new TusFileUploadService() + .withUploadUri(UPLOAD_URI) + .withStoragePath(storagePath.toAbsolutePath().toString()) + .withMaxUploadSize(1073741824L) + .withUploadExpirationPeriod(2L * 24 * 60 * 60 * 1000) + .withDownloadFeature(); + } +} diff --git a/src/test/java/me/desair/tus/server/ITTusFileUploadService.java b/src/test/java/me/desair/tus/server/ITTusFileUploadService.java index 53f6025d..f39f6112 100644 --- a/src/test/java/me/desair/tus/server/ITTusFileUploadService.java +++ b/src/test/java/me/desair/tus/server/ITTusFileUploadService.java @@ -1,1492 +1,74 @@ package me.desair.tus.server; -import static me.desair.tus.server.util.MapMatcher.hasSize; -import static org.hamcrest.CoreMatchers.allOf; import static org.hamcrest.CoreMatchers.is; -import static org.hamcrest.CoreMatchers.nullValue; import static org.hamcrest.MatcherAssert.assertThat; -import static org.hamcrest.Matchers.containsInAnyOrder; -import static org.hamcrest.Matchers.greaterThan; -import static org.hamcrest.collection.IsMapContaining.hasEntry; import static org.junit.Assert.assertFalse; -import static org.junit.Assert.assertNull; import static org.junit.Assert.assertTrue; -import static org.junit.Assert.fail; import jakarta.servlet.http.HttpServletResponse; import java.io.IOException; -import java.io.InputStream; -import java.nio.charset.StandardCharsets; import java.nio.file.Files; import java.nio.file.Path; import java.nio.file.Paths; -import java.text.DateFormat; -import java.text.SimpleDateFormat; -import java.util.Arrays; -import java.util.Locale; -import java.util.UUID; -import me.desair.tus.server.exception.TusException; import me.desair.tus.server.upload.UploadInfo; -import me.desair.tus.server.util.Utils; import org.apache.commons.io.FileUtils; -import org.apache.commons.io.IOUtils; import org.apache.commons.lang3.StringUtils; import org.junit.AfterClass; -import org.junit.Before; import org.junit.BeforeClass; import org.junit.Test; -import org.springframework.mock.web.MockHttpServletRequest; -import org.springframework.mock.web.MockHttpServletResponse; -/** Test cases for the {@link TusFileUploadService}. */ -public class ITTusFileUploadService { +/** + * Disk-backed integration test suite for {@link TusFileUploadService}. Extends {@link + * AbstractITTusFileUploadService} to run all protocol test use cases against disk storage. + */ +public class ITTusFileUploadService extends AbstractITTusFileUploadService { - protected static final String UPLOAD_URI = "/test/upload"; - protected static final String OWNER_KEY = "JOHN_DOE"; - - private static final DateFormat mockDateFormat = - new SimpleDateFormat("EEE, dd MMM yyyy HH:mm:ss zzz", Locale.US); - - protected MockHttpServletRequest servletRequest; - protected MockHttpServletResponse servletResponse; - - protected TusFileUploadService tusFileUploadService; - - protected static Path storagePath; - - @BeforeClass - public static void setupDataFolder() throws IOException { - storagePath = Paths.get("target", "tus", "data").toAbsolutePath(); - Files.createDirectories(storagePath); - } - - @AfterClass - public static void destroyDataFolder() throws IOException { - FileUtils.deleteDirectory(storagePath.toFile()); - } - - @Before - public void setUp() { - reset(); - tusFileUploadService = - new TusFileUploadService() - .withUploadUri(UPLOAD_URI) - .withStoragePath(storagePath.toAbsolutePath().toString()) - .withMaxUploadSize(1073741824L) - .withUploadExpirationPeriod(2L * 24 * 60 * 60 * 1000) - .withDownloadFeature() - .withChunkedTransferDecoding(true); - } - - protected void reset() { - servletRequest = new MockHttpServletRequest(); - servletRequest.setRemoteAddr("192.168.1.1"); - servletRequest.addHeader(HttpHeader.X_FORWARDED_FOR, "10.0.2.1, 123.231.12.4"); - servletResponse = new MockHttpServletResponse(); - } - - @Test - public void testSupportedHttpMethods() { - assertThat( - tusFileUploadService.getSupportedHttpMethods(), - containsInAnyOrder( - HttpMethod.HEAD, - HttpMethod.OPTIONS, - HttpMethod.PATCH, - HttpMethod.POST, - HttpMethod.PUT, - HttpMethod.DELETE, - HttpMethod.GET)); - - assertThat( - tusFileUploadService.getEnabledFeatures(), - containsInAnyOrder( - "core", - "creation", - "creation-with-upload", - "checksum", - "termination", - "download", - "expiration", - "concatenation", - "cors", - "resumable-uploads-for-http", - "http-digests")); - } - - @Test - public void testDisableFeature() throws Exception { - tusFileUploadService.disableTusExtension("download"); - tusFileUploadService.disableTusExtension("termination"); - tusFileUploadService.disableTusExtension("resumable-uploads-for-http"); - tusFileUploadService.disableTusExtension("http-digests"); - - assertThat( - tusFileUploadService.getSupportedHttpMethods(), - containsInAnyOrder(HttpMethod.HEAD, HttpMethod.OPTIONS, HttpMethod.PATCH, HttpMethod.POST)); - - assertThat( - tusFileUploadService.getEnabledFeatures(), - containsInAnyOrder( - "core", - "creation", - "creation-with-upload", - "checksum", - "expiration", - "concatenation", - "cors")); - - reset(); - servletRequest.setMethod("GET"); - servletRequest.setRequestURI(UPLOAD_URI + "/" + UUID.randomUUID()); - servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); - - tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); - assertResponseHeader(HttpHeader.CONTENT_LENGTH, "0"); - assertResponseStatus(HttpServletResponse.SC_METHOD_NOT_ALLOWED); - - reset(); - servletRequest.setMethod("DELETE"); - servletRequest.setRequestURI(UPLOAD_URI + "/" + UUID.randomUUID()); - servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); - - tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); - assertResponseHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); - assertResponseHeader(HttpHeader.CONTENT_LENGTH, "0"); - assertResponseStatus(HttpServletResponse.SC_METHOD_NOT_ALLOWED); - } - - @Test(expected = IllegalArgumentException.class) - public void testDisableCore() { - tusFileUploadService.disableTusExtension("core"); - } - - @Test(expected = NullPointerException.class) - public void testWithFileStoreServiceNull() throws Exception { - tusFileUploadService.withUploadStorageService(null); - } - - @Test - public void testProcessCompleteUpload() throws Exception { - String uploadContent = "This is my test upload content"; - - // Create upload - servletRequest.setMethod("POST"); - servletRequest.setRequestURI(UPLOAD_URI); - servletRequest.addHeader(HttpHeader.CONTENT_LENGTH, 0); - servletRequest.addHeader(HttpHeader.UPLOAD_LENGTH, uploadContent.getBytes().length); - servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); - servletRequest.addHeader( - HttpHeader.UPLOAD_METADATA, "filename d29ybGRfZG9taW5hdGlvbl9wbGFuLnBkZg=="); - - tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); - assertResponseHeaderNotBlank(HttpHeader.LOCATION); - assertResponseHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); - assertResponseHeader(HttpHeader.CONTENT_LENGTH, "0"); - assertResponseHeaderNotBlank(HttpHeader.UPLOAD_EXPIRES); - assertResponseStatus(HttpServletResponse.SC_CREATED); - - String location = - UPLOAD_URI - + StringUtils.substringAfter( - servletResponse.getHeader(HttpHeader.LOCATION), UPLOAD_URI); - - // Upload bytes - reset(); - servletRequest.setMethod("PATCH"); - servletRequest.setRequestURI(location); - servletRequest.addHeader(HttpHeader.CONTENT_TYPE, "application/offset+octet-stream"); - servletRequest.addHeader(HttpHeader.CONTENT_LENGTH, uploadContent.getBytes().length); - servletRequest.addHeader(HttpHeader.UPLOAD_OFFSET, 0); - servletRequest.addHeader(HttpHeader.UPLOAD_CHECKSUM, "sha1 Mfhm5HaSPUf+pUakdMxARo4rvfQ="); - servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); - servletRequest.setContent(uploadContent.getBytes()); - - tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); - assertResponseHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); - assertResponseHeader(HttpHeader.CONTENT_LENGTH, "0"); - assertResponseHeader(HttpHeader.UPLOAD_OFFSET, "" + uploadContent.getBytes().length); - assertResponseHeaderNotBlank(HttpHeader.UPLOAD_EXPIRES); - assertResponseStatus(HttpServletResponse.SC_NO_CONTENT); - - // Make sure cleanup does not interfere with this test - tusFileUploadService.cleanup(); - - // Check with HEAD request upload is complete - reset(); - servletRequest.setMethod("HEAD"); - servletRequest.setRequestURI(location); - servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); - - tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); - assertResponseHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); - assertResponseHeader(HttpHeader.CONTENT_LENGTH, "0"); - assertResponseHeader(HttpHeader.UPLOAD_OFFSET, "" + uploadContent.getBytes().length); - assertResponseHeader(HttpHeader.UPLOAD_LENGTH, "" + uploadContent.getBytes().length); - assertResponseHeaderNull(HttpHeader.UPLOAD_DEFER_LENGTH); - assertResponseHeader( - HttpHeader.UPLOAD_METADATA, "filename d29ybGRfZG9taW5hdGlvbl9wbGFuLnBkZg=="); - assertResponseStatus(HttpServletResponse.SC_NO_CONTENT); - - // Get upload info from service - UploadInfo info = tusFileUploadService.getUploadInfo(location, OWNER_KEY); - assertFalse(info.isUploadInProgress()); - assertThat(info.getLength(), is((long) uploadContent.getBytes().length)); - assertThat(info.getOffset(), is((long) uploadContent.getBytes().length)); - assertThat( - info.getMetadata(), allOf(hasSize(1), hasEntry("filename", "world_domination_plan.pdf"))); - assertThat(info.getCreatorIpAddresses(), is("10.0.2.1, 123.231.12.4, 192.168.1.1")); - - // Try retrieving the uploaded bytes without owner key - try { - tusFileUploadService.getUploadedBytes(location); - fail(); - } catch (TusException ex) { - assertThat(ex.getStatus(), is(404)); - } - - // Get uploaded bytes from service - try (InputStream uploadedBytes = tusFileUploadService.getUploadedBytes(location, OWNER_KEY)) { - assertThat( - IOUtils.toString(uploadedBytes, StandardCharsets.UTF_8), - is("This is my test upload content")); - } - - // Make sure cleanup does not interfere with this test - tusFileUploadService.cleanup(); - - // Download the upload - reset(); - servletRequest.setMethod("GET"); - servletRequest.setRequestURI(location); - - tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); - assertResponseHeader(HttpHeader.CONTENT_LENGTH, "" + uploadContent.getBytes().length); - assertResponseHeader( - HttpHeader.UPLOAD_METADATA, "filename d29ybGRfZG9taW5hdGlvbl9wbGFuLnBkZg=="); - assertResponseStatus(HttpServletResponse.SC_OK); - assertThat(servletResponse.getContentAsString(), is("This is my test upload content")); - - // Pretend that we processed the upload and that we can remove it - tusFileUploadService.deleteUpload(location, OWNER_KEY); - - // Check that the upload is really gone - reset(); - servletRequest.setMethod("HEAD"); - servletRequest.setRequestURI(location); - servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); - - tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); - assertResponseStatus(HttpServletResponse.SC_NOT_FOUND); - assertResponseHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); - assertResponseHeader(HttpHeader.CONTENT_LENGTH, "0"); - } - - @Test - public void testProcessZeroByteUpload() throws Exception { - // Create upload - servletRequest.setMethod("POST"); - servletRequest.setRequestURI(UPLOAD_URI); - servletRequest.addHeader(HttpHeader.CONTENT_LENGTH, 0); - servletRequest.addHeader(HttpHeader.UPLOAD_LENGTH, 0); - servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); - - tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); - assertResponseHeaderNotBlank(HttpHeader.LOCATION); - assertResponseHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); - assertResponseHeader(HttpHeader.CONTENT_LENGTH, "0"); - assertResponseHeaderNotBlank(HttpHeader.UPLOAD_EXPIRES); - assertResponseStatus(HttpServletResponse.SC_CREATED); - - String location = - UPLOAD_URI - + StringUtils.substringAfter( - servletResponse.getHeader(HttpHeader.LOCATION), UPLOAD_URI); - - // Get upload info from service - UploadInfo info = tusFileUploadService.getUploadInfo(location, OWNER_KEY); - assertFalse(info.isUploadInProgress()); - assertThat(info.getLength(), is(0L)); - assertThat(info.getOffset(), is(0L)); - - // Get uploaded bytes from service - try (InputStream uploadedBytes = tusFileUploadService.getUploadedBytes(location, OWNER_KEY)) { - assertThat(IOUtils.toString(uploadedBytes, StandardCharsets.UTF_8), is("")); - } - - // Make sure cleanup does not interfere with this test - tusFileUploadService.cleanup(); - - // Download the upload - reset(); - servletRequest.setMethod("GET"); - servletRequest.setRequestURI(location); - - tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); - assertResponseHeader(HttpHeader.CONTENT_LENGTH, "0"); - assertResponseStatus(HttpServletResponse.SC_OK); - assertThat(servletResponse.getContentAsString(), is("")); - } - - @Test - public void testTerminateViaHttpRequest() throws Exception { - String uploadContent = "This is my terminated test upload"; - - // Create upload - servletRequest.setMethod("POST"); - servletRequest.setRequestURI(UPLOAD_URI); - servletRequest.addHeader(HttpHeader.CONTENT_LENGTH, 0); - servletRequest.addHeader(HttpHeader.UPLOAD_LENGTH, uploadContent.getBytes().length); - servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); - servletRequest.addHeader( - HttpHeader.UPLOAD_METADATA, "filename d29ybGRfZG9taW5hdGlvbl9wbGFuLnBkZg=="); - - tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); - assertResponseHeaderNotBlank(HttpHeader.LOCATION); - assertResponseHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); - assertResponseHeader(HttpHeader.CONTENT_LENGTH, "0"); - assertResponseHeaderNotBlank(HttpHeader.UPLOAD_EXPIRES); - assertResponseStatus(HttpServletResponse.SC_CREATED); - - String location = - UPLOAD_URI - + StringUtils.substringAfter( - servletResponse.getHeader(HttpHeader.LOCATION), UPLOAD_URI); - - // Upload bytes - reset(); - servletRequest.setMethod("PATCH"); - servletRequest.setRequestURI(location); - servletRequest.addHeader(HttpHeader.CONTENT_TYPE, "application/offset+octet-stream"); - servletRequest.addHeader(HttpHeader.CONTENT_LENGTH, uploadContent.getBytes().length); - servletRequest.addHeader(HttpHeader.UPLOAD_OFFSET, 0); - servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); - servletRequest.setContent(uploadContent.getBytes()); - - tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); - assertResponseHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); - assertResponseHeader(HttpHeader.CONTENT_LENGTH, "0"); - assertResponseHeader(HttpHeader.UPLOAD_OFFSET, "" + uploadContent.getBytes().length); - assertResponseHeaderNotBlank(HttpHeader.UPLOAD_EXPIRES); - assertResponseStatus(HttpServletResponse.SC_NO_CONTENT); - - // Make sure cleanup does not interfere with this test - tusFileUploadService.cleanup(); - - // Download the upload to make sure it was uploaded correctly - reset(); - servletRequest.setMethod("GET"); - servletRequest.setRequestURI(location); - - tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); - assertResponseHeader(HttpHeader.CONTENT_LENGTH, "" + uploadContent.getBytes().length); - assertResponseHeader( - HttpHeader.UPLOAD_METADATA, "filename d29ybGRfZG9taW5hdGlvbl9wbGFuLnBkZg=="); - assertResponseStatus(HttpServletResponse.SC_OK); - assertThat(servletResponse.getContentAsString(), is("This is my terminated test upload")); - - // Terminate the upload so that the server can remove it - reset(); - servletRequest.setMethod("DELETE"); - servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); - servletRequest.setRequestURI(location); - - tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); - assertResponseHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); - assertResponseStatus(HttpServletResponse.SC_NO_CONTENT); - - // Check that the upload is really gone - reset(); - servletRequest.setMethod("HEAD"); - servletRequest.setRequestURI(location); - servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); - - tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); - assertResponseStatus(HttpServletResponse.SC_NOT_FOUND); - assertResponseHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); - assertResponseHeader(HttpHeader.CONTENT_LENGTH, "0"); - } - - @Test - public void testProcessUploadTwoParts() throws Exception { - String part1 = - "29\r\nThis is the first part of my test upload " - + "\r\n0\r\nUpload-Checksum: sha1 n5RQbRwM6UVAD+9iuHEmnN6HCGQ="; - String part2 = - "1C\r\nand this is the second part." - + "\r\n0\r\nUpload-Checksum: sha1 oNge323kGFKICxp+Me5xJgPvGEM="; - - // Create upload - servletRequest.setMethod("POST"); - servletRequest.setRequestURI(UPLOAD_URI); - servletRequest.addHeader(HttpHeader.CONTENT_LENGTH, 0); - servletRequest.addHeader(HttpHeader.UPLOAD_LENGTH, "69"); - servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); - servletRequest.addHeader( - HttpHeader.UPLOAD_METADATA, "filename d29ybGRfZG9taW5hdGlvbl9wbGFuLnBkZg=="); - - tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); - assertResponseHeaderNotBlank(HttpHeader.LOCATION); - assertResponseHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); - assertResponseHeader(HttpHeader.CONTENT_LENGTH, "0"); - assertResponseHeaderNotBlank(HttpHeader.UPLOAD_EXPIRES); - assertResponseStatus(HttpServletResponse.SC_CREATED); - - String location = - UPLOAD_URI - + StringUtils.substringAfter( - servletResponse.getHeader(HttpHeader.LOCATION), UPLOAD_URI); - - // Make sure cleanup does not interfere with this test - tusFileUploadService.cleanup(); - - // Upload part 1 bytes - reset(); - servletRequest.setMethod("PATCH"); - servletRequest.setRequestURI(location); - servletRequest.addHeader(HttpHeader.CONTENT_TYPE, "application/offset+octet-stream"); - servletRequest.addHeader(HttpHeader.CONTENT_LENGTH, "41"); - servletRequest.addHeader(HttpHeader.UPLOAD_OFFSET, 0); - servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); - servletRequest.addHeader(HttpHeader.TRANSFER_ENCODING, "chunked"); - servletRequest.setContent(part1.getBytes()); - - tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); - assertResponseStatus(HttpServletResponse.SC_NO_CONTENT); - assertResponseHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); - assertResponseHeader(HttpHeader.CONTENT_LENGTH, "0"); - assertResponseHeaderNotBlank(HttpHeader.UPLOAD_EXPIRES); - assertResponseHeader(HttpHeader.UPLOAD_OFFSET, "41"); - - // Check with service that upload is still in progress - UploadInfo info = tusFileUploadService.getUploadInfo(location, OWNER_KEY); - assertTrue(info.isUploadInProgress()); - assertThat(info.getLength(), is(69L)); - assertThat(info.getOffset(), is(41L)); - assertThat( - info.getMetadata(), allOf(hasSize(1), hasEntry("filename", "world_domination_plan.pdf"))); - assertThat(info.getCreatorIpAddresses(), is("10.0.2.1, 123.231.12.4, 192.168.1.1")); - - // Make sure cleanup does not interfere with this test - tusFileUploadService.cleanup(); - - // Verify that we cannot download an in-progress upload - reset(); - servletRequest.setMethod("GET"); - servletRequest.setRequestURI(location); - - tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); - assertResponseStatus(204); - assertResponseHeader(HttpHeader.CONTENT_LENGTH, "0"); - assertThat(servletResponse.getContentAsString(), is("")); - - // Upload part 2 bytes - reset(); - servletRequest.setMethod("PATCH"); - servletRequest.setRequestURI(location); - servletRequest.addHeader(HttpHeader.CONTENT_TYPE, "application/offset+octet-stream"); - servletRequest.addHeader(HttpHeader.CONTENT_LENGTH, "28"); - servletRequest.addHeader(HttpHeader.UPLOAD_OFFSET, "41"); - servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); - servletRequest.addHeader(HttpHeader.TRANSFER_ENCODING, "chunked"); - servletRequest.setContent(part2.getBytes()); - - tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); - assertResponseHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); - assertResponseHeader(HttpHeader.CONTENT_LENGTH, "0"); - assertResponseHeader(HttpHeader.UPLOAD_OFFSET, "69"); - assertResponseHeaderNotBlank(HttpHeader.UPLOAD_EXPIRES); - assertResponseStatus(HttpServletResponse.SC_NO_CONTENT); - - // Check with HEAD request upload is complete - reset(); - servletRequest.setMethod("HEAD"); - servletRequest.setRequestURI(location); - servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); - - tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); - assertResponseHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); - assertResponseHeader(HttpHeader.CONTENT_LENGTH, "0"); - assertResponseHeader(HttpHeader.UPLOAD_OFFSET, "69"); - assertResponseHeader(HttpHeader.UPLOAD_LENGTH, "69"); - assertResponseHeaderNull(HttpHeader.UPLOAD_DEFER_LENGTH); - assertResponseHeader( - HttpHeader.UPLOAD_METADATA, "filename d29ybGRfZG9taW5hdGlvbl9wbGFuLnBkZg=="); - assertResponseStatus(HttpServletResponse.SC_NO_CONTENT); - - // Get upload info from service - info = tusFileUploadService.getUploadInfo(location, OWNER_KEY); - assertFalse(info.isUploadInProgress()); - assertThat(info.getLength(), is(69L)); - assertThat(info.getOffset(), is(69L)); - assertThat( - info.getMetadata(), allOf(hasSize(1), hasEntry("filename", "world_domination_plan.pdf"))); - assertThat(info.getCreatorIpAddresses(), is("10.0.2.1, 123.231.12.4, 192.168.1.1")); - - // Get uploaded bytes from service - try (InputStream uploadedBytes = tusFileUploadService.getUploadedBytes(location, OWNER_KEY)) { - assertThat( - IOUtils.toString(uploadedBytes, StandardCharsets.UTF_8), - is("This is the first part of my test upload and this is the second part.")); - } - } - - @Test - public void testProcessUploadDeferredLength() throws Exception { - String part1 = "When sending this part, we don't know the length and "; - String part2 = "when sending this part, we know the length but the upload is not complete. "; - String part3 = "Finally when sending the third part, the upload is complete."; - - // Create upload - servletRequest.setMethod("POST"); - servletRequest.setRequestURI(UPLOAD_URI); - servletRequest.addHeader(HttpHeader.CONTENT_LENGTH, 0); - servletRequest.addHeader(HttpHeader.UPLOAD_DEFER_LENGTH, 1); - servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); - servletRequest.addHeader( - HttpHeader.UPLOAD_METADATA, "filename d29ybGRfZG9taW5hdGlvbl9wbGFuLnBkZg=="); - - tusFileUploadService.process(servletRequest, servletResponse); - assertResponseHeaderNotBlank(HttpHeader.LOCATION); - assertResponseHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); - assertResponseHeader(HttpHeader.CONTENT_LENGTH, "0"); - assertResponseHeaderNotBlank(HttpHeader.UPLOAD_EXPIRES); - assertResponseStatus(HttpServletResponse.SC_CREATED); - - Long expirationTimestampBefore = - Long.parseLong( - String.valueOf( - mockDateFormat - .parse(servletResponse.getHeader(HttpHeader.UPLOAD_EXPIRES)) - .getTime())); - - String location = - UPLOAD_URI - + StringUtils.substringAfter( - servletResponse.getHeader(HttpHeader.LOCATION), UPLOAD_URI); - - // Upload part 1 bytes - reset(); - servletRequest.setMethod("PATCH"); - servletRequest.setRequestURI(location); - servletRequest.addHeader(HttpHeader.CONTENT_TYPE, "application/offset+octet-stream"); - servletRequest.addHeader(HttpHeader.CONTENT_LENGTH, part1.getBytes().length); - servletRequest.addHeader(HttpHeader.UPLOAD_OFFSET, 0); - servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); - servletRequest.setContent(part1.getBytes()); - - tusFileUploadService.process(servletRequest, servletResponse); - assertResponseHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); - assertResponseHeader(HttpHeader.CONTENT_LENGTH, "0"); - assertResponseHeaderNotBlank(HttpHeader.UPLOAD_EXPIRES); - assertResponseHeader(HttpHeader.UPLOAD_OFFSET, "" + part1.getBytes().length); - assertResponseStatus(HttpServletResponse.SC_NO_CONTENT); - - // Check with service that upload is still in progress - UploadInfo info = tusFileUploadService.getUploadInfo(location, null); - assertTrue(info.isUploadInProgress()); - assertThat(info.getLength(), is(nullValue())); - assertThat(info.getOffset(), is((long) part1.getBytes().length)); - assertThat( - info.getMetadata(), allOf(hasSize(1), hasEntry("filename", "world_domination_plan.pdf"))); - assertThat(info.getCreatorIpAddresses(), is("10.0.2.1, 123.231.12.4, 192.168.1.1")); - - // Make sure cleanup does not interfere with this test - tusFileUploadService.cleanup(); - - // Check with HEAD request length is still not known - reset(); - servletRequest.setMethod("HEAD"); - servletRequest.setRequestURI(location); - servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); - - tusFileUploadService.process(servletRequest, servletResponse); - assertResponseHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); - assertResponseHeader(HttpHeader.CONTENT_LENGTH, "0"); - assertResponseHeader(HttpHeader.UPLOAD_OFFSET, "" + part1.getBytes().length); - assertResponseHeader(HttpHeader.UPLOAD_DEFER_LENGTH, "1"); - assertResponseHeader( - HttpHeader.UPLOAD_METADATA, "filename d29ybGRfZG9taW5hdGlvbl9wbGFuLnBkZg=="); - assertResponseStatus(HttpServletResponse.SC_NO_CONTENT); - - // Upload part 2 bytes with length - reset(); - servletRequest.setMethod("PATCH"); - servletRequest.setRequestURI(location); - servletRequest.addHeader(HttpHeader.CONTENT_TYPE, "application/offset+octet-stream"); - servletRequest.addHeader(HttpHeader.CONTENT_LENGTH, part2.getBytes().length); - servletRequest.addHeader(HttpHeader.UPLOAD_OFFSET, part1.getBytes().length); - servletRequest.addHeader(HttpHeader.UPLOAD_LENGTH, (part1 + part2 + part3).getBytes().length); - servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); - servletRequest.setContent(part2.getBytes()); - - tusFileUploadService.process(servletRequest, servletResponse); - assertResponseHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); - assertResponseHeader(HttpHeader.CONTENT_LENGTH, "0"); - assertResponseHeader(HttpHeader.UPLOAD_OFFSET, "" + (part1 + part2).getBytes().length); - assertResponseHeaderNotBlank(HttpHeader.UPLOAD_EXPIRES); - assertResponseStatus(HttpServletResponse.SC_NO_CONTENT); - - // Make sure cleanup does not interfere with this test - tusFileUploadService.cleanup(); - - // Check with HEAD request length is known - reset(); - servletRequest.setMethod("HEAD"); - servletRequest.setRequestURI(location); - servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); - - tusFileUploadService.process(servletRequest, servletResponse); - assertResponseHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); - assertResponseHeader(HttpHeader.CONTENT_LENGTH, "0"); - assertResponseHeader(HttpHeader.UPLOAD_OFFSET, "" + (part1 + part2).getBytes().length); - assertResponseHeader(HttpHeader.UPLOAD_LENGTH, "" + (part1 + part2 + part3).getBytes().length); - assertResponseHeader( - HttpHeader.UPLOAD_METADATA, "filename d29ybGRfZG9taW5hdGlvbl9wbGFuLnBkZg=="); - assertResponseHeaderNull(HttpHeader.UPLOAD_DEFER_LENGTH); - assertResponseStatus(HttpServletResponse.SC_NO_CONTENT); - - info = tusFileUploadService.getUploadInfo(location, null); - assertTrue(info.isUploadInProgress()); - assertThat(info.getLength(), is((long) (part1 + part2 + part3).getBytes().length)); - - // check that expiration timestamp was updated - assertThat(info.getExpirationTimestamp(), greaterThan(expirationTimestampBefore)); - - // Upload part 3 bytes - reset(); - servletRequest.setMethod("PATCH"); - servletRequest.setRequestURI(location); - servletRequest.addHeader(HttpHeader.CONTENT_TYPE, "application/offset+octet-stream"); - servletRequest.addHeader(HttpHeader.CONTENT_LENGTH, part3.getBytes().length); - servletRequest.addHeader(HttpHeader.UPLOAD_OFFSET, (part1 + part2).getBytes().length); - servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); - servletRequest.setContent(part3.getBytes()); - - tusFileUploadService.process(servletRequest, servletResponse); - assertResponseHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); - assertResponseHeader(HttpHeader.CONTENT_LENGTH, "0"); - assertResponseHeader(HttpHeader.UPLOAD_OFFSET, "" + (part1 + part2 + part3).getBytes().length); - assertResponseHeaderNotBlank(HttpHeader.UPLOAD_EXPIRES); - assertResponseStatus(HttpServletResponse.SC_NO_CONTENT); - - // Make sure cleanup does not interfere with this test - tusFileUploadService.cleanup(); - - // Get upload info from service - info = tusFileUploadService.getUploadInfo(location, null); - assertFalse(info.isUploadInProgress()); - assertThat(info.getLength(), is((long) (part1 + part2 + part3).getBytes().length)); - assertThat(info.getOffset(), is((long) (part1 + part2 + part3).getBytes().length)); - assertThat( - info.getMetadata(), allOf(hasSize(1), hasEntry("filename", "world_domination_plan.pdf"))); - - // Get uploaded bytes from service - try (InputStream uploadedBytes = tusFileUploadService.getUploadedBytes(location, null)) { - assertThat( - IOUtils.toString(uploadedBytes, StandardCharsets.UTF_8), - is( - "When sending this part, we don't know the length and " - + "when sending this part, we know the length but the upload is not complete. " - + "Finally when sending the third part, the upload is complete.")); - } - - // Make sure cleanup does not interfere with this test - tusFileUploadService.cleanup(); - - // Download the upload - reset(); - servletRequest.setMethod("GET"); - servletRequest.setRequestURI(location); - - tusFileUploadService.process(servletRequest, servletResponse, null); - assertResponseStatus(HttpServletResponse.SC_OK); - assertResponseHeader(HttpHeader.CONTENT_LENGTH, "" + (part1 + part2 + part3).getBytes().length); - assertResponseHeader( - HttpHeader.UPLOAD_METADATA, "filename d29ybGRfZG9taW5hdGlvbl9wbGFuLnBkZg=="); - assertThat( - servletResponse.getContentAsString(), - is( - "When sending this part, we don't know the length and " - + "when sending this part, we know the length but the upload is not complete. " - + "Finally when sending the third part, the upload is complete.")); - } - - @Test - public void testProcessUploadInvalidChecksumSecondPart() throws Exception { - String part1 = - "29\r\nThis is the first part of my test upload " - + "\r\n0\r\nUPLOAD-CHECKSUM: sha1 n5RQbRwM6UVAD+9iuHEmnN6HCGQ="; - String part2 = "1C\r\nand this is the second part." + "\r\n0\r\nupload-checksum: sha1 invalid"; - - // Create upload - servletRequest.setMethod("POST"); - servletRequest.setRequestURI(UPLOAD_URI); - servletRequest.addHeader(HttpHeader.CONTENT_LENGTH, 0); - servletRequest.addHeader(HttpHeader.UPLOAD_LENGTH, "69"); - servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); - servletRequest.addHeader( - HttpHeader.UPLOAD_METADATA, "filename d29ybGRfZG9taW5hdGlvbl9wbGFuLnBkZg=="); - - tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); - assertResponseHeaderNotBlank(HttpHeader.LOCATION); - assertResponseHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); - assertResponseHeader(HttpHeader.CONTENT_LENGTH, "0"); - assertResponseHeaderNotBlank(HttpHeader.UPLOAD_EXPIRES); - assertResponseStatus(HttpServletResponse.SC_CREATED); - - String location = - UPLOAD_URI - + StringUtils.substringAfter( - servletResponse.getHeader(HttpHeader.LOCATION), UPLOAD_URI); - - // Make sure cleanup does not interfere with this test - tusFileUploadService.cleanup(); - - // Upload part 1 bytes - reset(); - servletRequest.setMethod("PATCH"); - servletRequest.setRequestURI(location); - servletRequest.addHeader(HttpHeader.CONTENT_TYPE, "application/offset+octet-stream"); - servletRequest.addHeader(HttpHeader.CONTENT_LENGTH, "41"); - servletRequest.addHeader(HttpHeader.UPLOAD_OFFSET, 0); - servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); - servletRequest.addHeader(HttpHeader.TRANSFER_ENCODING, "chunked"); - servletRequest.setContent(part1.getBytes()); - - tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); - assertResponseStatus(HttpServletResponse.SC_NO_CONTENT); - assertResponseHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); - assertResponseHeader(HttpHeader.CONTENT_LENGTH, "0"); - assertResponseHeaderNotBlank(HttpHeader.UPLOAD_EXPIRES); - assertResponseHeader(HttpHeader.UPLOAD_OFFSET, "41"); - - Long expirationTimestampBefore = - Long.parseLong( - String.valueOf( - mockDateFormat - .parse(servletResponse.getHeader(HttpHeader.UPLOAD_EXPIRES)) - .getTime())); - - // Make sure cleanup does not interfere with this test - tusFileUploadService.cleanup(); - - // Upload part 2 bytes - reset(); - servletRequest.setMethod("PATCH"); - servletRequest.setRequestURI(location); - servletRequest.addHeader(HttpHeader.CONTENT_TYPE, "application/offset+octet-stream"); - servletRequest.addHeader(HttpHeader.CONTENT_LENGTH, "28"); - servletRequest.addHeader(HttpHeader.UPLOAD_OFFSET, "41"); - servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); - servletRequest.addHeader(HttpHeader.TRANSFER_ENCODING, "chunked"); - servletRequest.setContent(part2.getBytes()); - - tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); - - // We expect the server to return a checksum mismatch error - assertResponseStatus(460); - assertResponseHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); - assertResponseHeader(HttpHeader.CONTENT_LENGTH, "0"); - - // Check that upload info is still from the first patch - UploadInfo info = tusFileUploadService.getUploadInfo(location, OWNER_KEY); - assertTrue(info.isUploadInProgress()); - assertThat(info.getLength(), is(69L)); - assertThat(info.getOffset(), is(41L)); - assertThat( - info.getMetadata(), allOf(hasSize(1), hasEntry("filename", "world_domination_plan.pdf"))); - - // check that expiration timestamp was updated - assertThat(info.getExpirationTimestamp(), greaterThan(expirationTimestampBefore)); - - // We only stored the first valid part - try (InputStream uploadedBytes = tusFileUploadService.getUploadedBytes(location, OWNER_KEY)) { - assertThat( - IOUtils.toString(uploadedBytes, StandardCharsets.UTF_8), - is("This is the first part of my test upload ")); - } - - // Make sure cleanup does not interfere with this test - tusFileUploadService.cleanup(); - - // Terminate our in progress upload - reset(); - servletRequest.setMethod("DELETE"); - servletRequest.setRequestURI(location); - servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); - - tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); - - // We expect the server to return a no content code to indicate successful deletion - assertResponseStatus(HttpServletResponse.SC_NO_CONTENT); - assertResponseHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); - assertResponseHeader(HttpHeader.CONTENT_LENGTH, "0"); - - // Make sure cleanup does not interfere with this test - tusFileUploadService.cleanup(); - - // Check that the upload is really gone - reset(); - servletRequest.setMethod("HEAD"); - servletRequest.setRequestURI(location); - servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); - - tusFileUploadService.process(servletRequest, servletResponse); - assertResponseStatus(HttpServletResponse.SC_NOT_FOUND); - assertResponseHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); - assertResponseHeader(HttpHeader.CONTENT_LENGTH, "0"); - } - - @Test - public void testCleanupExpiredUpload() throws Exception { - // Set the expiration period to 500 ms - tusFileUploadService.withUploadExpirationPeriod(500L); - - String part1 = "This is the first part of my test upload"; - // Create upload - servletRequest.setMethod("POST"); - servletRequest.setRequestURI(UPLOAD_URI); - servletRequest.addHeader(HttpHeader.CONTENT_LENGTH, 0); - servletRequest.addHeader(HttpHeader.UPLOAD_LENGTH, part1.getBytes().length + 20L); - servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); - - tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); - assertResponseHeaderNotBlank(HttpHeader.LOCATION); - assertResponseHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); - assertResponseHeader(HttpHeader.CONTENT_LENGTH, "0"); - assertResponseHeaderNotBlank(HttpHeader.UPLOAD_EXPIRES); - assertResponseStatus(HttpServletResponse.SC_CREATED); - - String location = - UPLOAD_URI - + StringUtils.substringAfter( - servletResponse.getHeader(HttpHeader.LOCATION), UPLOAD_URI); - - // Upload part 1 bytes - reset(); - servletRequest.setMethod("PATCH"); - servletRequest.setRequestURI(location); - servletRequest.addHeader(HttpHeader.CONTENT_TYPE, "application/offset+octet-stream"); - servletRequest.addHeader(HttpHeader.CONTENT_LENGTH, part1.getBytes().length); - servletRequest.addHeader(HttpHeader.UPLOAD_OFFSET, 0); - servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); - servletRequest.setContent(part1.getBytes()); - - tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); - assertResponseStatus(HttpServletResponse.SC_NO_CONTENT); - assertResponseHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); - assertResponseHeader(HttpHeader.CONTENT_LENGTH, "0"); - assertResponseHeaderNotBlank(HttpHeader.UPLOAD_EXPIRES); - assertResponseHeader(HttpHeader.UPLOAD_OFFSET, "" + part1.getBytes().length); - - // Check with service that upload is still in progress - UploadInfo info = tusFileUploadService.getUploadInfo(location, OWNER_KEY); - assertTrue(info.isUploadInProgress()); - assertThat(info.getLength(), is(part1.getBytes().length + 20L)); - assertThat(info.getOffset(), is(Long.valueOf(part1.getBytes().length))); - - // Now wait until the upload expired and run the cleanup - Utils.sleep(1000L); - tusFileUploadService.cleanup(); - - // Check with HEAD request that the upload is gone - // If a Client does attempt to resume an upload which has since been removed by the Server, - // the Server SHOULD respond with the404 Not Found or 410 Gone status. - reset(); - servletRequest.setMethod("HEAD"); - servletRequest.setRequestURI(location); - servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); - - tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); - assertResponseHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); - assertResponseHeader(HttpHeader.CONTENT_LENGTH, "0"); - assertResponseStatus(HttpServletResponse.SC_NOT_FOUND); - } - - @Test - public void testConcatenationCompleted() throws Exception { - String part1 = - "29\r\nThis is the first part of my test upload " - + "\r\n0\r\nUpload-Checksum: sha1 n5RQbRwM6UVAD+9iuHEmnN6HCGQ="; - String part2 = - "1C\r\nand this is the second part." - + "\r\n0\r\nUpload-Checksum: sha1 oNge323kGFKICxp+Me5xJgPvGEM="; - - // Create first upload - servletRequest.setMethod("POST"); - servletRequest.setRequestURI(UPLOAD_URI); - servletRequest.addHeader(HttpHeader.CONTENT_LENGTH, 0); - servletRequest.addHeader(HttpHeader.UPLOAD_LENGTH, "41"); - servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); - servletRequest.addHeader(HttpHeader.UPLOAD_CONCAT, "partial"); - servletRequest.addHeader( - HttpHeader.UPLOAD_METADATA, "filename d29ybGRfZG9taW5hdGlvbl9wbGFuLnBkZg=="); - - tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); - assertResponseHeaderNotBlank(HttpHeader.LOCATION); - assertResponseHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); - assertResponseHeader(HttpHeader.CONTENT_LENGTH, "0"); - assertResponseHeaderNotBlank(HttpHeader.UPLOAD_EXPIRES); - assertResponseStatus(HttpServletResponse.SC_CREATED); - - String location1 = - UPLOAD_URI - + StringUtils.substringAfter( - servletResponse.getHeader(HttpHeader.LOCATION), UPLOAD_URI); - - // Make sure cleanup does not interfere with this test - tusFileUploadService.cleanup(); - - // Upload part 1 bytes - reset(); - servletRequest.setMethod("PATCH"); - servletRequest.setRequestURI(location1); - servletRequest.addHeader(HttpHeader.CONTENT_TYPE, "application/offset+octet-stream"); - servletRequest.addHeader(HttpHeader.CONTENT_LENGTH, "41"); - servletRequest.addHeader(HttpHeader.UPLOAD_OFFSET, 0); - servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); - servletRequest.addHeader(HttpHeader.TRANSFER_ENCODING, "chunked"); - servletRequest.setContent(part1.getBytes()); - - tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); - assertResponseStatus(HttpServletResponse.SC_NO_CONTENT); - assertResponseHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); - assertResponseHeader(HttpHeader.CONTENT_LENGTH, "0"); - assertResponseHeaderNotBlank(HttpHeader.UPLOAD_EXPIRES); - assertResponseHeader(HttpHeader.UPLOAD_OFFSET, "41"); - - // Make sure cleanup does not interfere with this test - tusFileUploadService.cleanup(); - - // Create the second upload - reset(); - servletRequest.setMethod("POST"); - servletRequest.setRequestURI(UPLOAD_URI); - servletRequest.addHeader(HttpHeader.CONTENT_LENGTH, 0); - servletRequest.addHeader(HttpHeader.UPLOAD_LENGTH, "28"); - servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); - servletRequest.addHeader(HttpHeader.UPLOAD_CONCAT, "partial"); - servletRequest.addHeader( - HttpHeader.UPLOAD_METADATA, "filename d29ybGRfZG9taW5hdGlvbl9wbGFuLnBkZg=="); - - tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); - assertResponseHeaderNotBlank(HttpHeader.LOCATION); - assertResponseHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); - assertResponseHeader(HttpHeader.CONTENT_LENGTH, "0"); - assertResponseHeaderNotBlank(HttpHeader.UPLOAD_EXPIRES); - assertResponseStatus(HttpServletResponse.SC_CREATED); - - String location2 = - UPLOAD_URI - + StringUtils.substringAfter( - servletResponse.getHeader(HttpHeader.LOCATION), UPLOAD_URI); - - // Upload part 2 bytes - reset(); - servletRequest.setMethod("PATCH"); - servletRequest.setRequestURI(location2); - servletRequest.addHeader(HttpHeader.CONTENT_TYPE, "application/offset+octet-stream"); - servletRequest.addHeader(HttpHeader.CONTENT_LENGTH, "28"); - servletRequest.addHeader(HttpHeader.UPLOAD_OFFSET, "0"); - servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); - servletRequest.addHeader(HttpHeader.TRANSFER_ENCODING, "chunked"); - servletRequest.setContent(part2.getBytes()); - - tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); - assertResponseHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); - assertResponseHeader(HttpHeader.CONTENT_LENGTH, "0"); - assertResponseHeader(HttpHeader.UPLOAD_OFFSET, "28"); - assertResponseHeaderNotBlank(HttpHeader.UPLOAD_EXPIRES); - assertResponseStatus(HttpServletResponse.SC_NO_CONTENT); - - // Create the final concatenated upload - reset(); - servletRequest.setMethod("POST"); - servletRequest.setRequestURI(UPLOAD_URI); - servletRequest.addHeader(HttpHeader.CONTENT_LENGTH, 0); - servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); - servletRequest.addHeader(HttpHeader.UPLOAD_CONCAT, "final ; " + location1 + " " + location2); - servletRequest.addHeader( - HttpHeader.UPLOAD_METADATA, - "filename d29ybGRfZG9taW5hdGlvbl9tYXBfY29uY2F0ZW5hdGVkLnBkZg=="); - - tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); - assertResponseHeaderNotBlank(HttpHeader.LOCATION); - assertResponseHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); - assertResponseHeader(HttpHeader.CONTENT_LENGTH, "0"); - assertResponseHeaderNotBlank(HttpHeader.UPLOAD_EXPIRES); - assertResponseStatus(HttpServletResponse.SC_CREATED); - - String location = - UPLOAD_URI - + StringUtils.substringAfter( - servletResponse.getHeader(HttpHeader.LOCATION), UPLOAD_URI); - - // Check with HEAD request upload is complete - reset(); - servletRequest.setMethod("HEAD"); - servletRequest.setRequestURI(location); - servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); - - tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); - assertResponseHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); - assertResponseHeader(HttpHeader.CONTENT_LENGTH, "0"); - assertResponseHeader(HttpHeader.UPLOAD_OFFSET, "69"); - assertResponseHeader(HttpHeader.UPLOAD_LENGTH, "69"); - assertResponseHeader(HttpHeader.UPLOAD_CONCAT, "final ; " + location1 + " " + location2); - assertResponseHeaderNull(HttpHeader.UPLOAD_DEFER_LENGTH); - assertResponseHeader( - HttpHeader.UPLOAD_METADATA, - "filename d29ybGRfZG9taW5hdGlvbl9tYXBfY29uY2F0ZW5hdGVkLnBkZg=="); - assertResponseStatus(HttpServletResponse.SC_NO_CONTENT); - - // Get upload info from service - UploadInfo info = tusFileUploadService.getUploadInfo(location, OWNER_KEY); - assertFalse(info.isUploadInProgress()); - assertThat(info.getLength(), is(69L)); - assertThat(info.getOffset(), is(69L)); - assertThat(info.isUploadInProgress(), is(false)); - assertThat( - info.getMetadata(), - allOf(hasSize(1), hasEntry("filename", "world_domination_map_concatenated.pdf"))); - assertThat(info.getCreatorIpAddresses(), is("10.0.2.1, 123.231.12.4, 192.168.1.1")); - - // Download the upload - reset(); - servletRequest.setMethod("GET"); - servletRequest.setRequestURI(location); - - tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); - assertResponseStatus(HttpServletResponse.SC_OK); - assertResponseHeader(HttpHeader.CONTENT_LENGTH, "69"); - assertResponseHeader( - HttpHeader.UPLOAD_METADATA, - "filename d29ybGRfZG9taW5hdGlvbl9tYXBfY29uY2F0ZW5hdGVkLnBkZg=="); - assertThat( - servletResponse.getContentAsString(), - is("This is the first part of my test upload and this is the second part.")); - - // Get uploaded bytes from service - try (InputStream uploadedBytes = tusFileUploadService.getUploadedBytes(location, OWNER_KEY)) { - assertThat( - IOUtils.toString(uploadedBytes, StandardCharsets.UTF_8), - is("This is the first part of my test upload and this is the second part.")); - } - } - - @Test - public void testConcatenationUnfinished() throws Exception { - String part1 = "When sending this part, the final upload was already created. "; - String part2 = "This is the second part of our concatenated upload. "; - String part3 = "Finally when sending the third part, the final upload is complete."; - - // Create upload part 1 - servletRequest.setMethod("POST"); - servletRequest.setRequestURI(UPLOAD_URI); - servletRequest.addHeader(HttpHeader.CONTENT_LENGTH, 0); - servletRequest.addHeader(HttpHeader.UPLOAD_DEFER_LENGTH, 1); - servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); - servletRequest.addHeader(HttpHeader.UPLOAD_CONCAT, "partial"); - servletRequest.addHeader(HttpHeader.UPLOAD_METADATA, "filename cGFydDEucGRm"); - - tusFileUploadService.process(servletRequest, servletResponse); - assertResponseHeaderNotBlank(HttpHeader.LOCATION); - assertResponseHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); - assertResponseHeader(HttpHeader.CONTENT_LENGTH, "0"); - assertResponseHeaderNotBlank(HttpHeader.UPLOAD_EXPIRES); - assertResponseStatus(HttpServletResponse.SC_CREATED); - - String location1 = - UPLOAD_URI - + StringUtils.substringAfter( - servletResponse.getHeader(HttpHeader.LOCATION), UPLOAD_URI); - - reset(); - // Create upload part 2 - servletRequest.setMethod("POST"); - servletRequest.setRequestURI(UPLOAD_URI); - servletRequest.addHeader(HttpHeader.CONTENT_LENGTH, 0); - servletRequest.addHeader(HttpHeader.UPLOAD_DEFER_LENGTH, 1); - servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); - servletRequest.addHeader(HttpHeader.UPLOAD_CONCAT, "partial"); - servletRequest.addHeader(HttpHeader.UPLOAD_METADATA, "filename cGFydDIucGRm"); - - tusFileUploadService.process(servletRequest, servletResponse); - assertResponseHeaderNotBlank(HttpHeader.LOCATION); - assertResponseHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); - assertResponseHeader(HttpHeader.CONTENT_LENGTH, "0"); - assertResponseHeaderNotBlank(HttpHeader.UPLOAD_EXPIRES); - assertResponseStatus(HttpServletResponse.SC_CREATED); - - String location2 = - UPLOAD_URI - + StringUtils.substringAfter( - servletResponse.getHeader(HttpHeader.LOCATION), UPLOAD_URI); - - reset(); - // Create upload part 3 - servletRequest.setMethod("POST"); - servletRequest.setRequestURI(UPLOAD_URI); - servletRequest.addHeader(HttpHeader.CONTENT_LENGTH, 0); - servletRequest.addHeader(HttpHeader.UPLOAD_DEFER_LENGTH, 1); - servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); - servletRequest.addHeader(HttpHeader.UPLOAD_CONCAT, "partial"); - servletRequest.addHeader(HttpHeader.UPLOAD_METADATA, "filename cGFydDMucGRm"); - - tusFileUploadService.process(servletRequest, servletResponse); - assertResponseHeaderNotBlank(HttpHeader.LOCATION); - assertResponseHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); - assertResponseHeader(HttpHeader.CONTENT_LENGTH, "0"); - assertResponseHeaderNotBlank(HttpHeader.UPLOAD_EXPIRES); - assertResponseStatus(HttpServletResponse.SC_CREATED); - - String location3 = - UPLOAD_URI - + StringUtils.substringAfter( - servletResponse.getHeader(HttpHeader.LOCATION), UPLOAD_URI); - - // Upload part 2 bytes - reset(); - servletRequest.setMethod("PATCH"); - servletRequest.setRequestURI(location2); - servletRequest.addHeader(HttpHeader.CONTENT_TYPE, "application/offset+octet-stream"); - servletRequest.addHeader(HttpHeader.CONTENT_LENGTH, part2.getBytes().length); - servletRequest.addHeader(HttpHeader.UPLOAD_LENGTH, part2.getBytes().length); - servletRequest.addHeader(HttpHeader.UPLOAD_OFFSET, 0); - servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); - servletRequest.setContent(part2.getBytes()); - - tusFileUploadService.process(servletRequest, servletResponse); - assertResponseHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); - assertResponseHeader(HttpHeader.CONTENT_LENGTH, "0"); - assertResponseHeaderNotBlank(HttpHeader.UPLOAD_EXPIRES); - assertResponseHeader(HttpHeader.UPLOAD_OFFSET, "" + part2.getBytes().length); - assertResponseStatus(HttpServletResponse.SC_NO_CONTENT); - - reset(); - // Create final upload - servletRequest.setMethod("POST"); - servletRequest.setRequestURI(UPLOAD_URI); - servletRequest.addHeader(HttpHeader.CONTENT_LENGTH, 0); - servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); - servletRequest.addHeader( - HttpHeader.UPLOAD_CONCAT, "final;" + location1 + " " + location2 + " " + location3); - servletRequest.addHeader(HttpHeader.UPLOAD_METADATA, "filename ZmluYWwucGRm"); - - tusFileUploadService.process(servletRequest, servletResponse); - assertResponseHeaderNotBlank(HttpHeader.LOCATION); - assertResponseHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); - assertResponseHeader(HttpHeader.CONTENT_LENGTH, "0"); - assertResponseHeaderNotBlank(HttpHeader.UPLOAD_EXPIRES); - assertResponseStatus(HttpServletResponse.SC_CREATED); - - String locationFinal = - UPLOAD_URI - + StringUtils.substringAfter( - servletResponse.getHeader(HttpHeader.LOCATION), UPLOAD_URI); - - // Check with HEAD request that length of final upload is undefined - reset(); - servletRequest.setMethod("HEAD"); - servletRequest.setRequestURI(locationFinal); - servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); - - tusFileUploadService.process(servletRequest, servletResponse); - assertResponseHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); - assertResponseHeader(HttpHeader.CONTENT_LENGTH, "0"); - assertResponseHeaderNull(HttpHeader.UPLOAD_OFFSET); - assertResponseHeaderNull(HttpHeader.UPLOAD_LENGTH); - assertResponseHeader(HttpHeader.UPLOAD_METADATA, "filename ZmluYWwucGRm"); - assertResponseHeader( - HttpHeader.UPLOAD_CONCAT, "final;" + location1 + " " + location2 + " " + location3); - assertResponseHeaderNull(HttpHeader.UPLOAD_DEFER_LENGTH); - assertResponseStatus(HttpServletResponse.SC_NO_CONTENT); - - // Verify that we cannot download an unfinished final upload - reset(); - servletRequest.setMethod("GET"); - servletRequest.setRequestURI(locationFinal); - - tusFileUploadService.process(servletRequest, servletResponse); - assertResponseStatus(204); - assertResponseHeader(HttpHeader.CONTENT_LENGTH, "0"); - assertThat(servletResponse.getContentAsString(), is("")); - - // Upload part 1 bytes - reset(); - servletRequest.setMethod("PATCH"); - servletRequest.setRequestURI(location1); - servletRequest.addHeader(HttpHeader.CONTENT_TYPE, "application/offset+octet-stream"); - servletRequest.addHeader(HttpHeader.CONTENT_LENGTH, part1.getBytes().length); - servletRequest.addHeader(HttpHeader.UPLOAD_LENGTH, part1.getBytes().length); - servletRequest.addHeader(HttpHeader.UPLOAD_OFFSET, 0); - servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); - servletRequest.setContent(part1.getBytes()); - - tusFileUploadService.process(servletRequest, servletResponse); - assertResponseHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); - assertResponseHeader(HttpHeader.CONTENT_LENGTH, "0"); - assertResponseHeaderNotBlank(HttpHeader.UPLOAD_EXPIRES); - assertResponseHeader(HttpHeader.UPLOAD_OFFSET, "" + part1.getBytes().length); - assertResponseStatus(HttpServletResponse.SC_NO_CONTENT); - - // Upload part 3 bytes - reset(); - servletRequest.setMethod("PATCH"); - servletRequest.setRequestURI(location3); - servletRequest.addHeader(HttpHeader.CONTENT_TYPE, "application/offset+octet-stream"); - servletRequest.addHeader(HttpHeader.CONTENT_LENGTH, part3.getBytes().length); - servletRequest.addHeader(HttpHeader.UPLOAD_LENGTH, part3.getBytes().length); - servletRequest.addHeader(HttpHeader.UPLOAD_OFFSET, 0); - servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); - servletRequest.setContent(part3.getBytes()); - - tusFileUploadService.process(servletRequest, servletResponse); - assertResponseHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); - assertResponseHeader(HttpHeader.CONTENT_LENGTH, "0"); - assertResponseHeaderNotBlank(HttpHeader.UPLOAD_EXPIRES); - assertResponseHeader(HttpHeader.UPLOAD_OFFSET, "" + part3.getBytes().length); - assertResponseStatus(HttpServletResponse.SC_NO_CONTENT); - - // Check with HEAD request length of final upload is known - reset(); - servletRequest.setMethod("HEAD"); - servletRequest.setRequestURI(locationFinal); - servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); - - tusFileUploadService.process(servletRequest, servletResponse); - assertResponseHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); - assertResponseHeader(HttpHeader.CONTENT_LENGTH, "0"); - assertResponseHeader(HttpHeader.UPLOAD_OFFSET, "" + (part1 + part2 + part3).getBytes().length); - assertResponseHeader(HttpHeader.UPLOAD_LENGTH, "" + (part1 + part2 + part3).getBytes().length); - assertResponseHeader(HttpHeader.UPLOAD_METADATA, "filename ZmluYWwucGRm"); - assertResponseHeader( - HttpHeader.UPLOAD_CONCAT, "final;" + location1 + " " + location2 + " " + location3); - assertResponseHeaderNull(HttpHeader.UPLOAD_DEFER_LENGTH); - assertResponseStatus(HttpServletResponse.SC_NO_CONTENT); - - // Download the upload - reset(); - servletRequest.setMethod("GET"); - servletRequest.setRequestURI(locationFinal); - - tusFileUploadService.process(servletRequest, servletResponse, null); - assertResponseStatus(HttpServletResponse.SC_OK); - assertResponseHeader(HttpHeader.CONTENT_LENGTH, "" + (part1 + part2 + part3).getBytes().length); - assertResponseHeader(HttpHeader.UPLOAD_METADATA, "filename ZmluYWwucGRm"); - assertThat( - servletResponse.getContentAsString(), - is( - "When sending this part, the final upload was already created. " - + "This is the second part of our concatenated upload. " - + "Finally when sending the third part, the final upload is complete.")); - - // Get uploaded bytes from service - try (InputStream uploadedBytes = tusFileUploadService.getUploadedBytes(locationFinal, null)) { - assertThat( - IOUtils.toString(uploadedBytes, StandardCharsets.UTF_8), - is( - "When sending this part, the final upload was already created. " - + "This is the second part of our concatenated upload. " - + "Finally when sending the third part, the final upload is complete.")); - } - } - - @Test - public void testChunkedDecodingDisabledAndRegexUploadUri() throws Exception { - String chunkedContent = - "1B;test=value\r\nThis upload looks chunked, \r\n" + "D\r\nbut it's not!\r\n" + "\r\n0\r\n"; - - // Create service without chunked decoding - tusFileUploadService = - new TusFileUploadService() - .withUploadUri("/users/[0-9]+/files/upload") - .withStoragePath(storagePath.toAbsolutePath().toString()) - .withDownloadFeature(); - - // Create upload - servletRequest.setMethod("POST"); - servletRequest.setRequestURI("/users/98765/files/upload"); - servletRequest.addHeader(HttpHeader.CONTENT_LENGTH, 0); - servletRequest.addHeader(HttpHeader.UPLOAD_LENGTH, "67"); - servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); - servletRequest.addHeader( - HttpHeader.UPLOAD_METADATA, "filename d29ybGRfZG9taW5hdGlvbl9wbGFuLnBkZg=="); - - tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); - assertResponseHeaderNotBlank(HttpHeader.LOCATION); - assertResponseHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); - assertResponseHeader(HttpHeader.CONTENT_LENGTH, "0"); - assertResponseHeaderNull(HttpHeader.UPLOAD_EXPIRES); - assertResponseStatus(HttpServletResponse.SC_CREATED); - - String location = servletResponse.getHeader(HttpHeader.LOCATION); - - // Upload content - reset(); - servletRequest.setMethod("PATCH"); - servletRequest.setRequestURI(location); - servletRequest.addHeader(HttpHeader.CONTENT_TYPE, "application/offset+octet-stream"); - servletRequest.addHeader(HttpHeader.CONTENT_LENGTH, "67"); - servletRequest.addHeader(HttpHeader.UPLOAD_OFFSET, 0); - servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); - servletRequest.addHeader(HttpHeader.TRANSFER_ENCODING, "chunked"); - servletRequest.setContent(chunkedContent.getBytes()); - - tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); - assertResponseStatus(HttpServletResponse.SC_NO_CONTENT); - assertResponseHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); - assertResponseHeader(HttpHeader.CONTENT_LENGTH, "0"); - assertResponseHeaderNull(HttpHeader.UPLOAD_EXPIRES); - assertResponseHeader(HttpHeader.UPLOAD_OFFSET, "67"); - - // Check with HEAD request upload is complete - reset(); - servletRequest.setMethod("HEAD"); - servletRequest.setRequestURI(location); - servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); - - tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); - assertResponseHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); - assertResponseHeader(HttpHeader.CONTENT_LENGTH, "0"); - assertResponseHeader(HttpHeader.UPLOAD_OFFSET, "67"); - assertResponseHeader(HttpHeader.UPLOAD_LENGTH, "67"); - assertResponseHeaderNull(HttpHeader.UPLOAD_DEFER_LENGTH); - assertResponseHeader( - HttpHeader.UPLOAD_METADATA, "filename d29ybGRfZG9taW5hdGlvbl9wbGFuLnBkZg=="); - assertResponseStatus(HttpServletResponse.SC_NO_CONTENT); - - // Get upload info from service - UploadInfo info = tusFileUploadService.getUploadInfo(location, OWNER_KEY); - assertFalse(info.isUploadInProgress()); - assertThat(info.getLength(), is(67L)); - assertThat(info.getOffset(), is(67L)); - assertThat( - info.getMetadata(), allOf(hasSize(1), hasEntry("filename", "world_domination_plan.pdf"))); - - // Get uploaded bytes from service - try (InputStream uploadedBytes = tusFileUploadService.getUploadedBytes(location, OWNER_KEY)) { - assertThat( - IOUtils.toString(uploadedBytes, StandardCharsets.UTF_8), - is( - "1B;test=value\r\nThis upload looks chunked, \r\n" - + "D\r\nbut it's not!\r\n" - + "\r\n0\r\n")); - } - } - - @Test - public void testOptions() throws Exception { - // Do options request and check response headers - servletRequest.setMethod("OPTIONS"); - - tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); + protected static Path storagePath; - assertResponseStatus(HttpServletResponse.SC_NO_CONTENT); - assertResponseHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); - assertResponseHeader(HttpHeader.CONTENT_LENGTH, "0"); - assertResponseHeader(HttpHeader.TUS_VERSION, "1.0.0"); - assertResponseHeader(HttpHeader.TUS_MAX_SIZE, "1073741824"); - assertResponseHeader( - HttpHeader.TUS_CHECKSUM_ALGORITHM, "md5", "sha1", "sha256", "sha384", "sha512"); - assertResponseHeader( - HttpHeader.TUS_EXTENSION, - "creation", - "creation-defer-length", - "creation-with-upload", - "checksum", - "checksum-trailer", - "termination", - "download", - "expiration", - "concatenation", - "concatenation-unfinished"); + @BeforeClass + public static void setupDataFolder() throws IOException { + storagePath = Paths.get("target", "tus", "data").toAbsolutePath(); + Files.createDirectories(storagePath); } - @Test - public void testHeadOnNonExistingUpload() throws Exception { - servletRequest.setMethod("HEAD"); - servletRequest.setRequestURI(UPLOAD_URI + "/" + UUID.randomUUID()); - servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); - - tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); - assertResponseStatus(HttpServletResponse.SC_NOT_FOUND); - assertResponseHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); - assertResponseHeader(HttpHeader.CONTENT_LENGTH, "0"); + @AfterClass + public static void destroyDataFolder() throws IOException { + FileUtils.deleteDirectory(storagePath.toFile()); } - @Test - public void testInvalidTusResumable() throws Exception { - servletRequest.setMethod("POST"); - servletRequest.setRequestURI(UPLOAD_URI); - servletRequest.addHeader(HttpHeader.CONTENT_LENGTH, 0); - servletRequest.addHeader(HttpHeader.UPLOAD_DEFER_LENGTH, 1); - servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "2.0.0"); - - tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); - assertResponseStatus(HttpServletResponse.SC_PRECONDITION_FAILED); - assertResponseHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); - assertResponseHeader(HttpHeader.CONTENT_LENGTH, "0"); + @Override + protected TusFileUploadService createTusFileUploadService() { + return createTusFileUploadService(UPLOAD_URI); } - @Test - public void testMaxUploadLengthExceeded() throws Exception { - tusFileUploadService.withMaxUploadSize(10L); - - String uploadContent = "This is upload is too long"; - - // Create upload - servletRequest.setMethod("POST"); - servletRequest.setRequestURI(UPLOAD_URI); - servletRequest.addHeader(HttpHeader.CONTENT_LENGTH, 0); - servletRequest.addHeader(HttpHeader.UPLOAD_LENGTH, uploadContent.getBytes().length); - servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); - - tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); - assertResponseStatus(HttpServletResponse.SC_REQUEST_ENTITY_TOO_LARGE); - assertResponseHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); - assertResponseHeader(HttpHeader.CONTENT_LENGTH, "0"); + @Override + protected TusFileUploadService createTusFileUploadService(String uploadUri) { + return new TusFileUploadService() + .withUploadUri(uploadUri) + .withStoragePath(storagePath.toAbsolutePath().toString()) + .withMaxUploadSize(1073741824L) + .withUploadExpirationPeriod(2L * 24 * 60 * 60 * 1000) + .withDownloadFeature() + .withChunkedTransferDecoding(true); } - @Test - public void testInvalidMethods() throws Exception { - servletRequest.setMethod("PUT"); - servletRequest.setRequestURI(UPLOAD_URI + "/" + UUID.randomUUID()); - servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); - - tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); - assertResponseStatus(HttpServletResponse.SC_METHOD_NOT_ALLOWED); - - reset(); - servletRequest.setMethod("CONNECT"); - servletRequest.setRequestURI(UPLOAD_URI + "/" + UUID.randomUUID()); - servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); - - tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); - assertResponseStatus(HttpServletResponse.SC_METHOD_NOT_ALLOWED); - assertResponseHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); - assertResponseHeader(HttpHeader.CONTENT_LENGTH, "0"); - - reset(); - servletRequest.setMethod("TRACE"); - servletRequest.setRequestURI(UPLOAD_URI + "/" + UUID.randomUUID()); - servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); - - tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); - assertResponseStatus(HttpServletResponse.SC_METHOD_NOT_ALLOWED); - assertResponseHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); - assertResponseHeader(HttpHeader.CONTENT_LENGTH, "0"); - } + // =============================================================================================== + // DISK-SPECIFIC STORAGE TESTS + // =============================================================================================== + /** + * Disk Storage Specific Test: Verify automatic file deduplication on upload completion, + * inspecting physical disk file structures, checking child data file non-existence, direct file + * manipulation on disk, and cleanup on parent deletion. + */ @Test - public void testFileDeduplicationEndToEnd() throws Exception { - // Enable deduplication feature + public void testAutomaticDeduplicationOnUploadCompletion() throws Exception { + // Step 1: Enable deduplication feature on disk storage tusFileUploadService.withUploadDeduplication(true); String uploadContent = "Deduplication integration test content"; - // 1. First upload (Parent) + // Step 2: Upload parent file servletRequest.setMethod("POST"); servletRequest.setRequestURI(UPLOAD_URI); servletRequest.addHeader(HttpHeader.CONTENT_LENGTH, 0); @@ -1511,7 +93,7 @@ public void testFileDeduplicationEndToEnd() throws Exception { tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); assertResponseStatus(HttpServletResponse.SC_NO_CONTENT); - // 2. Second upload (Child/Duplicate) + // Step 3: Upload identical duplicate child file reset(); servletRequest.setMethod("POST"); servletRequest.setRequestURI(UPLOAD_URI); @@ -1537,23 +119,23 @@ public void testFileDeduplicationEndToEnd() throws Exception { tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); assertResponseStatus(HttpServletResponse.SC_NO_CONTENT); - // 3. Verify deduplication succeeded + // Step 4: Verify deduplication metadata link (child duplicates parent) UploadInfo parentInfo = tusFileUploadService.getUploadInfo(parentLocation, OWNER_KEY); UploadInfo childInfo = tusFileUploadService.getUploadInfo(childLocation, OWNER_KEY); assertThat(childInfo.getDuplicatesUploadId(), is(parentInfo.getId())); - // Verify child upload physical data file does NOT exist + // Step 5: Verify child upload physical data file does NOT exist on disk Path childDataPath = storagePath.resolve("uploads").resolve(childInfo.getId().toString()).resolve("data"); assertFalse(Files.exists(childDataPath)); - // Verify parent upload physical data file DOES exist + // Step 6: Verify parent upload physical data file DOES exist on disk Path parentDataPath = storagePath.resolve("uploads").resolve(parentInfo.getId().toString()).resolve("data"); assertTrue(Files.exists(parentDataPath)); - // Verify child download retrieves parent's content + // Step 7: Verify downloading child retrieves parent's content reset(); servletRequest.setMethod("GET"); servletRequest.setRequestURI(childLocation); @@ -1561,12 +143,12 @@ public void testFileDeduplicationEndToEnd() throws Exception { assertResponseStatus(HttpServletResponse.SC_OK); assertThat(servletResponse.getContentAsString(), is(uploadContent)); - // Manipulate parent file directly on disk + // Step 8: Directly modify parent data file on physical disk String manipulatedContent = "manipulated content on disk"; Files.write( parentDataPath, manipulatedContent.getBytes(java.nio.charset.StandardCharsets.UTF_8)); - // Verify parent download retrieves the manipulated content + // Step 9: Verify parent and child downloads both return the manipulated content reset(); servletRequest.setMethod("GET"); servletRequest.setRequestURI(parentLocation); @@ -1574,7 +156,6 @@ public void testFileDeduplicationEndToEnd() throws Exception { assertResponseStatus(HttpServletResponse.SC_OK); assertThat(servletResponse.getContentAsString(), is(manipulatedContent)); - // Verify child download retrieves the same manipulated content reset(); servletRequest.setMethod("GET"); servletRequest.setRequestURI(childLocation); @@ -1582,433 +163,13 @@ public void testFileDeduplicationEndToEnd() throws Exception { assertResponseStatus(HttpServletResponse.SC_OK); assertThat(servletResponse.getContentAsString(), is(manipulatedContent)); - // 4. Delete parent upload + // Step 10: Delete parent upload and verify child download returns 404 tusFileUploadService.deleteUpload(parentLocation, OWNER_KEY); - // 5. Attempt child download: should return 404 Not Found since parent is deleted reset(); servletRequest.setMethod("GET"); servletRequest.setRequestURI(childLocation); tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); assertResponseStatus(HttpServletResponse.SC_NOT_FOUND); } - - @Test - public void testLockContentionAndHeadRelease() throws Exception { - // 1. Create upload resource - servletRequest.setMethod("POST"); - servletRequest.setRequestURI(UPLOAD_URI); - servletRequest.addHeader(HttpHeader.CONTENT_LENGTH, 0); - servletRequest.addHeader(HttpHeader.UPLOAD_LENGTH, 1000L); - servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); - tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); - String location = - UPLOAD_URI - + StringUtils.substringAfter( - servletResponse.getHeader(HttpHeader.LOCATION), UPLOAD_URI); - - // 2. Start a blocking PATCH in a background thread to hold the lock - final java.util.concurrent.CountDownLatch requestStarted = - new java.util.concurrent.CountDownLatch(1); - final java.util.concurrent.atomic.AtomicReference bgException = - new java.util.concurrent.atomic.AtomicReference<>(); - - InputStream blockingStream = - new InputStream() { - private volatile boolean closed = false; - - @Override - public int read() throws IOException { - requestStarted.countDown(); - synchronized (this) { - while (!closed) { - try { - this.wait(100); - } catch (InterruptedException e) { - Thread.currentThread().interrupt(); - throw new IOException("Interrupted", e); - } - } - } - throw new IOException("Stream closed"); - } - - @Override - public void close() throws IOException { - synchronized (this) { - closed = true; - this.notifyAll(); - } - } - }; - - final MockHttpServletRequest bgRequest = - new MockHttpServletRequest() { - @Override - public jakarta.servlet.ServletInputStream getInputStream() { - return new jakarta.servlet.ServletInputStream() { - @Override - public int read() throws IOException { - return blockingStream.read(); - } - - @Override - public void close() throws IOException { - blockingStream.close(); - } - - @Override - public boolean isFinished() { - return false; - } - - @Override - public boolean isReady() { - return true; - } - - @Override - public void setReadListener(jakarta.servlet.ReadListener readListener) {} - }; - } - }; - bgRequest.setMethod("PATCH"); - bgRequest.setRequestURI(location); - bgRequest.addHeader(HttpHeader.CONTENT_TYPE, "application/offset+octet-stream"); - bgRequest.addHeader(HttpHeader.CONTENT_LENGTH, 100); - bgRequest.addHeader(HttpHeader.UPLOAD_OFFSET, 0); - bgRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); - - Thread bgThread = - new Thread( - new Runnable() { - @Override - public void run() { - try { - MockHttpServletResponse bgResponse = new MockHttpServletResponse(); - tusFileUploadService.process(bgRequest, bgResponse, OWNER_KEY); - } catch (Exception e) { - e.printStackTrace(); - bgException.set(e); - } - } - }); - bgThread.start(); - - // Wait for the background thread to start reading (meaning it holds the lock) - requestStarted.await(2, java.util.concurrent.TimeUnit.SECONDS); - - // 3. Concurrent PATCH request must fail immediately with 423 - MockHttpServletRequest patchRequest = new MockHttpServletRequest(); - MockHttpServletResponse patchResponse = new MockHttpServletResponse(); - patchRequest.setMethod("PATCH"); - patchRequest.setRequestURI(location); - patchRequest.addHeader(HttpHeader.CONTENT_TYPE, "application/offset+octet-stream"); - patchRequest.addHeader(HttpHeader.CONTENT_LENGTH, 10); - patchRequest.addHeader(HttpHeader.UPLOAD_OFFSET, 0); - patchRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); - patchRequest.setContent(new byte[10]); - - tusFileUploadService.process(patchRequest, patchResponse, OWNER_KEY); - assertThat(patchResponse.getStatus(), is(423)); - - // 4. Concurrent HEAD request must interrupt background thread and succeed - MockHttpServletRequest headRequest = new MockHttpServletRequest(); - MockHttpServletResponse headResponse = new MockHttpServletResponse(); - headRequest.setMethod("HEAD"); - headRequest.setRequestURI(location); - headRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); - - tusFileUploadService.process(headRequest, headResponse, OWNER_KEY); - assertThat(headResponse.getStatus(), is(204)); - assertThat(headResponse.getHeader(HttpHeader.UPLOAD_OFFSET), is("0")); - - // Clean up - bgThread.join(2000); - } - - @Test - public void testCreationWithUploadOptions() throws Exception { - reset(); - servletRequest.setMethod("OPTIONS"); - servletRequest.setRequestURI(UPLOAD_URI); - tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); - assertResponseHeader( - HttpHeader.TUS_EXTENSION, - "creation", - "creation-defer-length", - "creation-with-upload", - "checksum", - "checksum-trailer", - "termination", - "download", - "expiration", - "concatenation", - "concatenation-unfinished"); - } - - @Test - public void testCreationWithUploadSuccess() throws Exception { - String uploadContent = "Initial data to upload"; - byte[] contentBytes = uploadContent.getBytes(StandardCharsets.UTF_8); - - reset(); - servletRequest.setMethod("POST"); - servletRequest.setRequestURI(UPLOAD_URI); - servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); - servletRequest.addHeader(HttpHeader.UPLOAD_LENGTH, contentBytes.length); - servletRequest.addHeader(HttpHeader.CONTENT_LENGTH, contentBytes.length); - servletRequest.addHeader(HttpHeader.CONTENT_TYPE, "application/offset+octet-stream"); - servletRequest.setContent(contentBytes); - - tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); - assertResponseStatus(HttpServletResponse.SC_CREATED); - assertResponseHeaderNotBlank(HttpHeader.LOCATION); - assertResponseHeader(HttpHeader.UPLOAD_OFFSET, String.valueOf(contentBytes.length)); - - String location = servletResponse.getHeader(HttpHeader.LOCATION); - - // Verify content in storage - try (InputStream is = tusFileUploadService.getUploadedBytes(location, OWNER_KEY)) { - String readContent = IOUtils.toString(is, StandardCharsets.UTF_8); - assertThat(readContent, is(uploadContent)); - } - } - - @Test - public void testCreationWithUploadDeferredLengthSuccess() throws Exception { - String uploadContent = "Initial data to upload with deferred length"; - byte[] contentBytes = uploadContent.getBytes(StandardCharsets.UTF_8); - - reset(); - servletRequest.setMethod("POST"); - servletRequest.setRequestURI(UPLOAD_URI); - servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); - servletRequest.addHeader(HttpHeader.UPLOAD_DEFER_LENGTH, "1"); - servletRequest.addHeader(HttpHeader.CONTENT_LENGTH, contentBytes.length); - servletRequest.addHeader(HttpHeader.CONTENT_TYPE, "application/offset+octet-stream"); - servletRequest.setContent(contentBytes); - - tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); - assertResponseStatus(HttpServletResponse.SC_CREATED); - assertResponseHeaderNotBlank(HttpHeader.LOCATION); - assertResponseHeader(HttpHeader.UPLOAD_OFFSET, String.valueOf(contentBytes.length)); - - String location = servletResponse.getHeader(HttpHeader.LOCATION); - - // Verify content in storage - try (InputStream is = tusFileUploadService.getUploadedBytes(location, OWNER_KEY)) { - String readContent = IOUtils.toString(is, StandardCharsets.UTF_8); - assertThat(readContent, is(uploadContent)); - } - } - - @Test - public void testCreationWithUploadInvalidContentType() throws Exception { - String uploadContent = "Initial data to upload"; - byte[] contentBytes = uploadContent.getBytes(StandardCharsets.UTF_8); - - reset(); - servletRequest.setMethod("POST"); - servletRequest.setRequestURI(UPLOAD_URI); - servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); - servletRequest.addHeader(HttpHeader.UPLOAD_LENGTH, contentBytes.length); - servletRequest.addHeader(HttpHeader.CONTENT_LENGTH, contentBytes.length); - servletRequest.addHeader(HttpHeader.CONTENT_TYPE, "application/octet-stream"); - servletRequest.setContent(contentBytes); - - tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); - assertResponseStatus(HttpServletResponse.SC_NOT_ACCEPTABLE); - } - - @Test - public void testCreationWithUploadExceedsLength() throws Exception { - String uploadContent = "Initial data to upload"; - byte[] contentBytes = uploadContent.getBytes(StandardCharsets.UTF_8); - - reset(); - servletRequest.setMethod("POST"); - servletRequest.setRequestURI(UPLOAD_URI); - servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); - servletRequest.addHeader(HttpHeader.UPLOAD_LENGTH, contentBytes.length - 5); - servletRequest.addHeader(HttpHeader.CONTENT_LENGTH, contentBytes.length); - servletRequest.addHeader(HttpHeader.CONTENT_TYPE, "application/offset+octet-stream"); - servletRequest.setContent(contentBytes); - - tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); - assertResponseStatus(HttpServletResponse.SC_BAD_REQUEST); - } - - @Test - public void testCreationWithUploadDisabled() throws Exception { - String uploadContent = "Initial data to upload"; - byte[] contentBytes = uploadContent.getBytes(StandardCharsets.UTF_8); - - tusFileUploadService.disableTusExtension("creation-with-upload"); - try { - reset(); - servletRequest.setMethod("POST"); - servletRequest.setRequestURI(UPLOAD_URI); - servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); - servletRequest.addHeader(HttpHeader.UPLOAD_LENGTH, contentBytes.length); - servletRequest.addHeader(HttpHeader.CONTENT_LENGTH, contentBytes.length); - servletRequest.addHeader(HttpHeader.CONTENT_TYPE, "application/offset+octet-stream"); - servletRequest.setContent(contentBytes); - - tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); - assertResponseStatus(HttpServletResponse.SC_BAD_REQUEST); - } finally { - // Restore for other tests - tusFileUploadService = - new TusFileUploadService() - .withUploadUri(UPLOAD_URI) - .withStoragePath(storagePath.toAbsolutePath().toString()) - .withMaxUploadSize(1073741824L) - .withUploadExpirationPeriod(2L * 24 * 60 * 60 * 1000) - .withDownloadFeature() - .withChunkedTransferDecoding(true); - } - } - - @Test - public void testCorsHeaders() throws Exception { - reset(); - servletRequest.setMethod("POST"); - servletRequest.setRequestURI(UPLOAD_URI); - servletRequest.addHeader("Origin", "https://example.com"); - servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); - servletRequest.addHeader(HttpHeader.UPLOAD_LENGTH, 100L); - - tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); - assertResponseHeader("Access-Control-Allow-Origin", "https://example.com"); - assertResponseHeaderNotBlank("Access-Control-Expose-Headers"); - } - - @Test - public void testCorsPreflight() throws Exception { - reset(); - servletRequest.setMethod("OPTIONS"); - servletRequest.setRequestURI(UPLOAD_URI); - servletRequest.addHeader("Origin", "https://example.com"); - servletRequest.addHeader("Access-Control-Request-Method", "PATCH"); - - tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); - assertResponseHeader("Access-Control-Allow-Origin", "https://example.com"); - assertResponseHeader("Access-Control-Allow-Methods", "POST, GET, HEAD, PATCH, DELETE, OPTIONS"); - assertResponseHeaderNotBlank("Access-Control-Allow-Headers"); - assertResponseHeader("Access-Control-Max-Age", "86400"); - } - - @Test - public void testTusVersionHeaderOn412() throws Exception { - reset(); - servletRequest.setMethod("POST"); - servletRequest.setRequestURI(UPLOAD_URI); - servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "2.0.0"); - - tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); - assertResponseStatus(HttpServletResponse.SC_PRECONDITION_FAILED); - assertResponseHeader(HttpHeader.TUS_VERSION, "1.0.0"); - } - - @Test - public void testModifyUploadLengthOnPatch() throws Exception { - // 1. Create upload with deferred length - reset(); - servletRequest.setMethod("POST"); - servletRequest.setRequestURI(UPLOAD_URI); - servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); - servletRequest.addHeader(HttpHeader.UPLOAD_DEFER_LENGTH, "1"); - tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); - assertResponseStatus(HttpServletResponse.SC_CREATED); - String location = servletResponse.getHeader(HttpHeader.LOCATION); - - // 2. Set length to 100 on PATCH - reset(); - servletRequest.setMethod("PATCH"); - servletRequest.setRequestURI(location); - servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); - servletRequest.addHeader(HttpHeader.UPLOAD_OFFSET, "0"); - servletRequest.addHeader(HttpHeader.UPLOAD_LENGTH, "100"); - servletRequest.addHeader(HttpHeader.CONTENT_TYPE, "application/offset+octet-stream"); - servletRequest.setContent("test content".getBytes(StandardCharsets.UTF_8)); - tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); - assertResponseStatus(HttpServletResponse.SC_NO_CONTENT); - - // 3. Try to change length to 200 on subsequent PATCH -> should return 400 - reset(); - servletRequest.setMethod("PATCH"); - servletRequest.setRequestURI(location); - servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); - servletRequest.addHeader(HttpHeader.UPLOAD_OFFSET, "12"); - servletRequest.addHeader(HttpHeader.UPLOAD_LENGTH, "200"); - servletRequest.addHeader(HttpHeader.CONTENT_TYPE, "application/offset+octet-stream"); - servletRequest.setContent("more content".getBytes(StandardCharsets.UTF_8)); - tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); - assertResponseStatus(HttpServletResponse.SC_BAD_REQUEST); - } - - @Test - public void testCreationWithUploadChecksumSuccess() throws Exception { - String uploadContent = "Initial data to upload with checksum"; - byte[] contentBytes = uploadContent.getBytes(StandardCharsets.UTF_8); - // Base64 hash for MD5 of "Initial data to upload with checksum" - // Base64 of MD5 bytes: aAMsB0BZbWXCBuPWG/ADyA== - String base64Checksum = "aAMsB0BZbWXCBuPWG/ADyA=="; - - reset(); - servletRequest.setMethod("POST"); - servletRequest.setRequestURI(UPLOAD_URI); - servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); - servletRequest.addHeader(HttpHeader.UPLOAD_LENGTH, contentBytes.length); - servletRequest.addHeader(HttpHeader.CONTENT_LENGTH, contentBytes.length); - servletRequest.addHeader(HttpHeader.CONTENT_TYPE, "application/offset+octet-stream"); - servletRequest.addHeader(HttpHeader.UPLOAD_CHECKSUM, "md5 " + base64Checksum); - servletRequest.setContent(contentBytes); - - tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); - assertResponseStatus(HttpServletResponse.SC_CREATED); - assertResponseHeader(HttpHeader.UPLOAD_OFFSET, String.valueOf(contentBytes.length)); - } - - @Test - public void testCreationWithUploadChecksumMismatch() throws Exception { - String uploadContent = "Initial data to upload with checksum"; - byte[] contentBytes = uploadContent.getBytes(StandardCharsets.UTF_8); - String invalidBase64Checksum = "aAMsB0BZbWXCBuPWG/ADyB=="; // changed A to B - - reset(); - servletRequest.setMethod("POST"); - servletRequest.setRequestURI(UPLOAD_URI); - servletRequest.addHeader(HttpHeader.TUS_RESUMABLE, "1.0.0"); - servletRequest.addHeader(HttpHeader.UPLOAD_LENGTH, contentBytes.length); - servletRequest.addHeader(HttpHeader.CONTENT_LENGTH, contentBytes.length); - servletRequest.addHeader(HttpHeader.CONTENT_TYPE, "application/offset+octet-stream"); - servletRequest.addHeader(HttpHeader.UPLOAD_CHECKSUM, "md5 " + invalidBase64Checksum); - servletRequest.setContent(contentBytes); - - tusFileUploadService.process(servletRequest, servletResponse, OWNER_KEY); - assertResponseStatus(460); // Checksum mismatch - } - - protected void assertResponseHeader(final String header, final String value) { - assertThat(servletResponse.getHeader(header), is(value)); - } - - protected void assertResponseHeader(final String header, final String... values) { - assertThat( - Arrays.asList(servletResponse.getHeader(header).split(",")), containsInAnyOrder(values)); - } - - protected void assertResponseHeaderNotBlank(final String header) { - assertTrue(StringUtils.isNotBlank(servletResponse.getHeader(header))); - } - - protected void assertResponseHeaderNull(final String header) { - assertNull(servletResponse.getHeader(header)); - } - - protected void assertResponseStatus(final int httpStatus) { - assertThat(servletResponse.getStatus(), is(httpStatus)); - } } diff --git a/src/test/java/me/desair/tus/server/ITTusFileUploadServiceCached.java b/src/test/java/me/desair/tus/server/ITTusFileUploadServiceCached.java index 4d619555..2cb9f630 100644 --- a/src/test/java/me/desair/tus/server/ITTusFileUploadServiceCached.java +++ b/src/test/java/me/desair/tus/server/ITTusFileUploadServiceCached.java @@ -19,7 +19,7 @@ public class ITTusFileUploadServiceCached extends ITTusFileUploadService { @Override @Before - public void setUp() { + public void setUp() throws Exception { super.setUp(); tusFileUploadService = tusFileUploadService diff --git a/src/test/java/me/desair/tus/server/TestUtils.java b/src/test/java/me/desair/tus/server/TestUtils.java new file mode 100644 index 00000000..d7659b64 --- /dev/null +++ b/src/test/java/me/desair/tus/server/TestUtils.java @@ -0,0 +1,107 @@ +package me.desair.tus.server; + +import io.minio.BucketExistsArgs; +import io.minio.MakeBucketArgs; +import io.minio.MinioClient; +import org.testcontainers.DockerClientFactory; +import org.testcontainers.containers.GenericContainer; + +/** + * Helper utility class for S3 integration tests running against Testcontainers MinIO using the + * MinIO Java SDK. Supports both Docker and Podman container engines automatically. + */ +public final class TestUtils { + + private TestUtils() { + // Utility class + } + + /** + * Check if a container runtime (Docker or Podman) is available locally. + * + * @return True if Docker or Podman is available for Testcontainers, false otherwise + */ + public static boolean isContainerRuntimeAvailable() { + try { + if (DockerClientFactory.instance().isDockerAvailable()) { + return true; + } + } catch (Throwable ignored) { + } + + try { + String userHome = System.getProperty("user.home", ""); + String[] possibleSockets = { + "/var/run/docker.sock", + "/run/podman/podman.sock", + userHome + "/.local/share/containers/podman/machine/podman-machine-default/podman.sock", + userHome + "/.local/share/containers/podman/machine/qemu/podman.sock" + }; + + for (String socketPath : possibleSockets) { + if (new java.io.File(socketPath).exists()) { + if (System.getProperty("DOCKER_HOST") == null) { + System.setProperty("DOCKER_HOST", "unix://" + socketPath); + } + System.setProperty("TESTCONTAINERS_RYUK_DISABLED", "true"); + try { + if (DockerClientFactory.instance().isDockerAvailable()) { + return true; + } + } catch (Throwable ignored) { + } + } + } + + Process process = new ProcessBuilder("podman", "info").start(); + if (process.waitFor() == 0) { + System.setProperty("TESTCONTAINERS_RYUK_DISABLED", "true"); + return DockerClientFactory.instance().isDockerAvailable(); + } + } catch (Throwable ignored) { + } + + return false; + } + + /** + * Create and configure a GenericContainer running MinIO for integration testing. + * + * @return A configured GenericContainer instance (not started yet) + */ + public static GenericContainer createMinioContainer() { + return new GenericContainer<>("minio/minio:RELEASE.2024-01-16T16-07-38Z") + .withExposedPorts(9000) + .withEnv("MINIO_ROOT_USER", "minioadmin") + .withEnv("MINIO_ROOT_PASSWORD", "minioadmin") + .withCommand("server /data"); + } + + /** + * Create a {@link MinioClient} configured to connect to the given MinIO container. + * + * @param minio The active MinIO Testcontainer + * @return Pre-configured MinioClient + */ + public static MinioClient createMinioClient(GenericContainer minio) { + String minioUrl = "http://" + minio.getHost() + ":" + minio.getMappedPort(9000); + return MinioClient.builder().endpoint(minioUrl).credentials("minioadmin", "minioadmin").build(); + } + + /** + * Ensures an S3 bucket exists using MinIO Client. + * + * @param minioClient The MinIO Client + * @param bucket The S3 bucket name + */ + public static void createBucket(MinioClient minioClient, String bucket) { + try { + boolean found = minioClient.bucketExists(BucketExistsArgs.builder().bucket(bucket).build()); + if (!found) { + minioClient.makeBucket(MakeBucketArgs.builder().bucket(bucket).build()); + } + } catch (Exception e) { + throw new RuntimeException("Failed to create bucket " + bucket, e); + } + } +} diff --git a/src/test/java/me/desair/tus/server/TusFileUploadServiceTest.java b/src/test/java/me/desair/tus/server/TusFileUploadServiceTest.java index 6d7257ae..55a2eee3 100644 --- a/src/test/java/me/desair/tus/server/TusFileUploadServiceTest.java +++ b/src/test/java/me/desair/tus/server/TusFileUploadServiceTest.java @@ -513,4 +513,13 @@ public void testGetRawInterimResponse() throws Exception { org.junit.Assert.assertNull( service.getRawInterimResponse((jakarta.servlet.http.HttpServletRequest) null, "owner-123")); } + + @Test + public void testWithJsonSerialization() { + TusFileUploadService service = new TusFileUploadService().withJsonSerialization(); + org.junit.Assert.assertTrue(service.getUploadStorageService().isJsonSerializationEnabled()); + + service.withJsonSerialization(false); + org.junit.Assert.assertFalse(service.getUploadStorageService().isJsonSerializationEnabled()); + } } diff --git a/src/test/java/me/desair/tus/server/upload/concatenation/VirtualConcatenationServiceTest.java b/src/test/java/me/desair/tus/server/upload/concatenation/VirtualConcatenationServiceTest.java index d75b78c5..769a092c 100644 --- a/src/test/java/me/desair/tus/server/upload/concatenation/VirtualConcatenationServiceTest.java +++ b/src/test/java/me/desair/tus/server/upload/concatenation/VirtualConcatenationServiceTest.java @@ -341,4 +341,43 @@ public void getConcatenatedBytesNotFound() throws Exception { concatenationService.getConcatenatedBytes(infoParent); } + + @Test(expected = UploadNotFoundException.class) + public void getPartialUploadsNotFound() throws Exception { + UploadInfo child1 = new UploadInfo(); + child1.setId(new UploadId(UUID.randomUUID())); + + UploadInfo infoParent = new UploadInfo(); + infoParent.setId(new UploadId(UUID.randomUUID())); + infoParent.setConcatenationPartIds( + java.util.Collections.singletonList(child1.getId().toString())); + + when(uploadStorageService.getUploadInfo(child1.getId().toString(), infoParent.getOwnerKey())) + .thenReturn(null); + + concatenationService.getPartialUploads(infoParent); + } + + @Test + public void testMergeHandlesUploadNotFoundExceptionOnUpdate() throws Exception { + UploadInfo child1 = new UploadInfo(); + child1.setId(new UploadId(UUID.randomUUID())); + child1.setLength(5L); + child1.setOffset(5L); + + UploadInfo infoParent = new UploadInfo(); + infoParent.setId(new UploadId(UUID.randomUUID())); + infoParent.setConcatenationPartIds( + java.util.Collections.singletonList(child1.getId().toString())); + + when(uploadStorageService.getUploadInfo(child1.getId().toString(), infoParent.getOwnerKey())) + .thenReturn(child1); + + org.mockito.Mockito.doThrow(new UploadNotFoundException("Parent upload missing")) + .when(uploadStorageService) + .update(infoParent); + + concatenationService.merge(infoParent); + assertThat(infoParent.getLength(), is(5L)); + } } diff --git a/src/test/java/me/desair/tus/server/upload/disk/DiskStorageServiceTest.java b/src/test/java/me/desair/tus/server/upload/disk/DiskStorageServiceTest.java index 533253ca..9a0666d8 100644 --- a/src/test/java/me/desair/tus/server/upload/disk/DiskStorageServiceTest.java +++ b/src/test/java/me/desair/tus/server/upload/disk/DiskStorageServiceTest.java @@ -972,5 +972,121 @@ public void setMaxAppendSize(Long maxAppendSize) {} assertThat(anonymousService.getMinSize(), is(nullValue())); anonymousService.setMinAppendSize(100L); anonymousService.setMinSize(100L); + assertThat(anonymousService.isJsonSerializationEnabled(), is(false)); + anonymousService.setJsonSerializationEnabled(true); + } + + @Test + public void testJsonSerialization() throws Exception { + storageService.setJsonSerializationEnabled(true); + assertThat(storageService.isJsonSerializationEnabled(), is(true)); + + UploadInfo info = new UploadInfo(); + info.setLength(1024L); + info.setEncodedMetadata("filename d29ybGQudHh0"); + + info = storageService.create(info, "owner-json"); + UploadInfo retrieved = storageService.getUploadInfo(info.getId()); + + assertThat(retrieved, is(notNullValue())); + assertThat(retrieved.getLength(), is(1024L)); + assertThat(retrieved.getOwnerKey(), is("owner-json")); + assertThat(retrieved.getFileName(), is("world.txt")); + + Path infoPath = getUploadInfoPath(info.getId()); + String fileContent = new String(Files.readAllBytes(infoPath), StandardCharsets.UTF_8); + assertThat(fileContent.contains("\"ownerKey\":\"owner-json\""), is(true)); + } + + @Test + public void testJsonSerializationFallbackAndInvalidFile() throws Exception { + storageService.setJsonSerializationEnabled(false); + UploadInfo info = new UploadInfo(); + info.setLength(2048L); + info = storageService.create(info, "owner-legacy"); + + storageService.setJsonSerializationEnabled(true); + UploadInfo fallbackRetrieved = storageService.getUploadInfo(info.getId()); + assertThat(fallbackRetrieved, is(notNullValue())); + assertThat(fallbackRetrieved.getLength(), is(2048L)); + + Path infoPath = getUploadInfoPath(info.getId()); + Files.write(infoPath, "corrupted-data".getBytes(StandardCharsets.UTF_8)); + UploadInfo corruptedRetrieved = storageService.getUploadInfo(info.getId()); + assertThat(corruptedRetrieved, is(nullValue())); + } + + @Test(expected = IllegalArgumentException.class) + public void testGetUploadInfoWithUnsafePathTraversalUploadId() throws Exception { + storageService.getUploadInfo(new UploadId("..")); + } + + @Test(expected = IllegalArgumentException.class) + public void testGetUploadedBytesWithUnsafePathTraversalUploadId() throws Exception { + storageService.getUploadedBytes(new UploadId("..")); + } + + @Test(expected = IllegalArgumentException.class) + public void testTerminateUploadWithUnsafePathTraversalUploadId() throws Exception { + UploadInfo info = new UploadInfo(); + info.setId(new UploadId("..")); + storageService.terminateUpload(info); + } + + @Test(expected = IllegalArgumentException.class) + public void testAppendWithUnsafePathTraversalUploadId() throws Exception { + UploadInfo info = new UploadInfo(); + info.setId(new UploadId("..")); + storageService.append(info, new java.io.ByteArrayInputStream(new byte[10])); + } + + @Test(expected = UploadNotFoundException.class) + public void testGetUploadedBytesMissingDataFileThrowsUploadNotFoundException() throws Exception { + UploadInfo info = new UploadInfo(); + info.setLength(100L); + info = storageService.create(info, null); + + info.setOffset(100L); + storageService.update(info); + + Path bytesPath = + storagePath.resolve("uploads").resolve(info.getId().toString()).resolve("data"); + Files.deleteIfExists(bytesPath); + + storageService.getUploadedBytes(info.getId()); + } + + @Test + public void testCopyUploadToWithNullOrInProgressOrMissingDataFile() throws Exception { + ByteArrayOutputStream baos = new ByteArrayOutputStream(); + + // Case 1: In-progress upload + UploadInfo inProgress = new UploadInfo(); + inProgress.setLength(100L); + inProgress.setOffset(50L); + inProgress = storageService.create(inProgress, null); + storageService.copyUploadTo(inProgress, baos); + + // Case 2: Missing data file for completed upload + UploadInfo missingData = new UploadInfo(); + missingData.setLength(10L); + missingData.setOffset(10L); + missingData = storageService.create(missingData, null); + Path dataPath = storagePath.resolve(missingData.getId().toString()).resolve("data"); + Files.deleteIfExists(dataPath); + + try { + storageService.copyUploadTo(missingData, baos); + } catch (UploadNotFoundException expected) { + } + } + + @Test(expected = IllegalArgumentException.class) + public void testUnsafeChildDataFileDeletionThrowsIllegalArgumentException() throws Exception { + UploadInfo child = new UploadInfo(); + child.setId(new UploadId("..")); + child.setDuplicatesUploadId(new UploadId("parent-123")); + + storageService.update(child); } } diff --git a/src/test/java/me/desair/tus/server/upload/disk/FileBasedLockTest.java b/src/test/java/me/desair/tus/server/upload/disk/FileBasedLockTest.java index 0d39344a..345f1c3c 100644 --- a/src/test/java/me/desair/tus/server/upload/disk/FileBasedLockTest.java +++ b/src/test/java/me/desair/tus/server/upload/disk/FileBasedLockTest.java @@ -91,6 +91,41 @@ public void testLockIoException() throws UploadAlreadyLockedException, IOExcepti lock.close(); } + @Test(expected = UploadAlreadyLockedException.class) + public void testOverlappingLockFileChannelCloseException() throws Exception { + UUID test = UUID.randomUUID(); + Path path = storagePath.resolve(test.toString()); + FileBasedLock lock = + new FileBasedLock("/test/upload/" + test.toString(), path) { + @Override + protected FileChannel createFileChannel() throws IOException { + FileChannel mockChannel = createFileChannelMock(); + doReturn(null).when(mockChannel).tryLock(anyLong(), anyLong(), anyBoolean()); + org.mockito.Mockito.doThrow(new IOException("Close error")).when(mockChannel).close(); + return mockChannel; + } + }; + } + + @Test + public void testReleaseIOException() throws Exception { + UUID test = UUID.randomUUID(); + Path path = storagePath.resolve(test.toString()); + FileBasedLock lock = + new FileBasedLock("/test/upload/" + test.toString(), path) { + @Override + protected FileChannel createFileChannel() throws IOException { + FileChannel mockChannel = createFileChannelMock(); + doReturn(org.mockito.Mockito.mock(java.nio.channels.FileLock.class)) + .when(mockChannel) + .tryLock(anyLong(), anyLong(), anyBoolean()); + org.mockito.Mockito.doThrow(new IOException("Close error")).when(mockChannel).close(); + return mockChannel; + } + }; + lock.release(); + } + private FileChannel createFileChannelMock() throws IOException { return spy(FileChannel.class); } diff --git a/src/test/java/me/desair/tus/server/upload/s3/ITS3LockingService.java b/src/test/java/me/desair/tus/server/upload/s3/ITS3LockingService.java new file mode 100644 index 00000000..58248985 --- /dev/null +++ b/src/test/java/me/desair/tus/server/upload/s3/ITS3LockingService.java @@ -0,0 +1,82 @@ +package me.desair.tus.server.upload.s3; + +import static org.junit.Assert.assertFalse; +import static org.junit.Assert.assertNotNull; +import static org.junit.Assert.assertTrue; + +import io.minio.MinioClient; +import me.desair.tus.server.TestUtils; +import me.desair.tus.server.exception.UploadAlreadyLockedException; +import me.desair.tus.server.upload.UploadId; +import me.desair.tus.server.upload.UploadLock; +import org.junit.AfterClass; +import org.junit.Before; +import org.junit.BeforeClass; +import org.junit.Test; +import org.testcontainers.containers.GenericContainer; + +public class ITS3LockingService { + + private static GenericContainer minio; + private static MinioClient minioClient; + private static final String BUCKET = "test-locking-service-bucket"; + + private S3LockingService lockingService; + + @BeforeClass + public static void setUpClass() { + org.junit.Assume.assumeTrue( + "Container runtime is not available; skipping Testcontainers MinIO test", + TestUtils.isContainerRuntimeAvailable()); + + minio = TestUtils.createMinioContainer(); + minio.start(); + + minioClient = TestUtils.createMinioClient(minio); + TestUtils.createBucket(minioClient, BUCKET); + } + + @AfterClass + public static void tearDownClass() { + if (minio != null) { + minio.stop(); + } + } + + @Before + public void setUp() { + org.junit.Assume.assumeTrue(TestUtils.isContainerRuntimeAvailable()); + lockingService = new S3LockingService(minioClient, BUCKET); + } + + @Test + public void testLockAcquireAndRelease() throws Exception { + String uri = "/test/upload/24249a5b-01a4-4bf8-b67a-364273bb5a21"; + UploadId id = new UploadId("24249a5b-01a4-4bf8-b67a-364273bb5a21"); + UploadLock lock = lockingService.lockUploadByUri(uri); + assertNotNull(lock); + + // Verify upload is locked + assertTrue(lockingService.isLocked(id)); + + // Release lock + lock.release(); + + // Verify lock is released + assertFalse(lockingService.isLocked(id)); + } + + @Test(expected = UploadAlreadyLockedException.class) + public void testConcurrentLockFails() throws Exception { + String uri = "/test/upload/24249a5b-01a4-4bf8-b67a-364273bb5a22"; + UploadLock lock1 = lockingService.lockUploadByUri(uri); + assertNotNull(lock1); + + try { + // Second lock attempt on same URI should throw UploadAlreadyLockedException + lockingService.lockUploadByUri(uri); + } finally { + lock1.release(); + } + } +} diff --git a/src/test/java/me/desair/tus/server/upload/s3/ITS3RufhProtocol.java b/src/test/java/me/desair/tus/server/upload/s3/ITS3RufhProtocol.java new file mode 100644 index 00000000..a064290b --- /dev/null +++ b/src/test/java/me/desair/tus/server/upload/s3/ITS3RufhProtocol.java @@ -0,0 +1,58 @@ +package me.desair.tus.server.upload.s3; + +import io.minio.MinioClient; +import me.desair.tus.server.AbstractITRufhProtocol; +import me.desair.tus.server.TestUtils; +import me.desair.tus.server.TusFileUploadService; +import org.junit.AfterClass; +import org.junit.BeforeClass; + +/** + * End-to-end integration test suite verifying the IETF Resumable Uploads for HTTP (RUFH) protocol + * implementation backed by {@link S3StorageService} and {@link S3LockingService} on MinIO using the + * MinIO Java SDK. + */ +public class ITS3RufhProtocol extends AbstractITRufhProtocol { + + private static org.testcontainers.containers.GenericContainer minio; + private static MinioClient minioClient; + private static final String BUCKET = "test-rufh-s3-bucket"; + + @BeforeClass + public static void setUpClass() { + org.junit.Assume.assumeTrue( + "Container runtime is not available; skipping Testcontainers MinIO test", + TestUtils.isContainerRuntimeAvailable()); + + minio = TestUtils.createMinioContainer(); + minio.start(); + + minioClient = TestUtils.createMinioClient(minio); + TestUtils.createBucket(minioClient, BUCKET); + } + + @AfterClass + public static void tearDownClass() { + if (minio != null) { + minio.stop(); + } + } + + @Override + protected TusFileUploadService createTusFileUploadService() { + org.junit.Assume.assumeTrue(TestUtils.isContainerRuntimeAvailable()); + + S3StorageService s3Storage = new S3StorageService(minioClient, BUCKET); + S3LockingService s3Locking = new S3LockingService(minioClient, BUCKET); + S3ConcatenationService s3Concat = new S3ConcatenationService(minioClient, BUCKET, s3Storage); + s3Storage.setUploadConcatenationService(s3Concat); + + return new TusFileUploadService() + .withUploadUri(UPLOAD_URI) + .withUploadStorageService(s3Storage) + .withUploadLockingService(s3Locking) + .withMaxUploadSize(1073741824L) + .withUploadExpirationPeriod(2L * 24 * 60 * 60 * 1000) + .withDownloadFeature(); + } +} diff --git a/src/test/java/me/desair/tus/server/upload/s3/ITS3StorageService.java b/src/test/java/me/desair/tus/server/upload/s3/ITS3StorageService.java new file mode 100644 index 00000000..544147a0 --- /dev/null +++ b/src/test/java/me/desair/tus/server/upload/s3/ITS3StorageService.java @@ -0,0 +1,181 @@ +package me.desair.tus.server.upload.s3; + +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertNotNull; +import static org.junit.Assert.assertNull; + +import io.minio.MinioClient; +import java.io.ByteArrayInputStream; +import java.io.InputStream; +import java.nio.charset.StandardCharsets; +import me.desair.tus.server.TestUtils; +import me.desair.tus.server.checksum.ChecksumAlgorithm; +import me.desair.tus.server.upload.UploadInfo; +import org.apache.commons.codec.digest.DigestUtils; +import org.apache.commons.io.IOUtils; +import org.junit.AfterClass; +import org.junit.Before; +import org.junit.BeforeClass; +import org.junit.Test; +import org.testcontainers.containers.GenericContainer; + +public class ITS3StorageService { + + private static GenericContainer minio; + private static MinioClient minioClient; + private static final String BUCKET = "test-storage-service-bucket"; + + private S3StorageService storageService; + + @BeforeClass + public static void setUpClass() { + org.junit.Assume.assumeTrue( + "Container runtime is not available; skipping Testcontainers MinIO test", + TestUtils.isContainerRuntimeAvailable()); + + minio = TestUtils.createMinioContainer(); + minio.start(); + + minioClient = TestUtils.createMinioClient(minio); + TestUtils.createBucket(minioClient, BUCKET); + } + + @AfterClass + public static void tearDownClass() { + if (minio != null) { + minio.stop(); + } + } + + @Before + public void setUp() { + org.junit.Assume.assumeTrue(TestUtils.isContainerRuntimeAvailable()); + storageService = new S3StorageService(minioClient, BUCKET); + } + + @Test + public void testFullUploadLifecycle() throws Exception { + UploadInfo info = new UploadInfo(); + info.setLength(11L); + + info = storageService.create(info, "owner1"); + assertNotNull(info.getId()); + assertNotNull(info.getStorageUploadId()); + assertEquals("owner1", info.getOwnerKey()); + + // Append data + ByteArrayInputStream bais = + new ByteArrayInputStream("hello world".getBytes(StandardCharsets.UTF_8)); + info = storageService.append(info, bais); + assertEquals(Long.valueOf(11), info.getOffset()); + + // Verify uploaded bytes + try (InputStream is = storageService.getUploadedBytes(info.getId())) { + assertNotNull(is); + assertEquals("hello world", IOUtils.toString(is, StandardCharsets.UTF_8)); + } + + // Verify getUploadInfo by URI and ID + UploadInfo fetched = storageService.getUploadInfo(info.getId()); + assertNotNull(fetched); + assertEquals(Long.valueOf(11), fetched.getOffset()); + + // Terminate upload + storageService.terminateUpload(info); + assertNull(storageService.getUploadInfo(info.getId())); + } + + @Test + public void testDeduplicationOnS3() throws Exception { + storageService.setUploadDeduplicationEnabled(true); + + byte[] content = "S3 Deduplicated Content".getBytes(StandardCharsets.UTF_8); + String sha1Base64 = + org.apache.commons.codec.binary.Base64.encodeBase64String(DigestUtils.sha1(content)); + + // Parent upload + UploadInfo parent = new UploadInfo(); + parent.setLength((long) content.length); + parent.setChecksum(sha1Base64); + parent.setChecksumAlgorithm(ChecksumAlgorithm.SHA1); + parent = storageService.create(parent, "owner1"); + + storageService.append(parent, new ByteArrayInputStream(content)); + + // Look up by checksum + UploadInfo found = storageService.getUploadInfoByChecksum(sha1Base64, ChecksumAlgorithm.SHA1); + assertNotNull(found); + assertEquals(parent.getId(), found.getId()); + } + + @Test + public void testMultipartChunkedUploadAndFinalizationOnS3() throws Exception { + UploadInfo info = new UploadInfo(); + info.setLength(20L); + info = storageService.create(info, "owner-multipart"); + + // Part 1: append 10 bytes (creates an incomplete .part file) + byte[] part1 = "1234567890".getBytes(StandardCharsets.UTF_8); + info = storageService.append(info, new ByteArrayInputStream(part1)); + assertEquals(Long.valueOf(10L), info.getOffset()); + + // Part 2: append 10 bytes (completes the upload, triggers leftover .part finalization on real + // S3) + byte[] part2 = "abcdefghij".getBytes(StandardCharsets.UTF_8); + info = storageService.append(info, new ByteArrayInputStream(part2)); + assertEquals(Long.valueOf(20L), info.getOffset()); + + // Verify uploaded bytes on real S3 + try (InputStream is = storageService.getUploadedBytes(info.getId())) { + assertNotNull(is); + assertEquals("1234567890abcdefghij", IOUtils.toString(is, StandardCharsets.UTF_8)); + } + } + + @Test + public void testTruncationOnRealS3() throws Exception { + // Scenario A: Truncate in-progress upload with .part object + UploadInfo info1 = new UploadInfo(); + info1.setLength(50L); + info1 = storageService.create(info1, "owner-trunc1"); + + byte[] bytes1 = + "Hello, World! This is an in-progress payload.".getBytes(StandardCharsets.UTF_8); + info1 = storageService.append(info1, new ByteArrayInputStream(bytes1)); + assertEquals(Long.valueOf(bytes1.length), info1.getOffset()); + + storageService.removeLastNumberOfBytes(info1, 10L); + assertEquals(Long.valueOf(bytes1.length - 10), info1.getOffset()); + + // Scenario B: Truncate completed upload + UploadInfo info2 = new UploadInfo(); + info2.setLength(12L); + info2 = storageService.create(info2, "owner-trunc2"); + info2 = + storageService.append( + info2, new ByteArrayInputStream("Hello World!".getBytes(StandardCharsets.UTF_8))); + assertEquals(Long.valueOf(12L), info2.getOffset()); + + storageService.removeLastNumberOfBytes(info2, 6L); + assertEquals(Long.valueOf(6L), info2.getOffset()); + } + + @Test + public void testCleanupExpiredUploadsOnS3() throws Exception { + UploadInfo info = new UploadInfo(); + info.setLength(100L); + info = storageService.create(info, "owner-exp"); + + // Set expiration in the past + info.setExpirationTimestamp(System.currentTimeMillis() - 60000L); + storageService.update(info); + + assertNotNull(storageService.getUploadInfo(info.getId())); + + // Run cleanup on real S3 + storageService.cleanupExpiredUploads(null); + + // Verify upload was cleaned up from real S3 + assertNull(storageService.getUploadInfo(info.getId())); + } +} diff --git a/src/test/java/me/desair/tus/server/upload/s3/ITS3TusFileUploadService.java b/src/test/java/me/desair/tus/server/upload/s3/ITS3TusFileUploadService.java new file mode 100644 index 00000000..af3c0436 --- /dev/null +++ b/src/test/java/me/desair/tus/server/upload/s3/ITS3TusFileUploadService.java @@ -0,0 +1,67 @@ +package me.desair.tus.server.upload.s3; + +import io.minio.MinioClient; +import me.desair.tus.server.AbstractITTusFileUploadService; +import me.desair.tus.server.ProtocolVersion; +import me.desair.tus.server.TestUtils; +import me.desair.tus.server.TusFileUploadService; +import org.junit.AfterClass; +import org.junit.BeforeClass; +import org.testcontainers.containers.GenericContainer; + +/** + * End-to-end integration test suite verifying {@link TusFileUploadService} backed by {@link + * S3StorageService} and {@link S3LockingService} on MinIO using the MinIO Java SDK. Extends {@link + * AbstractITTusFileUploadService} to run all Tus 1.0.0 protocol use cases against S3 storage. + */ +public class ITS3TusFileUploadService extends AbstractITTusFileUploadService { + + private static GenericContainer minio; + private static MinioClient minioClient; + private static final String BUCKET = "test-service-s3-bucket"; + + @BeforeClass + public static void setUpClass() { + org.junit.Assume.assumeTrue( + "Container runtime is not available; skipping Testcontainers MinIO test", + TestUtils.isContainerRuntimeAvailable()); + + minio = TestUtils.createMinioContainer(); + minio.start(); + + minioClient = TestUtils.createMinioClient(minio); + TestUtils.createBucket(minioClient, BUCKET); + } + + @AfterClass + public static void tearDownClass() { + if (minio != null) { + minio.stop(); + } + } + + @Override + protected TusFileUploadService createTusFileUploadService() { + return createTusFileUploadService(UPLOAD_URI); + } + + @Override + protected TusFileUploadService createTusFileUploadService(String uploadUri) { + org.junit.Assume.assumeTrue(TestUtils.isContainerRuntimeAvailable()); + + S3StorageService s3Storage = new S3StorageService(minioClient, BUCKET); + S3LockingService s3Locking = new S3LockingService(minioClient, BUCKET); + S3ConcatenationService s3Concat = new S3ConcatenationService(minioClient, BUCKET, s3Storage); + s3Storage.setUploadConcatenationService(s3Concat); + + return new TusFileUploadService() + .withUploadUri(uploadUri) + .withUploadStorageService(s3Storage) + .withUploadLockingService(s3Locking) + .withMaxUploadSize(1073741824L) + .withUploadExpirationPeriod(2L * 24 * 60 * 60 * 1000) + .withSupportedProtocolVersions(ProtocolVersion.TUS_1_0_0) + .withDownloadFeature() + .withChunkedTransferDecoding(true); + } +} diff --git a/src/test/java/me/desair/tus/server/upload/s3/S3ConcatenationServiceTest.java b/src/test/java/me/desair/tus/server/upload/s3/S3ConcatenationServiceTest.java new file mode 100644 index 00000000..90a91cbf --- /dev/null +++ b/src/test/java/me/desair/tus/server/upload/s3/S3ConcatenationServiceTest.java @@ -0,0 +1,248 @@ +package me.desair.tus.server.upload.s3; + +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertNotNull; +import static org.junit.Assert.assertNull; +import static org.junit.Assert.assertTrue; + +import io.minio.ComposeObjectArgs; +import io.minio.MinioClient; +import io.minio.PutObjectArgs; +import java.io.ByteArrayInputStream; +import java.io.IOException; +import java.io.InputStream; +import java.nio.file.Paths; +import java.util.Arrays; +import java.util.Collections; +import java.util.List; +import me.desair.tus.server.exception.UploadNotFoundException; +import me.desair.tus.server.upload.UploadId; +import me.desair.tus.server.upload.UploadInfo; +import me.desair.tus.server.upload.UploadStorageService; +import org.junit.Before; +import org.junit.Test; +import org.mockito.Mockito; + +public class S3ConcatenationServiceTest { + + private MinioClient minioClient; + private UploadStorageService storageService; + private S3ConcatenationService concatenationService; + + @Before + public void setUp() { + minioClient = Mockito.mock(MinioClient.class); + storageService = Mockito.mock(UploadStorageService.class); + concatenationService = + new S3ConcatenationService( + minioClient, + "test-bucket", + "tus-uploads/", + storageService, + Paths.get(System.getProperty("java.io.tmpdir"))); + } + + @Test + public void testConstructorsAndSetters() { + S3ConcatenationService service1 = new S3ConcatenationService(minioClient, "test-bucket"); + S3ConcatenationService service2 = + new S3ConcatenationService(minioClient, "test-bucket", storageService); + service1.setUploadStorageService(storageService); + + assertNotNull(service1); + assertNotNull(service2); + } + + @Test + public void testMergeEarlyReturnConditions() throws Exception { + // Null upload info + concatenationService.merge(null); + + // Upload not in progress (offset == length) + UploadInfo infoNotInProgress = new UploadInfo(); + infoNotInProgress.setLength(100L); + infoNotInProgress.setOffset(100L); + infoNotInProgress.setConcatenationPartIds(Arrays.asList("/part-1")); + concatenationService.merge(infoNotInProgress); + + // Null concatenation part IDs + UploadInfo infoNullParts = new UploadInfo(); + infoNullParts.setLength(100L); + infoNullParts.setOffset(50L); + infoNullParts.setConcatenationPartIds(null); + concatenationService.merge(infoNullParts); + } + + @Test + public void testGetPartialUploads() throws Exception { + UploadInfo p1 = new UploadInfo(); + p1.setId(new UploadId("part-1")); + p1.setLength(10L * 1024 * 1024); + + UploadInfo p2 = new UploadInfo(); + p2.setId(new UploadId("part-2")); + p2.setLength(10L * 1024 * 1024); + + Mockito.when(storageService.getUploadInfo("/part-1", "owner-1")).thenReturn(p1); + Mockito.when(storageService.getUploadInfo("/part-2", "owner-1")).thenReturn(p2); + + UploadInfo finalUpload = new UploadInfo(); + finalUpload.setId(new UploadId("final-1")); + finalUpload.setOwnerKey("owner-1"); + finalUpload.setConcatenationPartIds(Arrays.asList("/part-1", "/part-2")); + + List partials = concatenationService.getPartialUploads(finalUpload); + assertNotNull(partials); + assertEquals(2, partials.size()); + + // Empty part list + finalUpload.setConcatenationPartIds(Collections.emptyList()); + assertTrue(concatenationService.getPartialUploads(finalUpload).isEmpty()); + } + + @Test(expected = UploadNotFoundException.class) + public void testGetPartialUploadsChildNotFound() throws Exception { + UploadInfo finalUpload = new UploadInfo(); + finalUpload.setId(new UploadId("final-1")); + finalUpload.setOwnerKey("owner-1"); + finalUpload.setConcatenationPartIds(Arrays.asList("/missing-part")); + + Mockito.when(storageService.getUploadInfo("/missing-part", "owner-1")).thenReturn(null); + + concatenationService.getPartialUploads(finalUpload); + } + + @Test + public void testMergePartialUploadsServerSideCopy() throws Exception { + UploadInfo p1 = new UploadInfo(); + p1.setId(new UploadId("part-1")); + p1.setLength(10L * 1024 * 1024); + p1.setOffset(10L * 1024 * 1024); + p1.setStorageUploadId("custom-part-1-key"); + + Mockito.when(storageService.getUploadInfo("/part-1", "owner-1")).thenReturn(p1); + + UploadInfo finalUpload = new UploadInfo(); + finalUpload.setId(new UploadId("final-1")); + finalUpload.setOwnerKey("owner-1"); + finalUpload.setConcatenationPartIds(Arrays.asList("/part-1")); + + concatenationService.merge(finalUpload); + assertEquals(Long.valueOf(10L * 1024 * 1024), finalUpload.getLength()); + assertEquals(Long.valueOf(10L * 1024 * 1024), finalUpload.getOffset()); + assertEquals("tus-uploads/final-1", finalUpload.getStorageUploadId()); + } + + @Test + public void testMergePartialUploadsStreamingReupload() throws Exception { + UploadInfo p1 = new UploadInfo(); + p1.setId(new UploadId("small-part-1")); + p1.setLength(100L); // < 5MB minPartSize + p1.setOffset(100L); + + Mockito.when(storageService.getUploadInfo("/small-part-1", "owner-1")).thenReturn(p1); + Mockito.when(storageService.getUploadedBytes(new UploadId("small-part-1"))) + .thenReturn(new ByteArrayInputStream(new byte[100])); + + UploadInfo finalUpload = new UploadInfo(); + finalUpload.setId(new UploadId("final-streaming")); + finalUpload.setOwnerKey("owner-1"); + finalUpload.setConcatenationPartIds(Arrays.asList("/small-part-1")); + + concatenationService.merge(finalUpload); + assertEquals(Long.valueOf(100L), finalUpload.getLength()); + } + + @Test(expected = IOException.class) + public void testMergeServerSideCopyFails() throws Exception { + UploadInfo p1 = new UploadInfo(); + p1.setId(new UploadId("part-1")); + p1.setLength(10L * 1024 * 1024); + p1.setOffset(10L * 1024 * 1024); + + Mockito.when(storageService.getUploadInfo("/part-1", "owner-1")).thenReturn(p1); + Mockito.when(minioClient.composeObject(Mockito.any(ComposeObjectArgs.class))) + .thenThrow(new RuntimeException("Compose error")); + + UploadInfo finalUpload = new UploadInfo(); + finalUpload.setId(new UploadId("final-1")); + finalUpload.setOwnerKey("owner-1"); + finalUpload.setConcatenationPartIds(Arrays.asList("/part-1")); + + concatenationService.merge(finalUpload); + } + + @Test(expected = IOException.class) + public void testMergeStreamingReuploadFails() throws Exception { + UploadInfo p1 = new UploadInfo(); + p1.setId(new UploadId("small-part-1")); + p1.setLength(100L); + p1.setOffset(100L); + + Mockito.when(storageService.getUploadInfo("/small-part-1", "owner-1")).thenReturn(p1); + Mockito.when(minioClient.putObject(Mockito.any(PutObjectArgs.class))) + .thenThrow(new RuntimeException("PutObject error")); + + UploadInfo finalUpload = new UploadInfo(); + finalUpload.setId(new UploadId("final-streaming")); + finalUpload.setOwnerKey("owner-1"); + finalUpload.setConcatenationPartIds(Arrays.asList("/small-part-1")); + + concatenationService.merge(finalUpload); + } + + @Test + public void testMergeHandlesStorageServiceUpdateException() throws Exception { + UploadInfo p1 = new UploadInfo(); + p1.setId(new UploadId("part-1")); + p1.setLength(10L * 1024 * 1024); + p1.setOffset(10L * 1024 * 1024); + + Mockito.when(storageService.getUploadInfo("/part-1", "owner-1")).thenReturn(p1); + Mockito.doThrow(new UploadNotFoundException("Not found")) + .when(storageService) + .update(Mockito.any(UploadInfo.class)); + + UploadInfo finalUpload = new UploadInfo(); + finalUpload.setId(new UploadId("final-1")); + finalUpload.setOwnerKey("owner-1"); + finalUpload.setConcatenationPartIds(Arrays.asList("/part-1")); + + concatenationService.merge(finalUpload); + } + + @Test + public void testGetConcatenatedBytesTriggersMergeWhenStorageUploadIdIsNull() throws Exception { + UploadInfo p1 = new UploadInfo(); + p1.setId(new UploadId("part-1")); + p1.setLength(10L * 1024 * 1024); + p1.setOffset(10L * 1024 * 1024); + + Mockito.when(storageService.getUploadInfo("/part-1", "owner-1")).thenReturn(p1); + Mockito.when(storageService.getUploadedBytes(new UploadId("final-1"))) + .thenReturn(new ByteArrayInputStream(new byte[10])); + + UploadInfo finalUpload = new UploadInfo(); + finalUpload.setId(new UploadId("final-1")); + finalUpload.setOwnerKey("owner-1"); + finalUpload.setConcatenationPartIds(Arrays.asList("/part-1")); + + InputStream result = concatenationService.getConcatenatedBytes(finalUpload); + assertNotNull(result); + } + + @Test(expected = IOException.class) + public void testGetConcatenatedBytesWithoutStorageService() throws Exception { + S3ConcatenationService standalone = new S3ConcatenationService(minioClient, "test-bucket"); + UploadInfo info = new UploadInfo(); + info.setId(new UploadId("concat-1")); + info.setStorageUploadId("tus-uploads/concat-1"); + + standalone.getConcatenatedBytes(info); + } + + @Test + public void testGetConcatenatedBytesNull() throws Exception { + assertNull(concatenationService.getConcatenatedBytes(null)); + } +} diff --git a/src/test/java/me/desair/tus/server/upload/s3/S3LockingServiceTest.java b/src/test/java/me/desair/tus/server/upload/s3/S3LockingServiceTest.java new file mode 100644 index 00000000..2f0c1e4b --- /dev/null +++ b/src/test/java/me/desair/tus/server/upload/s3/S3LockingServiceTest.java @@ -0,0 +1,272 @@ +package me.desair.tus.server.upload.s3; + +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertFalse; +import static org.junit.Assert.assertNotNull; +import static org.junit.Assert.assertNull; +import static org.junit.Assert.assertTrue; + +import io.minio.GetObjectArgs; +import io.minio.GetObjectResponse; +import io.minio.ListObjectsArgs; +import io.minio.MinioClient; +import io.minio.PutObjectArgs; +import io.minio.Result; +import io.minio.StatObjectArgs; +import io.minio.errors.ErrorResponseException; +import io.minio.messages.ErrorResponse; +import io.minio.messages.Item; +import java.io.ByteArrayInputStream; +import java.io.IOException; +import java.io.InputStream; +import java.io.Serializable; +import java.nio.charset.StandardCharsets; +import me.desair.tus.server.upload.UploadId; +import me.desair.tus.server.upload.UploadLock; +import me.desair.tus.server.util.InterruptibleInputStream; +import org.junit.Before; +import org.junit.Test; +import org.mockito.Mockito; + +public class S3LockingServiceTest { + + private MinioClient minioClient; + private S3LockingService lockingService; + + @Before + public void setUp() { + minioClient = Mockito.mock(MinioClient.class); + lockingService = new S3LockingService(minioClient, "test-bucket"); + } + + @Test + public void testLockUploadByUriSuccessAndLockMethods() throws Exception { + Mockito.when(minioClient.putObject(Mockito.any(PutObjectArgs.class))).thenReturn(null); + + UploadLock lock = + lockingService.lockUploadByUri("/files/upload/24249a5b-01a4-4bf8-b67a-364273bb5a2e"); + assertNotNull(lock); + assertEquals("/files/upload/24249a5b-01a4-4bf8-b67a-364273bb5a2e", lock.getUploadUri()); + + if (lock instanceof S3UploadLock) { + S3UploadLock s3Lock = (S3UploadLock) lock; + assertNotNull(s3Lock.getHolderId()); + s3Lock.release(); + } + + lock.close(); + } + + @Test + public void testLockUploadByUriInvalidUri() throws Exception { + UploadLock lock = lockingService.lockUploadByUri("/invalid-uri"); + assertNull(lock); + } + + @Test + public void testIsLocked() throws Exception { + assertFalse(lockingService.isLocked((UploadId) null)); + + // Case 1: Lock object missing -> returns false + ErrorResponse errorResponse = Mockito.mock(ErrorResponse.class); + Mockito.when(errorResponse.code()).thenReturn("NoSuchKey"); + ErrorResponseException noSuchKeyEx = new ErrorResponseException(errorResponse, null, null); + + Mockito.when(minioClient.getObject(Mockito.any(GetObjectArgs.class))).thenThrow(noSuchKeyEx); + + UploadId uploadId = new UploadId("24249a5b-01a4-4bf8-b67a-364273bb5a2e"); + assertFalse(lockingService.isLocked(uploadId)); + + // Case 2: Active lock object present -> returns true + long futureExpiry = System.currentTimeMillis() + 600000L; + String lockJson = "{\"holderId\":\"h1\",\"expiresAt\":" + futureExpiry + "}"; + + Mockito.when(minioClient.getObject(Mockito.any(GetObjectArgs.class))) + .thenAnswer( + invocation -> + new GetObjectResponse( + null, + "test-bucket", + "us-east-1", + "tus-locks/24249a5b.lock", + new ByteArrayInputStream(lockJson.getBytes(StandardCharsets.UTF_8)))); + + assertTrue(lockingService.isLocked(uploadId)); + } + + @Test + public void testIsLockedReturnsFalseOnGenericMinioException() throws Exception { + Mockito.when(minioClient.getObject(Mockito.any(GetObjectArgs.class))) + .thenThrow(new RuntimeException("GetObject failure")); + + UploadId uploadId = new UploadId("24249a5b-01a4-4bf8-b67a-364273bb5a2e"); + assertFalse(lockingService.isLocked(uploadId)); + } + + @Test + public void testIsLockedReturnsFalseOnErrorResponseNon404() throws Exception { + ErrorResponse errorResponse = Mockito.mock(ErrorResponse.class); + Mockito.when(errorResponse.code()).thenReturn("AccessDenied"); + ErrorResponseException accessDeniedEx = new ErrorResponseException(errorResponse, null, null); + + Mockito.when(minioClient.getObject(Mockito.any(GetObjectArgs.class))).thenThrow(accessDeniedEx); + + UploadId uploadId = new UploadId("24249a5b-01a4-4bf8-b67a-364273bb5a2e"); + assertFalse(lockingService.isLocked(uploadId)); + } + + @Test + public void testLockAcquisitionFailures() throws Exception { + // Generic Exception during putObject + Mockito.when(minioClient.putObject(Mockito.any(PutObjectArgs.class))) + .thenThrow(new RuntimeException("S3 unreachable")); + + try { + lockingService.lockUploadByUri("/files/upload/24249a5b-01a4-4bf8-b67a-364273bb5a2e"); + } catch (Exception expected) { + } + } + + @Test + public void testRegisterInputStreamAndRequestLockReleaseWithInterruption() throws Exception { + lockingService.setIdFactory(new me.desair.tus.server.upload.UuidUploadIdFactory()); + + // Test with InterruptibleInputStream + InterruptibleInputStream interruptibleStream = Mockito.mock(InterruptibleInputStream.class); + + lockingService.registerInputStream( + "/files/upload/24249a5b-01a4-4bf8-b67a-364273bb5a2e", interruptibleStream); + + // Mock statObject to succeed (stop key exists) + Mockito.when(minioClient.statObject(Mockito.any(StatObjectArgs.class))) + .thenReturn(Mockito.mock(io.minio.StatObjectResponse.class)); + + lockingService.requestLockRelease("/files/upload/24249a5b-01a4-4bf8-b67a-364273bb5a2e"); + Mockito.verify(interruptibleStream).interrupt(); + + // Test with standard InputStream + InputStream standardStream = Mockito.mock(InputStream.class); + lockingService.registerInputStream( + "/files/upload/24249a5b-01a4-4bf8-b67a-364273bb5a2e", standardStream); + lockingService.requestLockRelease("/files/upload/24249a5b-01a4-4bf8-b67a-364273bb5a2e"); + Mockito.verify(standardStream).close(); + + lockingService.requestLockRelease(null); + } + + @Test + public void testCleanupStaleLocksWithExpiredItem() throws Exception { + Item item = Mockito.mock(Item.class); + Mockito.when(item.objectName()).thenReturn("tus-locks/expired.lock"); + + Result result = new Result<>(item); + Iterable> results = java.util.Collections.singletonList(result); + + Mockito.when(minioClient.listObjects(Mockito.any(ListObjectsArgs.class))).thenReturn(results); + + // Expired lock JSON + String expiredJson = + "{\"holderId\":\"h1\",\"expiresAt\":" + (System.currentTimeMillis() - 1000) + "}"; + GetObjectResponse response = + new GetObjectResponse( + null, + "test-bucket", + "us-east-1", + "tus-locks/expired.lock", + new ByteArrayInputStream(expiredJson.getBytes(StandardCharsets.UTF_8))); + + Mockito.when(minioClient.getObject(Mockito.any(GetObjectArgs.class))).thenReturn(response); + + lockingService.cleanupStaleLocks(); + } + + @Test(expected = IOException.class) + public void testCleanupStaleLocksThrowsIOExceptionOnMinioFailure() throws Exception { + Mockito.when(minioClient.listObjects(Mockito.any(ListObjectsArgs.class))) + .thenThrow(new RuntimeException("ListObjects failure")); + + lockingService.cleanupStaleLocks(); + } + + @Test(expected = me.desair.tus.server.exception.UploadAlreadyLockedException.class) + public void testLockUploadByUriThrowsUploadAlreadyLockedExceptionWhenPutObjectFails() + throws Exception { + Mockito.when(minioClient.putObject(Mockito.any(PutObjectArgs.class))) + .thenThrow(new RuntimeException("PutObject failure")); + + lockingService.lockUploadByUri("/files/upload/24249a5b-01a4-4bf8-b67a-364273bb5a2e"); + } + + @Test + public void testWriteStopSignalHandlesMinioException() throws Exception { + lockingService.setIdFactory(new me.desair.tus.server.upload.UuidUploadIdFactory()); + Mockito.when(minioClient.putObject(Mockito.any(PutObjectArgs.class))) + .thenThrow(new RuntimeException("PutObject failure for stop signal")); + + lockingService.requestLockRelease("/files/upload/24249a5b-01a4-4bf8-b67a-364273bb5a2e"); + } + + @Test + public void testDeleteObjectQuietlyHandlesMinioException() throws Exception { + Mockito.doThrow(new RuntimeException("RemoveObject failure")) + .when(minioClient) + .removeObject(Mockito.any(io.minio.RemoveObjectArgs.class)); + + UploadLock lock = + lockingService.lockUploadByUri("/files/upload/24249a5b-01a4-4bf8-b67a-364273bb5a2e"); + if (lock != null) { + lock.close(); + } + } + + @Test + public void testCheckStopSignalForEntryWithNullUploadId() throws Exception { + lockingService.setIdFactory( + new me.desair.tus.server.upload.UploadIdFactory() { + @Override + public me.desair.tus.server.upload.UploadId readUploadId(String text) { + return null; + } + + @Override + public me.desair.tus.server.upload.UploadId createId() { + return null; + } + + @Override + public String getUploadUri() { + return "/"; + } + + @Override + protected Serializable getIdValueIfValid(String extractedUrlId) { + throw new UnsupportedOperationException("Unimplemented method 'getIdValueIfValid'"); + } + }); + InputStream mockStream = Mockito.mock(InputStream.class); + lockingService.registerInputStream("/invalid-uri", mockStream); + lockingService.requestLockRelease("/invalid-uri"); + } + + @Test + public void testInterruptStreamStandardStreamCloseException() throws Exception { + lockingService.setIdFactory(new me.desair.tus.server.upload.UuidUploadIdFactory()); + InputStream brokenStream = Mockito.mock(InputStream.class); + Mockito.doThrow(new IOException("Close error")).when(brokenStream).close(); + + lockingService.registerInputStream( + "/files/upload/24249a5b-01a4-4bf8-b67a-364273bb5a2e", brokenStream); + lockingService.requestLockRelease("/files/upload/24249a5b-01a4-4bf8-b67a-364273bb5a2e"); + } + + @Test + public void testSanitizePrefixNullOrEmpty() throws Exception { + S3LockingService serviceWithEmptyPrefix = + new S3LockingService(minioClient, "test-bucket", "", 30000L, 0L); + assertNotNull(serviceWithEmptyPrefix); + + S3LockingService serviceWithNullPrefix = + new S3LockingService(minioClient, "test-bucket", null, 30000L, 0L); + assertNotNull(serviceWithNullPrefix); + } +} diff --git a/src/test/java/me/desair/tus/server/upload/s3/S3StorageServiceTest.java b/src/test/java/me/desair/tus/server/upload/s3/S3StorageServiceTest.java new file mode 100644 index 00000000..22d03209 --- /dev/null +++ b/src/test/java/me/desair/tus/server/upload/s3/S3StorageServiceTest.java @@ -0,0 +1,1261 @@ +package me.desair.tus.server.upload.s3; + +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertNotNull; +import static org.junit.Assert.assertNull; +import static org.junit.Assert.assertTrue; +import static org.mockito.ArgumentMatchers.any; +import static org.mockito.Mockito.doThrow; +import static org.mockito.Mockito.mock; +import static org.mockito.Mockito.when; + +import io.minio.ComposeObjectArgs; +import io.minio.GetObjectArgs; +import io.minio.GetObjectResponse; +import io.minio.ListObjectsArgs; +import io.minio.MinioClient; +import io.minio.PutObjectArgs; +import io.minio.Result; +import io.minio.StatObjectArgs; +import io.minio.StatObjectResponse; +import io.minio.errors.ErrorResponseException; +import io.minio.messages.ErrorResponse; +import io.minio.messages.Item; +import java.io.ByteArrayInputStream; +import java.io.ByteArrayOutputStream; +import java.io.IOException; +import java.io.InputStream; +import java.util.Arrays; +import me.desair.tus.server.checksum.ChecksumAlgorithm; +import me.desair.tus.server.exception.MinUploadLengthNotReachedException; +import me.desair.tus.server.upload.UploadId; +import me.desair.tus.server.upload.UploadInfo; +import me.desair.tus.server.upload.UploadLockingService; +import me.desair.tus.server.util.UploadInfoJsonSerializer; +import org.junit.Before; +import org.junit.Test; + +public class S3StorageServiceTest { + + private MinioClient minioClient; + private S3StorageService storageService; + + @Before + public void setUp() { + minioClient = mock(MinioClient.class); + storageService = new S3StorageService(minioClient, "test-bucket"); + } + + @Test + public void testCreateUpload() throws Exception { + UploadInfo info = new UploadInfo(); + info.setId(new UploadId("24249a5b-01a4-4bf8-b67a-364273bb5a2e")); + info.setLength(1024L); + + UploadInfo created = storageService.create(info, "owner-1"); + + assertNotNull(created); + assertEquals("24249a5b-01a4-4bf8-b67a-364273bb5a2e", created.getStorageUploadId()); + assertEquals("owner-1", created.getOwnerKey()); + assertEquals( + "tus-uploads/24249a5b-01a4-4bf8-b67a-364273bb5a2e", storageService.getS3ObjectKey(created)); + } + + @Test + public void testNullTemporaryDirectoryConstructor() { + java.nio.file.Path nullPath = null; + S3StorageService serviceWithNullTmp = + new S3StorageService( + minioClient, + "test-bucket", + "tus-uploads/", + "tus-uploads/", + "checksums/", + "locks/", + nullPath); + assertNotNull(serviceWithNullTmp); + } + + @Test + public void testGetS3ObjectKeyByUri() throws Exception { + UploadInfo info = new UploadInfo(); + info.setId(new UploadId("24249a5b-01a4-4bf8-b67a-364273bb5a2e")); + info.setStorageUploadId("tus-uploads/custom-key-123"); + info.setOwnerKey("owner-1"); + + String json = UploadInfoJsonSerializer.serialize(info); + when(minioClient.getObject(any(GetObjectArgs.class))) + .thenAnswer(invocation -> mockGetObjectResponse(json.getBytes())); + + storageService.setIdFactory(new me.desair.tus.server.upload.UuidUploadIdFactory()); + + String keyByUri = + storageService.getS3ObjectKey("/files/upload/24249a5b-01a4-4bf8-b67a-364273bb5a2e"); + assertEquals("tus-uploads/custom-key-123", keyByUri); + + String keyByUriAndOwner = + storageService.getS3ObjectKey( + "/files/upload/24249a5b-01a4-4bf8-b67a-364273bb5a2e", "owner-1"); + assertEquals("tus-uploads/custom-key-123", keyByUriAndOwner); + } + + @Test + public void testGetS3ObjectKeyByUriExceptions() throws Exception { + when(minioClient.getObject(any(GetObjectArgs.class))) + .thenThrow(new RuntimeException("MinIO failure")); + + storageService.setIdFactory(new me.desair.tus.server.upload.UuidUploadIdFactory()); + + assertNull(storageService.getS3ObjectKey("/files/upload/24249a5b-01a4-4bf8-b67a-364273bb5a2e")); + assertNull( + storageService.getS3ObjectKey( + "/files/upload/24249a5b-01a4-4bf8-b67a-364273bb5a2e", "owner-1")); + } + + @Test(expected = me.desair.tus.server.exception.UploadNotFoundException.class) + public void testGetUploadedBytesNotFound() throws Exception { + ErrorResponse errorResponse = mock(ErrorResponse.class); + when(errorResponse.code()).thenReturn("NoSuchKey"); + ErrorResponseException ex = new ErrorResponseException(errorResponse, null, null); + when(minioClient.getObject(any(GetObjectArgs.class))).thenThrow(ex); + + storageService.getUploadedBytes(new UploadId("missing-123")); + } + + @Test + public void testGetUploadedBytesDuplicate() throws Exception { + UploadInfo child = new UploadInfo(); + child.setId(new UploadId("child-123")); + child.setDuplicatesUploadId(new UploadId("parent-456")); + + UploadInfo parent = new UploadInfo(); + parent.setId(new UploadId("parent-456")); + + String childJson = UploadInfoJsonSerializer.serialize(child); + String parentJson = UploadInfoJsonSerializer.serialize(parent); + + when(minioClient.getObject(any(GetObjectArgs.class))) + .thenAnswer( + invocation -> { + GetObjectArgs args = invocation.getArgument(0); + if (args.object().contains("child-123")) { + return mockGetObjectResponse(childJson.getBytes()); + } + if (args.object().contains("parent-456.info")) { + return mockGetObjectResponse(parentJson.getBytes()); + } + return mockGetObjectResponse("parent-data".getBytes()); + }); + + InputStream stream = storageService.getUploadedBytes(new UploadId("child-123")); + assertNotNull(stream); + } + + @Test + public void testGetUploadedBytesConcatenatedUnmerged() throws Exception { + UploadInfo info = new UploadInfo(); + info.setId(new UploadId("concat-123")); + info.setUploadType(me.desair.tus.server.upload.UploadType.CONCATENATED); + info.setStorageUploadId(null); + + UploadInfo mergedInfo = new UploadInfo(); + mergedInfo.setId(new UploadId("concat-123")); + mergedInfo.setStorageUploadId("tus-uploads/concat-123"); + + String jsonBefore = UploadInfoJsonSerializer.serialize(info); + String jsonAfter = UploadInfoJsonSerializer.serialize(mergedInfo); + + java.util.concurrent.atomic.AtomicInteger infoCallCount = + new java.util.concurrent.atomic.AtomicInteger(); + + when(minioClient.getObject(any(GetObjectArgs.class))) + .thenAnswer( + invocation -> { + GetObjectArgs args = invocation.getArgument(0); + if (args.object().endsWith(".info")) { + if (infoCallCount.getAndIncrement() == 0) { + return mockGetObjectResponse(jsonBefore.getBytes()); + } + return mockGetObjectResponse(jsonAfter.getBytes()); + } + return mockGetObjectResponse("merged-bytes".getBytes()); + }); + + S3ConcatenationService mockConcat = mock(S3ConcatenationService.class); + storageService.setUploadConcatenationService(mockConcat); + + InputStream stream = storageService.getUploadedBytes(new UploadId("concat-123")); + assertNotNull(stream); + } + + @Test(expected = me.desair.tus.server.exception.MaxAppendSizeExceededException.class) + public void testAppendExceedsMaxAppendSizeLimit() throws Exception { + UploadInfo info = new UploadInfo(); + info.setId(new UploadId("24249a5b-01a4-4bf8-b67a-364273bb5a2e")); + info.setLength(10000L); + + String json = UploadInfoJsonSerializer.serialize(info); + when(minioClient.getObject(any(GetObjectArgs.class))) + .thenAnswer(invocation -> mockGetObjectResponse(json.getBytes())); + + storageService.setMaxAppendSize(50L); + storageService.append(info, new ByteArrayInputStream(new byte[100])); + } + + @Test(expected = MinUploadLengthNotReachedException.class) + public void testAppendBelowMinSize() throws Exception { + UploadInfo info = new UploadInfo(); + info.setId(new UploadId("24249a5b-01a4-4bf8-b67a-364273bb5a2e")); + info.setLength(1000L); + + String json = UploadInfoJsonSerializer.serialize(info); + GetObjectResponse stream = mockGetObjectResponse(json.getBytes()); + + when(minioClient.getObject(any(GetObjectArgs.class))).thenReturn(stream); + + storageService.setMinSize(2000L); + storageService.append(info, new ByteArrayInputStream(new byte[100])); + } + + @Test(expected = IOException.class) + public void testAppendThrowsIOExceptionOnStreamError() throws Exception { + UploadInfo info = new UploadInfo(); + info.setId(new UploadId("24249a5b-01a4-4bf8-b67a-364273bb5a2e")); + info.setLength(1000L); + + String json = UploadInfoJsonSerializer.serialize(info); + when(minioClient.getObject(any(GetObjectArgs.class))) + .thenAnswer(invocation -> mockGetObjectResponse(json.getBytes())); + + InputStream brokenStream = mock(InputStream.class); + when(brokenStream.read(any(byte[].class))).thenThrow(new IOException("Read failed")); + + storageService.append(info, brokenStream); + } + + @Test + public void testGetUploadInfoReturnsNullForMissingKey() throws Exception { + ErrorResponse errorResponse = mock(ErrorResponse.class); + when(errorResponse.code()).thenReturn("NoSuchKey"); + + ErrorResponseException ex = new ErrorResponseException(errorResponse, null, null); + + when(minioClient.getObject(any(GetObjectArgs.class))).thenThrow(ex); + + UploadInfo result = + storageService.getUploadInfo(new UploadId("24249a5b-01a4-4bf8-b67a-364273bb5a2e")); + assertNull(result); + } + + @Test(expected = IOException.class) + public void testGetUploadInfoThrowsIOExceptionOnGenericException() throws Exception { + when(minioClient.getObject(any(GetObjectArgs.class))) + .thenThrow(new RuntimeException("Storage failure")); + + storageService.getUploadInfo(new UploadId("24249a5b-01a4-4bf8-b67a-364273bb5a2e")); + } + + @Test(expected = IOException.class) + public void testGetUploadInfoThrowsIOExceptionOnErrorResponseNon404() throws Exception { + ErrorResponse errorResponse = mock(ErrorResponse.class); + when(errorResponse.code()).thenReturn("AccessDenied"); + + ErrorResponseException ex = new ErrorResponseException(errorResponse, null, null); + when(minioClient.getObject(any(GetObjectArgs.class))).thenThrow(ex); + + storageService.getUploadInfo(new UploadId("24249a5b-01a4-4bf8-b67a-364273bb5a2e")); + } + + @Test + public void testCopyUploadToAndRemoveLastBytes() throws Exception { + UploadInfo info = new UploadInfo(); + info.setId(new UploadId("24249a5b-01a4-4bf8-b67a-364273bb5a2e")); + info.setLength(100L); + info.setOffset(100L); + + String json = UploadInfoJsonSerializer.serialize(info); + byte[] payload = new byte[100]; + + when(minioClient.getObject(any(GetObjectArgs.class))) + .thenAnswer( + invocation -> { + GetObjectArgs args = invocation.getArgument(0); + if (args.object().endsWith(".info")) { + return mockGetObjectResponse(json.getBytes()); + } + return mockGetObjectResponse(payload); + }); + + ByteArrayOutputStream baos = new ByteArrayOutputStream(); + storageService.copyUploadTo(info, baos); + assertEquals(100, baos.size()); + + when(minioClient.statObject(any())).thenReturn(mock(StatObjectResponse.class)); + + // Verify removeLastNumberOfBytes updates offset + storageService.removeLastNumberOfBytes(info, 5); + assertEquals(Long.valueOf(95L), info.getOffset()); + + // Test removeLastNumberOfBytes with byteCount <= 0 + storageService.removeLastNumberOfBytes(info, 0); + } + + @Test + public void testTruncateIncompletePartPartial() throws Exception { + UploadInfo info = new UploadInfo(); + info.setId(new UploadId("24249a5b-01a4-4bf8-b67a-364273bb5a2e")); + info.setOffset(100L); + info.setLength(1000L); + + byte[] partBytes = new byte[100]; + StatObjectResponse mockHead = mock(StatObjectResponse.class); + when(mockHead.size()).thenReturn(100L); + + when(minioClient.statObject(any(StatObjectArgs.class))).thenReturn(mockHead); + when(minioClient.getObject(any(GetObjectArgs.class))) + .thenAnswer(invocation -> mockGetObjectResponse(partBytes)); + + storageService.removeLastNumberOfBytes(info, 5); + assertEquals(Long.valueOf(95L), info.getOffset()); + } + + @Test + public void testCalculateAndSetOffsetWhenCompletedObjectExists() throws Exception { + String json = "{\"id\":\"24249a5b-01a4-4bf8-b67a-364273bb5a2e\",\"length\":1000}"; + StatObjectResponse mockHead = mock(StatObjectResponse.class); + when(mockHead.size()).thenReturn(1000L); + + when(minioClient.getObject(any(GetObjectArgs.class))) + .thenAnswer(invocation -> mockGetObjectResponse(json.getBytes())); + when(minioClient.statObject(any(StatObjectArgs.class))).thenReturn(mockHead); + + UploadInfo fetched = + storageService.getUploadInfo(new UploadId("24249a5b-01a4-4bf8-b67a-364273bb5a2e")); + assertNotNull(fetched); + } + + @Test + public void testGetUploadInfoWithNullOffsetCalculatesOffset() throws Exception { + String json = + "{\"id\":\"24249a5b-01a4-4bf8-b67a-364273bb5a2e\",\"length\":1000,\"offset\":null}"; + StatObjectResponse mockHead = mock(StatObjectResponse.class); + when(mockHead.size()).thenReturn(500L); + + when(minioClient.getObject(any(GetObjectArgs.class))) + .thenAnswer(invocation -> mockGetObjectResponse(json.getBytes())); + + when(minioClient.statObject(any(StatObjectArgs.class))) + .thenAnswer( + invocation -> { + StatObjectArgs args = invocation.getArgument(0); + + // Check if objects ends with ".part" + if (args.object().endsWith(".part")) { + // Simulate that the part does not exist by throwing a NoSuchKey exception + ErrorResponse errorResponse = mock(ErrorResponse.class); + when(errorResponse.code()).thenReturn("NoSuchKey"); + throw new ErrorResponseException(errorResponse, null, null); + } + + return mockHead; + }); + + UploadInfo fetched = + storageService.getUploadInfo(new UploadId("24249a5b-01a4-4bf8-b67a-364273bb5a2e")); + assertNotNull(fetched); + assertEquals(Long.valueOf(500L), fetched.getOffset()); + } + + @Test + public void testAppendCompletingUploadWithLeftoverPart() throws Exception { + UploadInfo info = new UploadInfo(); + info.setId(new UploadId("24249a5b-01a4-4bf8-b67a-364273bb5a2e")); + info.setLength(100L); + info.setOffset(50L); + + String jsonBefore = UploadInfoJsonSerializer.serialize(info); + info.setOffset(100L); + String jsonAfter = UploadInfoJsonSerializer.serialize(info); + info.setOffset(50L); + + byte[] payload = new byte[50]; + java.util.concurrent.atomic.AtomicInteger infoCallCount = + new java.util.concurrent.atomic.AtomicInteger(); + + when(minioClient.getObject(any(GetObjectArgs.class))) + .thenAnswer( + invocation -> { + GetObjectArgs args = invocation.getArgument(0); + if (args.object().endsWith(".info")) { + if (infoCallCount.getAndIncrement() == 0) { + return mockGetObjectResponse(jsonBefore.getBytes()); + } + return mockGetObjectResponse(jsonAfter.getBytes()); + } + return mockGetObjectResponse(payload); + }); + + when(minioClient.statObject(any(StatObjectArgs.class))) + .thenAnswer( + invocation -> { + StatObjectArgs args = invocation.getArgument(0); + if (args.object().endsWith(".part")) { + StatObjectResponse resp = mock(StatObjectResponse.class); + when(resp.size()).thenReturn(50L); + return resp; + } + ErrorResponse errorResponse = mock(ErrorResponse.class); + when(errorResponse.code()).thenReturn("NoSuchKey"); + throw new ErrorResponseException(errorResponse, null, null); + }); + + storageService.append(info, new ByteArrayInputStream(payload)); + } + + @Test + public void testFetchS3ByteStreamWithOffsetAndLengthRange() throws Exception { + UploadInfo info = new UploadInfo(); + info.setId(new UploadId("24249a5b-01a4-4bf8-b67a-364273bb5a2e")); + String json = UploadInfoJsonSerializer.serialize(info); + + when(minioClient.getObject(any(GetObjectArgs.class))) + .thenAnswer( + invocation -> { + GetObjectArgs args = invocation.getArgument(0); + if (args.object().endsWith(".info")) { + return mockGetObjectResponse(json.getBytes()); + } + return mockGetObjectResponse("ranged-payload".getBytes()); + }); + + InputStream stream = storageService.getUploadedBytes(info.getId()); + assertNotNull(stream); + } + + @Test(expected = IOException.class) + public void testGetUploadInfoByChecksumThrowsIOExceptionOnErrorResponseNon404() throws Exception { + storageService.setUploadDeduplicationEnabled(true); + + ErrorResponse errorResponse = mock(ErrorResponse.class); + when(errorResponse.code()).thenReturn("AccessDenied"); + ErrorResponseException ex = new ErrorResponseException(errorResponse, null, null); + + when(minioClient.getObject(any(GetObjectArgs.class))).thenThrow(ex); + + storageService.getUploadInfoByChecksum("abc123hash", ChecksumAlgorithm.SHA256); + } + + @Test(expected = IOException.class) + public void testGetUploadInfoByChecksumThrowsIOExceptionOnGenericException() throws Exception { + storageService.setUploadDeduplicationEnabled(true); + + when(minioClient.getObject(any(GetObjectArgs.class))) + .thenThrow(new RuntimeException("MinIO failure")); + + storageService.getUploadInfoByChecksum("abc123hash", ChecksumAlgorithm.SHA256); + } + + @Test + public void testDeduplicationChecksumLookupSelfCleaningWhenParentMissing() throws Exception { + storageService.setUploadDeduplicationEnabled(true); + + when(minioClient.getObject(any(GetObjectArgs.class))) + .thenReturn(mockGetObjectResponse("stale-parent-456".getBytes())); + + ErrorResponse errorResponse = mock(ErrorResponse.class); + when(errorResponse.code()).thenReturn("NoSuchKey"); + ErrorResponseException noSuchKey = new ErrorResponseException(errorResponse, null, null); + when(minioClient.statObject(any(StatObjectArgs.class))).thenThrow(noSuchKey); + + UploadInfo match = + storageService.getUploadInfoByChecksum("stalehash", ChecksumAlgorithm.SHA256); + assertNull(match); + } + + @Test + public void testDeduplicationChecksumLookup() throws Exception { + storageService.setUploadDeduplicationEnabled(true); + + UploadInfo parentInfo = new UploadInfo(); + parentInfo.setId(new UploadId("parent-123")); + parentInfo.setLength(5000L); + + String json = UploadInfoJsonSerializer.serialize(parentInfo); + + java.util.Map objectData = new java.util.HashMap<>(); + objectData.put("checksums/sha256/abc123hash", "parent-123".getBytes()); + objectData.put("tus-uploads/checksums/sha256/abc123hash", "parent-123".getBytes()); + objectData.put("tus-uploads/parent-123.info", json.getBytes()); + + when(minioClient.getObject(any(GetObjectArgs.class))) + .thenAnswer( + invocation -> { + GetObjectArgs args = invocation.getArgument(0); + byte[] data = objectData.get(args.object()); + if (data != null) { + return mockGetObjectResponse(data); + } + return mockGetObjectResponse(json.getBytes()); + }); + + when(minioClient.statObject(any(StatObjectArgs.class))) + .thenReturn(mock(StatObjectResponse.class)); + + UploadInfo match = + storageService.getUploadInfoByChecksum("abc123hash", ChecksumAlgorithm.SHA256); + assertNotNull(match); + assertEquals(new UploadId("parent-123"), match.getId()); + } + + @Test + public void testConfigurationSettersAndGetters() { + storageService.setMaxUploadSize(5000L); + assertEquals(5000L, storageService.getMaxUploadSize()); + + storageService.setMaxAppendSize(3000L); + assertEquals(Long.valueOf(3000L), storageService.getMaxAppendSize()); + + storageService.setMinAppendSize(100L); + assertEquals(Long.valueOf(100L), storageService.getMinAppendSize()); + + storageService.setMinSize(50L); + assertEquals(Long.valueOf(50L), storageService.getMinSize()); + + storageService.setUploadExpirationPeriod(86400000L); + assertEquals(Long.valueOf(86400000L), storageService.getUploadExpirationPeriod()); + + storageService.setUploadDeduplicationEnabled(true); + assertTrue(storageService.isUploadDeduplicationEnabled()); + + storageService.setIdFactory(new me.desair.tus.server.upload.UuidUploadIdFactory()); + + S3ConcatenationService concat = new S3ConcatenationService(minioClient, "test-bucket"); + storageService.setUploadConcatenationService(concat); + assertEquals(concat, storageService.getUploadConcatenationService()); + + assertNotNull(storageService.getUploadUri()); + } + + @Test + public void testNullUploadOperations() throws Exception { + assertNull(storageService.getUploadInfo((UploadId) null)); + assertNull(storageService.getUploadInfo((String) null, null)); + assertNull(storageService.getS3ObjectKey((UploadInfo) null)); + assertNull(storageService.getS3ObjectKey((String) null)); + + storageService.update(null); + storageService.removeLastNumberOfBytes(null, 100); + storageService.terminateUpload(null); + + assertNull(storageService.getUploadInfoByChecksum(null, null)); + assertNull(storageService.getUploadInfoByChecksum("abc", ChecksumAlgorithm.SHA256)); + } + + @Test(expected = me.desair.tus.server.exception.MinAppendSizeNotMetException.class) + public void testAppendThrowsMinAppendSizeNotMetException() throws Exception { + UploadInfo info = new UploadInfo(); + info.setId(new UploadId("24249a5b-01a4-4bf8-b67a-364273bb5a2e")); + info.setLength(1000L); + + String json = UploadInfoJsonSerializer.serialize(info); + when(minioClient.getObject(any(GetObjectArgs.class))) + .thenAnswer(invocation -> mockGetObjectResponse(json.getBytes())); + + storageService.setMinAppendSize(500L); + storageService.append(info, new ByteArrayInputStream(new byte[100])); + } + + @Test(expected = me.desair.tus.server.exception.MaxUploadLengthExceededException.class) + public void testAppendThrowsMaxUploadLengthExceededException() throws Exception { + UploadInfo info = new UploadInfo(); + info.setId(new UploadId("24249a5b-01a4-4bf8-b67a-364273bb5a2e")); + info.setLength(2000L); + + String json = UploadInfoJsonSerializer.serialize(info); + when(minioClient.getObject(any(GetObjectArgs.class))) + .thenAnswer(invocation -> mockGetObjectResponse(json.getBytes())); + + storageService.setMaxUploadSize(1000L); + storageService.append(info, new ByteArrayInputStream(new byte[100])); + } + + @Test(expected = me.desair.tus.server.exception.UploadNotFoundException.class) + public void testGetUploadedBytesByUriNotFoundThrowsException() throws Exception { + ErrorResponse errorResponse = mock(ErrorResponse.class); + when(errorResponse.code()).thenReturn("NoSuchKey"); + ErrorResponseException ex = new ErrorResponseException(errorResponse, null, null); + when(minioClient.getObject(any(GetObjectArgs.class))).thenThrow(ex); + + storageService.getUploadedBytes("/files/upload/non-existent-id", null); + } + + @Test(expected = me.desair.tus.server.exception.UploadNotFoundException.class) + public void testAppendByUploadIdNotFoundThrowsException() throws Exception { + ErrorResponse errorResponse = mock(ErrorResponse.class); + when(errorResponse.code()).thenReturn("NoSuchKey"); + ErrorResponseException ex = new ErrorResponseException(errorResponse, null, null); + when(minioClient.getObject(any(GetObjectArgs.class))).thenThrow(ex); + + UploadInfo info = new UploadInfo(); + info.setId(new UploadId("non-existent-id")); + + storageService.append(info, new ByteArrayInputStream(new byte[100])); + } + + @Test(expected = me.desair.tus.server.exception.UploadNotFoundException.class) + public void testCopyUploadToNotFoundThrowsUploadNotFoundException() throws Exception { + UploadInfo info = new UploadInfo(); + info.setId(new UploadId("24249a5b-01a4-4bf8-b67a-364273bb5a2e")); + info.setOffset(100L); + + ErrorResponse errorResponse = mock(ErrorResponse.class); + when(errorResponse.code()).thenReturn("NoSuchKey"); + ErrorResponseException ex = new ErrorResponseException(errorResponse, null, null); + when(minioClient.getObject(any(GetObjectArgs.class))).thenThrow(ex); + + ByteArrayOutputStream baos = new ByteArrayOutputStream(); + storageService.copyUploadTo(info, baos); + } + + @Test + public void testCleanupExpiredUploads() throws Exception { + UploadInfo expiredInfo = new UploadInfo(); + UploadId expiredId = new UploadId("expired-123"); + expiredInfo.setId(expiredId); + expiredInfo.setExpirationTimestamp(System.currentTimeMillis() - 10000L); + + String json = UploadInfoJsonSerializer.serialize(expiredInfo); + + Item item = mock(Item.class); + when(item.objectName()).thenReturn("tus-uploads/expired-123.info"); + Result result = new Result<>(item); + when(minioClient.listObjects(any(ListObjectsArgs.class))) + .thenReturn(java.util.Collections.singletonList(result)); + + when(minioClient.getObject(any(GetObjectArgs.class))) + .thenAnswer(invocation -> mockGetObjectResponse(json.getBytes())); + + UploadLockingService mockLocking = mock(UploadLockingService.class); + when(mockLocking.isLocked(expiredId)).thenReturn(false); + + storageService.cleanupExpiredUploads(mockLocking); + } + + @Test(expected = IOException.class) + public void testCleanupExpiredUploadsThrowsIOExceptionOnMinioFailure() throws Exception { + when(minioClient.listObjects(any(ListObjectsArgs.class))) + .thenThrow(new RuntimeException("MinIO failure")); + + storageService.cleanupExpiredUploads(null); + } + + @Test + public void testFinalizeCompletedUploadWithMultipleParts() throws Exception { + UploadInfo info = new UploadInfo(); + UploadId id = new UploadId("multi-part-123"); + info.setId(id); + info.setLength(100L); + info.setOffset(0L); + + String json = UploadInfoJsonSerializer.serialize(info); + + Item item1 = mock(Item.class); + when(item1.objectName()).thenReturn("tus-uploads/multi-part-123.part.00001"); + Item item2 = mock(Item.class); + when(item2.objectName()).thenReturn("tus-uploads/multi-part-123.part.00002"); + + when(minioClient.listObjects(any(ListObjectsArgs.class))) + .thenReturn(Arrays.asList(new Result<>(item1), new Result<>(item2))); + + when(minioClient.getObject(any(GetObjectArgs.class))) + .thenAnswer(invocation -> mockGetObjectResponse(json.getBytes())); + + storageService.append(info, new ByteArrayInputStream(new byte[100])); + } + + @Test + public void testFinalizeCompletedUploadWithLeftoverIncompletePart() throws Exception { + UploadInfo info = new UploadInfo(); + UploadId id = new UploadId("leftover-part-123"); + info.setId(id); + info.setLength(50L); + info.setOffset(0L); + + String json = UploadInfoJsonSerializer.serialize(info); + + StatObjectResponse leftoverHead = mock(StatObjectResponse.class); + when(leftoverHead.size()).thenReturn(50L); + + when(minioClient.statObject(any(StatObjectArgs.class))) + .thenAnswer( + invocation -> { + StatObjectArgs args = invocation.getArgument(0); + if (args.object().endsWith(".part")) { + return leftoverHead; + } + ErrorResponse err = mock(ErrorResponse.class); + when(err.code()).thenReturn("NoSuchKey"); + throw new ErrorResponseException(err, null, null); + }); + + when(minioClient.getObject(any(GetObjectArgs.class))) + .thenAnswer( + invocation -> { + GetObjectArgs args = invocation.getArgument(0); + if (args.object().endsWith(".info")) { + return mockGetObjectResponse(json.getBytes()); + } + return mockGetObjectResponse(new byte[50]); + }); + + storageService.append(info, new ByteArrayInputStream(new byte[50])); + } + + @Test + public void testFinalizeCompletedUploadZeroLength() throws Exception { + UploadInfo info = new UploadInfo(); + UploadId id = new UploadId("zero-len-123"); + info.setId(id); + info.setLength(0L); + info.setOffset(0L); + + String json = UploadInfoJsonSerializer.serialize(info); + when(minioClient.getObject(any(GetObjectArgs.class))) + .thenAnswer(invocation -> mockGetObjectResponse(json.getBytes())); + + storageService.append(info, new ByteArrayInputStream(new byte[0])); + } + + @Test + public void testTruncateFromCompletedObject() throws Exception { + UploadInfo info = new UploadInfo(); + UploadId id = new UploadId("trunc-completed-123"); + info.setId(id); + info.setLength(100L); + info.setOffset(100L); + + StatObjectResponse mockHead = mock(StatObjectResponse.class); + when(mockHead.size()).thenReturn(100L); + when(minioClient.statObject(any(StatObjectArgs.class))).thenReturn(mockHead); + + when(minioClient.getObject(any(GetObjectArgs.class))) + .thenAnswer(invocation -> mockGetObjectResponse(new byte[100])); + + storageService.removeLastNumberOfBytes(info, 30L); + assertEquals(Long.valueOf(70L), info.getOffset()); + } + + @Test + public void testTruncateFromIncompletePartByteCountGreaterThanPartSize() throws Exception { + UploadInfo info = new UploadInfo(); + UploadId id = new UploadId("trunc-inc-123"); + info.setId(id); + info.setOffset(50L); + + ErrorResponse noSuchKeyErr = mock(ErrorResponse.class); + when(noSuchKeyErr.code()).thenReturn("NoSuchKey"); + ErrorResponseException noSuchKeyEx = new ErrorResponseException(noSuchKeyErr, null, null); + + StatObjectResponse partHead = mock(StatObjectResponse.class); + when(partHead.size()).thenReturn(50L); + + when(minioClient.statObject(any(StatObjectArgs.class))) + .thenAnswer( + invocation -> { + StatObjectArgs args = invocation.getArgument(0); + if (args.object().endsWith(".part")) { + return partHead; + } + throw noSuchKeyEx; + }); + + storageService.removeLastNumberOfBytes(info, 100L); + assertEquals(Long.valueOf(0L), info.getOffset()); + } + + @Test + public void testTruncateFromIncompletePartPartialBytes() throws Exception { + UploadInfo info = new UploadInfo(); + UploadId id = new UploadId("trunc-part-123"); + info.setId(id); + info.setOffset(100L); + + ErrorResponse noSuchKeyErr = mock(ErrorResponse.class); + when(noSuchKeyErr.code()).thenReturn("NoSuchKey"); + ErrorResponseException noSuchKeyEx = new ErrorResponseException(noSuchKeyErr, null, null); + + StatObjectResponse partHead = mock(StatObjectResponse.class); + when(partHead.size()).thenReturn(100L); + + when(minioClient.statObject(any(StatObjectArgs.class))) + .thenAnswer( + invocation -> { + StatObjectArgs args = invocation.getArgument(0); + if (args.object().endsWith(".part")) { + return partHead; + } + throw noSuchKeyEx; + }); + + when(minioClient.getObject(any(GetObjectArgs.class))) + .thenAnswer(invocation -> mockGetObjectResponse(new byte[100])); + + storageService.removeLastNumberOfBytes(info, 30L); + assertEquals(Long.valueOf(70L), info.getOffset()); + } + + @Test + public void testTerminateUploadWithChecksumAndParts() throws Exception { + UploadInfo info = new UploadInfo(); + UploadId id = new UploadId("term-123"); + info.setId(id); + info.setChecksum("hash123"); + info.setChecksumAlgorithm(ChecksumAlgorithm.SHA256); + + Item item1 = mock(Item.class); + when(item1.objectName()).thenReturn("tus-uploads/term-123.part.00001"); + when(minioClient.listObjects(any(ListObjectsArgs.class))) + .thenReturn(java.util.Collections.singletonList(new Result<>(item1))); + + storageService.terminateUpload(info); + } + + @Test + public void testFetchS3ByteStreamIncompletePartFallback() throws Exception { + UploadInfo info = new UploadInfo(); + UploadId id = new UploadId("fallback-123"); + info.setId(id); + info.setOffset(50L); + + ErrorResponse noSuchKeyErr = mock(ErrorResponse.class); + when(noSuchKeyErr.code()).thenReturn("NoSuchKey"); + ErrorResponseException noSuchKeyEx = new ErrorResponseException(noSuchKeyErr, null, null); + + when(minioClient.getObject(any(GetObjectArgs.class))) + .thenAnswer( + invocation -> { + GetObjectArgs args = invocation.getArgument(0); + if (args.object().endsWith(".info")) { + return mockGetObjectResponse(UploadInfoJsonSerializer.serialize(info).getBytes()); + } + if (args.object().endsWith(".part")) { + return mockGetObjectResponse("part-data".getBytes()); + } + throw noSuchKeyEx; + }); + + InputStream stream = storageService.getUploadedBytes(id); + assertNotNull(stream); + } + + @Test + public void testFetchS3ByteStreamZeroOffsetFallback() throws Exception { + UploadInfo info = new UploadInfo(); + UploadId id = new UploadId("zero-offset-123"); + info.setId(id); + info.setOffset(0L); + + ErrorResponse noSuchKeyErr = mock(ErrorResponse.class); + when(noSuchKeyErr.code()).thenReturn("NoSuchKey"); + ErrorResponseException noSuchKeyEx = new ErrorResponseException(noSuchKeyErr, null, null); + + when(minioClient.getObject(any(GetObjectArgs.class))) + .thenAnswer( + invocation -> { + GetObjectArgs args = invocation.getArgument(0); + if (args.object().endsWith(".info")) { + return mockGetObjectResponse(UploadInfoJsonSerializer.serialize(info).getBytes()); + } + throw noSuchKeyEx; + }); + + InputStream stream = storageService.getUploadedBytes(id); + assertNotNull(stream); + assertEquals(0, stream.available()); + } + + @Test + public void testPutChecksumIndexAndObjectExistsExceptions() throws Exception { + storageService.setUploadDeduplicationEnabled(true); + + UploadInfo info = new UploadInfo(); + info.setId(new UploadId("chk-123")); + info.setLength(100L); + info.setOffset(100L); + info.setChecksum("hashabc"); + info.setChecksumAlgorithm(ChecksumAlgorithm.SHA256); + + when(minioClient.putObject(any(PutObjectArgs.class))) + .thenAnswer( + invocation -> { + PutObjectArgs args = invocation.getArgument(0); + if (args.object().contains("checksums")) { + throw new RuntimeException("Checksum put failure"); + } + return null; + }); + + storageService.update(info); + } + + @Test(expected = IOException.class) + public void testUpdateThrowsIOExceptionOnMinioFailure() throws Exception { + UploadInfo info = new UploadInfo(); + info.setId(new UploadId("upd-err-123")); + when(minioClient.putObject(any(PutObjectArgs.class))) + .thenThrow(new RuntimeException("PutObject failure")); + storageService.update(info); + } + + @Test + public void testCreateWhenUpdateThrowsUploadNotFoundException() throws Exception { + S3StorageService spyService = org.mockito.Mockito.spy(storageService); + UploadInfo info = new UploadInfo(); + org.mockito.Mockito.doThrow( + new me.desair.tus.server.exception.UploadNotFoundException("Not found")) + .when(spyService) + .update(any(UploadInfo.class)); + UploadInfo created = spyService.create(info, "owner"); + assertNotNull(created); + } + + @Test(expected = IOException.class) + public void testGetUploadInfoByChecksumWithNonNoSuchKeyError() throws Exception { + storageService.setUploadDeduplicationEnabled(true); + ErrorResponse err = mock(ErrorResponse.class); + when(err.code()).thenReturn("AccessDenied"); + ErrorResponseException ex = new ErrorResponseException(err, null, null); + + when(minioClient.getObject(any(GetObjectArgs.class))).thenThrow(ex); + + storageService.getUploadInfoByChecksum("hash", ChecksumAlgorithm.SHA1); + } + + @Test(expected = IOException.class) + public void testGetUploadInfoByChecksumWithGenericException() throws Exception { + storageService.setUploadDeduplicationEnabled(true); + when(minioClient.getObject(any(GetObjectArgs.class))) + .thenThrow(new RuntimeException("MinIO error")); + + storageService.getUploadInfoByChecksum("hash", ChecksumAlgorithm.SHA1); + } + + @Test + public void testPrepareStreamWithExistingIncompletePartGenericException() throws Exception { + UploadInfo info = new UploadInfo(); + UploadId id = new UploadId("prep-err-123"); + info.setId(id); + info.setLength(100L); + info.setOffset(0L); + + when(minioClient.getObject(any(GetObjectArgs.class))) + .thenAnswer( + inv -> { + GetObjectArgs args = inv.getArgument(0); + if (args.object().endsWith(".info")) { + return mockGetObjectResponse(UploadInfoJsonSerializer.serialize(info).getBytes()); + } + throw new RuntimeException("GetObject error"); + }); + + when(minioClient.statObject(any(StatObjectArgs.class))) + .thenThrow(new RuntimeException("Stat error")); + + ByteArrayInputStream bais = new ByteArrayInputStream(new byte[10]); + storageService.append(info, bais); + } + + @Test(expected = IOException.class) + public void testUploadChunkToS3ThrowsIOException() throws Exception { + UploadInfo info = new UploadInfo(); + info.setId(new UploadId("chunk-err-123")); + info.setLength(100L); + info.setOffset(0L); + + when(minioClient.getObject(any(GetObjectArgs.class))) + .thenAnswer( + inv -> { + GetObjectArgs args = inv.getArgument(0); + if (args.object().endsWith(".info")) { + return mockGetObjectResponse(UploadInfoJsonSerializer.serialize(info).getBytes()); + } + throw new RuntimeException("GetObject error"); + }); + + when(minioClient.putObject(any(PutObjectArgs.class))) + .thenAnswer( + inv -> { + PutObjectArgs args = inv.getArgument(0); + if (args.object().endsWith(".info")) { + return null; + } + throw new RuntimeException("Chunk put failure"); + }); + + storageService.append(info, new ByteArrayInputStream(new byte[10])); + } + + @Test(expected = IOException.class) + public void testFinalizeCompletedUploadSinglePartComposeException() throws Exception { + UploadInfo info = new UploadInfo(); + info.setId(new UploadId("single-compose-err")); + info.setLength(10L); + info.setOffset(0L); + + ErrorResponse noSuchKeyErr = mock(ErrorResponse.class); + when(noSuchKeyErr.code()).thenReturn("NoSuchKey"); + ErrorResponseException noSuchKeyEx = new ErrorResponseException(noSuchKeyErr, null, null); + + when(minioClient.getObject(any(GetObjectArgs.class))) + .thenAnswer( + inv -> { + GetObjectArgs args = inv.getArgument(0); + if (args.object().endsWith(".info")) { + return mockGetObjectResponse(UploadInfoJsonSerializer.serialize(info).getBytes()); + } + throw noSuchKeyEx; + }); + + StatObjectResponse statRes = mock(StatObjectResponse.class); + when(statRes.size()).thenReturn(10L); + + when(minioClient.statObject(any(StatObjectArgs.class))) + .thenAnswer( + inv -> { + StatObjectArgs args = inv.getArgument(0); + if (args.object().endsWith(".part")) { + throw noSuchKeyEx; + } + return statRes; + }); + + Item item1 = mock(Item.class); + when(item1.objectName()).thenReturn("tus-uploads/single-compose-err.part.00001"); + when(minioClient.listObjects(any(ListObjectsArgs.class))) + .thenReturn(Arrays.asList(new Result<>(item1))); + + doThrow(new RuntimeException("Compose error")) + .when(minioClient) + .composeObject(any(io.minio.ComposeObjectArgs.class)); + + storageService.append(info, new ByteArrayInputStream(new byte[10])); + } + + @Test(expected = IOException.class) + public void testFinalizeCompletedUploadMultipartComposeException() throws Exception { + UploadInfo info = new UploadInfo(); + info.setId(new UploadId("multi-compose-err")); + info.setLength(20L); + info.setOffset(0L); + + ErrorResponse noSuchKeyErr = mock(ErrorResponse.class); + when(noSuchKeyErr.code()).thenReturn("NoSuchKey"); + ErrorResponseException noSuchKeyEx = new ErrorResponseException(noSuchKeyErr, null, null); + + when(minioClient.getObject(any(GetObjectArgs.class))) + .thenAnswer( + inv -> { + GetObjectArgs args = inv.getArgument(0); + if (args.object().endsWith(".info")) { + return mockGetObjectResponse(UploadInfoJsonSerializer.serialize(info).getBytes()); + } + throw noSuchKeyEx; + }); + + StatObjectResponse statRes = mock(StatObjectResponse.class); + when(statRes.size()).thenReturn(10L); + + when(minioClient.statObject(any(StatObjectArgs.class))) + .thenAnswer( + inv -> { + StatObjectArgs args = inv.getArgument(0); + if (args.object().endsWith(".part")) { + throw noSuchKeyEx; + } + return statRes; + }); + + Item item1 = mock(Item.class); + when(item1.objectName()).thenReturn("tus-uploads/multi-compose-err.part.00001"); + Item item2 = mock(Item.class); + when(item2.objectName()).thenReturn("tus-uploads/multi-compose-err.part.00002"); + + when(minioClient.listObjects(any(ListObjectsArgs.class))) + .thenReturn(Arrays.asList(new Result<>(item1), new Result<>(item2))); + + doThrow(new RuntimeException("Multipart compose error")) + .when(minioClient) + .composeObject(any(ComposeObjectArgs.class)); + + storageService.append(info, new ByteArrayInputStream(new byte[20])); + } + + @Test(expected = IOException.class) + public void testFinalizeCompletedUploadZeroBytePutException() throws Exception { + UploadInfo info = new UploadInfo(); + info.setId(new UploadId("zero-byte-err")); + info.setLength(0L); + info.setOffset(0L); + + when(minioClient.getObject(any(GetObjectArgs.class))) + .thenAnswer( + inv -> { + GetObjectArgs args = inv.getArgument(0); + if (args.object().endsWith(".info")) { + return mockGetObjectResponse(UploadInfoJsonSerializer.serialize(info).getBytes()); + } + throw new RuntimeException("GetObject error"); + }); + + when(minioClient.putObject(any(PutObjectArgs.class))) + .thenAnswer( + inv -> { + PutObjectArgs args = inv.getArgument(0); + if (args.object().endsWith(".info")) { + return null; + } + throw new RuntimeException("Zero byte put error"); + }); + + storageService.append(info, new ByteArrayInputStream(new byte[0])); + } + + @Test(expected = me.desair.tus.server.exception.UploadNotFoundException.class) + public void testFetchS3ByteStreamGenericExceptionOnObjectKey() throws Exception { + UploadInfo info = new UploadInfo(); + UploadId id = new UploadId("fetch-err-123"); + info.setId(id); + info.setOffset(10L); + + when(minioClient.getObject(any(GetObjectArgs.class))) + .thenAnswer( + inv -> { + GetObjectArgs args = inv.getArgument(0); + if (args.object().endsWith(".info")) { + return mockGetObjectResponse(UploadInfoJsonSerializer.serialize(info).getBytes()); + } + throw new RuntimeException("GetObject failure"); + }); + + storageService.getUploadedBytes(id); + } + + @Test(expected = IOException.class) + public void testTruncateFromCompletedObjectThrowsIOException() throws Exception { + UploadInfo info = new UploadInfo(); + info.setId(new UploadId("trunc-err-123")); + info.setLength(20L); + info.setOffset(20L); + + when(minioClient.getObject(any(GetObjectArgs.class))) + .thenAnswer( + inv -> { + GetObjectArgs args = inv.getArgument(0); + if (args.object().endsWith(".info")) { + return mockGetObjectResponse(UploadInfoJsonSerializer.serialize(info).getBytes()); + } + throw new RuntimeException("GetObject completed object failure"); + }); + + storageService.removeLastNumberOfBytes(info, 5L); + } + + @Test + public void testTruncateFromIncompletePartExceptionHandling() throws Exception { + UploadInfo info = new UploadInfo(); + info.setId(new UploadId("trunc-part-err")); + info.setLength(20L); + info.setOffset(10L); + + when(minioClient.statObject(any(StatObjectArgs.class))) + .thenThrow(new RuntimeException("Stat object error")); + + storageService.removeLastNumberOfBytes(info, 5L); + } + + @Test + public void testCalculateCurrentOffsetIncompletePartHeadException() throws Exception { + UploadInfo info = new UploadInfo(); + UploadId id = new UploadId("offset-head-err"); + info.setId(id); + + when(minioClient.getObject(any(GetObjectArgs.class))) + .thenAnswer( + inv -> { + GetObjectArgs args = inv.getArgument(0); + if (args.object().endsWith(".info")) { + return mockGetObjectResponse(UploadInfoJsonSerializer.serialize(info).getBytes()); + } + throw new RuntimeException("GetObject error"); + }); + + when(minioClient.statObject(any(StatObjectArgs.class))) + .thenThrow(new RuntimeException("Head error")); + + UploadInfo fetched = storageService.getUploadInfo(id); + } + + @Test + public void testFetchExistingPartKeysExceptionIgnored() throws Exception { + UploadInfo info = new UploadInfo(); + info.setId(new UploadId("list-err-123")); + info.setLength(10L); + info.setOffset(0L); + + when(minioClient.getObject(any(GetObjectArgs.class))) + .thenAnswer( + inv -> { + GetObjectArgs args = inv.getArgument(0); + if (args.object().endsWith(".info")) { + return mockGetObjectResponse(UploadInfoJsonSerializer.serialize(info).getBytes()); + } + throw new RuntimeException("GetObject error"); + }); + + when(minioClient.listObjects(any(ListObjectsArgs.class))) + .thenThrow(new RuntimeException("List objects error")); + + storageService.append(info, new ByteArrayInputStream(new byte[10])); + } + + @Test + public void testCalcOptimalPartSizeForVeryLargeUpload() throws Exception { + UploadInfo info = new UploadInfo(); + info.setId(new UploadId("large-upload-123")); + info.setLength(50000000000L); + info.setOffset(0L); + + UploadInfo created = storageService.create(info, "owner"); + assertNotNull(created); + } + + @Test + public void testDeleteObjectQuietlyNullAndException() throws Exception { + org.mockito.Mockito.doThrow(new RuntimeException("Remove object error")) + .when(minioClient) + .removeObject(any(io.minio.RemoveObjectArgs.class)); + + storageService.terminateUpload(null); + + UploadInfo info = new UploadInfo(); + info.setId(new UploadId("del-err-123")); + storageService.terminateUpload(info); + } + + @Test + public void testSanitizePrefixNullOrEmptyInS3StorageService() throws Exception { + java.nio.file.Path tmpDir = java.nio.file.Paths.get(System.getProperty("java.io.tmpdir")); + + S3StorageService s1 = new S3StorageService(minioClient, "bucket", "", "", "", "", tmpDir); + assertNotNull(s1); + + S3StorageService s2 = + new S3StorageService(minioClient, "bucket", null, null, null, null, tmpDir); + assertNotNull(s2); + } + + private GetObjectResponse mockGetObjectResponse(byte[] bytes) { + return new GetObjectResponse( + null, "test-bucket", "us-east-1", "object-key", new ByteArrayInputStream(bytes)); + } +} diff --git a/src/test/java/me/desair/tus/server/upload/s3/S3UploadLockTest.java b/src/test/java/me/desair/tus/server/upload/s3/S3UploadLockTest.java new file mode 100644 index 00000000..cbff78ae --- /dev/null +++ b/src/test/java/me/desair/tus/server/upload/s3/S3UploadLockTest.java @@ -0,0 +1,128 @@ +package me.desair.tus.server.upload.s3; + +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertNotNull; +import static org.mockito.ArgumentMatchers.any; +import static org.mockito.Mockito.mock; + +import io.minio.MinioClient; +import io.minio.PutObjectArgs; +import io.minio.RemoveObjectArgs; +import java.io.InputStream; +import java.util.concurrent.ConcurrentHashMap; +import java.util.concurrent.ConcurrentMap; +import org.junit.Before; +import org.junit.Test; +import org.mockito.Mockito; + +public class S3UploadLockTest { + + private MinioClient minioClient; + private ConcurrentMap inputStreamMap; + + @Before + public void setUp() { + minioClient = mock(MinioClient.class); + inputStreamMap = new ConcurrentHashMap<>(); + } + + @Test + public void testLockGettersReleaseAndRenewLease() throws Exception { + InputStream mockStream = mock(InputStream.class); + inputStreamMap.put("/files/upload-1", mockStream); + + S3UploadLock lock = + new S3UploadLock( + minioClient, + "test-bucket", + "tus-locks/upload-1.lock", + "tus-locks/upload-1.stop", + "holder-123", + 60000L, + "/files/upload-1", + inputStreamMap); + + assertEquals("holder-123", lock.getHolderId()); + assertEquals("/files/upload-1", lock.getUploadUri()); + + // Explicitly call renewLease() to verify lease renewal + lock.renewLease(); + + lock.release(); + assertNotNull(lock); + } + + @Test + public void testRenewLeaseExceptionHandling() throws Exception { + Mockito.doThrow(new RuntimeException("PutObject failed")) + .when(minioClient) + .putObject(any(PutObjectArgs.class)); + + S3UploadLock lock = + new S3UploadLock( + minioClient, + "test-bucket", + "tus-locks/upload-1.lock", + "tus-locks/upload-1.stop", + "holder-123", + 60000L, + "/files/upload-1", + inputStreamMap); + + lock.renewLease(); + } + + @Test + public void testLockDeleteQuietlyWithNullKeysAndExceptionHandling() throws Exception { + Mockito.doThrow(new RuntimeException("Remove failed")) + .when(minioClient) + .removeObject(any(RemoveObjectArgs.class)); + + S3UploadLock lockWithNullKeys = + new S3UploadLock( + minioClient, + "test-bucket", + null, + null, + "holder-123", + 60000L, + "/files/upload-1", + inputStreamMap); + + lockWithNullKeys.close(); + + S3UploadLock lockWithKeys = + new S3UploadLock( + minioClient, + "test-bucket", + "tus-locks/upload-1.lock", + "tus-locks/upload-1.stop", + "holder-123", + 60000L, + "/files/upload-1", + inputStreamMap); + + lockWithKeys.close(); + } + + @Test + public void testCloseHeartbeatExecutorShutdownException() throws Exception { + java.util.concurrent.ScheduledExecutorService mockExecutor = + mock(java.util.concurrent.ScheduledExecutorService.class); + Mockito.doThrow(new RuntimeException("Shutdown error")).when(mockExecutor).shutdownNow(); + + S3UploadLock lock = + new S3UploadLock( + minioClient, + "test-bucket", + "tus-locks/upload-1.lock", + "tus-locks/upload-1.stop", + "holder-123", + 60000L, + "/files/upload-1", + inputStreamMap, + mockExecutor); + + lock.close(); + } +} diff --git a/src/test/java/me/desair/tus/server/upload/s3/S3UtilsTest.java b/src/test/java/me/desair/tus/server/upload/s3/S3UtilsTest.java new file mode 100644 index 00000000..19262143 --- /dev/null +++ b/src/test/java/me/desair/tus/server/upload/s3/S3UtilsTest.java @@ -0,0 +1,41 @@ +package me.desair.tus.server.upload.s3; + +import static org.junit.Assert.assertEquals; + +import io.minio.errors.ErrorResponseException; +import io.minio.messages.ErrorResponse; +import org.junit.Test; + +public class S3UtilsTest { + + @Test + public void testParseErrorResponseNull() { + assertEquals(S3ErrorType.UNKNOWN, S3Utils.parseErrorResponse(null)); + } + + @Test + public void testParseErrorResponseCodes() throws Exception { + assertEquals( + S3ErrorType.NO_SUCH_KEY, S3Utils.parseErrorResponse(createExceptionWithCode("NoSuchKey"))); + assertEquals( + S3ErrorType.NO_SUCH_KEY, + S3Utils.parseErrorResponse(createExceptionWithCode("NoSuchBucket"))); + assertEquals( + S3ErrorType.PRECONDITION_FAILED, + S3Utils.parseErrorResponse(createExceptionWithCode("PreconditionFailed"))); + assertEquals( + S3ErrorType.CONFLICT, + S3Utils.parseErrorResponse(createExceptionWithCode("ObjectAlreadyExists"))); + assertEquals( + S3ErrorType.ACCESS_DENIED, + S3Utils.parseErrorResponse(createExceptionWithCode("AccessDenied"))); + assertEquals( + S3ErrorType.UNKNOWN, S3Utils.parseErrorResponse(createExceptionWithCode("InternalError"))); + } + + private ErrorResponseException createExceptionWithCode(String code) { + ErrorResponse errorResponse = org.mockito.Mockito.mock(ErrorResponse.class); + org.mockito.Mockito.when(errorResponse.code()).thenReturn(code); + return new ErrorResponseException(errorResponse, null, null); + } +} diff --git a/src/test/java/me/desair/tus/server/util/UploadInfoJsonSerializerTest.java b/src/test/java/me/desair/tus/server/util/UploadInfoJsonSerializerTest.java new file mode 100644 index 00000000..9a8a5d33 --- /dev/null +++ b/src/test/java/me/desair/tus/server/util/UploadInfoJsonSerializerTest.java @@ -0,0 +1,70 @@ +package me.desair.tus.server.util; + +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertNotNull; +import static org.junit.Assert.assertNull; + +import java.io.ByteArrayInputStream; +import java.io.ByteArrayOutputStream; +import java.io.InputStream; +import java.nio.charset.StandardCharsets; +import me.desair.tus.server.upload.UploadId; +import me.desair.tus.server.upload.UploadInfo; +import org.junit.Test; + +public class UploadInfoJsonSerializerTest { + + @Test + public void testSerializeAndDeserializeUploadInfo() throws Exception { + UploadInfo info = new UploadInfo(); + info.setId(new UploadId("24249a5b-01a4-4bf8-b67a-364273bb5a2e")); + info.setLength(1024L); + info.setOffset(512L); + info.setOwnerKey("owner-1"); + info.setStorageUploadId("custom-storage-id"); + + String json = UploadInfoJsonSerializer.serialize(info); + assertNotNull(json); + + UploadInfo deserialized = UploadInfoJsonSerializer.deserialize(json); + assertNotNull(deserialized); + assertEquals("24249a5b-01a4-4bf8-b67a-364273bb5a2e", deserialized.getId().toString()); + assertEquals(Long.valueOf(1024L), deserialized.getLength()); + assertEquals(Long.valueOf(512L), deserialized.getOffset()); + assertEquals("owner-1", deserialized.getOwnerKey()); + assertEquals("custom-storage-id", deserialized.getStorageUploadId()); + + // Test InputStream overload + UploadInfo fromStream = + UploadInfoJsonSerializer.deserialize( + new ByteArrayInputStream(json.getBytes(StandardCharsets.UTF_8))); + assertNotNull(fromStream); + assertEquals("24249a5b-01a4-4bf8-b67a-364273bb5a2e", fromStream.getId().toString()); + + // Test OutputStream overload + ByteArrayOutputStream baos = new ByteArrayOutputStream(); + UploadInfoJsonSerializer.serializeToStream(info, baos); + UploadInfo fromStream2 = + UploadInfoJsonSerializer.deserialize(new ByteArrayInputStream(baos.toByteArray())); + assertNotNull(fromStream2); + assertEquals("24249a5b-01a4-4bf8-b67a-364273bb5a2e", fromStream2.getId().toString()); + } + + @Test + public void testNullAndEmptyHandling() throws Exception { + assertNull(UploadInfoJsonSerializer.serialize(null)); + assertNull(UploadInfoJsonSerializer.deserialize((String) null)); + assertNull(UploadInfoJsonSerializer.deserialize((InputStream) null)); + assertNull(UploadInfoJsonSerializer.deserialize("")); + + UploadInfo emptyIdInfo = UploadInfoJsonSerializer.deserialize("{\"id\":\"\"}"); + assertNotNull(emptyIdInfo); + assertNull(emptyIdInfo.getId()); + + try { + UploadInfoJsonSerializer.deserialize("invalid-json"); + } catch (Exception expected) { + // expected + } + } +}