feat(rufh): add Python conformity test suite and resolve RUFH draft-12 compliance failures

.gitignore

@@ -172,3 +172,4 @@ buildNumber.properties
 __pycache__/
 .venv/
 *.pyc
+CONFORMITY_TEST_IMPROVEMENTS.md

AGENTS.md

@@ -70,6 +70,7 @@ Completed parent uploads are indexed by checksum under the `<storagePath>/checks
 
 ### 10. Unit Test Coverage & Pragmatic Testing
 - Unit test coverage must remain high for all new feature logic, handlers, validators, and core workflows.
+- **Mandatory Test Addition Rule**: Whenever any functional change, feature implementation, or protocol fix is added, corresponding unit tests MUST ALWAYS be added automatically to prove the fix/feature. Compliance unit tests MUST contain section references and verbatim specification quotes in method Javadocs based on the official specification.
 - Do not use reflection to test private helper methods. Always test code through public API boundaries instead of bypassing encapsulation.
 - Compliance unit tests in `me.desair.tus.server.rufh` MUST contain verbatim specification quotes in method Javadocs based on the official specification.
 - Coverage should focus on meaningful domain logic and contract behavior. Do not over-complicate test suites, write brittle reflection hacks, or add unnatural code structures solely to hit 100% JaCoCo coverage on defensive catch blocks or trivial fallbacks.
@@ -109,6 +110,12 @@ Whenever a new setter or configuration property (such as `setMinAppendSize`, `se
 - `TusFileUploadService.withUploadStorageService(...)` MUST be updated to copy the setting from the old `UploadStorageService` instance to the new one.
 - `ThreadLocalCachedStorageAndLockingService` MUST delegate the setter and getter methods to `storageServiceDelegate`.
 
+### 15. Typed Exceptions & HttpServletResponse Status Codes
+- Do NOT throw generic `TusException` directly when throwing protocol errors or request validation failures.
+- Always throw specific typed exceptions from the `me.desair.tus.server.exception` package (e.g., `UploadNotFoundException`, `InvalidUploadMetadataException`, `UploadLengthExceededException`, `InvalidHttpDigestException`).
+- If a new error condition is introduced, create a new typed exception class in `me.desair.tus.server.exception` that extends `TusException`.
+- Typed exception constructors MUST use `jakarta.servlet.http.HttpServletResponse` HTTP status code constants (e.g., `HttpServletResponse.SC_BAD_REQUEST`, `HttpServletResponse.SC_CONFLICT`, `HttpServletResponse.SC_REQUEST_ENTITY_TOO_LARGE`) when calling `super(status, message)`.
+
 ## IETF Resumable Uploads for HTTP (RUFH) Spec Maintenance & Update Playbook
 
 ### 1. Spec Diff Review
@@ -135,3 +142,64 @@ When updating the IETF protocol implementation for a new draft revision, follow
    ```bash
    mvn verify -Pcheck-coverage -Djacoco.compare.branch=master -q
    ```
+
+### 4. Conformity Test Suite Maintenance & Subagent Isolation
+Whenever a new draft revision of the RUFH specification is published, the repository's Python conformity test suite (`scripts/rufh_conformity_test.py`) MUST be reviewed and updated by a separate, dedicated subagent.
+- **Strict Isolation Rule**: The subagent tasked with updating `scripts/rufh_conformity_test.py` MUST ONLY consult the official IETF specification document (and RFC 9530) and MUST NOT inspect the Java server implementation code under `src/main/java/`. This ensures the conformity test suite remains an independent, unbiased specification benchmark.
+
+### 5. Conformity Test Suite Audit — Repeatable Procedure
+Use this procedure to audit `scripts/rufh_conformity_test.py` against the current (or a new) specification revision. The goal is to identify untested MUST/SHOULD/MAY requirements and produce an actionable improvement report.
+
+#### 5.1 Inputs
+- **Specification document**: The full text of the target draft revision, e.g.:
+  `https://www.ietf.org/archive/id/draft-ietf-httpbis-resumable-upload-<REV>.txt`
+- **Test suite**: `scripts/rufh_conformity_test.py` (read it in full).
+- **Previous audit report** (if any): `CONFORMITY_TEST_IMPROVEMENTS.md` in the project root.
+
+#### 5.2 Isolation Rules
+- **Do NOT read any Java source code** under `src/main/java/` during the audit. The audit must be purely spec-vs-test-script.
+- The only project files to read are `scripts/rufh_conformity_test.py` and optionally `CONFORMITY_TEST_IMPROVEMENTS.md`.
+- You may read the specification document, RFC 9530 (HTTP Digests), RFC 9651 (Structured Fields), and RFC 9457 (Problem Details) for normative context.
+
+#### 5.3 Audit Methodology (Clause-by-Clause)
+Walk through every normative section of the specification in order. For each section:
+
+1. **Extract every requirement** containing MUST, MUST NOT, SHOULD, SHOULD NOT, or MAY (per RFC 2119 / RFC 8174 semantics).
+2. **For each requirement**, search the test suite for a test that exercises it:
+   - Check if the test sends the right request (method, headers, body).
+   - Check if the test asserts the correct response behavior (status code, headers, body content).
+   - Note whether the test covers both the positive (conformant) and negative (non-conformant input) cases.
+3. **Classify the finding**:
+   - ✅ **Covered** — a test exists and its assertions match the requirement.
+   - ✅ **Partial** — a test exists but assertions are incomplete or only cover one case.
+   - ❌ **Missing** — no test covers this requirement.
+4. **For partial/missing items**, write a concrete recommendation: test method name, spec section, request/response to send, and assertions to make.
+
+The sections to audit (for draft-12) are:
+- §4.1.1 (Offset), §4.1.2 (Completeness), §4.1.3 (Length), §4.1.4 (Limits)
+- §4.2 (Upload Creation): §4.2.1 (Client Behavior), §4.2.2 (Server Behavior)
+- §4.3 (Offset Retrieval): §4.3.1 (Client Behavior), §4.3.2 (Server Behavior)
+- §4.4 (Upload Append): §4.4.1 (Client Behavior), §4.4.2 (Server Behavior)
+- §4.5 (Upload Cancellation): §4.5.1, §4.5.2 (Server Behavior)
+- §4.6 (Concurrency), §4.7 (Retry)
+- §5 (Status Code 104)
+- §6 (Media Type application/partial-upload)
+- §7.1 (Mismatching Offset problem type), §7.2 (Inconsistent Length problem type)
+- §10.1 (Optimistic Upload Creation), §10.1.1 (Upgrading), §10.2 (Careful Upload Creation)
+
+#### 5.4 Output Format
+Produce a Markdown report saved as `CONFORMITY_TEST_IMPROVEMENTS.md` in the project root (overwrite the previous version). The report MUST contain:
+
+1. **Executive Summary** — overall coverage assessment.
+2. **Critical Gaps** (🔴) — untested MUST-level requirements, with spec quotes and recommended test methods.
+3. **Important Gaps** (🟡) — untested SHOULD-level requirements or incomplete assertions.
+4. **Minor Improvements** (🔵) — edge cases, test quality improvements, spec alignment.
+5. **Existing Test Corrections** — any tests with incorrect or overly permissive assertions.
+6. **Recommended New Test Methods** — organized by test class, with spec section, method name, and description.
+7. **Summary Matrix** — table with columns: Spec Section, Requirement Level, Currently Tested (✅/✅ Partial/❌), Gap Description.
+
+#### 5.5 How to Invoke This Audit
+Request the audit with a prompt like:
+> Perform a strict conformity audit of `scripts/rufh_conformity_test.py` against the draft-12 specification at `https://www.ietf.org/archive/id/draft-ietf-httpbis-resumable-upload-12.txt`. Follow the audit procedure in AGENTS.md §5. Do NOT inspect any Java implementation code.
+
+To audit against a newer draft, replace the draft number in the URL.

CHANGELOG.md

@@ -5,13 +5,16 @@ All notable changes to this project will be documented in this file.
 ## [2.0.0]
 
 ### Added
-- **IETF Resumable Uploads for HTTP (RUFH) Protocol Compliance**: Implemented full support for the official IETF Resumable Uploads for HTTP specification (`draft-ietf-httpbis-resumable-upload`).
+- **IETF Resumable Uploads for HTTP (RUFH) Protocol**: Implemented full support for the official IETF Resumable Uploads for HTTP specification (`draft-ietf-httpbis-resumable-upload-12`).
 - **Dual Protocol Auto-Detection**: Added transparent protocol routing in `TusFileUploadService` supporting both legacy `TUS_1_0_0` (`Tus-Resumable: 1.0.0`) and `RUFH` (`ProtocolVersion.RUFH`) clients concurrently on the same endpoint.
 - **RFC 9651 Structured Header Fields**: Implemented RFC 9651 parsing and serialization for `Upload-Offset`, `Upload-Complete`, `Upload-Length`, and `Upload-Limit` dictionary headers.
 - **RFC 7807 Problem Details JSON**: Added support for standard `application/problem+json` error responses (`mismatching-upload-offset`, `completed-upload`, `inconsistent-upload-length`).
-- **Dedicated Compliance & Security Test Suites**: Added comprehensive, spec-quoted unit tests under package `me.desair.tus.server.ietf` and security tests under `me.desair.tus.server.ietf.security` verifying Path Traversal protection, DoS limits, CRLF sanitization, and lock safety.
+- **Dedicated Compliance Test Suites**: Added comprehensive, spec-quoted end-to-end tests using a dedicated Python script `scripts/rufh_conformity_test.py` with documentation on how to run the tests in `docs/CONFORMITY_TESTING.md`.
 - **User Migration & Interim Responses Documentation**: Added `docs/MIGRATION.md` and `docs/INTERIM_RESPONSES.md` detailing migration strategies, HTTP 104 status frames under IETF RUFH, Tomcat/Servlet container limitations, cached reflection optimizations, and Spring Boot Tomcat Valve integration.
 
+### Breaking
+- **Downloads**: In order to support both the Tus protocol and RUFH protocol, the unofficial download extension will not return a HTTP status code `204` for uploads that are still in progress and will not contain the response header `Tus-Resumable`. Removed the `UploadInProgressException` class.
+
 ## [1.0.0-3.3]
 
 ### Added

README.md

@@ -86,8 +86,8 @@ The first step is to create a `TusFileUploadService` object using its constructo
 * `withChunkedTransferDecoding`: You can enable or disable the decoding of chunked HTTP requests by this library. Enable this feature in case the web container in which this service is running does not decode chunked transfers itself. By default, chunked decoding via this library is disabled (as modern frameworks tend to already do this for you).
 * `withThreadLocalCache(Boolean)`: Optionally you can enable (or disable) an in-memory (thread local) cache of upload request data to reduce load on the storage backend and potentially increase performance when processing upload requests.
 * `withUploadExpirationPeriod(Long)`: You can set the number of milliseconds after which an upload is considered as expired and available for cleanup. Applies to both Tus 1.0.0 (`Upload-Expires` response header) and IETF RUFH (`max-age` parameter in `Upload-Limit` response header).
-* `getRawInterimResponse(HttpServletRequest, String)`: Helper method that inspects an incoming request and returns the raw HTTP 104 interim response frame string (`HTTP/1.1 104 Upload Resumption Supported\r\nLocation: ...\r\nUpload-Offset: 0\r\n\r\n`) if applicable, or `null` otherwise. Useful for web container extensions (such as Tomcat Valves) that flush 1xx interim responses directly to client sockets.
-* `withDownloadFeature()`: Enable the unofficial `download` extension that also allows you to download uploaded bytes.
+* `withDownloadFeature()`: Enable the unofficial `download` extension that allows clients to download uploaded bytes via `GET`. This feature is disabled by default.
+  * **Disclaimer**: Enabling the download extension for `GET` requests may interfere with IETF RUFH `GET` offset retrieval conformity (Section 4.3 of draft-12), as RUFH specifies `GET` requests for offset retrieval returning `204 No Content`.
 * `withUploadDeduplication(Boolean)`: Enable duplicate file processing based on the checksum hash. If enabled, the server will scan previous completed uploads for a file with the same checksum. If a duplicate is found, the new upload will link to the existing file (`duplicatesUploadId`), skipping redundant disk storage writes and saving disk space.
   * **Disclaimer**: If duplicate file processing is enabled, the duplicate (child) upload depends directly on the original (parent) upload file. If the original parent upload is deleted or terminated, any duplicate child uploads pointing to it will no longer be downloadable (returning `404 Not Found`).
 * `addTusExtension(TusExtension)`: Add a custom (application-specific) extension that implements the `me.desair.tus.server.TusExtension` interface. For example you can add your own extension that checks authentication and authorization policies within your application for the user doing the upload.
@@ -135,10 +135,12 @@ After having processed the uploaded bytes on the server backend (e.g. copy them
 
 Next to removing uploads after they have been completed and processed by the backend, it is also recommended to schedule a regular maintenance task to clean up any expired uploads or locks. Cleaning up expired uploads and locks can be achieved using the `me.desair.tus.server.TusFileUploadService.cleanup()` method.
 
-## Compatible Client Implementations
+## Compatible Client Implementations & Conformity Testing
 This server implementation has been tested with:
 - **Tus 1.0.0 Clients**: Tested with [Uppy](https://uppy.io/) and `tus-js-client`.
-- **IETF Resumable Uploads Clients**: For now, the implementation has only been tested with the [RUFH conformity tests of the IETF hackathon](https://github.com/tus/ietf-hackathon).
+- **IETF Resumable Uploads Clients & Conformity Tests**: The implementation has been thoroughly tested with our own built-in RUFH conformity test suite (`scripts/rufh_conformity_test.py`) validating compliance with draft-12 of the RUFH protocol specification and RFC 9530 HTTP Digests, as well as the community [RUFH conformity tests from the IETF hackathon](https://github.com/tus/ietf-hackathon).
+
+For detailed instructions on running our native conformity test suite and interpreting results, see the **[Conformity Testing Guide (docs/CONFORMITY_TESTING.md)](docs/CONFORMITY_TESTING.md)**.
 
 This repository also contains comprehensive automated integration test suites (`ITTusFileUploadService`, `IetfProtocolCreationTest`, `IetfProtocolAppendTest`, `IetfProtocolHeadTest`, `IetfProtocolCancellationTest`) validating both protocol specifications.
 

docs/CONFORMITY_TESTING.md

@@ -1,6 +1,8 @@
 # Conformity Testing Guide (IETF Resumable Uploads for HTTP)
 
-This guide describes how to manually execute the RUFH conformity tests against a locally running instance of the Spring Boot demo server using the community conformity testing suite.
+This guide describes how to execute the RUFH (Resumable Uploads for HTTP) conformity tests against a locally running instance of the Spring Boot demo server or any RUFH compliant server endpoint.
+
+The test suite validates compliance with [draft-ietf-httpbis-resumable-upload-12](https://www.ietf.org/archive/id/draft-ietf-httpbis-resumable-upload-12.txt) and [RFC 9530 HTTP Digests](https://www.rfc-editor.org/rfc/rfc9530.html).
 
 ---
 
@@ -10,14 +12,14 @@ First, compile and install the core `tus-java-server` library to your local Mave
 
 ```bash
 # In the root of the tus-java-server repository
-mvn clean install
+mvn clean install -DskipTests
 ```
 
 ---
 
 ## 2. Start the Demo Server
 
-1. Update the dependency version in the demo project if necessary. In `tus-java-server-spring-demo` project's `spring-boot-rest/pom.xml`, verify it points to the locally built snapshot version:
+1. Verify the dependency in `tus-java-server-spring-demo` project's `spring-boot-rest/pom.xml` points to the snapshot version:
    ```xml
    <dependency>
      <groupId>me.desair.tus</groupId>
@@ -26,42 +28,64 @@ mvn clean install
    </dependency>
    ```
 
-2. Build and start the Spring Boot REST demo server:
+2. Build and start the Spring Boot REST demo server with a `1 KB` maximum upload size parameter (`--tus.server.max-upload-size=1024`) to enable full limit discovery & limit enforcement verification:
    ```bash
    cd ../tus-java-server-spring-demo
-   mvn clean package
-   java -jar spring-boot-rest/target/spring-boot-rest-0.0.1-SNAPSHOT.jar
+   mvn clean package -DskipTests
+   java -jar spring-boot-rest/target/spring-boot-rest-0.0.1-SNAPSHOT.jar --tus.server.max-upload-size=1024
    ```
 
    The server will start on port `8080` with the upload endpoint exposed at:
    `http://localhost:8080/test/api/upload`
 
 ---
 
-## 3. Clone and Run the Conformity Tester
+## 3. Run the Built-In RUFH Conformity Test Suite
 
-The conformity tests are written in Python using `pytest` and are maintained by the community under the `ietf-hackathon` repository.
+The repository includes its own native Python conformity test suite located at `scripts/rufh_conformity_test.py`. It requires `pytest` and `requests`.
 
-1. Clone the repository and navigate to the tests directory:
-   ```bash
-   git clone https://github.com/tus/ietf-hackathon.git
-   cd ietf-hackathon/tests
-   ```
+### Prerequisites
+Install Python dependencies if not already installed:
+```bash
+pip install pytest requests
+```
 
-2. Set up a Python virtual environment and activate it:
-   ```bash
-   python3 -m venv venv
-   source venv/bin/activate
-   ```
+### Running the Test Suite
 
-3. Install required Python packages:
-   ```bash
-   pip install -r requirements.txt
-   ```
+You can execute the test suite using Python directly or via PyTest:
 
-4. Run the conformity tests pointing to your locally running Spring Boot endpoint:
-   ```bash
-   pytest --url http://localhost:8080/test/api/upload
-   ```
+#### Option A: Running directly with Python (Recommended for structured AI / Agent reporting)
+```bash
+python3 scripts/rufh_conformity_test.py --url http://localhost:8080/test/api/upload
+```
+
+#### Option B: Running with PyTest
+```bash
+pytest scripts/rufh_conformity_test.py --url http://localhost:8080/test/api/upload
+```
+
+---
+
+## 4. Understanding Test Results & AI Agent Remediation
+
+When executed, the script produces a structured summary report detailing:
+
+1. **Total Tests Executed**: Count of total specification compliance tests run.
+2. **Passed Tests**: Number of tests matching draft-12 specification requirements.
+3. **Failed Tests**: Detailed list of failing tests including test method names, exact error tracebacks, expected status codes/headers, and corresponding RFC section references.
+4. **104 Interim Responses**: Count of tests where `HTTP/1.1 104 Upload Resumption Supported` interim responses were detected from the server socket.
 
-   All tests should pass, certifying that the server implementation conforms to the IETF Resumable Uploads for HTTP (RUFH) specification.
+AI agents and developers can analyze the detailed failure breakdown in the script's console output to pinpoint specific compliance gaps and adjust server logic accordingly.
+
+---
+
+## 5. Running Community (IETF Hackathon) Tests
+
+Alternatively, you can also run the external community test suite from the `ietf-hackathon` repository:
+
+```bash
+git clone https://github.com/tus/ietf-hackathon.git
+cd ietf-hackathon/tests
+pip install -r requirements.txt
+pytest --url http://localhost:8080/test/api/upload
+```

docs/INTERIM_RESPONSES.md

@@ -87,7 +87,7 @@ public class TusInterimResponseTomcatValve extends ValveBase {
           // Step 2: Write raw HTTP 104 bytes directly to Tomcat's underlying SocketWrapperBase
           boolean written = writeToSocketWrapper(response, bytes);
           if (written) {
-            LOG.info(
+            LOG.debug(
                 "Emitted raw HTTP 104 Interim Response via Tomcat SocketWrapper for request URI: {}",
                 request.getRequestURI());
           } else {