[diwu-sf]

The problem

Fixes #14773
CRC32C validation is effectively disabled for ALL normal HTTP downloads through reader(), readAllBytes(), and downloadTo(). Damaged data can be returned without a checksum error.

GCS sends x-goog-stored-content-encoding and x-goog-stored-content-length to describe how a file is stored. The client treats either header as a sign that GCS changed the file before sending it, so it skips the checksum check even when the bytes are unchanged.

How to reproduce

Use com.google.cloud:google-cloud-storage:2.75.0 to download this public Landsat metadata file. It is 8,454 bytes and needs no credentials.

Set a breakpoint in HttpStorageRpcHasherHelper.shouldValidate() and run:

import com.google.cloud.NoCredentials;
import com.google.cloud.storage.Storage;
import com.google.cloud.storage.StorageOptions;

public class LandsatRepro {
  public static void main(String[] args) throws Exception {
    String scene = "LC08_L1GT_044034_20130330_20170310_01_T2";
    String object = "LC08/01/044/034/" + scene + "/" + scene + "_MTL.txt";

    try (Storage storage = StorageOptions.http()
        .setProjectId("landsat-crc32c-repro")
        .setCredentials(NoCredentials.getInstance())
        .build()
        .getService()) {
      byte[] data = storage.readAllBytes("gcp-public-data-landsat", object);
      System.out.println("Downloaded " + data.length + " bytes");
    }
  }
}

The full download returns HTTP 200 with these headers:

Content-Length: 8454
x-goog-stored-content-length: 8454
x-goog-stored-content-encoding: identity
x-goog-hash: crc32c=TeBDMA==,md5=ziZNd8uikTtDCkBTgS5Riw==

There is no Content-Encoding header. GCS sends the file unchanged and provides its checksum, but shouldValidate() returns false. It should return true so the client checks the downloaded bytes.

I confirmed this against the live Landsat file with version 2.75.0. All three download methods returned the same 8,454 bytes, and a separately calculated CRC32C matched the checksum from GCS. This reproduction shows the skipped check; the damaged-data tests below show the consequence.

What this PR changes

The client now runs the check on normal downloads. It skips the check only when the data it receives is not the stored file:

• GCS unzipped the file before sending it. The client compares how the file is stored with how it was sent, instead of only looking for the headers.
• The client unzipped the file itself. The HTTP library unzips a download marked gzip, GZIP or x-gzip. The client now uses the same rule, so it never checks unzipped data against the checksum of the zipped file.
• Only part of the file was downloaded. This includes a download that resumes after the connection drops. reader() used to keep the checksum from the first, incomplete attempt and compare it with the checksum for the whole file. That failed even though the data was fine. It now drops that checksum when it resumes.

How it was tested

Unit tests. The tests cover normal downloads, unzipped downloads, partial downloads, and resumed downloads with and without a checksum from GCS on the second request. They also cover uppercase GZIP and spaces or tabs around encoding values, with and without client decompression. All 42 tests in the three test classes pass.

mvn -pl java-storage/google-cloud-storage \
  -Dtest=HttpStorageRpcHasherHelperTest,HttpStorageRpcChecksumTest,ApiaryUnbufferedReadableByteChannelTest \
  test

Real GCS downloads. I downloaded these public files with the client, through readAllBytes(), downloadTo() and reader(), both with and without the client's unzip option:

File	What kind of file
cloud-samples-data/storage/static-hosting/index.html	small plain file
chromium-clang-format/76267ebd69846fb255d926fb71e2f70a538a9e22	1.4 MB plain file
apache-beam-samples/gcs/	empty file
apache-beam-samples/gcs/bigfile.txt.gz	zipped data stored as a plain file
apache-beam-samples/gcs/bigfile_with_encoding_plain.txt.gz	zipped file that GCS unzips when asked
apache-beam-samples/gcs/bigfile_with_encoding.txt.gz	zipped file that GCS always unzips
istio-prow/api-postsubmit/1/build-log.txt	zipped file that GCS never unzips

• Normal downloads: all 42 downloads gave the right data, both before and after this change.
• Damaged data: I put a small local proxy in front of GCS that changes one byte in each download. Before this change, the damaged data was accepted every time. After it, all 18 damaged downloads of the plain files were rejected.
• Dropped connections: the same proxy cut each download off halfway, so the client had to resume. With this change, all 16 resumed reader() downloads succeeded with the right data.
• Uppercase GZIP: the proxy changed gzip to GZIP in the headers. With this change, every download method succeeded.

The file that GCS always unzips still can't be checked. GCS sends the checksum of the zipped file but delivers the unzipped data, so there is nothing to compare against. This is the same as before.

I have not seen GCS send GZIP in uppercase on its own. Handling it keeps the client in step with the HTTP library.

[google-cla]

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

[gemini-code-assist]

Code Review

This pull request updates the CRC32C validation logic for Google Cloud Storage reads, ensuring that resumed reads correctly skip full-object checksum validation. It also refines transcoding detection by comparing stored content encoding against actual content encoding and normalizes encoding strings when checking for client-side decompression. Feedback suggests trimming both the stored and actual encoding strings in isTranscoded to prevent comparison failures caused by leading or trailing whitespaces.

[gemini-code-assist]

If the Content-Encoding header contains leading or trailing whitespace (e.g., " GZIP " or "gzip "), the case-insensitive comparison storedEncoding.equalsIgnoreCase(...) will fail because storedEncoding (e.g., "gzip") is not trimmed. This causes isTranscoded to incorrectly return true even when the client requested raw bytes (returnRaw = true) and the data is not transcoded. As a result, checksum validation will be skipped. Trimming both storedEncoding and encoding before comparison ensures robust and correct validation behavior.

Suggested change

	return (storedEncoding != null
	&& !storedEncoding.equalsIgnoreCase(encoding == null ? "identity" : encoding))
	|| isDecompressedByClient(response);
	return (storedEncoding != null
	&& !storedEncoding.trim().equalsIgnoreCase(encoding == null ? "identity" : encoding.trim()))
	|| isDecompressedByClient(response);