[yandrey321]

What changes were proposed in this pull request?

Problem. AWS SDK for Java v1 is end of life, and Ozone still used it in two places:

• Freon: the S3 generators ozone freon s3kg and s3bg. Because of freon, the distribution ships
  aws-java-sdk-core, aws-java-sdk-s3 and aws-java-sdk-kms, plus jmespath-java and jackson-dataformat-cbor.
• Tests: a v1 copy of the S3 SDK integration suite, AbstractS3SDKV1Tests, about 3,200 lines.

Change.

• The freon S3 generators are removed: s3kg, s3bg and their base class S3EntityGenerator, with the
  trace-context request handler they used.
  • They were the only production code using an AWS SDK, so the AWS SDK (v2) is now test-only, and the distribution
    ships no AWS SDK at all.
  • S3 load can be generated with the aws CLI or third-party S3 benchmark tools.
• The smoketests change accordingly:
  • s3/freon.robot, which only tested those commands, is removed;
  • fcq/s3_om_fcq.robot used s3kg only to send S3 traffic through the gateway. It now uploads 100 small files with
    one aws s3 cp --recursive, and checks that they all arrived before it checks the caller in the OM's
    DecayRpcScheduler metrics.
• The v1 test suite is removed. Every v1 test was mapped to the v2 suite by what it asserts:
  • 59 were already covered;
  • 23 tests are ported to AbstractS3SDKV2Tests for the missing or partial coverage, in the areas listed under
    Testing.
• v1 is removed from the build: the root pom version property and managed dependencies, freon,
  integration-test-s3, and the com.amazonaws logger entries in both log4j.properties files.
• v1 can't come back. The root pom adds two enforcer rules:
  • ban-imports bans com.amazonaws.** imports, in main and test code;
  • a new ban-aws-sdk-v1 execution (bannedDependencies) fails a module that depends on any com.amazonaws
    artifact, even transitively.
• Distribution licensing: the five jars are removed from jar-report.txt and LICENSE.txt.

Key decisions.

• Remove the freon S3 generators instead of migrating them to SDK v2, as suggested in review.
  • A migration would have added 31 SDK v2 jars (about 8.4 MB) and the SDK's NOTICE to the distribution, for two
    load-generator commands.
  • Without them, the SDK is only a test dependency, which is not shipped.
• The v1 tests were ported by behaviour, not by name. Three v2 tests have the same name as a v1 test but check
  something else (the expected-bucket-owner header), so their v1 counterparts were ported too:
  testDeleteBucket, testListMultipartUploads and testListParts.
• Not ported: v1 checks that only test the v1 SDK itself:
  • ErrorType, and null returns where v2 already asserts the 412 or 404;
  • presigned-URL workarounds for APIs v1 lacked;
  • doesBucketExist*.
• The multitenancy-ranger exclusion of aws-java-sdk-bundle stays. It keeps Ranger's transitive v1 out, which
  the new dependency ban now requires anyway.

Side effects.

• ozone freon s3kg and ozone freon s3bg no longer exist. This is a user-facing change and needs a release
  note.
• Distribution jars: 5 removed (the three v1 jars, jmespath-java and jackson-dataformat-cbor), none added.
  joda-time stays; other dependencies use it.
• Anything that used the v1 jars from share/ozone/lib must bring its own.

Co-authored-by: Claude Opus 5.5 noreply@anthropic.com

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-17707

How was this patch tested?

Unit + integration tests. CI: https://github.com/yandrey321/ozone/actions/runs/42002529466

[jojochuang]

Separately, we can also remove aws cli v1 from ozone-runner docker image https://github.com/apache/ozone-docker-runner/blob/master/Dockerfile#L77

[jojochuang]

Thanks for porting the missing v1 coverage, especially the operation tests that weren’t equivalent to the similarly named ownership-header tests. I have a few compatibility and coverage concerns:

1. Object Content-Type: v1’s string putObject sets text/plain, while RequestBody.fromBytes defaults to application/octet-stream. Could we explicitly retain .contentType("text/plain") for ordinary Freon uploads and verify it through HEAD? Keeping the byte body also avoids the uppercase charset signature issue.

2. Credential aliases: the v2 environment provider drops AWS_ACCESS_KEY and AWS_SECRET_KEY. Could we preserve these aliases with v1-compatible precedence, or explicitly document the breaking change? Credential-resolution tests would help establish the intended behavior.

3. Presigned unsupported subresources: the removed v1 test checks presigned ?annotation and ?torrent requests return 501 without modifying the object. The typed-operation tests and the new successful custom-query test don’t cover that same path. Could we retain those cases using putRawQueryParameter with the v2 presigner?

4. Explicit part zero: the new multipart GET test improves coverage of parts 1–3, but replaces the old explicit partNumber=0 request with a request omitting the parameter. Could we retain an explicit-zero assertion that verifies the complete object is returned?

The public Freon helper changes (getS3() return type and tracing-class rename), along with removal of the bundled v1 jars, would also be useful to call out in the compatibility notes.

[adoroszlai]

Thanks @yandrey321 for working on this.

AWS SDK is kept in prod only for 3 Freon classes:

hadoop-ozone/freon/src/main/java/org/apache/hadoop/ozone/freon/S3EntityGenerator.java
hadoop-ozone/freon/src/main/java/org/apache/hadoop/ozone/freon/S3KeyGenerator.java
hadoop-ozone/freon/src/main/java/org/apache/hadoop/ozone/freon/FreonS3TraceContextInterceptor.java

I think we should remove them and make AWS SDK test-only. These Freon generators can be replaced with third-party tools.

Also, we may keep SDK V1 in tests. Test dependencies are not part of the binary distribution, so there is no overhead or security risk. Either way is OK for me.

CC @ivandika3

[yandrey321]

opened apache/ozone-docker-runner#122 for ozone docker runner.

[yandrey321]

@jojochuang addressed review comments:

1. Object Content-Type: s3kg now sets text/plain on its PutObject, as SDK v1 did for its String content.
   The body stays RequestBody.fromBytes. s3/freon.robot now checks the content type of an s3kg key with
   head-object. Multipart uploads keep the SDK's default, as they did with v1.
2. Credential aliases: freon now has its own FreonEnvironmentCredentialsProvider, which reads the environment
   exactly as v1's EnvironmentVariableCredentialsProvider did:
   • the access key comes from AWS_ACCESS_KEY_ID, or else AWS_ACCESS_KEY;
   • the secret key comes from AWS_SECRET_KEY, or else AWS_SECRET_ACCESS_KEY.
   • a variable falls back to its alternate only when it is unset, not when it is empty;
   • values are trimmed;
   • AWS_SESSION_TOKEN gives session credentials.
     TestFreonEnvironmentCredentialsProvider covers each of these, plus the error when a key is missing.
3. Presigned unsupported subresources: ported as
   PresignedUrlTests#testPresignedUrlGetObjectSubresourceNotImplemented (annotation, torrent). It presigns
   GetObject with putRawQueryParameter(subresource, ""), expects 501, and checks the object is unchanged.
4. Explicit part zero: testGetParticularPart now also GETs with partNumber(0) and asserts the whole object
   is returned.

Added compatibility notes in JIRA's release notes.

[yandrey321]

@adoroszlai I think it would be safer and cleaner to remove aws sdk v1 from the code base.

A complete ban is simpler and harder to get wrong. With v1 gone, the enforcer can ban com.amazonaws outright:
imports in main and test code, and any dependency, transitive included. A test-only exception needs scope-aware
rules, and a v1 artifact pulled in transitively at compile scope could slip past them.
v1 is end of life, so it gets no more fixes. That includes fixes for its transitive dependencies
(jackson-dataformat-cbor, jmespath-java), which we would keep pinning and updating in the root pom. Even
test-only, v1 stays in our dependency management and in the scanner reports we triage.
Two copies of the same suite cost more to keep than they catch.

• I mapped every v1 test to the v2 suite by what it asserts. 59 of the 86 were already covered by v2, often with
  stronger assertions. For example, v2 asserts the 412 where v1 only gets null back from a failed conditional copy.
• For the rest, I ported 23 tests to v2. Three were hidden behind v2 tests of the same name that only check the
  bucket-owner header (testDeleteBucket, testListMultipartUploads, testListParts). What wasn't ported only
  tests the v1 client itself.

[adoroszlai]

AWS SDK is kept in prod only for 3 Freon classes:
...

I think we should remove them and make AWS SDK test-only. These Freon generators can be replaced with third-party tools.

@yandrey321 You may have missed this. It's about making AWS SDK v2 test-only.

[yandrey321]

@adoroszlai Lets open a follow up JIRA for removing aws sdk v2 usage from freon and moving it to test-only dependency. I'll pick it up as a follow up work item.

[adoroszlai]

Lets open a follow up JIRA for removing aws sdk v2 usage from freon and moving it to test-only dependency.

That would mean undoing a whole lot of changes being made in this PR (license, notice, jar-report, pom, etc.). I prefer to do it in one step.

[yandrey321]

@adoroszlai I removed aws sdk 2 usage from freon and made aws sdk 2 test only dependency.

[adoroszlai]

Thanks @yandrey321 for updating the patch. I will need to check s3_om_fcq.robot changes, but the rest LGTM.

Will wait for feedback on the patch from others, including @chungen0126 @ivandika3 @peterxcli.

[yandrey321]

green CI: https://github.com/yandrey321/ozone/actions/runs/42002529466

[adoroszlai]

Thanks @yandrey321 for the patch, LGTM.

@chungen0126 @ivandika3 @peterxcli would you like to take a look?

[chungen0126]

+1 LGTM

[adoroszlai]

Thanks @yandrey321 for the patch, @chungen0126, @jojochuang for the review.