[sreejasahithi]

What changes were proposed in this pull request?

Run smoketest/s3/ with both aws (v1) and aws2 (v2)

The runner version is updated by : #11431

What is the link to the Apache JIRA

HDDS-16467

How was this patch tested?

Green CI: https://github.com/sreejasahithi/ozone/actions/runs/38346577758

[adoroszlai]

@sreejasahithi as @jojochuang noticed (see #11569), this significantly increases wall-clock CI time (+30 minutes?), since the runs affected are on the critical path. While we may drop v1 support in the future, I don't think such change should be merged to master. We should do the parallel version in the first commit, not as follow-up.

We can decrease both wall-clock and total CI time by running v1 and v2 tests on different backend configurations, instead of a full matrix. For example I would guess that HA or non-HA cluster behaves the same, since the differences are probably at the S3 Gateway-side, not in OM/SCM. So we can run e.g. v1 with non-HA and v2 with HA (to reflect the importance of v2). We have similar optimization for filesystem tests vs. bucket types vs. HA/non-HA:

hadoop-ozone/dist/src/main/compose/ozone/test.sh
55:execute_robot_test scm -v SCHEME:o3fs -v BUCKET_TYPE:bucket -N ozonefs-o3fs-bucket ozonefs/ozonefs.robot

hadoop-ozone/dist/src/main/compose/ozone-ha/test.sh
38:execute_robot_test ${SCM} -v SCHEME:ofs -v BUCKET_TYPE:link -N ozonefs-ofs-link ozonefs/ozonefs.robot

hadoop-ozone/dist/src/main/compose/ozonesecure/test.sh
44:execute_robot_test scm -v SCHEME:ofs -v BUCKET_TYPE:bucket -N ozonefs-ofs-bucket ozonefs/ozonefs.robot

hadoop-ozone/dist/src/main/compose/ozonesecure-ha/test.sh
45:execute_robot_test s3g -v SCHEME:o3fs -v BUCKET_TYPE:link -N ozonefs-o3fs-link ozonefs/ozonefs.robot

I have some work in progress to drop existing duplicates (HDDS-16818). Instead of dropping all duplicate S3 tests, we may convert some of them to v2.

[sreejasahithi]

@sreejasahithi as @jojochuang noticed (see #11569), this significantly increases wall-clock CI time (+30 minutes?), since the runs affected are on the critical path. While we may drop v1 support in the future, I don't think such change should be merged to master. We should do the parallel version in the first commit, not as follow-up.

We can decrease both wall-clock and total CI time by running v1 and v2 tests on different backend configurations, instead of a full matrix. For example I would guess that HA or non-HA cluster behaves the same, since the differences are probably at the S3 Gateway-side, not in OM/SCM. So we can run e.g. v1 with non-HA and v2 with HA (to reflect the importance of v2). We have similar optimization for filesystem tests vs. bucket types vs. HA/non-HA:

hadoop-ozone/dist/src/main/compose/ozone/test.sh
55:execute_robot_test scm -v SCHEME:o3fs -v BUCKET_TYPE:bucket -N ozonefs-o3fs-bucket ozonefs/ozonefs.robot

hadoop-ozone/dist/src/main/compose/ozone-ha/test.sh
38:execute_robot_test ${SCM} -v SCHEME:ofs -v BUCKET_TYPE:link -N ozonefs-ofs-link ozonefs/ozonefs.robot

hadoop-ozone/dist/src/main/compose/ozonesecure/test.sh
44:execute_robot_test scm -v SCHEME:ofs -v BUCKET_TYPE:bucket -N ozonefs-ofs-bucket ozonefs/ozonefs.robot

hadoop-ozone/dist/src/main/compose/ozonesecure-ha/test.sh
45:execute_robot_test s3g -v SCHEME:o3fs -v BUCKET_TYPE:link -N ozonefs-o3fs-link ozonefs/ozonefs.robot

I have some work in progress to drop existing duplicates (HDDS-16818). Instead of dropping all duplicate S3 tests, we may convert some of them to v2.

Thanks for the feedback, I understand the concern about both the wall clock CI time and the overall CI resource usage. I will look into splitting the v1 and v2 tests across the existing backend configurations rather than running the full matrix.
and address this in the current PR rather than deferring it to a follow up.

[sreejasahithi]

Problem with the First Version of This PR

The initial compose wiring used execute_s3_smoketests() in testlib.sh, which ran the full smoketest/s3/ tree twice per call (-v AWS_CLI:aws, then -v AWS_CLI:aws2). Every compose leg that already ran s3/ once now ran it twice on the same cluster configuration.

This causes approximately 30 minutes of extra wall-clock time

Why Not the Parallel Matrix (#11569)?

#11569 tried to recover wall-clock time by splitting acceptance into matrix legs (OZONE_AWS_CLI=aws|aws2).

That restores per-job duration, but still runs the full S3 suite twice in CI and adds concurrent runner load on the shared Apache queue. Review feedback was to avoid that plumbing and a full matrix, we should not pay 2× the total work, whether sequentially or in parallel.

Proposed Approach: Partition CLI Across Existing Compose Legs (OzoneFS-Style)

We already split coverage across different acceptance environments using Robot variables instead of rerunning the same suite everywhere. For example, OzoneFS uses SCHEME / BUCKET_TYPE across different legs:

• ozone/test.sh — SCHEME:o3fs, BUCKET_TYPE:bucket
• ozone-ha/test.sh — SCHEME:ofs, BUCKET_TYPE:link
• ozonesecure/test.sh — SCHEME:ofs, BUCKET_TYPE:bucket
• ozonesecure-ha/test.sh — SCHEME:o3fs, BUCKET_TYPE:link

Proposal: Treat AWS_CLI the same way. Each existing full-s3/ invocation runs once, with either aws or aws2 passed as -v AWS_CLI:.... Both CLIs are covered across CI legs, not within each leg.

1. Remove execute_s3_smoketests from hadoop-ozone/dist/src/main/compose/testlib.sh.

2. Replace each execute_s3_smoketests ... call with a single execute_robot_test ... -v AWS_CLI:<cli> ... s3, using the assignment table below.

3. EC legs: Export AWS_CLI=... in ozone/test-ec.sh and ozonesecure/test-ec.sh. In common/ec-test.sh, use:

   execute_robot_test scm -v AWS_CLI:${AWS_CLI:-aws} -v BUCKET:erasure s3

No new acceptance suites and no workflow changes. The number of full s3/ invocations remains the same as on master before this PR.

How to Split v1 vs v2

Pattern A - One CLI per script (most legs)

#suite:	Script	Proposed CLI
HA-unsecure	ozone-ha/test-haproxy-s3g.sh	aws
secure	ozonesecure/test.sh	aws
HA-secure	ozonesecure-ha/test.sh	aws2
HA-secure	ozonesecure-ha/test-haproxy-s3g.sh	aws
misc	ozonesecure/test-vault.sh	aws

Pattern B - EC pair (unsecure vs secure)

Script	Proposed configuration
ozone/test-ec.sh	export AWS_CLI=aws
ozonesecure/test-ec.sh	export AWS_CLI=aws2

Pattern C - HA-unsecure layout loop (split across existing runs, no extra invocations)

ozone-ha/test.sh already runs the full s3/ suite three times: OBJECT_STORE, LEGACY, and FILE_SYSTEM_OPTIMIZED. The no-bucket-type tests are excluded after the first pass.

Proposal: Add a case on layout to select the CLI:

Pass	BUCKET_LAYOUT	Proposed CLI	Notes
1	OBJECT_STORE	aws2	Full directory, including no-bucket-type tests
2	LEGACY	aws	Excluded tags
3	FILE_SYSTEM_OPTIMIZED	aws2	Excluded tags

Expected totals: 10 full s3/ invocations — 6 × aws and 4 × aws2, versus 20 invocations if every current call site retained the double-run behavior.

Why This Split?

1. CI cost: One CLI per invocation brings wall-clock time and total work back to one pass per leg, addressing the double-run issue on the critical path.
2. Reuse existing dimensions: The only job that already performs multiple full s3/ runs is the HA-unsecure layout loop. Assigning aws to LEGACY and aws2 to OBJECT_STORE and FILE_SYSTEM_OPTIMIZED splits v1/v2 within that existing loop, with zero extra Robot invocations. This is the closest equivalent to using different backend knobs instead of running everything twice.
3. v2 on selected HA/secure paths: aws2 covers HA-unsecure OBJECT_STORE (full suite) and FILE_SYSTEM_OPTIMIZED, the HA-secure main S3 run, and secure EC.
4. Default v1 elsewhere: Legs that historically ran one full s3/ suite with implicit aws remain on aws unless explicitly listed above.

Trade-off

Each Robot case runs under one CLI on one backend configuration. We do not run every test with both CLIs on the same cluster. Regressions that occur only with the other CLI on the same backend would require a different leg or a targeted test.

Does this partition look reasonable before I commit and push the changes? I am happy to adjust the aws vs. aws2 mapping, especially across HA layouts, secure-HA, and EC, if you prefer a different split.

cc @adoroszlai @jojochuang

[adoroszlai]

Thanks @sreejasahithi, sounds good to me.

[adoroszlai]

@sreejasahithi #11761 removed some duplicate S3 combinations. ozone-ha/test.sh no longer runs all 3 layouts, so "Pattern C" no longer applies. But the plan is still good.

[sreejasahithi]

@jojochuang Could you please also review the plan.

[sreejasahithi]

Given that HDDS-17707 is moving towards removing AWS CLI/SDK v1 entirely rather than maintaining support for both versions, I wanted to highlight that this PR (HDDS-16467) may no longer be necessary.

• HDDS-17707. Remove AWS SDK for Java v1 #11939 (merged) removed AWS SDK for Java v1.
• HDDS-17707. Remove AWS CLI v1 from ozone-runner ozone-docker-runner#122 (open) removes AWS CLI v1 and makes aws point to CLI v2.

Once the runner image is bumped, every S3 smoketest runs on v2 automatically, so the dual-run / AWS_CLI split this PR adds is no longer needed. If any v2 behaviour differences that break the smoketests will be taken care in the runner-bump PR.

CC: @adoroszlai @jojochuang what do you think? Does it make sense to close this PR given the direction of HDDS-17707?

[adoroszlai]

I wanted to highlight that this PR (HDDS-16467) may no longer be necessary.

Agree. Thanks @sreejasahithi for working on this.