Skip to content

HDDS-17707. Remove AWS CLI v1 from ozone-runner - #122

Open
yandrey321 wants to merge 2 commits into
apache:masterfrom
yandrey321:HDDS-17707
Open

yandrey321 wants to merge 2 commits into
apache:masterfrom
yandrey321:HDDS-17707

Conversation

@yandrey321

Copy link
Copy Markdown

What changes were proposed in this pull request?

Problem. The ozone-runner image installs AWS CLI v1 (awscli==1.38.15, from pip) as aws. AWS CLI v1 is end of
life, like the AWS SDK for Java v1 that apache/ozone#11939 (HDDS-17707) removes from Ozone. HDDS-16466 already added
AWS CLI v2 next to it, as aws2, but the Ozone smoketests still call aws, so they still run v1.

Change.

  • v1 is gone: awscli is dropped from the pip install. robotframework and boto3 stay; boto3 brings its own
    botocore.
  • aws is AWS CLI v2: the same pinned build (AWS_CLI_V2_VERSION=2.27.50, the official zip) is now linked as
    both aws and aws2. aws2 is kept as an alias for anything that started using it after HDDS-16466.
  • AWS_PAGER="": CLI v2 can send its output through a pager; a test container must never do that.
  • A build check: the build fails if aws --version is not CLI v2.

Key decisions.

  • Keep the aws name instead of switching the smoketests to aws2. The existing tests (commonawslib.robot and
    the S3, security and FCQ suites) work unchanged, and new tests don't need to know which CLI version they get.
  • Keep aws2: it costs one symlink, and removing it could break a test that started using it.

Side effects.

  • Ozone acceptance tests run aws as CLI v2 once Ozone bumps docker.ozone-runner.version to an image with this
    change. That bump should run the full acceptance suite.

  • Behaviour differences of CLI v2 that the Ozone smoketests could notice:

    • default request checksums on uploads;
    • some output details;
    • --cli-binary-format for blob arguments.

    Any failures should be fixed in the Ozone PR that bumps the image. The S3 Gateway already passes the AWS SDK v2
    integration suite, which sends the same default checksums.

  • Ozone release branches are not affected: they pin their own runner image versions.

Co-authored-by: Claude Opus 5.5 noreply@anthropic.com

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-17707

How was this patch tested?

  • Image build: docker build succeeds (locally on aarch64; the CI workflow builds the image too). The new
    version check passes in the build.
  • In the built image:
    • aws --version and aws2 --version both report aws-cli/2.27.50, from /usr/local/bin/aws and
      /usr/local/bin/aws2;
    • AWS_PAGER is empty;
    • pip3 list has no awscli, and still has boto3 1.37.15 (with botocore 1.37.38) and robotframework
      6.1.1.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
@adoroszlai

Copy link
Copy Markdown
Contributor

Thanks @yandrey321 for the patch. Please enable workflows in your fork.

https://github.com/yandrey321/ozone-docker-runner/actions

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants