feat(sync): add --force-upload flag to control S3 upload dedup - #9008
feat(sync): add --force-upload flag to control S3 upload dedup#9008vishwakt wants to merge 2 commits into
Conversation
`sam sync` previously hardcoded `force_upload=True` in three places, which bypassed the SHA-based dedup in `S3Uploader.upload_with_dedup` and caused every sync to re-upload all artifacts even when nothing had changed. Issue aws#8168 reports a 20%+ (90+ second) speedup from toggling this off in a fork. Expose `--force-upload` on `sam sync` (default False, matching `sam deploy`'s default) and thread it through: - PackageContext and DeployContext, replacing the two hardcoded `True` values in `samcli/commands/sync/command.py`. - The S3Uploader instantiation in ZipFunctionSyncFlow now reads `force_upload` from `self._deploy_context.force_upload` so the flag actually controls the oversized-zip upload path too. Schema regenerated. Unit tests updated to assert the flag flows through PackageContext, DeployContext, and the ZipFunctionSyncFlow S3 upload path for both True and False values. Closes aws#8168
Resolve conflicts with aws#9106 (--express) and language extensions support: keep force_upload plumbed through PackageContext/DeployContext alongside the new language_extensions and express parameters; regenerate schema.
| @image_repository_option | ||
| @image_repositories_option | ||
| @s3_bucket_option(disable_callback=True) # pylint: disable=E1120 | ||
| @force_upload_option |
There was a problem hiding this comment.
[BUG] The new option will not appear in sam sync --help because it was never registered in samcli/commands/sync/core/options.py.
SyncCommand.format_options delegates to CoreCommand._format_options(..., formatting_options=OPTIONS_INFO), and that helper only renders parameters whose name is present in one of the groups:
for param in params
if param.name in options.get("option_names", {}).keys()Since force_upload is absent from REQUIRED_OPTIONS, AWS_CREDENTIAL_OPTION_NAMES, INFRASTRUCTURE_OPTION_NAMES, CONFIGURATION_OPTION_NAMES, ADDITIONAL_OPTIONS, and OTHER_OPTIONS, the flag is silently dropped from the help output. It is also missing from ALL_OPTIONS, which the formatter uses for column justification.
This is the reason sam deploy and sam package register it explicitly — samcli/commands/deploy/core/options.py:39 and samcli/commands/package/core/options.py both list "force_upload" under their DEPLOYMENT_OPTIONS group. No existing unit test catches this: tests/unit/commands/sync/core/test_command.py stubs get_params with a fixed list, so the help assertions pass either way.
Because the PR flips the effective default for sam sync, an undiscoverable opt-out flag is a concrete usability problem: users who hit a regression have no way to find --force-upload from the CLI. Sync has no DEPLOYMENT_OPTIONS group, so the closest match to the decorator placement is the infrastructure group:
INFRASTRUCTURE_OPTION_NAMES: List[str] = [
"parameter_overrides",
"capabilities",
"s3_bucket",
"force_upload",
"s3_prefix",
...
]Consider also adding a tests/unit/commands/sync/core/test_options.py mirroring the existing test_all_options_formatted test that other commands have (deploy, package, build, logs, etc.), which asserts sorted(ALL_OPTIONS) == sorted(command_options + ["help"]). That test is what would have caught this, and sync is currently one of the few commands missing it.
Which issue(s) does this change fix?
Closes #8168
Why is this change necessary?
sam synccurrently hardcodesforce_upload=Truein three places. That bypasses the SHA-based dedup insideS3Uploader.upload_with_dedup, so every sync re-zips and re-uploads every artifact to S3 even when nothing has actually changed. The original issue reports a 20%+ (~90s) speedup per run from disabling this on a fork.How does it address the issue?
Exposes
--force-uploadonsam sync, defaulting toFalse(matching the default onsam deployandsam package), and threads the flag through every code path that was previously hardcoded:samcli/commands/sync/command.py— replacesforce_upload=TrueinPackageContext(...)andDeployContext(...)with the new option.samcli/lib/sync/flows/zip_function_sync_flow.py— theS3Uploaderconstructed for oversized lambda zips now readsforce_uploadfromself._deploy_context.force_uploadinstead of hardcodingTrue. This was the third code path that defeated the dedup.force_upload_option()decorator fromsamcli/commands/_utils/options.py, so help text, click metadata, and config-file plumbing are consistent withsam deploy/sam package.schema/samcli.jsonwas regenerated viapython -m schema.make_schemaand now listsforce_uploadas a sync parameter.Users who actually want the old behavior can pass
--force-upload.What side effects does this change have?
--force-upload,sam syncwill now skip uploading artifacts whose SHA already exists in S3 — which is what the issue requester (and the maintainer who welcomed a PR) asked for. Backward-compatible toggle via the new flag.Tests
tests/unit/commands/sync/test_command.py— existing assertions updated fromforce_upload=Truetoforce_upload=self.force_upload, plus a new parameterized test (test_force_upload_flag_propagates_to_package_and_deploy_contexts) covering bothTrueandFalseand asserting the value reaches bothPackageContextandDeployContext.tests/unit/lib/sync/flows/test_zip_function_sync_flow.py— new parameterized test (test_s3_uploader_uses_deploy_context_force_upload) confirming theS3Uploaderis constructed withdeploy_context.force_uploadfor both values.tests/unit/commands/samconfig/test_samconfig.py—do_cliarg-order assertion updated for the new positional parameter.black --check setup.py samcli tests schemaandruff check samcli schemaare both clean locally.Mandatory Checklist
sam deploy/sam package)tests/integration/sync/)make prpasses locally (black --check,ruff check,python -m schema.make_schema, full unit suite)make update-reproducible-reqsif dependencies were changed (no dep changes)force_upload_option; no separate docs file in this repo for this command)By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.