Skip to content

Validate copy sources with HEAD instead of downloading the blob - #2680

Open
Andrew Gaul (gaul) wants to merge 2 commits into
Azure:mainfrom
gaul:fix-copy-gzip-source
Open

Validate copy sources with HEAD instead of downloading the blob#2680
Andrew Gaul (gaul) wants to merge 2 commits into
Azure:mainfrom
gaul:fix-copy-gzip-source

Conversation

@gaul

Copy link
Copy Markdown
Contributor

startCopyFromURL and copyFromURL validate the copy source by fetching ?comp=metadata through axios. Azurite serves that request as a full Blob_Download (issue #646), so every validated copy downloaded the entire source only to discard it, and a source blob declaring Content-Encoding: gzip over bytes that are not really gzip made axios decompression fail and turned every copy of that blob into a 500.

Validate with a bodiless HEAD (Get Blob Properties) request instead. Error details live only in response bodies, and reading an archived source must fail the copy like the download did, so those cases repeat the request as the previous comp=metadata GET, now with axios decompression disabled.

Copilot AI lite review requested due to automatic review settings July 24, 2026 00:19

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR optimizes blob copy-source validation in Azurite by switching from a metadata GET (which Azurite currently serves as a full blob download) to a bodiless HEAD request, avoiding unnecessary data transfer and preventing axios decompression failures when the source declares Content-Encoding: gzip over non-gzip bytes.

Changes:

  • Update copy-source validation to use HEAD (Get Blob Properties) and retry with GET ?comp=metadata only when needed, with axios decompression disabled on the retry.
  • Add a SAS-based regression test ensuring copy succeeds when the source declares Content-Encoding: gzip but the payload is not gzipped.
  • Document the behavior change in ChangeLog.md.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

File Description
tests/blob/sas.test.ts Adds regression coverage for copy with misleading Content-Encoding: gzip on the source blob.
src/blob/handlers/BlobHandler.ts Switches copy-source validation to HEAD, with a conditional GET ?comp=metadata retry to preserve error details and avoid decompression issues.
ChangeLog.md Notes the optimization/fix in the upcoming release section.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread tests/blob/sas.test.ts Outdated
Comment on lines +601 to +603
await blob2.beginCopyFromURL(blob1.url);

const properties = await blob2.getProperties();

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

Copilot AI review requested due to automatic review settings July 24, 2026 00:30

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated no new comments.

@jainakanksha-msft

Copy link
Copy Markdown
Member

Andrew Gaul (@gaul) , could you please refresh your PR with main, and address the review comments if any to move this PR forward.

Andrew Gaul (gaul) and others added 2 commits August 13, 2026 09:43
startCopyFromURL and copyFromURL validate the copy source by fetching
<source>?comp=metadata through axios.  Azurite serves that request as
a full Blob_Download (issue Azure#646), so every validated copy downloaded
the entire source only to discard it, and a source blob declaring
Content-Encoding: gzip over bytes that are not really gzip made axios
decompression fail and turned every copy of that blob into a 500.

Validate with a bodiless HEAD (Get Blob Properties) request instead.
Error details live only in response bodies, and reading an archived
source must fail the copy like the download did, so those cases repeat
the request as the previous comp=metadata GET, now with axios
decompression disabled.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
beginCopyFromURL resolves once the poller is created, not when the
copy finishes, so poll to completion like the neighboring tests.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Copilot AI review requested due to automatic review settings August 13, 2026 16:48
@gaul

Copy link
Copy Markdown
Contributor Author

Andrew Gaul (Andrew Gaul (@gaul)) , could you please refresh your PR with main, and address the review comments if any to move this PR forward.

Done.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated no new comments.

Suppressed comments (1)

src/blob/handlers/BlobHandler.ts:744

  • The archive-tier special-case here forces a GET ?comp=metadata even when the HEAD succeeds. This validation helper doesn’t have the destination tier option available, so it can incorrectly block scenarios that the metadata store explicitly allows (e.g., same-account copy from an Archive blob when a destination tier is specified), and it also relies on ?comp=metadata behaving like a download to fail (but BlobHandler.getProperties() handles comp=metadata as a 200 and does not enforce Archive restrictions).

Consider limiting the retry GET to non-200 responses (auth/404/etc.) and let the actual copy path enforce Archive semantics based on the real copy options.

    if (
      validationResponse.status !== 200 ||
      validationResponse.headers["x-ms-access-tier"] === "Archive"
    ) {
      // Error details live only in response bodies, and reading an

@gaul

Copy link
Copy Markdown
Contributor Author

Suppressed comments (1)

Re: the suppressed comment on src/blob/handlers/BlobHandler.ts:744 — the key premise doesn't hold, so I've left the archive-tier check as is.

GET <blob>?comp=metadata is not served by getProperties() here. Azurite serves it as a full blob download — that's issue #646, the quirk this PR works around — and the download path does enforce Archive restrictions. Measured against an archived blob on this branch:

HEAD                → 200, x-ms-access-tier: Archive
GET ?comp=metadata  → 409, <Code>BlobArchived</Code>

So the archive-tier check is load-bearing rather than redundant. HEAD returns 200 for an archived source, so without the special case an archived source would pass validation where it previously failed. tests/blob/sas.test.ts pins this: "Copy blob across accounts should fail if source is archived" asserts 409 CannotVerifyCopySource. I applied the suggestion (retry the GET only on non-200) and that test fails with BlobArchived instead — the copy is still rejected, but by the metadata store rather than the validation helper, changing the error code clients see.

On the concern that this can block a same-account copy from an Archive blob when a destination tier is specified: that's an accurate description of the behavior, but it is pre-existing rather than introduced by this PR. validateCopySource already ran for same-account copies whenever the source URL carries ?sig= (see the call site above), and the previous code issued the same ?comp=metadata GET unconditionally. Running exactly that scenario — same account, Archive source, tier: "Hot" — against main without this PR returns the same 409 CannotVerifyCopySource. This PR preserves that behavior deliberately; the archive-tier check exists precisely to keep the outcome identical after moving the happy path to HEAD.

There is a genuine inconsistency in that area worth noting separately: LokiBlobMetadataStore.startCopyFromURL() permits a same-account Archive source when a destination tier is given, while validateCopySource rejects it first. Resolving that means establishing what the real service returns in that case, so it belongs in its own change rather than this one.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants