Skip to content

fix(azure-storage): per-account queue and table namespaces, storage root routing by host, Service Bus RFC1123 schedule - #1436

Merged
NitinKumar004 merged 2 commits into
developmentfrom
fix/azure-storage-dataplane-isolation
Oct 4, 2026
Merged

NitinKumar004 merged 2 commits into
developmentfrom
fix/azure-storage-dataplane-isolation

Conversation

@NitinKumar004

Copy link
Copy Markdown
Collaborator

Plan (fast mode)

Rows: AZSTO-07, AZSTO-08, AZSTO-06 (queue + table), AZMSG-04. Re-checked on origin/development 07ee8a5 first.

  • AZSTO-07: since fix(azure): persist resource groups across restart, blob virtual-host account routing, subscription-scoped RG purge #1408, GET /?comp=list on {acct}.blob.core.windows.net already lists containers (the Queue handler declines other services' hosts). Still broken on the bare host (https://127.0.0.1:4568/): server/azure/queue/handler.go Matches claimed every root comp=list, so azblob ListContainers got <Queues>. Azurite gives each service its own port, and real Azure gives each its own host, so neither has to share a root. Fix: on a bare host, root List and the service calls go to Queue only when a Queue client sends them (Azure SDK user agent: azsdk-go-azqueue, azsdk-python-storage-queue, azsdk-net-Storage.Queues, ...). Everything else goes to Blob.
  • AZSTO-08: server/azure/cosmosdb/handler.go Matches took any root request without comp=list as the Cosmos account probe. The probe never carries a query, so Cosmos now declines a root request with comp= or restype=. Blob serves Get/Set Blob Service Properties through the existing BlobServiceConfig (delete retention and CORS, shared with ARM blobServices/default) and Get Account Information. Queue and Table answer the same calls on their own hosts (azurearm.ServeStorageServiceOp: Get reports the defaults, Set validates the XML and returns 202). Source: Blob/Queue/Table "Get/Set Service Properties" and "Get Account Information" REST docs.
  • AZSTO-06: the Queue and Table handlers had a single namespace. They now resolve the account the same way blob does (fix(azure): persist resource groups across restart, blob virtual-host account routing, subscription-scoped RG purge #1408): the {acct}.queue|table.core.windows.net host when it names an existing storage account, otherwise a path-style /{acct}/ prefix (Azurite/IP style). Driver keys are {acct}/{name}, with bare names for the default cloudemu account and the bare host, so snapshots taken before this change load into the default account unchanged. Listings are filtered per account. Event Grid StorageQueue delivery tries {account from resourceId}/{queue} first, then the default account.
  • AZMSG-04: server/azure/servicebus/dataplane.go parsed only RFC3339 and used time.Until (wall clock). It now accepts RFC1123 (the documented format), RFC1123Z, RFC3339 and ISO 8601 without a zone. The absolute instant goes to the provider as SendMessageInput.ScheduledEnqueueTime, and the Service Bus mock holds the message until then on config.Clock.

Tests (each fails on the old code): server/azure/storage_dataplane_isolation_test.go (queue/table isolation by host and path, bare-host list routing, service properties and account info by host, blob puts not taken by the path peel) and TestDataPlaneScheduledEnqueueTimeFormats (FakeClock, 3 formats). TestStorageHostRoutesListToItsService now expects List Containers on the bare host.

Verification

  • Gates: go build ./..., vet, -race on touched packages, server/azure/..., providers/azure/..., persist/..., server/wire/... green. golangci-lint --new-from-rev 0 issues. go generate produces no diff.
  • Real-SDK e2e against serve (azqueue, aztables, azblob, Service Bus REST): 20/20 pass on this branch, and 14 fail on the origin/development binary. Covered: the same queue/table name in two accounts stays isolated (path-style and vhost), the default account doesn't list them, azblob ListContainers works on the account host and the bare host, blob service GetProperties/SetProperties/GetAccountInfo work, and a message with an RFC1123 ScheduledEnqueueTimeUtc 5s ahead gets 204 right away and 200 after 6s.
  • TF (binary): azurerm_storage_queue/azurerm_storage_table in two accounts can't run yet, and this PR doesn't cause that. azurerm v4 creates queues over ARM queueServices/default/queues, which cloudemu answers with 501 (not modeled). It checks tables over the real {acct}.table.core.windows.net DNS on 443, which the container can't resolve. Both happen on origin/development as well. Follow-up row below. Docker: pending Gate 2.

Deferred (next PR)

  • ARM queueServices/default/queues and tableServices/default/tables CRUD backed by the per-account queue/table data plane, so azurerm_storage_queue/table work (server/azure/storageaccount/service_settings.go serveServiceChild).
  • Deleting a storage account does not yet delete its queues and tables (DeleteStorageAccount covers blob only).
  • Queue/Table Set Service Properties is validated but not stored. Get Service Stats (comp=stats) and user-delegation key are not implemented.
  • Functions queueTrigger for a queue in a non-default account: the trigger sees the {acct}/{queue} key, so a binding to the bare queue name doesn't fire.

…oot routing by host, Service Bus RFC1123 schedule

@NitinKumar004 NitinKumar004 left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Request changes on the bare-host user-agent routing (AZSTO-07). The rest checked out.

Built b4b41ac and ran serve with the kit cert, then sent GET /?comp=list to the bare host with different User-Agents. A container ctr1 and a queue q1 existed in the default account.

User-Agent Listed Expected
azsdk-go-azqueue, azsdk-python-storage-queue, azsdk-java/js/net queue SDKs, az CLI CommandName/storage.queue.list q1 q1
azsdk-go-azblob, azsdk-python-storage-blob ctr1 ctr1
curl (no UA) ctr1 ambiguous
Legacy Azure-Storage/2.1.0 (Python...) queue client ctr1 q1
myqueue-worker azsdk-go-azblob/v1.6.0 q1 ctr1

The last row is the problem. isQueueClient (server/azure/queue/account.go) uses strings.Contains(lower(UA), "queue"). Any blob or table client whose application ID, telemetry prefix or az CommandName contains "queue" is routed to the Queue handler on the bare host. That includes ?restype=service&comp=properties and ?restype=account. A service called queue-processor using azblob with ApplicationID set would get queue results back from ListContainers.

Queue clients with no recognisable UA fall through to Blob. This now differs from before the PR, when the bare host always meant Queue.

Suggested change:

  • Match only the SDK product token, for example azsdk-(go-azqueue|python-storage-queue|js-storage-queue|java-azure-storage-queue|net-Storage\.Queues). Do not match free text from the application ID or az command name.
  • Keep Blob as the bare-host default. Document in the handler comment and in the docs that {account}.queue.core.windows.net (or a path-style /{account}/) is the reliable way to reach queues. Legacy or UA-less clients should use that form.

Other checks, all fine:

  • Queue and table isolation (AZSTO-06): queueKey for the default account returns the bare name, so existing snapshots and keys stay readable. Queues and tables on acct1.queue and acct1.table were separate from the bare-host ones. A message sent through acct1 was not visible on the bare host. Table t1 existed in both namespaces independently.
  • Event Grid delivery: it tries {account}/{queue} first, then the bare name, so default-account subscriptions still deliver. I did not run the Functions queue-trigger test locally. The existing queueTrigger test in server/azure/functions_blob_trigger_test.go should cover it, and CI is still running.
  • Cosmos root decline (AZSTO-08): GET / with no query still returns the account JSON (azcosmos-style probe). Only roots with comp= or restype= are declined. Bare-host blob service properties returned the default StorageServiceProperties. Persistence is covered by a unit test that I did not run locally.
  • Service Bus schedule (AZMSG-04): parseScheduledEnqueueTime accepts RFC1123, RFC1123Z, RFC3339Nano and a no-zone ISO form. visibleAt is max(now+delay, scheduled) using the provider's now, so the schedule follows config.Clock. I read the code but did not run the curl RFC1123 schedule live; please confirm with the unit test in dataplane_message_test.go.

CI was still pending at review time (Test, Lint, Race, Contrib).

@NitinKumar004
NitinKumar004 marked this pull request as ready for review October 4, 2026 09:28
@NitinKumar004
NitinKumar004 merged commit 4bc3cfb into development Oct 4, 2026
22 of 23 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant