Repository navigation
fix(azure-storage): per-account queue and table namespaces, storage root routing by host, Service Bus RFC1123 schedule - #1436
Conversation
…oot routing by host, Service Bus RFC1123 schedule
NitinKumar004
left a comment
There was a problem hiding this comment.
Request changes on the bare-host user-agent routing (AZSTO-07). The rest checked out.
Built b4b41ac and ran serve with the kit cert, then sent GET /?comp=list to the bare host with different User-Agents. A container ctr1 and a queue q1 existed in the default account.
| User-Agent | Listed | Expected |
|---|---|---|
azsdk-go-azqueue, azsdk-python-storage-queue, azsdk-java/js/net queue SDKs, az CLI CommandName/storage.queue.list |
q1 | q1 |
| azsdk-go-azblob, azsdk-python-storage-blob | ctr1 | ctr1 |
| curl (no UA) | ctr1 | ambiguous |
Legacy Azure-Storage/2.1.0 (Python...) queue client |
ctr1 | q1 |
myqueue-worker azsdk-go-azblob/v1.6.0 |
q1 | ctr1 |
The last row is the problem. isQueueClient (server/azure/queue/account.go) uses strings.Contains(lower(UA), "queue"). Any blob or table client whose application ID, telemetry prefix or az CommandName contains "queue" is routed to the Queue handler on the bare host. That includes ?restype=service&comp=properties and ?restype=account. A service called queue-processor using azblob with ApplicationID set would get queue results back from ListContainers.
Queue clients with no recognisable UA fall through to Blob. This now differs from before the PR, when the bare host always meant Queue.
Suggested change:
- Match only the SDK product token, for example
azsdk-(go-azqueue|python-storage-queue|js-storage-queue|java-azure-storage-queue|net-Storage\.Queues). Do not match free text from the application ID or az command name. - Keep Blob as the bare-host default. Document in the handler comment and in the docs that
{account}.queue.core.windows.net(or a path-style/{account}/) is the reliable way to reach queues. Legacy or UA-less clients should use that form.
Other checks, all fine:
- Queue and table isolation (AZSTO-06):
queueKeyfor the default account returns the bare name, so existing snapshots and keys stay readable. Queues and tables onacct1.queueandacct1.tablewere separate from the bare-host ones. A message sent throughacct1was not visible on the bare host. Tablet1existed in both namespaces independently. - Event Grid delivery: it tries
{account}/{queue}first, then the bare name, so default-account subscriptions still deliver. I did not run the Functions queue-trigger test locally. The existing queueTrigger test inserver/azure/functions_blob_trigger_test.goshould cover it, and CI is still running. - Cosmos root decline (AZSTO-08):
GET /with no query still returns the account JSON (azcosmos-style probe). Only roots withcomp=orrestype=are declined. Bare-host blob service properties returned the defaultStorageServiceProperties. Persistence is covered by a unit test that I did not run locally. - Service Bus schedule (AZMSG-04):
parseScheduledEnqueueTimeaccepts RFC1123, RFC1123Z, RFC3339Nano and a no-zone ISO form.visibleAtismax(now+delay, scheduled)using the provider'snow, so the schedule followsconfig.Clock. I read the code but did not run the curl RFC1123 schedule live; please confirm with the unit test indataplane_message_test.go.
CI was still pending at review time (Test, Lint, Race, Contrib).
Plan (fast mode)
Rows: AZSTO-07, AZSTO-08, AZSTO-06 (queue + table), AZMSG-04. Re-checked on origin/development 07ee8a5 first.
GET /?comp=liston{acct}.blob.core.windows.netalready lists containers (the Queue handler declines other services' hosts). Still broken on the bare host (https://127.0.0.1:4568/):server/azure/queue/handler.goMatches claimed every rootcomp=list, so azblob ListContainers got<Queues>. Azurite gives each service its own port, and real Azure gives each its own host, so neither has to share a root. Fix: on a bare host, root List and the service calls go to Queue only when a Queue client sends them (Azure SDK user agent: azsdk-go-azqueue, azsdk-python-storage-queue, azsdk-net-Storage.Queues, ...). Everything else goes to Blob.server/azure/cosmosdb/handler.goMatches took any root request withoutcomp=listas the Cosmos account probe. The probe never carries a query, so Cosmos now declines a root request withcomp=orrestype=. Blob serves Get/Set Blob Service Properties through the existingBlobServiceConfig(delete retention and CORS, shared with ARMblobServices/default) and Get Account Information. Queue and Table answer the same calls on their own hosts (azurearm.ServeStorageServiceOp: Get reports the defaults, Set validates the XML and returns 202). Source: Blob/Queue/Table "Get/Set Service Properties" and "Get Account Information" REST docs.{acct}.queue|table.core.windows.nethost when it names an existing storage account, otherwise a path-style/{acct}/prefix (Azurite/IP style). Driver keys are{acct}/{name}, with bare names for the defaultcloudemuaccount and the bare host, so snapshots taken before this change load into the default account unchanged. Listings are filtered per account. Event Grid StorageQueue delivery tries{account from resourceId}/{queue}first, then the default account.server/azure/servicebus/dataplane.goparsed only RFC3339 and usedtime.Until(wall clock). It now accepts RFC1123 (the documented format), RFC1123Z, RFC3339 and ISO 8601 without a zone. The absolute instant goes to the provider asSendMessageInput.ScheduledEnqueueTime, and the Service Bus mock holds the message until then onconfig.Clock.Tests (each fails on the old code):
server/azure/storage_dataplane_isolation_test.go(queue/table isolation by host and path, bare-host list routing, service properties and account info by host, blob puts not taken by the path peel) andTestDataPlaneScheduledEnqueueTimeFormats(FakeClock, 3 formats).TestStorageHostRoutesListToItsServicenow expects List Containers on the bare host.Verification
go build ./..., vet,-raceon touched packages,server/azure/...,providers/azure/...,persist/...,server/wire/...green. golangci-lint--new-from-rev0 issues.go generateproduces no diff.serve(azqueue, aztables, azblob, Service Bus REST): 20/20 pass on this branch, and 14 fail on the origin/development binary. Covered: the same queue/table name in two accounts stays isolated (path-style and vhost), the default account doesn't list them, azblob ListContainers works on the account host and the bare host, blob service GetProperties/SetProperties/GetAccountInfo work, and a message with an RFC1123 ScheduledEnqueueTimeUtc 5s ahead gets 204 right away and 200 after 6s.azurerm_storage_queue/azurerm_storage_tablein two accounts can't run yet, and this PR doesn't cause that. azurerm v4 creates queues over ARMqueueServices/default/queues, which cloudemu answers with 501 (not modeled). It checks tables over the real{acct}.table.core.windows.netDNS on 443, which the container can't resolve. Both happen on origin/development as well. Follow-up row below. Docker: pending Gate 2.Deferred (next PR)
queueServices/default/queuesandtableServices/default/tablesCRUD backed by the per-account queue/table data plane, so azurerm_storage_queue/table work (server/azure/storageaccount/service_settings.goserveServiceChild).DeleteStorageAccountcovers blob only).comp=stats) and user-delegation key are not implemented.{acct}/{queue}key, so a binding to the bare queue name doesn't fire.