diff --git a/CHANGELOG.md b/CHANGELOG.md index ec74be7b..0fcacc5f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,9 +8,12 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). ### Upgrade Notes -#### Agents exporting through the delegated (OBO) route: grant Observability API permissions +#### Agents that still need Observability API permissions -Agents that export telemetry through the delegated (OBO) route need `Agent365.Observability.OtelWrite` granted as both a **delegated** and an **application** permission on the blueprint app. Requires Global Administrator. +`a365 setup all` no longer requests `Agent365.Observability.OtelWrite` for blueprint agents (#501) or AI Teammates (#510). Agents that export telemetry with an app-only token through the S2S endpoint don't need it: the service authorizes a blueprint agent by its agent registration and an AI Teammate by its blueprint's registration. Grant it manually on the blueprint app (requires Global Administrator) only when: + +- the agent exports through the delegated (OBO) route: grant the **delegated** permission. +- app-only exports return 403 `insufficient_scope`: grant the **application** permission, which agent identities inherit. **Option A — Entra portal** (existing blueprints that already have the Observability inheritable-permission entry): @@ -20,13 +23,11 @@ Agents that export telemetry through the delegated (OBO) route need `Agent365.Ob - GCC Moderate: `2c672ad5-b104-44ed-8069-bb68dd138546` - GCC High: `009c6bd0-82e4-4466-95b3-4c996521f3d7` - DoD: `a9e04047-c6a7-430b-a7ae-faf8f8eed1b7` -3. **Delegated permissions** > select `Agent365.Observability.OtelWrite` > **Add permissions** -4. Repeat step 2 > **Application permissions** > select `Agent365.Observability.OtelWrite` > **Add permissions** +3. Delegated route: **Delegated permissions** > select `Agent365.Observability.OtelWrite` > **Add permissions** +4. 403 `insufficient_scope`: repeat step 2 > **Application permissions** > select `Agent365.Observability.OtelWrite` > **Add permissions** 5. **Grant admin consent for \** > confirm -**Option B — CLI**: `a365 setup admin` has been removed in this release. For blueprint agents created after this release, run `a365 setup permissions custom --resource-app-id --scopes Agent365.Observability.OtelWrite` to stamp the inheritable permission and request consent. For AI Teammates, the `a365 setup all` summary also prints the PowerShell steps for the application permission. - -Blueprint agents that export telemetry through the app-only S2S endpoint don't need these permissions, and `a365 setup all` no longer requests them for blueprint agents (#501). +**Option B — CLI**: `a365 setup admin` has been removed in this release. For blueprints that lack the inheritable-permission entry, including those created after this release, run `a365 setup permissions custom --resource-app-id --scopes Agent365.Observability.OtelWrite` to stamp it and request consent for the delegated permission. The CLI does not grant the application permission; use Option A for it. ### Added - `a365 develop-mcp grant-agents-access --agent-blueprint-id --mcp-server-name ` reports which agent instances of a blueprint are missing the permission to call a BYO MCP server, and prompts you to select which ones to grant it to (#500). @@ -91,6 +92,7 @@ Blueprint agents that export telemetry through the app-only S2S endpoint don't n - `a365 publish` no longer crashes when `manifest.json` has a non-string `name.short` value (#478). - `setup all --agent-registration-only` now exits non-zero and reports errors when the requested agent registration step fails (#478). - `a365 setup all` now exits with code 1 when blueprint-agent registration fails or cannot be verified, including `--agent-registration-only` runs with unverifiable existing registrations (#501). +- Non-admin `a365 setup all` runs now include the Observability API in the combined admin consent URL when `customBlueprintPermissions` adds it back (#510). - `a365 develop-mcp grant-agents-access --device-code` no longer prompts repeatedly within a single command, and no longer fails in embedded or remote terminals where the sign-in prompt could not be displayed (#500). - Setup no longer fails to detect the Agent 365 CLI application in tenants where it is not yet provisioned, and reports lookup errors instead of silently switching your configured client app (#489). - The first-party Agent 365 CLI app now uses device code authentication when Windows Account Manager is unavailable, avoiding unsupported browser-response errors in WSL, macOS, and Linux (#489). @@ -138,6 +140,7 @@ Blueprint agents that export telemetry through the app-only S2S endpoint don't n ### Changed - `a365 setup all` no longer requests Observability API permissions for blueprint agents; registered agents that export telemetry through the app-only S2S endpoint need no admin consent (#501). +- `a365 setup all` no longer requests Observability API permissions for AI Teammates, and `a365 setup permissions bot` and `a365 create-instance` no longer request them for any agent, because the service authorizes app-only telemetry exports by registration (#510). - Hardened token storage: the CLI no longer writes access tokens to a plaintext file — they live only in the OS-protected MSAL cache (DPAPI/Keychain/owner-only file). Any legacy plaintext cache is removed automatically; sign-in prompts are unchanged. - `develop-mcp register-external-mcp-server` now sets `exit code 1` on failure paths (validation errors, tenant detection failure, Graph unavailable, Entra app creation failure, MCP-Platform AddMcpServer failure). Previously these paths logged an error and exited `0`, which made the command's success/failure status undetectable from scripts and CI. Successful dry-run and user-initiated cancellation at the y/N prompt continue to exit `0`. - Admin consent canary path (when the caller lacks `DelegatedPermissionGrant.Read.All`) no longer prompts for Enter immediately. The CLI now polls every 5 seconds, prints a friendly progress message at 30 seconds, and responds promptly to Enter or Ctrl+C. The previous jargon-heavy message about `oauth2PermissionGrants` was rewritten in plain English; technical details are demoted to `Debug`. diff --git a/docs/agent365-guided-setup/a365-observability-instructions.md b/docs/agent365-guided-setup/a365-observability-instructions.md index f509abf4..20e2a2bf 100644 --- a/docs/agent365-guided-setup/a365-observability-instructions.md +++ b/docs/agent365-guided-setup/a365-observability-instructions.md @@ -10,9 +10,11 @@ Add Agent 365 observability to your agent at any point after `a365 setup all` ha > > **Prerequisite:** `a365 setup all` must have completed successfully so `AgentId` and `TenantId` values are present in `appsettings.json` (written by setup) or `a365.generated.config.json`. If neither source has these values, run `a365 setup all` first. -> **Source of truth:** For non-AI-Teammate blueprint agents, use the `plugins/agent365/skills/instrument-observability` skill from [microsoft/agent365-skills](https://github.com/microsoft/agent365-skills). The skill contains the current, verified code for the app-only S2S exporter in .NET, Node.js, and Python. +> **Source of truth:** For every agent, including AI Teammates, use the `plugins/agent365/skills/instrument-observability` skill from [microsoft/agent365-skills](https://github.com/microsoft/agent365-skills). The skill contains the current, verified code for the app-only S2S exporter in .NET, Node.js, and Python. -> **Blueprint telemetry model:** Every non-AI-Teammate blueprint agent exports telemetry through the S2S route (`/observabilityService/...`) with an app-only token for the exporting agent identity. Set the SDK route flag (`UseS2SEndpoint`, `useS2SEndpoint`, or `a365_use_s2s_endpoint`), wire an app-only token resolver, and do **not** use delegated Observability token caches or per-turn delegated OBS token refresh. Registered blueprint agents do not need `Agent365.Observability.OtelWrite`; `a365 setup all` no longer requests it for blueprint agents. +> **Telemetry model:** Every agent, including AI Teammates, exports telemetry through the S2S route (`/observabilityService/...`) with an app-only token for the exporting agent identity. Set the SDK route flag (`UseS2SEndpoint`, `useS2SEndpoint`, or `a365_use_s2s_endpoint`), wire an app-only token resolver, and do **not** use delegated Observability token caches or per-turn delegated OBS token refresh. Agents do not need `Agent365.Observability.OtelWrite`: the service authorizes a registered agent, or an AI Teammate through its blueprint's registration, and `a365 setup all` no longer requests the permission. +> +> **AI Teammates:** use the same app-only S2S exporter as other agents. The delegated route (hosting-package token cache) still works only if you grant the delegated `OtelWrite` permission yourself; see [OtelWrite App Role Assignment](#otelwrite-app-role-assignment). --- @@ -75,11 +77,13 @@ If `agentType` and `authMode` are already present in the detection cache (from a Store `agentType` (`ai-teammate` = AI Teammate, or `system-agent` = Agent (Non AI Teammate)) and `authMode`: - **AI Teammate:** `user-delegated` (OBO as signed-in user) or `agentic-identity` (OBO as agent's own M365 identity) -- **Agent (Non AI Teammate):** workload auth may be `obo`, `s2s`, or `both`, but telemetry wiring is always the app-only S2S exporter described above. +- **Agent (Non AI Teammate):** workload auth may be `obo`, `s2s`, or `both`. + +For both agent kinds, telemetry wiring is always the app-only S2S exporter described above. **Update `.a365-workspace-detection.json`** — merge `agentType` and `authMode` into the existing cache file, preserving all other fields (`agentStack`, `programmingLanguage`, `usesTeamsOrCopilot`, `detectedAt`). Use the **Write** tool to write the merged object back. -For non-AI-Teammate agents, `authMode` controls workload tokens and permission grants only; it must not switch telemetry back to a delegated-route exporter. Phases 3–5 must select the S2S route and app-only token resolver in every auth mode. +`authMode` controls workload tokens and permission grants only; it must not switch telemetry back to a delegated-route exporter. Phases 3–5 must select the S2S route and app-only token resolver in every auth mode. **TaskUpdate** — Mark complete: "Determine agent type and authentication mode" @@ -116,13 +120,13 @@ For non-AI-Teammate agents, `authMode` controls workload tokens and permission g 1. **Bash** — Run package installation (path-dependent): - **AI Teammate / legacy delegated path only**: + **Legacy delegated route only** (`a365 setup all` no longer grants the delegated `OtelWrite` scope this route needs; grant it first with `a365 setup permissions custom` or `customBlueprintPermissions`, see [OtelWrite App Role Assignment](#otelwrite-app-role-assignment)): ```bash dotnet add package Microsoft.Agents.A365.Observability.Runtime dotnet add package Microsoft.Agents.A365.Observability.Hosting ``` - **Non-AI-Teammate blueprint agents — use the app-only S2S distro in every workload auth mode** (preferred; do NOT also add Runtime/Hosting): + **All agents, including AI Teammates — use the app-only S2S distro in every workload auth mode** (preferred; do NOT also add Runtime/Hosting): ```bash dotnet add package Microsoft.OpenTelemetry --version 1.0.0-beta.1 dotnet add package Azure.Identity @@ -177,7 +181,7 @@ For non-AI-Teammate agents, `authMode` controls workload tokens and permission g npm install @microsoft/agents-a365-observability npm install @microsoft/agents-a365-runtime npm install @microsoft/agents-a365-observability-hosting - # Non-AI-Teammate blueprint telemetry uses app-only S2S in every workload auth mode. + # Telemetry uses app-only S2S for every agent, including AI Teammates. npm install @azure/msal-node @azure/identity ``` @@ -206,10 +210,10 @@ For non-AI-Teammate agents, `authMode` controls workload tokens and permission g 1. **Bash** — Run package installation (unified distro + S2S deps): ```bash pip3 install microsoft-opentelemetry 2>/dev/null || pip install microsoft-opentelemetry - # Non-AI-Teammate blueprint telemetry uses app-only S2S in every workload auth mode. + # Telemetry uses app-only S2S for every agent, including AI Teammates. pip3 install msal azure-identity httpx 2>/dev/null || pip install msal azure-identity httpx ``` - > **AI Teammate / legacy delegated path:** follow the AI Teammate sample's delegated hosting package guidance instead of the blueprint app-only resolver. + > **Legacy delegated route only** (`a365 setup all` no longer grants the delegated `OtelWrite` scope this route needs; grant it first with `a365 setup permissions custom` or `customBlueprintPermissions`, see [OtelWrite App Role Assignment](#otelwrite-app-role-assignment)): follow the sample's delegated hosting package guidance instead of the app-only resolver. 2. **Optional auto-instrumentation extensions** — ask the user which AI framework they use and install accordingly: ```bash @@ -245,9 +249,8 @@ For non-AI-Teammate agents, `authMode` controls workload tokens and permission g 2. **Edit** — Add observability wiring following the reference pattern in `dotnet-observability.md`: - Add using directives for the observability namespaces - - **Non-AI-Teammate blueprint agents:** follow the .NET reference in [microsoft/agent365-skills](https://github.com/microsoft/agent365-skills) (`plugins/agent365/skills/instrument-observability/references/dotnet-observability.md`). The required shape is `UseMicrosoftOpenTelemetry(...)` with `o.Agent365.UseS2SEndpoint = true` (or `o.Agent365.Exporter.UseS2SEndpoint = true` on older distros) plus an app-only token resolver for the exporting agent identity. - - **Do not** call `AddAgenticTracingExporter()` for non-AI-Teammate blueprint telemetry; it wires the delegated route and delegated token cache. - - **AI Teammate only:** keep the existing hosting-package pattern if that is what the AI Teammate sample uses. + - **All agents, including AI Teammates:** follow the .NET reference in [microsoft/agent365-skills](https://github.com/microsoft/agent365-skills) (`plugins/agent365/skills/instrument-observability/references/dotnet-observability.md`). The required shape is `UseMicrosoftOpenTelemetry(...)` with `o.Agent365.UseS2SEndpoint = true` (or `o.Agent365.Exporter.UseS2SEndpoint = true` on older distros) plus an app-only token resolver for the exporting agent identity. + - **Do not** call `AddAgenticTracingExporter()`; it wires the delegated route and delegated token cache. Keep an existing hosting-package pattern only on the legacy delegated route (`a365 setup all` no longer grants the delegated `OtelWrite` scope this route needs; grant it first with `a365 setup permissions custom` or `customBlueprintPermissions`, see [OtelWrite App Role Assignment](#otelwrite-app-role-assignment)). - Optionally register baggage middleware only when it does not introduce a delegated Observability token dependency. - Mark all new lines with: `// A365 Observability — best-effort instrumentation (verify against official sample)` @@ -260,7 +263,7 @@ For non-AI-Teammate agents, `authMode` controls workload tokens and permission g 2. **Edit** — Add observability initialization following the reference pattern in `nodejs-observability.md`: - Initialize `useMicrosoftOpenTelemetry({ a365: { enabled: true, useS2SEndpoint: true, tokenResolver } })` **before** any LLM/framework imports. - Implement `tokenResolver` as an app-only resolver for the exporting agent identity, using the current Node reference in [microsoft/agent365-skills](https://github.com/microsoft/agent365-skills). - - **Do not** read from `AgenticTokenCacheInstance` for non-AI-Teammate blueprint telemetry and do not call `RefreshObservabilityToken` to obtain exporter tokens; those produce delegated tokens for the wrong route. + - **Do not** read from `AgenticTokenCacheInstance` for exporter tokens and do not call `RefreshObservabilityToken` to obtain exporter tokens; those produce delegated tokens for the wrong route. - Optionally register `BaggageMiddleware` only for baggage propagation; it must not be used as the exporter token source. - Mark all new lines with: `// A365 Observability — best-effort instrumentation (verify against official sample)` @@ -273,7 +276,7 @@ For non-AI-Teammate agents, `authMode` controls workload tokens and permission g 2. **Edit** — Add observability configuration following the reference pattern in `python-observability.md`: - Add `from microsoft.opentelemetry import use_microsoft_opentelemetry` and call `use_microsoft_opentelemetry(enable_a365=True, a365_token_resolver=...)` with `service_name` and `service_namespace` - Pass `a365_use_s2s_endpoint=True` and an app-only `a365_token_resolver` for the exporting agent identity, using the current Python reference in [microsoft/agent365-skills](https://github.com/microsoft/agent365-skills). - - **Do not** wire `a365_token_resolver` to a delegated `AgenticTokenCache` token for non-AI-Teammate blueprint telemetry. + - **Do not** wire `a365_token_resolver` to a delegated `AgenticTokenCache` token. - Optionally register baggage helpers only for baggage propagation; they must not perform a delegated Observability token exchange. - Mark all new lines with: `# A365 Observability — best-effort instrumentation (verify against official sample)` @@ -291,7 +294,7 @@ For non-AI-Teammate agents, `authMode` controls workload tokens and permission g > baggage propagation automatically for every request. > **Auth mode note:** All three `authMode` values use `authHandlerName: "AGENTIC"` in the -> workload-auth code, but telemetry export for non-AI-Teammate blueprint agents is always app-only S2S. +> workload-auth code, but telemetry export is always app-only S2S, including for AI Teammates. > Do not add delegated Observability token refresh to a message handler. ### For .NET AgentFramework @@ -301,7 +304,7 @@ For non-AI-Teammate agents, `authMode` controls workload tokens and permission g 2. **Edit** — Follow the current .NET pattern from the `instrument-observability` skill: - Use `new BaggageBuilder().FromTurnContext(turnContext).Build()` as baggage context. - Use the app-only `AgentDetails` / `CallerDetails` pattern from the reference when creating manual scopes. - - **Do not** inject `IExporterTokenCache` or call `RegisterObservability` for non-AI-Teammate blueprint telemetry; the exporter token is supplied by the app-only resolver configured in Phase 3. + - **Do not** inject `IExporterTokenCache` or call `RegisterObservability`; the exporter token is supplied by the app-only resolver configured in Phase 3. - Add inline comment: `// A365 telemetry: app-only S2S route; workload auth mode: {authMode}` Mark all new lines with: `// A365 Observability — best-effort instrumentation (verify against official sample)` @@ -315,7 +318,7 @@ For non-AI-Teammate agents, `authMode` controls workload tokens and permission g 2. **Edit** — Add BaggageBuilder context following the reference pattern in `nodejs-observability.md`: - Import `BaggageBuilder` from `@microsoft/agents-a365-observability` - Import `BaggageBuilderUtils` from `@microsoft/agents-a365-observability-hosting` - - **Do not** call `AgenticTokenCacheInstance.RefreshObservabilityToken` for non-AI-Teammate blueprint telemetry. The `tokenResolver` passed to `useMicrosoftOpenTelemetry()` supplies an app-only token for the S2S route. + - **Do not** call `AgenticTokenCacheInstance.RefreshObservabilityToken`. The `tokenResolver` passed to `useMicrosoftOpenTelemetry()` supplies an app-only token for the S2S route. - Use `BaggageBuilderUtils.fromTurnContext(new BaggageBuilder(), context).build()` to build baggage automatically from TurnContext - Wrap the handler body in `await baggageScope.run(async () => { ... })` - Add inline comment: `// A365 telemetry: app-only S2S route; workload auth mode: {authMode}` @@ -330,7 +333,7 @@ For non-AI-Teammate agents, `authMode` controls workload tokens and permission g 2. **Edit** — Add BaggageBuilder context following the reference pattern in `python-observability.md`: - Import `BaggageBuilder` from `microsoft.opentelemetry.a365.core` - Import `populate` from `microsoft.opentelemetry.a365.hosting.scope_helpers.populate_baggage` - - **Do not** call `cache_agentic_token`, `register_observability`, or `exchange_token(...get_observability_authentication_scope...)` for non-AI-Teammate blueprint telemetry. The `a365_token_resolver` configured in Phase 3 supplies an app-only token for the S2S route. + - **Do not** call `cache_agentic_token`, `register_observability`, or `exchange_token(...get_observability_authentication_scope...)`. The `a365_token_resolver` configured in Phase 3 supplies an app-only token for the S2S route. - Use `populate(builder, turn_context)` to auto-populate baggage, then `with builder.build():` - Wrap existing agent logic inside the baggage scope - Add inline comment: `# A365 telemetry: app-only S2S route; workload auth mode: {authMode}` @@ -346,31 +349,31 @@ For non-AI-Teammate agents, `authMode` controls workload tokens and permission g **TaskCreate** — "Implement agentic token resolver with caching" -For non-AI-Teammate blueprint agents, Phase 5 means "ensure the app-only S2S token resolver from the `instrument-observability` skill is present." Do not create or use delegated Observability token caches. For AI Teammate agents using the hosting packages, the built-in delegated token cache may still be the sample's expected pattern. +For every agent, including AI Teammates, Phase 5 means "ensure the app-only S2S token resolver from the `instrument-observability` skill is present." Do not create or use delegated Observability token caches. The built-in delegated token caches below apply only to the legacy delegated route, which needs the delegated `OtelWrite` grant added manually. -### For .NET AgentFramework (AI Teammate / legacy delegated path only) +### For .NET AgentFramework (legacy delegated route only) 1. `AddAgenticTracingExporter()` (registered in Phase 3) provides the `IExporterTokenCache` DI instance — no additional token resolver class needed. 2. In the agent class, inject `IExporterTokenCache` in the constructor and call `RegisterObservability(...)` per turn (already done in Phase 4). -### For .NET AgentFramework (blueprint agents) +### For .NET AgentFramework (all agents, including AI Teammates) Use the app-only token resolver pattern from [microsoft/agent365-skills](https://github.com/microsoft/agent365-skills), set the S2S route flag, and keep delegated OBS token registration out of the handler. -### For Node.js (AI Teammate / legacy delegated path only) +### For Node.js (legacy delegated route only) -`AgenticTokenCacheInstance` from `@microsoft/agents-a365-observability-hosting` handles delegated token caching automatically. Do not use it for non-AI-Teammate blueprint telemetry. +`AgenticTokenCacheInstance` from `@microsoft/agents-a365-observability-hosting` handles delegated token caching automatically. Use it only on the legacy delegated route. -### For Node.js (blueprint agents) +### For Node.js (all agents, including AI Teammates) Create or reuse the app-only resolver from the `instrument-observability` skill and pass it to `useMicrosoftOpenTelemetry({ a365: { useS2SEndpoint: true, tokenResolver } })`. -### For Python (AI Teammate / legacy delegated path only) +### For Python (legacy delegated route only) -`AgenticTokenCache` from `microsoft.opentelemetry.a365.hosting.token_cache_helpers` handles delegated token caching automatically. Do not use it for non-AI-Teammate blueprint telemetry. +`AgenticTokenCache` from `microsoft.opentelemetry.a365.hosting.token_cache_helpers` handles delegated token caching automatically. Use it only on the legacy delegated route. -### For Python (blueprint agents) +### For Python (all agents, including AI Teammates) Create or reuse the app-only resolver from the `instrument-observability` skill and pass it to `use_microsoft_opentelemetry(..., a365_use_s2s_endpoint=True, a365_token_resolver=...)`. @@ -743,26 +746,26 @@ This skill is safe to rerun. On subsequent runs: ### OtelWrite App Role Assignment -> **Blueprint agents:** `a365 setup all` does not request Observability API permissions for blueprint agents in any auth mode — the S2S endpoint authorizes registered agent instances without the `OtelWrite` role, so no admin consent is needed. Setup exits with code 1 if registration fails; retry with `a365 setup all --agent-registration-only`. Grant `OtelWrite` manually (steps below) only if the agent still exports through the delegated (OBO) route. Permissions granted by earlier runs are not revoked. - -For **AI Teammate** agents, `a365 setup all` still **attempts** to grant the `Agent365.Observability.OtelWrite` application role to the **blueprint service principal**, which requires **Global Administrator** privileges. If the logged-in user is not a Global Admin, the assignment fails with 403 and trace exports can return HTTP 403 from the observability service. +> **No Observability permissions by default:** `a365 setup all` does not request Observability API permissions for blueprint agents (in any auth mode) or AI Teammates. Agents export telemetry with an app-only token through the S2S endpoint, which authorizes a blueprint agent by its agent registration and an AI Teammate by its blueprint's registration, so no admin consent is needed. For blueprint agents, setup exits with code 1 if registration fails; retry with `a365 setup all --agent-registration-only` (AI Teammates don't use this flag). Permissions granted by earlier runs are not revoked. -**The CLI prints a PowerShell admin consent script** in its output when the assignment fails. When running `a365 setup all`, **always scan the output for this script block** and display it to the user in a fenced code block so they can copy it and hand it to a Global Admin. +Grant `Agent365.Observability.OtelWrite` manually (requires **Global Administrator**) only when: +- the agent still exports through the delegated (OBO) route: grant the **delegated** permission. +- app-only trace exports return HTTP 403 `insufficient_scope`: the service did not authorize the agent by registration. Grant the **application** permission on the blueprint; agent identities inherit it. -For a **blueprint agent still exporting through the delegated (OBO) route**, prefer the CLI opt-back-in path (requires Global Administrator): +To restore the delegated permission, prefer the CLI opt-back-in path: ```bash a365 setup permissions custom --resource-app-id --scopes Agent365.Observability.OtelWrite ``` -This stamps the Observability inheritable-permission entry on the blueprint and requests admin consent. The Entra portal path below works for existing blueprints only when that inheritable-permission entry already exists. +This stamps the Observability inheritable-permission entry on the blueprint and requests admin consent. Agent identities inherit the blueprint's Observability permissions only through this entry, so the Entra portal and PowerShell paths below work only when it already exists; run this command first on blueprints that lack it. -If the AI Teammate script was not captured, grant the permission manually via Entra portal (requires Global Administrator): +To grant either permission in the Entra portal: 1. [Entra portal](https://entra.microsoft.com) > App registrations > select Blueprint app > API permissions 2. Add a permission > APIs my organization uses > search the Observability app ID for your cloud (commercial: `9b975845-388f-4429-889e-eab1ef63949c`; see `src/Microsoft.Agents.A365.DevTools.Cli/design.md#environment-variable-overrides` for other clouds) -3. Add both **Delegated** and **Application** `Agent365.Observability.OtelWrite` > Grant admin consent +3. Add **Delegated** `Agent365.Observability.OtelWrite` for the delegated route, or **Application** `Agent365.Observability.OtelWrite` for a 403 `insufficient_scope` > Grant admin consent -Alternatively for **AI Teammates**, look up the Blueprint and Observability enterprise applications and create the application-role assignment the S2S route accepts: +Alternatively, for a 403 `insufficient_scope`, look up the Blueprint and Observability enterprise applications and create the application-role assignment the S2S route accepts: ```powershell $blueprintSp = Get-MgServicePrincipal -Filter "appId eq ''" @@ -772,7 +775,6 @@ New-MgServicePrincipalAppRoleAssignment -ServicePrincipalId $blueprintSp.Id -Pri ``` - `` is the resource application ID, not the service-principal object ID. -- For agents provisioned before CLI 1.1, this manual step is still required ### Node.js and .NET SDK `/otlp/` URL Path Bug diff --git a/docs/agent365-guided-setup/references/dotnet-observability.md b/docs/agent365-guided-setup/references/dotnet-observability.md index b92dd52e..ddda0a81 100644 --- a/docs/agent365-guided-setup/references/dotnet-observability.md +++ b/docs/agent365-guided-setup/references/dotnet-observability.md @@ -190,7 +190,7 @@ public static class ObservabilityServiceExtensions > - `true` (production) — MSI → Blueprint FIC → Agent Identity → API > - `false` (local dev) — Client Secret → Blueprint FIC → Agent Identity → API > -> **Note:** `a365 setup all` no longer requests `Agent365.Observability.OtelWrite` for blueprint agents. Registered blueprint agents that use the app-only S2S endpoint need no Observability admin consent. Agents that still export through the delegated route must opt back in with `a365 setup permissions custom --resource-app-id --scopes Agent365.Observability.OtelWrite` (commercial: `9b975845-388f-4429-889e-eab1ef63949c`; other clouds are listed under [Environment Variable Overrides](../../../src/Microsoft.Agents.A365.DevTools.Cli/design.md#environment-variable-overrides)). +> **Note:** `a365 setup all` no longer requests `Agent365.Observability.OtelWrite` for blueprint agents or AI Teammates. Agents that use the app-only S2S endpoint need no Observability admin consent: the service authorizes a blueprint agent by its agent registration and an AI Teammate by its blueprint's registration. Agents that still export through the delegated route must opt back in with `a365 setup permissions custom --resource-app-id --scopes Agent365.Observability.OtelWrite` (commercial: `9b975845-388f-4429-889e-eab1ef63949c`; other clouds are listed under [Environment Variable Overrides](../../../src/Microsoft.Agents.A365.DevTools.Cli/design.md#environment-variable-overrides)). ```csharp using Azure.Core; @@ -944,7 +944,7 @@ The `a365 setup` command (as of April 2026) automatically writes the following t | S2S: token service skipped at startup | Placeholder or missing `Agent365Observability` credentials | Run `a365 setup all` or populate `TenantId`, `AgentId`, `ClientId`, and `ClientSecret` (when `UseManagedIdentity` is `false`) | | S2S: 401 on export | Token acquired for wrong scope/app or the exporter is not on the S2S route | Verify FMI Hop 3 scope is `api://9b975845-388f-4429-889e-eab1ef63949c/.default`, set the S2S route flag, and ensure the token is app-only for the exporting agent identity | | S2S: FMI Hop 1+2 fails | Blueprint credentials wrong or `.WithFmiPath(agentId)` target incorrect | Check `ClientId` (Blueprint app ID) and `ClientSecret` in appsettings; verify `AgentId` matches the Agent Identity app ID | -| S2S: 403 `insufficient_scope` on export | Agent instance is not registered, or AI Teammate OtelWrite app-role grant is incomplete | Blueprint agents: run `a365 setup all --agent-registration-only` and retry. AI Teammates: complete the OtelWrite application-role PowerShell step printed by `a365 setup all --aiteammate` | +| S2S: 403 `insufficient_scope` on export | The service could not authorize the agent by registration: a blueprint agent's instance is not registered, or an AI Teammate's blueprint registration was not accepted | Blueprint agents: run `a365 setup all --agent-registration-only` and retry. AI Teammates: have a Global Administrator grant the `OtelWrite` application role on the blueprint (see [OtelWrite App Role Assignment](../a365-observability-instructions.md#otelwrite-app-role-assignment)) | | S2S: MSI fails locally | No Managed Identity available in dev | Set `UseManagedIdentity: false` in appsettings.Development.json, ensure `ClientSecret` is populated | | S2S: `UseMicrosoftOpenTelemetry` not found | Unified distro not installed | Run `dotnet add package Microsoft.OpenTelemetry --version 1.0.0-beta.1` | | S2S: Runtime `FileNotFoundException` for `Microsoft.Extensions.Logging v10.0.0` | `Microsoft.OpenTelemetry` v1.0.0-beta.1 depends on v10 logging | (1) Upgrade TFM to `net9.0`. (2) Run `dotnet add package Microsoft.Extensions.Logging --version "10.0.4"` — use the **stable** version, not a preview; specifying a preview causes NU1605 downgrade errors because `Microsoft.Agents.A365.Observability.Hosting` requires `>= 10.0.4`. | diff --git a/docs/agent365-guided-setup/references/nodejs-observability.md b/docs/agent365-guided-setup/references/nodejs-observability.md index 683f0ad8..08ce22d8 100644 --- a/docs/agent365-guided-setup/references/nodejs-observability.md +++ b/docs/agent365-guided-setup/references/nodejs-observability.md @@ -111,7 +111,7 @@ No OBO user token is required. > standard client-credential request. This workaround will be removed once MSAL ships native > `fmiPath` support for the client-secret credential path. -> **Note:** `a365 setup all` no longer requests `Agent365.Observability.OtelWrite` for blueprint agents. Registered blueprint agents that use the app-only S2S endpoint need no Observability admin consent. Agents that still export through the delegated route must opt back in with `a365 setup permissions custom --resource-app-id --scopes Agent365.Observability.OtelWrite` (commercial: `9b975845-388f-4429-889e-eab1ef63949c`; other clouds are listed under [Environment Variable Overrides](../../../src/Microsoft.Agents.A365.DevTools.Cli/design.md#environment-variable-overrides)). +> **Note:** `a365 setup all` no longer requests `Agent365.Observability.OtelWrite` for blueprint agents or AI Teammates. Agents that use the app-only S2S endpoint need no Observability admin consent: the service authorizes a blueprint agent by its agent registration and an AI Teammate by its blueprint's registration. Agents that still export through the delegated route must opt back in with `a365 setup permissions custom --resource-app-id --scopes Agent365.Observability.OtelWrite` (commercial: `9b975845-388f-4429-889e-eab1ef63949c`; other clouds are listed under [Environment Variable Overrides](../../../src/Microsoft.Agents.A365.DevTools.Cli/design.md#environment-variable-overrides)). > **IMPORTANT — SDK `useS2SEndpoint` bug (v0.1.0-beta.1):** The `@microsoft/opentelemetry` > distro does **not** pass `useS2SEndpoint` to `Agent365Exporter`. The exporter defaults @@ -1053,7 +1053,7 @@ setLogger({ | `extensions-openai` install fails / peer dep error | Missing `@openai/agents` peer dep | Run `npm install @openai/agents@^0.7.0` first; this is the OpenAI Agents SDK, not the `openai` package | | S2S: AADSTS82001 or AADSTS1002012 | Direct MSAL client credentials not supported | Use the 3-hop FMI chain: Blueprint → FMI path → Agent Identity → Observability API token. | | S2S: 401 on export | Token scope mismatch, delegated token, or wrong route | Ensure Hop 3 scope is `api://9b975845-388f-4429-889e-eab1ef63949c/.default`, set `useS2SEndpoint: true`, and use an app-only token for the exporting agent identity | -| S2S: 403 `insufficient_scope` on `observabilityService/` endpoint | Agent instance is not registered, or AI Teammate OtelWrite app-role grant is incomplete | Blueprint agents: run `a365 setup all --agent-registration-only` and retry. AI Teammates: complete the OtelWrite application-role PowerShell step printed by `a365 setup all --aiteammate` | +| S2S: 403 `insufficient_scope` on `observabilityService/` endpoint | The service could not authorize the agent by registration: a blueprint agent's instance is not registered, or an AI Teammate's blueprint registration was not accepted | Blueprint agents: run `a365 setup all --agent-registration-only` and retry. AI Teammates: have a Global Administrator grant the `OtelWrite` application role on the blueprint (see [OtelWrite App Role Assignment](../a365-observability-instructions.md#otelwrite-app-role-assignment)) | | S2S: MSI fails locally | No Managed Identity in dev | Set `AGENT365_USE_MANAGED_IDENTITY=false` and provide `AGENT365_CLIENT_SECRET` | | S2S: token resolver never called | `RefreshObservabilityToken` called for S2S | Remove `AgenticTokenCacheInstance.RefreshObservabilityToken` — not used in S2S; token comes from `a365.tokenResolver` in `useMicrosoftOpenTelemetry(...)` | | `fromTurnContext` not found on `BaggageBuilder` | Static method is on `BaggageBuilderUtils`, not `BaggageBuilder` | Use `BaggageBuilderUtils.fromTurnContext(new BaggageBuilder(), context)` | diff --git a/docs/agent365-guided-setup/references/python-observability.md b/docs/agent365-guided-setup/references/python-observability.md index 184a55ac..9480d82f 100644 --- a/docs/agent365-guided-setup/references/python-observability.md +++ b/docs/agent365-guided-setup/references/python-observability.md @@ -64,7 +64,7 @@ No OBO user token is required. > **⚠️ Known Issue (msal v1.34.0):** Python MSAL does NOT properly support `fmi_path` as a parameter to `acquire_token_for_client()`. Passing it causes `TypeError: Session.request() got an unexpected keyword argument 'fmi_path'`. Use **direct HTTP POST** to the token endpoint with `fmi_path` as a form parameter for Hop 1+2 (same workaround as Node.js). MSAL is fine for Hop 3 (no `fmi_path` needed). -> **Note:** `a365 setup all` no longer requests `Agent365.Observability.OtelWrite` for blueprint agents. Registered blueprint agents that use the app-only S2S endpoint need no Observability admin consent. Agents that still export through the delegated route must opt back in with `a365 setup permissions custom --resource-app-id --scopes Agent365.Observability.OtelWrite` (commercial: `9b975845-388f-4429-889e-eab1ef63949c`; other clouds are listed under [Environment Variable Overrides](../../../src/Microsoft.Agents.A365.DevTools.Cli/design.md#environment-variable-overrides)). +> **Note:** `a365 setup all` no longer requests `Agent365.Observability.OtelWrite` for blueprint agents or AI Teammates. Agents that use the app-only S2S endpoint need no Observability admin consent: the service authorizes a blueprint agent by its agent registration and an AI Teammate by its blueprint's registration. Agents that still export through the delegated route must opt back in with `a365 setup permissions custom --resource-app-id --scopes Agent365.Observability.OtelWrite` (commercial: `9b975845-388f-4429-889e-eab1ef63949c`; other clouds are listed under [Environment Variable Overrides](../../../src/Microsoft.Agents.A365.DevTools.Cli/design.md#environment-variable-overrides)). #### Step 1 — Create `observability/token_cache.py` @@ -872,6 +872,6 @@ python -c "from microsoft.opentelemetry import use_microsoft_opentelemetry; from | S2S: `TypeError: Session.request() got an unexpected keyword argument 'fmi_path'` | MSAL Python v1.34.0 bug | Use direct HTTP POST to `https://login.microsoftonline.com/{tenantId}/oauth2/v2.0/token` with `fmi_path` as form data instead of MSAL `acquire_token_for_client(fmi_path=...)`. MSAL is still used for Hop 3 (no `fmi_path` needed). | | S2S: `InferenceCallDetails.__init__() got an unexpected keyword argument 'operation_name'` | Python SDK uses camelCase kwargs | Use `operationName=`, `providerName=`, `inputTokens=`, `outputTokens=`, `finishReasons=` (camelCase, NOT snake_case) | | S2S: HTTP 400 TenantIdInvalid from exporter | Token not yet acquired when exporter first fires | Ensure `acquire_initial_token()` runs in lifespan BEFORE monitor starts. The `a365_token_resolver` returns `""` when no cached token exists, causing 400. | -| S2S: HTTP 403 `insufficient_scope: Required app role: Agent365.Observability.OtelWrite` | Agent instance is not registered, or AI Teammate OtelWrite app-role grant is incomplete | Blueprint agents: run `a365 setup all --agent-registration-only` and retry. AI Teammates: complete the OtelWrite application-role PowerShell step printed by `a365 setup all --aiteammate` | +| S2S: HTTP 403 `insufficient_scope: Required app role: Agent365.Observability.OtelWrite` | The service could not authorize the agent by registration: a blueprint agent's instance is not registered, or an AI Teammate's blueprint registration was not accepted | Blueprint agents: run `a365 setup all --agent-registration-only` and retry. AI Teammates: have a Global Administrator grant the `OtelWrite` application role on the blueprint (see [OtelWrite App Role Assignment](../a365-observability-instructions.md#otelwrite-app-role-assignment)) | | S2S: FMI Hop 3 returns `AADSTS700024` | Agent Identity has no FMI credential | Verify `a365 setup all` completed successfully — it creates the federated credential on the Agent Identity | | S2S: HTTP 200 but `rejectedSpans > 0` | Missing baggage context (tenant_id/agent_id) | Ensure `BaggageBuilder().tenant_id(...).agent_id(...).build()` wraps all scope code — without it, spans lack identity and are rejected | diff --git a/src/Microsoft.Agents.A365.DevTools.Cli/Commands/CreateInstanceCommand.cs b/src/Microsoft.Agents.A365.DevTools.Cli/Commands/CreateInstanceCommand.cs index 397b86c0..7691d7b2 100644 --- a/src/Microsoft.Agents.A365.DevTools.Cli/Commands/CreateInstanceCommand.cs +++ b/src/Microsoft.Agents.A365.DevTools.Cli/Commands/CreateInstanceCommand.cs @@ -168,24 +168,7 @@ public static Command CreateCommand(ILogger logger, IConf if (!botApiGrantOk) logger.LogWarning("Failed to create/update oauth2PermissionGrant for agent identity to Messaging Bot API."); - var observabilityApiAppId = ConfigConstants.GetObservabilityApiAppId(instanceConfig.Environment); - var observabilityApiResourceSpObjectId = await graphApiService.EnsureServicePrincipalForAppIdAsync( - instanceConfig.TenantId, - observabilityApiAppId) - ?? throw new InvalidOperationException( - $"Failed to resolve service principal for Observability API (appId {observabilityApiAppId})."); - - // Grant oauth2PermissionGrants: *agent identity SP* -> Observability API SP - var observabilityApiGrantOk = await graphApiService.CreateOrUpdateOauth2PermissionGrantAsync( - instanceConfig.TenantId, - agenticAppSpObjectId, - observabilityApiResourceSpObjectId, - new[] { "user_impersonation", ConfigConstants.ObservabilityApiOtelWriteScope }); - - if (!observabilityApiGrantOk) - logger.LogWarning("Failed to create/update oauth2PermissionGrant for agent identity to Observability API."); - - var adminConsentGrantOk = mcpGrantOk && botApiGrantOk && observabilityApiGrantOk; + var adminConsentGrantOk = mcpGrantOk && botApiGrantOk; if (!adminConsentGrantOk) { logger.LogError("Admin consent for Agent Identity completed with errors. One or more required API grants failed and follow-up action is required."); diff --git a/src/Microsoft.Agents.A365.DevTools.Cli/Commands/SetupCommand.cs b/src/Microsoft.Agents.A365.DevTools.Cli/Commands/SetupCommand.cs index e2d9b33c..4414c619 100644 --- a/src/Microsoft.Agents.A365.DevTools.Cli/Commands/SetupCommand.cs +++ b/src/Microsoft.Agents.A365.DevTools.Cli/Commands/SetupCommand.cs @@ -52,7 +52,7 @@ public static Command CreateCommand( " a365 setup requirements Validate prerequisites\n" + " a365 setup blueprint Create the agent blueprint (Entra app)\n" + " a365 setup permissions mcp MCP server permissions (when using MCP tools)\n" + - " a365 setup permissions bot Bot API + Observability + Power Platform\n" + + " a365 setup permissions bot Bot API + Power Platform\n" + " a365 setup permissions custom Inline custom resource permission (--resource-app-id, --scopes)\n" + " a365 setup permissions copilotstudio Copilot Studio agent permissions\n\n" + "Roles required:\n" + diff --git a/src/Microsoft.Agents.A365.DevTools.Cli/Commands/SetupSubcommands/AllSubcommand.cs b/src/Microsoft.Agents.A365.DevTools.Cli/Commands/SetupSubcommands/AllSubcommand.cs index 49ace519..4ed82a6b 100644 --- a/src/Microsoft.Agents.A365.DevTools.Cli/Commands/SetupSubcommands/AllSubcommand.cs +++ b/src/Microsoft.Agents.A365.DevTools.Cli/Commands/SetupSubcommands/AllSubcommand.cs @@ -397,11 +397,9 @@ effectiveAuthModeForValidation is not ("obo" or "s2s" or "both")) return; } - // Registered blueprint agents export telemetry app-only over S2S without OtelWrite in every auth - // mode, so blueprint setup never requests it; AI Teammate setup (including an AI Teammate config - // kept for a dry run) is unchanged. - var skipObservabilityPermissions = nonDwConfig is not null - && (aiTeammateFlag == false || nonDwConfig.IsBlueprintAgent); + // Agents export telemetry app-only over S2S, authorized by their registration (AI Teammates: their + // blueprint's), so setup never requests OtelWrite for either agent type. + const bool skipObservabilityPermissions = true; if (nonDwConfig is not null) { @@ -593,7 +591,9 @@ await RequirementsSubcommand.RunChecksOrExitAsync( isM365: isM365, skipSpProvisioning: skipSpProvisioning, messagingEndpointOverride: messagingEndpointFlag, - nonInteractive: Console.IsInputRedirected); + nonInteractive: Console.IsInputRedirected, + skipObservabilityPermissions: skipObservabilityPermissions); + setupResults.ObservabilityPermissionsSkipped = ctx.ObservabilityPermissionsEffectivelySkipped; // Step 1: Infrastructure (optional, DW only) await ExecuteInfrastructureStepAsync(ctx); @@ -602,6 +602,8 @@ await RequirementsSubcommand.RunChecksOrExitAsync( await ExecuteBlueprintStepAsync(ctx); // Step 3: Configure all permissions in a batch. + if (ctx.ObservabilityPermissionsEffectivelySkipped) + logger.LogInformation("Observability API permissions not requested: AI Teammates export telemetry with an app-only token authorized by the blueprint's registration."); var (specs, mcpResourceAppId, mcpScopes, mcpScopesByAudience, mcpServerNamesByAudience) = await BuildPermissionSpecsAsync(ctx); await ExecuteBatchPermissionsStepAsync( diff --git a/src/Microsoft.Agents.A365.DevTools.Cli/Commands/SetupSubcommands/PermissionsSubcommand.cs b/src/Microsoft.Agents.A365.DevTools.Cli/Commands/SetupSubcommands/PermissionsSubcommand.cs index 090be96d..98496fe1 100644 --- a/src/Microsoft.Agents.A365.DevTools.Cli/Commands/SetupSubcommands/PermissionsSubcommand.cs +++ b/src/Microsoft.Agents.A365.DevTools.Cli/Commands/SetupSubcommands/PermissionsSubcommand.cs @@ -305,7 +305,6 @@ private static Command CreateBotSubcommand( logger.LogInformation("Would configure Bot API permissions:"); logger.LogInformation(" - Blueprint: {BlueprintId}", dryRunConfig.AgentBlueprintId); logger.LogInformation(" - Messaging Bot API: {Scope}", ConfigConstants.MessagingBotApiAdminConsentScope); - logger.LogInformation(" - Observability API: {OtelScope} (delegated + application)", ConfigConstants.ObservabilityApiOtelWriteScope); logger.LogInformation(" - Power Platform API: Connectivity.Connections.Read"); logger.LogInformation("No changes made. Run without --dry-run to execute."); return; @@ -764,11 +763,13 @@ public static async Task ConfigureBotPermissionsAsync( try { + // Telemetry is authorized by registration, so this step requests no Observability API permissions. var specs = new List( SetupHelpers.GetFixedApiPermissionSpecs( setInheritable: true, isM365: true, - setupConfig.Environment)); + setupConfig.Environment, + includeObservability: false)); var localResults = setupResults ?? new SetupResults(); var (_, _, consentGranted, adminConsentUrl) = await BatchPermissionsOrchestrator.ConfigureAllPermissionsAsync( @@ -786,35 +787,22 @@ public static async Task ConfigureBotPermissionsAsync( await configService.SaveStateAsync(setupConfig); - // BlueprintS2SOutcome == Granted means S2S succeeded; any other value (NotApplicable, - // Failed) treats this as "not in place". Consent state is checked before S2S so the - // non-admin path still shows "consent required" rather than the S2S warning (S2S is - // never attempted when consent isn't granted). - var s2sFailed = localResults.BlueprintS2SOutcome != Models.GrantOutcome.Granted; - logger.LogInformation(""); - if (!s2sFailed && consentGranted) + if (consentGranted) { logger.LogInformation("Bot API permissions configured successfully"); } - else if (!consentGranted) + else { logger.LogInformation("Bot API permissions configured; admin consent required"); LogAdminConsentNextSteps(logger, adminConsentUrl, localResults, specs); } - else - { - logger.LogWarning( - "Bot API permissions configured, but S2S app role assignment failed. " + - "Re-run 'a365 setup permissions bot' as {Roles} to retry.", - AuthenticationConstants.S2SGrantRequiredRoles); - } logger.LogInformation(""); if (!iSetupAll) { logger.LogInformation("Next step: Run 'a365 publish' to package your agent for upload to the Microsoft 365 Admin Center."); } - return consentGranted && !s2sFailed; + return consentGranted; } catch (Exception ex) { diff --git a/src/Microsoft.Agents.A365.DevTools.Cli/Commands/SetupSubcommands/README.md b/src/Microsoft.Agents.A365.DevTools.Cli/Commands/SetupSubcommands/README.md index 2c4aaebc..b83c08c6 100644 --- a/src/Microsoft.Agents.A365.DevTools.Cli/Commands/SetupSubcommands/README.md +++ b/src/Microsoft.Agents.A365.DevTools.Cli/Commands/SetupSubcommands/README.md @@ -94,11 +94,11 @@ a365 setup all --authmode both ### Observability permissions -For blueprint agents, `setup all` does not request `Agent365.Observability.OtelWrite` in any auth mode. Registered agents export telemetry with an app-only token through the S2S endpoint, which authorizes them by their agent registration, so no Observability admin consent is needed. Registration is then the agent's only authorization, so a registration failure is reported as an error (exit code 1). OtelWrite was the only app role setup requested. Unless another permission adds an app role, `--authmode s2s` and `both` have nothing to assign for blueprint agents, and the setup summary reports the S2S grant as not required. +`setup all` does not request `Agent365.Observability.OtelWrite` for blueprint agents (in any auth mode) or AI Teammates, and `setup permissions bot` and `create-instance` do not request it for any agent. Agents export telemetry with an app-only token through the S2S endpoint, which authorizes a blueprint agent by its agent registration and an AI Teammate by its blueprint's registration, so no Observability admin consent is needed. For blueprint agents, registration is then the agent's only authorization, so a registration failure is reported as an error (exit code 1). OtelWrite was the only app role setup requested. Unless another permission adds an app role, `--authmode s2s` and `both` have nothing to assign for blueprint agents, and the setup summary reports the S2S grant as not required. -Agents whose SDK still exports through the delegated (OBO) route need `OtelWrite`; grant it manually (see the CHANGELOG upgrade note). AI Teammate setup is unchanged. Re-running setup does not revoke permissions granted earlier. +Agents whose SDK still exports through the delegated (OBO) route need `OtelWrite`, as do agents whose app-only exports return 403 `insufficient_scope`; grant it manually (see the CHANGELOG upgrade note). Re-running setup does not revoke permissions granted earlier. -To opt a blueprint agent back into delegated-route Observability, add the Observability API to `customBlueprintPermissions` or run `a365 setup permissions custom --resource-app-id --scopes Agent365.Observability.OtelWrite` (commercial: `9b975845-388f-4429-889e-eab1ef63949c`; other clouds are listed under [Environment Variable Overrides](../../design.md#environment-variable-overrides)); the custom path stamps inheritable permissions and requires admin-run consent (custom permissions are not included in the non-admin combined consent URL). +To opt an agent back into delegated-route Observability, add the Observability API to `customBlueprintPermissions` or run `a365 setup permissions custom --resource-app-id --scopes Agent365.Observability.OtelWrite` (commercial: `9b975845-388f-4429-889e-eab1ef63949c`; other clouds are listed under [Environment Variable Overrides](../../design.md#environment-variable-overrides)). Both stamp inheritable permissions. A non-admin `setup all` run includes the Observability API in the combined consent URL; `setup permissions custom` requires admin-run consent. --- diff --git a/src/Microsoft.Agents.A365.DevTools.Cli/Commands/SetupSubcommands/SetupContext.cs b/src/Microsoft.Agents.A365.DevTools.Cli/Commands/SetupSubcommands/SetupContext.cs index 57632ed7..cb05e47e 100644 --- a/src/Microsoft.Agents.A365.DevTools.Cli/Commands/SetupSubcommands/SetupContext.cs +++ b/src/Microsoft.Agents.A365.DevTools.Cli/Commands/SetupSubcommands/SetupContext.cs @@ -101,8 +101,8 @@ internal sealed class SetupContext public bool NonInteractive { get; } /// - /// When true, Observability API permissions are omitted from every grant and consent URL (non-DW blueprint only), - /// making agent registration the agent's only Observability authorization. + /// When true, the default Observability API spec is omitted from grants and consent URLs. + /// customBlueprintPermissions can still request it; use for the effective state. /// public bool SkipObservabilityPermissions { get; } diff --git a/src/Microsoft.Agents.A365.DevTools.Cli/Commands/SetupSubcommands/SetupHelpers.cs b/src/Microsoft.Agents.A365.DevTools.Cli/Commands/SetupSubcommands/SetupHelpers.cs index 741c3f6a..737944ac 100644 --- a/src/Microsoft.Agents.A365.DevTools.Cli/Commands/SetupSubcommands/SetupHelpers.cs +++ b/src/Microsoft.Agents.A365.DevTools.Cli/Commands/SetupSubcommands/SetupHelpers.cs @@ -101,15 +101,20 @@ internal static ResourcePermissionSpec[] GetFixedApiPermissionSpecs( /// Scope names are matched case-insensitively, consistent with custom-permission validation. /// internal static bool CustomPermissionsRequestObservability(Agent365Config config) + => CustomPermissionsRequestObservability(config.CustomBlueprintPermissions, config.Environment); + + internal static bool CustomPermissionsRequestObservability( + IEnumerable? customPermissions, + string? environment) { string? expectedAppId = null; - foreach (var customPerm in config.CustomBlueprintPermissions ?? new List()) + foreach (var customPerm in customPermissions ?? Enumerable.Empty()) { var (isValid, _) = customPerm.Validate(); if (!isValid || !ConfigConstants.IsObservabilityApiAppId(customPerm.ResourceAppId)) continue; - expectedAppId ??= ConfigConstants.GetObservabilityApiAppId(config.Environment); + expectedAppId ??= ConfigConstants.GetObservabilityApiAppId(environment); if (string.Equals(customPerm.ResourceAppId, expectedAppId, StringComparison.OrdinalIgnoreCase) && customPerm.Scopes.Contains(ConfigConstants.ObservabilityApiOtelWriteScope, StringComparer.OrdinalIgnoreCase)) { @@ -1219,8 +1224,8 @@ public static void DisplaySetupSummary(SetupResults results, ILogger logger, str { nextStepLines.Add(() => logger.LogInformation(" 1. Run 'a365 setup permissions mcp' to configure MCP permissions")); nextStepLines.Add(() => logger.LogInformation(results.IsM365 - ? " 2. Run 'a365 setup permissions bot' to configure Bot API, Observability, and Power Platform permissions" - : " 2. Run 'a365 setup permissions bot' to configure Observability and Power Platform permissions")); + ? " 2. Run 'a365 setup permissions bot' to configure Bot API and Power Platform permissions" + : " 2. Run 'a365 setup permissions bot' to configure Power Platform permissions")); } if (nextStepLines.Count > 0) @@ -1631,7 +1636,8 @@ internal static void ApplyConsentUrlsIfNeeded( if (ctx.Results.TenantWideConsentOutcome == Models.GrantOutcome.Granted || string.IsNullOrWhiteSpace(ctx.Config.AgentBlueprintId)) return; - var includeObservability = !ctx.SkipObservabilityPermissions; + // A custom-permission opt-back-in still needs Observability in the non-admin consent URL. + var includeObservability = !ctx.ObservabilityPermissionsEffectivelySkipped; var consentResourceNames = PopulateAdminConsentUrls(ctx.Config, mcpResourceAppId, mcpScopes, isM365, mcpScopesByAudience, mcpAudienceDisplayNames, includeObservability); ctx.Results.ConsentUrlsSavedToPath = ctx.GeneratedConfigPath; ctx.Results.ConsentResourceNames.AddRange(consentResourceNames); @@ -1704,7 +1710,11 @@ internal static void PrintDwSetupAllDryRunPlan( } // 4. Inheritable Permissions - logger.LogInformation(DryRunRow(4, "Inheritable Permissions") + "configure for Microsoft Graph, Agent 365 Tools, Messaging Bot API, Observability API, Power Platform API"); + logger.LogInformation(DryRunRow(4, "Inheritable Permissions") + "configure for Microsoft Graph, Agent 365 Tools, Messaging Bot API, Power Platform API"); + if (config is null || !CustomPermissionsRequestObservability(config)) + logger.LogInformation(sub + "Observability API not requested (telemetry is authorized by the blueprint's registration)"); + else + logger.LogInformation(sub + "Observability API requested by customBlueprintPermissions"); // 5. Blueprint Permission Grants logger.LogInformation(DryRunRow(5, "Blueprint Permission Grants") + "admin approval required — see 'Action Required' in setup output"); diff --git a/src/Microsoft.Agents.A365.DevTools.Cli/Commands/SetupSubcommands/SetupResults.cs b/src/Microsoft.Agents.A365.DevTools.Cli/Commands/SetupSubcommands/SetupResults.cs index 088761c2..a5f8bc94 100644 --- a/src/Microsoft.Agents.A365.DevTools.Cli/Commands/SetupSubcommands/SetupResults.cs +++ b/src/Microsoft.Agents.A365.DevTools.Cli/Commands/SetupSubcommands/SetupResults.cs @@ -294,8 +294,8 @@ public class SetupResults public bool PermissionGrantsSkipped { get; set; } /// - /// True when Observability API permissions were not requested (blueprint agents). - /// Registration failure is then an error, and the admin consent walkthrough omits Observability API. + /// True when Observability API permissions were not requested. For blueprint agents, registration + /// failure is then an error; the admin consent walkthrough omits Observability API. /// public bool ObservabilityPermissionsSkipped { get; set; } diff --git a/src/Microsoft.Agents.A365.DevTools.Cli/Constants/ConfigConstants.cs b/src/Microsoft.Agents.A365.DevTools.Cli/Constants/ConfigConstants.cs index 2168ee4c..6399c86f 100644 --- a/src/Microsoft.Agents.A365.DevTools.Cli/Constants/ConfigConstants.cs +++ b/src/Microsoft.Agents.A365.DevTools.Cli/Constants/ConfigConstants.cs @@ -198,17 +198,30 @@ public static string GetAgent365ToolsResourceAppId(string environment) /// Returns the Observability resource Application ID for the selected cloud. /// public static string GetObservabilityApiAppId(string? environment) - => NormalizeEnvironmentKey(environment) switch + { + EnsureUnambiguousEnvironment(environment); + return NormalizeEnvironmentKey(environment) switch { "GCC" or "GCC_MODERATE" => GccObservabilityApiAppId, "GCC_HIGH" => GccHighObservabilityApiAppId, "DOD" => DodObservabilityApiAppId, - "AZUREUSGOVERNMENT" => throw new ArgumentException( - "AzureUSGovernment does not distinguish GCC Moderate, GCC High, and DoD. " + - "Set the environment to gcc, gcc-high, or dod.", - nameof(environment)), _ => ObservabilityApiAppId, }; + } + + /// + /// Rejects AzureUSGovernment, which does not say which US government cloud's resources to use. + /// + public static void EnsureUnambiguousEnvironment(string? environment) + { + if (NormalizeEnvironmentKey(environment) == "AZUREUSGOVERNMENT") + { + throw new ArgumentException( + "AzureUSGovernment does not distinguish GCC Moderate, GCC High, and DoD. " + + "Set the environment to gcc, gcc-high, or dod.", + nameof(environment)); + } + } /// /// Returns the Observability resource identifier URI for the selected cloud. diff --git a/src/Microsoft.Agents.A365.DevTools.Cli/Services/A365CreateInstanceRunner.cs b/src/Microsoft.Agents.A365.DevTools.Cli/Services/A365CreateInstanceRunner.cs index b775c64e..88867a73 100644 --- a/src/Microsoft.Agents.A365.DevTools.Cli/Services/A365CreateInstanceRunner.cs +++ b/src/Microsoft.Agents.A365.DevTools.Cli/Services/A365CreateInstanceRunner.cs @@ -1,8 +1,10 @@ // Copyright (c) Microsoft Corporation. // Licensed under the MIT License. +using Microsoft.Agents.A365.DevTools.Cli.Commands.SetupSubcommands; using Microsoft.Agents.A365.DevTools.Cli.Constants; using Microsoft.Agents.A365.DevTools.Cli.Helpers; +using Microsoft.Agents.A365.DevTools.Cli.Models; using Microsoft.Agents.A365.DevTools.Cli.Services.Helpers; using Microsoft.Agents.A365.DevTools.Cli.Services.Internal; using Microsoft.Extensions.Logging; @@ -152,10 +154,9 @@ string GetConfig(string name) => // token acquisition target the correct national cloud (commercial by default). var configuredGraphBaseUrl = GetConfig("graphBaseUrl"); var configuredAuthorityHost = GetConfig("authorityHost"); - string observabilityResourceAppId; try { - observabilityResourceAppId = ConfigConstants.GetObservabilityApiAppId(environment); + ConfigConstants.EnsureUnambiguousEnvironment(environment); _graphService.GraphBaseUrl = ConfigConstants.GetGraphBaseUrl( environment, string.IsNullOrWhiteSpace(configuredGraphBaseUrl) ? null : configuredGraphBaseUrl); @@ -353,13 +354,6 @@ string GetConfig(string name) => "McpServers.Mail.All", "McpServersMetadata.Read.All" }), - [observabilityResourceAppId] = ( - "Observability API", - new HashSet(StringComparer.OrdinalIgnoreCase) - { - "user_impersonation", - ConfigConstants.ObservabilityApiOtelWriteScope - }), [ConfigConstants.MessagingBotApiAppId] = ( "Messaging Bot API", new HashSet(StringComparer.OrdinalIgnoreCase) @@ -375,7 +369,10 @@ string GetConfig(string name) => }), }; - // Merge additional scopes from resourceConsents in the generated config + // Merge additional scopes from resourceConsents in the generated config. + // Observability entries retained from earlier setups are skipped unless + // customBlueprintPermissions opts back in (#510). + var includeObservability = CustomPermissionsRequestObservability(config, environment); if (instance.TryGetPropertyValue("resourceConsents", out var consentsNode) && consentsNode is JsonArray consentsArray) { @@ -387,6 +384,14 @@ string GetConfig(string name) => var resourceAppId = obj["resourceAppId"]?.GetValue(); if (string.IsNullOrWhiteSpace(resourceAppId)) continue; + if (!includeObservability && ConfigConstants.IsObservabilityApiAppId(resourceAppId)) + { + _logger.LogInformation( + "Skipping saved Observability API consent {ResourceAppId}; add it to customBlueprintPermissions to grant it.", + resourceAppId); + continue; + } + var resourceName = obj["resourceName"]?.GetValue() ?? "(unknown)"; if (!requiredPermissions.TryGetValue(resourceAppId, out var entry)) @@ -1031,6 +1036,26 @@ await File.WriteAllTextAsync( _logger.LogInformation("Saved instance state to {Path}", path); } + // Matches ConfigService, which reads config keys case-insensitively. + private static readonly JsonSerializerOptions CaseInsensitiveJsonOptions = new() { PropertyNameCaseInsensitive = true }; + + private bool CustomPermissionsRequestObservability(JsonObject config, string environment) + { + if (!config.TryGetPropertyValue("customBlueprintPermissions", out var node) || node is not JsonArray) + return false; + + try + { + var customPermissions = node.Deserialize>(CaseInsensitiveJsonOptions); + return SetupHelpers.CustomPermissionsRequestObservability(customPermissions, environment); + } + catch (JsonException ex) + { + _logger.LogWarning(ex, "Could not parse customBlueprintPermissions; Observability API permissions will not be granted."); + return false; + } + } + /// /// Verify that a service principal exists in Azure AD for the given app ID. /// This is critical before creating an agent user that references the identity as a parent. diff --git a/src/Microsoft.Agents.A365.DevTools.Cli/design.md b/src/Microsoft.Agents.A365.DevTools.Cli/design.md index 28141334..12771cb5 100644 --- a/src/Microsoft.Agents.A365.DevTools.Cli/design.md +++ b/src/Microsoft.Agents.A365.DevTools.Cli/design.md @@ -458,7 +458,7 @@ return await new CommandLineBuilder(rootCommand) | Agent Type | Flag | What it creates | |---|---|---| -| **AI Teammate agent** | `--aiteammate` | Azure infra + Agent Blueprint + batch permissions (5 resources) + messaging endpoint | +| **AI Teammate agent** | `--aiteammate` | Azure infra + Agent Blueprint + batch permissions (4 resources) + messaging endpoint | | **Custom Engine Agent / Blueprint** (default) | omit `--aiteammate` | Agent Blueprint + batch permissions (Graph + A365 Tools only) + Agent Instance (Graph API) | Non-DW blueprint agents do not use Azure Bot Service, so there is no infrastructure step, no manifest zip, and no messaging endpoint registration. The final step is `POST /beta/agentRegistry/agentInstances` instead. @@ -475,7 +475,7 @@ AllSubcommand.ExecuteAsync ├── DW path: │ ExecuteInfrastructureStepAsync(ctx) ← DW only │ ExecuteBlueprintStepAsync(ctx) ← shared - │ ExecuteBatchPermissionsStepAsync(ctx, dwSpecs) ← shared (5 resources) + │ ExecuteBatchPermissionsStepAsync(ctx, dwSpecs) ← shared (4 resources) │ ExecuteMessagingEndpointStepAsync(ctx) ← DW only │ └── Non-DW path: @@ -496,9 +496,10 @@ The non-DW spec list is a strict subset of the DW list: | Microsoft Graph (delegated) | ✓ | — | | Agent 365 Tools (delegated) | ✓ | — | | Messaging Bot API | ✓ | — | -| Observability API | ✓ | — | | Power Platform API | ✓ | ✓ | +Neither list includes the Observability API: telemetry is exported with app-only tokens that the service authorizes by registration. Adding it to `customBlueprintPermissions` requests its delegated scope. + --- ## Cross-References diff --git a/src/Tests/Microsoft.Agents.A365.DevTools.Cli.Tests/Commands/AllSubcommandTests.cs b/src/Tests/Microsoft.Agents.A365.DevTools.Cli.Tests/Commands/AllSubcommandTests.cs index 432272d6..efe88a04 100644 --- a/src/Tests/Microsoft.Agents.A365.DevTools.Cli.Tests/Commands/AllSubcommandTests.cs +++ b/src/Tests/Microsoft.Agents.A365.DevTools.Cli.Tests/Commands/AllSubcommandTests.cs @@ -493,6 +493,30 @@ public void ApplyConsentUrlsIfNeeded_AdminRun_CustomObservabilityPermission_Keep because: "custom Observability permissions opt back into the resource, so setup must not clear the saved hand-off URL"); } + [Fact] + public void ApplyConsentUrlsIfNeeded_NonAdminRun_CustomObservabilityPermission_HandsOffObservabilityConsent() + { + var ctx = BuildPermissionsContext( + skipObservabilityPermissions: true, + customPermissions: + [ + new CustomResourcePermission + { + ResourceAppId = ConfigConstants.ObservabilityApiAppId, + ResourceName = "Observability API", + Scopes = [ConfigConstants.ObservabilityApiOtelWriteScope], + } + ]); + + SetupHelpers.ApplyConsentUrlsIfNeeded( + ctx, McpConstants.WorkIQToolsProdAppId, ctx.Config.AgentApplicationScopes, new[] { "McpServers.Mail.All" }, isM365: false); + + ctx.Results.ConsentResourceNames.Should().Contain("Observability API", + because: "an opted-back-in Observability permission is stamped on the blueprint, so a non-admin run must hand its consent to an administrator"); + ctx.Results.CombinedConsentUrl.Should().Contain(ConfigConstants.ObservabilityApiAppId, + because: "the summary prints the combined consent URL, which must request the opted-in Observability scope"); + } + public static TheoryData CustomObservabilityCases() => new() { { diff --git a/src/Tests/Microsoft.Agents.A365.DevTools.Cli.Tests/Commands/PermissionsSubcommandTests.cs b/src/Tests/Microsoft.Agents.A365.DevTools.Cli.Tests/Commands/PermissionsSubcommandTests.cs index bee6b981..3d1bc4a1 100644 --- a/src/Tests/Microsoft.Agents.A365.DevTools.Cli.Tests/Commands/PermissionsSubcommandTests.cs +++ b/src/Tests/Microsoft.Agents.A365.DevTools.Cli.Tests/Commands/PermissionsSubcommandTests.cs @@ -3,6 +3,7 @@ using FluentAssertions; using Microsoft.Agents.A365.DevTools.Cli.Commands.SetupSubcommands; +using Microsoft.Agents.A365.DevTools.Cli.Constants; using Microsoft.Agents.A365.DevTools.Cli.Helpers; using Microsoft.Agents.A365.DevTools.Cli.Models; using Microsoft.Agents.A365.DevTools.Cli.Services; @@ -773,9 +774,9 @@ public async Task ConfigureCustomPermissionsAsync_WithEmptyList_SkipsGracefully( #endregion #region ConfigureBotPermissionsAsync — Issue 402 Regression Tests - // Regression tests for the localResults fix: S2S failure was invisible when setupResults=null. + // Regression tests for the localResults fix: failures must be visible even when setupResults=null. - private Agent365Config ArrangeAdminPath(bool s2sSucceeds) + private Agent365Config ArrangeAdminPath() { var config = new Agent365Config { @@ -803,11 +804,6 @@ private Agent365Config ArrangeAdminPath(bool s2sSucceeds) Arg.Any>(), Arg.Any?>(), Arg.Any()) .Returns(Task.FromResult<(bool ok, bool alreadyExists, string? error)>((true, false, null))); - _mockBlueprintService.GrantAppRoleAssignmentAsync( - Arg.Any(), Arg.Any(), Arg.Any(), - Arg.Any>(), Arg.Any?>(), Arg.Any()) - .Returns(Task.FromResult(new Microsoft.Agents.A365.DevTools.Cli.Models.AppRoleGrantResult(AllSucceeded: s2sSucceeds, AllAlreadyAssigned: false))); - _mockGraphApiService.GraphGetAsync( Arg.Any(), Arg.Any(), Arg.Any(), Arg.Any?>()) @@ -830,11 +826,15 @@ private Agent365Config ArrangeAdminPath(bool s2sSucceeds) return config; } + /// + /// Telemetry is authorized by registration, so the bot step requests no Observability API permissions. That + /// removes the only app role it requested, so an app role grant that would fail is never attempted. + /// [Fact] - public async Task ConfigureBotPermissionsAsync_AdminPath_WhenS2SFailsAndNoExternalSetupResults_ReturnsFalse() + public async Task ConfigureBotPermissionsAsync_AdminPath_RequestsNoObservabilityApiOrAppRoles() { - // Arrange — admin user, OAuth2 grants succeed, S2S grant fails, no external SetupResults. - var config = ArrangeAdminPath(s2sSucceeds: false); + // Arrange — admin user, OAuth2 grants succeed. + var config = ArrangeAdminPath(); // Act — no external SetupResults, mirrors how the CLI handler calls this method. var result = await PermissionsSubcommand.ConfigureBotPermissionsAsync( @@ -849,10 +849,18 @@ public async Task ConfigureBotPermissionsAsync_AdminPath_WhenS2SFailsAndNoExtern setupResults: null); // Assert - result.Should().BeFalse( - because: "the method must create a local SetupResults so S2S failure is captured " + - "even when called without an external setupResults (the CLI handler path)"); - _mockLogger.Received().Log( + result.Should().BeTrue( + because: "with no app role requested there is no S2S grant left that could fail the step"); + await _mockBlueprintService.Received().SetInheritablePermissionsAsync( + Arg.Any(), Arg.Any(), PowerPlatformConstants.PowerPlatformApiResourceAppId, + Arg.Any>(), Arg.Any?>(), Arg.Any()); + await _mockBlueprintService.DidNotReceive().SetInheritablePermissionsAsync( + Arg.Any(), Arg.Any(), ConfigConstants.ObservabilityApiAppId, + Arg.Any>(), Arg.Any?>(), Arg.Any()); + await _mockBlueprintService.DidNotReceive().GrantAppRoleAssignmentAsync( + Arg.Any(), Arg.Any(), Arg.Any(), + Arg.Any>(), Arg.Any?>(), Arg.Any()); + _mockLogger.DidNotReceive().Log( LogLevel.Warning, Arg.Any(), Arg.Is(o => (o.ToString() ?? string.Empty).Contains("S2S app role assignment failed")), @@ -940,46 +948,11 @@ public async Task ConfigureBotPermissionsAsync_WhenGraphAuthFails_LogsConsentReq Arg.Any>()); } - [Fact] - public async Task ConfigureBotPermissionsAsync_AdminPath_WhenS2SFails_EmitsS2SWarningAndSuppressesSuccessLog() - { - // Arrange — admin user, OAuth2 grants succeed, S2S grant fails. - var config = ArrangeAdminPath(s2sSucceeds: false); - - // Act - _ = await PermissionsSubcommand.ConfigureBotPermissionsAsync( - "config.json", - _mockLogger, - _mockConfigService, - _mockExecutor, - config, - _mockGraphApiService, - _mockBlueprintService, - iSetupAll: false, - setupResults: null); - - // Assert — S2S failure warning must be logged. - _mockLogger.Received().Log( - LogLevel.Warning, - Arg.Any(), - Arg.Is(o => (o.ToString() ?? string.Empty).Contains("S2S app role assignment failed")), - Arg.Any(), - Arg.Any>()); - - // Assert — success message must not be logged when S2S failed. - _mockLogger.DidNotReceive().Log( - LogLevel.Information, - Arg.Any(), - Arg.Is(o => (o.ToString() ?? string.Empty).Contains("configured successfully")), - Arg.Any(), - Arg.Any>()); - } - [Fact] public async Task ConfigureBotPermissionsAsync_AdminPath_WhenAllSucceeds_ReturnsTrueAndLogsSuccess() { - // Arrange — admin user, all OAuth2 grants succeed, S2S grant succeeds. - var config = ArrangeAdminPath(s2sSucceeds: true); + // Arrange — admin user, all OAuth2 grants and inheritable permissions succeed. + var config = ArrangeAdminPath(); // Act var result = await PermissionsSubcommand.ConfigureBotPermissionsAsync( @@ -994,7 +967,7 @@ public async Task ConfigureBotPermissionsAsync_AdminPath_WhenAllSucceeds_Returns setupResults: null); // Assert - result.Should().BeTrue(because: "all grants and S2S succeed — method must return true"); + result.Should().BeTrue(because: "every permission the bot step requests was granted, so the method must return true"); _mockLogger.Received().Log( LogLevel.Information, Arg.Any(), @@ -1004,10 +977,10 @@ public async Task ConfigureBotPermissionsAsync_AdminPath_WhenAllSucceeds_Returns } [Fact] - public async Task ConfigureBotPermissionsAsync_AdminPath_WhenS2SFailsWithExternalSetupResults_PopulatesS2SFlagAndReturnsFalse() + public async Task ConfigureBotPermissionsAsync_WithExternalSetupResults_RecordsNoS2SAttempt() { - // Arrange — S2S fails, non-null SetupResults passed in (AllSubcommand path). - var config = ArrangeAdminPath(s2sSucceeds: false); + // Arrange — non-null SetupResults passed in (AllSubcommand path). + var config = ArrangeAdminPath(); var externalResults = new SetupResults(); // Act @@ -1023,10 +996,10 @@ public async Task ConfigureBotPermissionsAsync_AdminPath_WhenS2SFailsWithExterna setupResults: externalResults); // Assert - result.Should().BeFalse( - because: "S2S failure must propagate via the external SetupResults reference"); - externalResults.BlueprintS2SOutcome.Should().NotBe(Cli.Models.GrantOutcome.Granted, - because: "the orchestrator must record BlueprintS2SOutcome as not-Granted (Failed or NotApplicable) on the caller's SetupResults when S2S grants did not succeed"); + result.Should().BeTrue( + because: "the bot step requests no app role, so it cannot fail on an S2S grant"); + externalResults.BlueprintS2SOutcome.Should().Be(Cli.Models.GrantOutcome.NotApplicable, + because: "the caller's summary must not report an S2S grant that this step never requested"); } #endregion diff --git a/src/Tests/Microsoft.Agents.A365.DevTools.Cli.Tests/Commands/SetupCommandTests.cs b/src/Tests/Microsoft.Agents.A365.DevTools.Cli.Tests/Commands/SetupCommandTests.cs index 2af4e503..dcd49757 100644 --- a/src/Tests/Microsoft.Agents.A365.DevTools.Cli.Tests/Commands/SetupCommandTests.cs +++ b/src/Tests/Microsoft.Agents.A365.DevTools.Cli.Tests/Commands/SetupCommandTests.cs @@ -3,6 +3,7 @@ using FluentAssertions; using Microsoft.Agents.A365.DevTools.Cli.Commands; +using Microsoft.Agents.A365.DevTools.Cli.Constants; using Microsoft.Agents.A365.DevTools.Cli.Models; using Microsoft.Agents.A365.DevTools.Cli.Services; using Microsoft.Agents.A365.DevTools.Cli.Services.Requirements; @@ -777,10 +778,11 @@ public async Task SetupAll_BlueprintAgent_AppRoleAuthModes_OmitObservabilityApi( } /// - /// AI Teammate setup is unchanged: its plan still requests Observability API permissions. + /// AI Teammates also export telemetry app-only over S2S, authorized by the blueprint's registration, so + /// their plan must not request Observability API permissions either. /// [Fact] - public async Task SetupAll_AiTeammate_Plan_KeepsObservabilityApi() + public async Task SetupAll_AiTeammate_Plan_OmitsObservabilityApi() { _mockConfigService.LoadAsync(Arg.Any(), Arg.Any()).Returns(Task.FromResult(BlueprintConfig())); var parser = new CommandLineBuilder(BuildSetupCommand()).Build(); @@ -788,20 +790,26 @@ public async Task SetupAll_AiTeammate_Plan_KeepsObservabilityApi() var result = await parser.InvokeAsync("all --aiteammate true --dry-run", new TestConsole()); result.Should().Be(0, because: "an AI Teammate dry run is valid"); + _mockLogger.DidNotReceive().Log( + LogLevel.Information, + Arg.Any(), + Arg.Is(o => o.ToString()!.Contains("Inheritable Permissions") && o.ToString()!.Contains("Observability")), + Arg.Any(), + Arg.Any>()); _mockLogger.Received().Log( LogLevel.Information, Arg.Any(), - Arg.Is(o => o.ToString()!.Contains("Inheritable Permissions") && o.ToString()!.Contains("Observability API")), + Arg.Is(o => o.ToString()!.Contains("Observability API not requested")), Arg.Any(), Arg.Any>()); } /// - /// A dry run keeps an AI Teammate config even without --aiteammate; the plan must still treat it as an - /// AI Teammate and not claim Observability API permissions are skipped. + /// A dry run keeps an AI Teammate config even without --aiteammate; AI Teammates no longer get Observability API + /// permissions, so that plan must also say they are not requested. /// [Fact] - public async Task SetupAll_AiTeammateConfig_DryRunWithoutFlag_DoesNotSkipObservabilityApi() + public async Task SetupAll_AiTeammateConfig_DryRunWithoutFlag_OmitsObservabilityApi() { var config = new Agent365Config { @@ -817,6 +825,49 @@ public async Task SetupAll_AiTeammateConfig_DryRunWithoutFlag_DoesNotSkipObserva var result = await parser.InvokeAsync("all --dry-run", new TestConsole()); result.Should().Be(0, because: "a dry run with an AI Teammate config is valid"); + _mockLogger.Received().Log( + LogLevel.Information, + Arg.Any(), + Arg.Is(o => o.ToString()!.Contains("Observability API not requested")), + Arg.Any(), + Arg.Any>()); + } + + /// + /// customBlueprintPermissions can opt an AI Teammate back into the Observability API, so the plan must say so. + /// + [Fact] + public async Task SetupAll_AiTeammate_CustomObservabilityOptIn_PlanShowsObservabilityRequested() + { + var config = new Agent365Config + { + TenantId = "tenant", + AgentIdentityDisplayName = "agent", + AgentBlueprintDisplayName = "TestBlueprint", + DeploymentProjectPath = ".", + AiTeammate = true, + CustomBlueprintPermissions = new List + { + new() + { + ResourceAppId = ConfigConstants.ObservabilityApiAppId, + ResourceName = "Observability", + Scopes = new List { ConfigConstants.ObservabilityApiOtelWriteScope }, + }, + }, + }; + _mockConfigService.LoadAsync(Arg.Any(), Arg.Any()).Returns(Task.FromResult(config)); + var parser = new CommandLineBuilder(BuildSetupCommand()).Build(); + + var result = await parser.InvokeAsync("all --aiteammate true --dry-run", new TestConsole()); + + result.Should().Be(0, because: "an AI Teammate dry run with custom permissions is valid"); + _mockLogger.Received().Log( + LogLevel.Information, + Arg.Any(), + Arg.Is(o => o.ToString()!.Contains("Observability API requested by customBlueprintPermissions")), + Arg.Any(), + Arg.Any>()); _mockLogger.DidNotReceive().Log( LogLevel.Information, Arg.Any(), diff --git a/src/Tests/Microsoft.Agents.A365.DevTools.Cli.Tests/Constants/ConfigConstantsTests.cs b/src/Tests/Microsoft.Agents.A365.DevTools.Cli.Tests/Constants/ConfigConstantsTests.cs index d568c048..a25e4167 100644 --- a/src/Tests/Microsoft.Agents.A365.DevTools.Cli.Tests/Constants/ConfigConstantsTests.cs +++ b/src/Tests/Microsoft.Agents.A365.DevTools.Cli.Tests/Constants/ConfigConstantsTests.cs @@ -98,6 +98,30 @@ public void GetObservabilityApiAppId_WithAmbiguousAzureGovernmentCloud_Throws() because: "the Azure CLI government cloud name cannot distinguish GCC Moderate, GCC High, and DoD resources"); } + [Theory] + [InlineData(null)] + [InlineData("prod")] + [InlineData("gcc")] + [InlineData("gcc-high")] + [InlineData("dod")] + public void EnsureUnambiguousEnvironment_AcceptsSpecificClouds(string? environment) + { + var act = () => ConfigConstants.EnsureUnambiguousEnvironment(environment); + + act.Should().NotThrow( + because: "each of these environments identifies the resources of exactly one cloud"); + } + + [Fact] + public void EnsureUnambiguousEnvironment_WithAzureGovernmentCloud_ThrowsWithGuidance() + { + var act = () => ConfigConstants.EnsureUnambiguousEnvironment("AzureUSGovernment"); + + act.Should().Throw( + because: "create-instance must stop before granting permissions in a cloud it cannot identify") + .WithMessage("*Set the environment to gcc, gcc-high, or dod.*"); + } + [Theory] [InlineData("http://graph.example")] [InlineData("https://user@graph.example")] diff --git a/src/Tests/Microsoft.Agents.A365.DevTools.Cli.Tests/Helpers/SetupHelpersDisplaySetupSummaryTests.cs b/src/Tests/Microsoft.Agents.A365.DevTools.Cli.Tests/Helpers/SetupHelpersDisplaySetupSummaryTests.cs index b4f3f316..ed23925a 100644 --- a/src/Tests/Microsoft.Agents.A365.DevTools.Cli.Tests/Helpers/SetupHelpersDisplaySetupSummaryTests.cs +++ b/src/Tests/Microsoft.Agents.A365.DevTools.Cli.Tests/Helpers/SetupHelpersDisplaySetupSummaryTests.cs @@ -860,7 +860,7 @@ public void DisplaySetupSummary_BlueprintOnly_EmitsPermissionsNextSteps() logger.AllOutput.Should().Contain("a365 setup permissions mcp", because: "the blueprint summary must point to the remaining MCP permissions step"); logger.AllOutput.Should().Contain("a365 setup permissions bot", - because: "the blueprint summary must point to the bot/observability permissions step"); + because: "the blueprint summary must point to the bot permissions step"); } [Fact] diff --git a/src/Tests/Microsoft.Agents.A365.DevTools.Cli.Tests/Services/A365CreateInstanceRunnerTests.cs b/src/Tests/Microsoft.Agents.A365.DevTools.Cli.Tests/Services/A365CreateInstanceRunnerTests.cs index 9394a292..339dd9f9 100644 --- a/src/Tests/Microsoft.Agents.A365.DevTools.Cli.Tests/Services/A365CreateInstanceRunnerTests.cs +++ b/src/Tests/Microsoft.Agents.A365.DevTools.Cli.Tests/Services/A365CreateInstanceRunnerTests.cs @@ -2,6 +2,7 @@ // Licensed under the MIT License. using FluentAssertions; +using Microsoft.Agents.A365.DevTools.Cli.Constants; using Microsoft.Agents.A365.DevTools.Cli.Services; using Microsoft.Extensions.Logging; using Microsoft.Extensions.Logging.Abstractions; @@ -92,6 +93,141 @@ await graph.DidNotReceive().CreateOrUpdateOauth2PermissionGrantAsync( Arg.Any?>()); } + [Fact] + public async Task RunAsync_IdentityStep_DoesNotGrantObservabilityApiScopes() + { + var configPath = Path.Combine(_testDirectory, "a365.config.json"); + var generatedConfigPath = Path.Combine(_testDirectory, "a365.generated.config.json"); + await File.WriteAllTextAsync( + configPath, + """ + { + "tenantId": "11111111-1111-1111-1111-111111111111", + "environment": "prod" + } + """); + await File.WriteAllTextAsync( + generatedConfigPath, + """ + { + "agentBlueprintId": "22222222-2222-2222-2222-222222222222", + "agentBlueprintClientSecret": "test-secret", + "AgenticAppId": "33333333-3333-3333-3333-333333333333", + "AgenticUserId": "44444444-4444-4444-4444-444444444444" + } + """); + + var logger = Substitute.For>(); + var executor = Substitute.For(NullLogger.Instance); + var graph = Substitute.For(NullLogger.Instance, executor); + graph.LookupServicePrincipalByAppIdAsync( + Arg.Any(), + "33333333-3333-3333-3333-333333333333", + Arg.Any(), + Arg.Any?>()) + .Returns("agent-sp-object-id"); + graph.GetOauth2PermissionGrantsAsync(Arg.Any(), "agent-sp-object-id", Arg.Any()) + .Returns(Task.FromResult(new List<(string resourceId, string scope, string consentType)>())); + graph.EnsureServicePrincipalForAppIdAsync( + Arg.Any(), Arg.Any(), Arg.Any(), Arg.Any?>(), Arg.Any()) + .Returns("resource-sp-object-id"); + graph.CreateOrUpdateOauth2PermissionGrantAsync( + Arg.Any(), Arg.Any(), Arg.Any(), + Arg.Any>(), Arg.Any(), Arg.Any?>()) + .Returns(true); + + var runner = new A365CreateInstanceRunner(logger, executor, graph); + + var succeeded = await runner.RunAsync(configPath, generatedConfigPath, step: "identity"); + + succeeded.Should().BeTrue( + because: "an existing agent identity with readable grants completes the identity step"); + await graph.Received().EnsureServicePrincipalForAppIdAsync( + Arg.Any(), + PowerPlatformConstants.PowerPlatformApiResourceAppId, + Arg.Any(), + Arg.Any?>(), + Arg.Any()); + await graph.DidNotReceive().EnsureServicePrincipalForAppIdAsync( + Arg.Any(), + Arg.Is(appId => ConfigConstants.IsObservabilityApiAppId(appId)), + Arg.Any(), + Arg.Any?>(), + Arg.Any()); + } + + [Theory] + [InlineData(false)] + [InlineData(true)] + public async Task RunAsync_IdentityStep_GrantsSavedObservabilityConsentOnlyWhenCustomPermissionsOptIn(bool optIn) + { + var configPath = Path.Combine(_testDirectory, "a365.config.json"); + var generatedConfigPath = Path.Combine(_testDirectory, "a365.generated.config.json"); + // PascalCase nested keys on purpose: setup reads config case-insensitively, so create-instance must too. + var customPermissions = optIn + ? $$""" + , + "customBlueprintPermissions": [ + { "ResourceAppId": "{{ConfigConstants.ObservabilityApiAppId}}", "ResourceName": "Observability", "Scopes": [ "{{ConfigConstants.ObservabilityApiOtelWriteScope}}" ] } + ] + """ + : string.Empty; + await File.WriteAllTextAsync( + configPath, + $$""" + { + "tenantId": "11111111-1111-1111-1111-111111111111", + "environment": "prod"{{customPermissions}} + } + """); + // Configs set up by earlier CLI versions keep an Observability entry in resourceConsents. + await File.WriteAllTextAsync( + generatedConfigPath, + $$""" + { + "agentBlueprintId": "22222222-2222-2222-2222-222222222222", + "agentBlueprintClientSecret": "test-secret", + "AgenticAppId": "33333333-3333-3333-3333-333333333333", + "AgenticUserId": "44444444-4444-4444-4444-444444444444", + "resourceConsents": [ + { "resourceAppId": "{{ConfigConstants.ObservabilityApiAppId}}", "resourceName": "Observability", "scopes": [ "{{ConfigConstants.ObservabilityApiOtelWriteScope}}" ] } + ] + } + """); + + var logger = Substitute.For>(); + var executor = Substitute.For(NullLogger.Instance); + var graph = Substitute.For(NullLogger.Instance, executor); + graph.LookupServicePrincipalByAppIdAsync( + Arg.Any(), + "33333333-3333-3333-3333-333333333333", + Arg.Any(), + Arg.Any?>()) + .Returns("agent-sp-object-id"); + graph.GetOauth2PermissionGrantsAsync(Arg.Any(), "agent-sp-object-id", Arg.Any()) + .Returns(Task.FromResult(new List<(string resourceId, string scope, string consentType)>())); + graph.EnsureServicePrincipalForAppIdAsync( + Arg.Any(), Arg.Any(), Arg.Any(), Arg.Any?>(), Arg.Any()) + .Returns("resource-sp-object-id"); + graph.CreateOrUpdateOauth2PermissionGrantAsync( + Arg.Any(), Arg.Any(), Arg.Any(), + Arg.Any>(), Arg.Any(), Arg.Any?>()) + .Returns(true); + + var runner = new A365CreateInstanceRunner(logger, executor, graph); + + var succeeded = await runner.RunAsync(configPath, generatedConfigPath, step: "identity"); + + succeeded.Should().BeTrue( + because: "a saved Observability consent must not block the identity step"); + await graph.Received(optIn ? 1 : 0).EnsureServicePrincipalForAppIdAsync( + Arg.Any(), + ConfigConstants.ObservabilityApiAppId, + Arg.Any(), + Arg.Any?>(), + Arg.Any()); + } + [Theory] [InlineData("\"graphBaseUrl\": \"https://graph.example/us\"", "Graph base URL")] [InlineData("\"authorityHost\": \"http://login.example.com\"", "Authority host")]