Reuse Azure Monitor offline storage for Agent365

.github/workflows/pr-validation.yml

@@ -112,6 +112,32 @@ jobs:
       - name: Run tests
         run: tox -e ${{ matrix.tox-env }}
 
+  agent365-storage:
+    name: Agent365 storage (${{ matrix.os }})
+    runs-on: ${{ matrix.os }}
+    strategy:
+      fail-fast: false
+      matrix:
+        os: [ubuntu-latest, macos-latest, windows-latest]
+    steps:
+      - name: Check out repository
+        uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
+      - name: Set up Python
+        uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
+        with:
+          python-version: "3.10"
+      - name: Install package and pinned storage provider
+        run: python -m pip install -e ".[test]" "azure-monitor-opentelemetry-exporter==1.0.0b58"
+      - name: Exercise real filesystem storage and replay
+        run: >-
+          python -m pytest
+          tests/a365/test_persistent_storage.py
+          tests/a365/test_exporter.py
+          tests/a365/test_durable_restart.py
+          tests/a365/test_handler.py
+          tests/a365/test_replay_coordinator.py
+          tests/test_distro.py -q
+
   build:
     name: Build wheel
     runs-on: ubuntu-latest

A365_DOCUMENTATION.md

@@ -43,7 +43,7 @@ use_microsoft_opentelemetry(
 | `a365_exporter_timeout_ms` | `int` | `30000` | — | Timeout for a single A365 export operation in milliseconds. |
 | `a365_max_export_batch_size` | `int` | `512` | — | Maximum batch size for a single A365 export operation. |
 | `a365_exporter_disable_offline_storage` | `bool` | `False` | — | Disable durable offline storage. When `True`, failed exports are not persisted to disk and at-least-once delivery is not guaranteed. Defaults to `False` (storage enabled). |
-| `a365_exporter_storage_directory` | `str` | `None` | — | Custom directory for durable offline storage. When `None`, a platform default path is used. Choose a path that only the current user or service account can read, because stored OTLP payloads may contain prompts or completions when sensitive-data capture is enabled. |
+| `a365_exporter_storage_directory` | `str` | `None` | — | Root for an `a365` cache using Azure Monitor's offline storage provider. Configure separate sibling roots for A365 and Azure Monitor; shared or overlapping cache paths are unsupported and are not rejected automatically. `None` uses Azure Monitor's default path resolver with an Agent365 namespace. See [offline storage](#durable-delivery-offline-storage) for security and upgrade guidance. |
 
 ### Resource / Service Name
 
@@ -95,18 +95,40 @@ Key defaults and limits:
 |---|---|
 | Enabled by default | Yes (when exporter is active) |
 | Retention window | 2 days |
-| Maximum storage size | 50 MB |
+| Maximum storage size | 50 MiB (upstream capacity check before each write; one write can cross the limit) |
 | Storage path | Platform default (see below) |
 
-The default path is a sub-directory of the platform's local app data folder,
-derived from the user, executable, and working directory. Processes for the
-same application can share the SQLite queue; leases coordinate replay across
-those processes.
+Agent365 uses the **same `LocalFileStorage` provider as Azure Monitor**, not a
+separate SQLite queue. The provider owns atomic JSON-blob writes, rename-based
+leases, expiration, capacity checks, maintenance, and platform permissions.
+Agent365 only adapts identity envelopes and its authenticated replay protocol.
+Processes using the same cache can coordinate replay through those file leases.
+
+For an explicitly configured root, Agent365 records live in `<root>/a365/`.
+Azure Monitor uses its configured storage directory directly. When both
+exporters are enabled, configure **separate sibling roots**, as in the example
+below, so neither effective cache directory contains the other. Reusing the
+same root does not isolate capacity: Azure Monitor recursively counts files
+in its directory, including an `a365` child. Configurations that resolve to
+the same effective cache directory can also cause either exporter to claim
+or delete the other's incompatible records. **Shared or overlapping cache paths
+are unsupported and are not detected or rejected automatically.** Customers
+must keep their resolved paths separate, including any filesystem aliases.
+
+With no configured root, Agent365 calls Azure Monitor's
+`_get_storage_directory("agent365")`: the resulting cache is
+under the system temporary directory, in
+`Microsoft-AzureMonitor-<hash>/opentelemetry-python-agent365`. The upstream hash
+includes the namespace, user, process name, and application directory. Reusing
+the same resulting path preserves blob replay across restarts. Temporary-directory
+cleanup by the operating system can remove queued records; configure persistent
+storage when restart durability must survive that cleanup.
 
 **Replay behavior:**
 
 - Each queued record stores only its identity (tenant, agent, agentic user, S2S flag) and payload — not a fixed URL. On replay, the exporter reconstructs the export endpoint from the exporter's *current* configuration and re-resolves a fresh bearer token, so records queued before an endpoint or credential change are still delivered correctly, and any endpoint that would not resolve to HTTPS is rejected rather than replayed.
 - Records that are permanently rejected by the service (e.g. `400 Bad Request`) or that fail internal validation (unsupported schema version, blank tenant/agent id, or blank payload — "poison" records) are discarded from the durable queue instead of being retried forever.
+- Unreadable blobs are logged and retained for retry or retention expiry. The provider does not distinguish malformed JSON from transient read errors, so an unreadable blob is not treated as a proven poison record.
 - `shutdown()` is drain-safe: it signals the background replay loop to stop and blocks until any in-flight replay send actually finishes before closing the durable store and HTTP session, so an in-progress send is never left holding a closed resource and no accepted span is dropped mid-flight. Concurrent `shutdown()` callers all wait for that same cleanup to complete.
 
 **To disable durable storage** (no disk writes, best-effort delivery only):
@@ -119,17 +141,89 @@ use_microsoft_opentelemetry(
 )
 ```
 
-**To specify a custom storage directory:**
+**To configure both exporters with separate storage roots:**
 
 ```python
 use_microsoft_opentelemetry(
     enable_a365=True,
     a365_enable_observability_exporter=True,
-    a365_exporter_storage_directory="/var/lib/my-agent/telemetry",
+    enable_azure_monitor=True,
+    a365_exporter_storage_directory=r"C:\MyApp\Telemetry\Agent365",
+    azure_monitor_exporter_storage_directory=r"C:\MyApp\Telemetry\AzureMonitor",
 )
 ```
 
-> **Security note:** Stored OTLP payloads are unencrypted. When sensitive-data capture is enabled (`enable_sensitive_data=True`), payloads may include prompts, completions, or tool arguments. The exporter restricts the queue directory to the current user and administrators on Windows; on POSIX systems it enforces mode `0700` on the directory and `0600` on the SQLite database and sidecar files.
+The resulting cache directories are `C:\MyApp\Telemetry\Agent365\a365` and
+`C:\MyApp\Telemetry\AzureMonitor`. On Unix, use absolute sibling roots such as
+`/var/lib/my-agent/telemetry/Agent365` and
+`/var/lib/my-agent/telemetry/AzureMonitor`, provisioned with appropriate
+service-account ownership and permissions. Configure the Azure Monitor
+connection string or credential as usual. The same independent exporter-root
+configuration is supported by the .NET SDK; Python's existing `a365` child
+layout and provider permission handling remain unchanged.
+
+> **Security note:** Stored OTLP payloads are unencrypted. With sensitive-data capture enabled (`enable_sensitive_data=True`), payloads may include prompts, completions, or tool arguments. Do not configure a shared temporary directory such as `/tmp` as a customer storage root: a predictable child in a shared parent can be replaced or precreated. Normally provision a private, service-account-owned `0700` root and review its ACLs, ancestors, and protection against path replacement. Cache permissions do not isolate data from root, administrators, or other processes using the same account.
+
+The implementation inspected is Azure Monitor Python exporter **1.0.0b58**, pinned
+in `uv.lock`, at commit
+[`83907dc9ed2eb8c742b353b5f2cf276e039026da`](https://github.com/Azure/azure-sdk-for-python/blob/83907dc9ed2eb8c742b353b5f2cf276e039026da/sdk/monitor/azure-monitor-opentelemetry-exporter/azure/monitor/opentelemetry/exporter/_storage.py).
+Unlike the .NET provider, this Python provider creates its cache directory and,
+on Linux/macOS, opens the final directory without following a symlink, checks
+that the owner is the current user or root, and applies `0700` with `fchmod`.
+It creates blob temporary files exclusively with mode `0600` before renaming
+them. On Windows it grants the current user and administrators access and
+removes inherited ACLs using `icacls`. Agent365 neither duplicates nor bypasses
+these upstream cache protections.
+
+On Linux/macOS, Agent365 additionally checks the explicitly configured root
+before initializing the provider on its `a365` child. The configured path is
+normalized with `os.path.abspath`, matching the provider, before root validation
+and legacy-backlog detection. The root must be a real
+directory (not a symlink), owned by the current user or root, and not writable
+by group or others. Unsafe roots disable offline delivery without permission
+repair. A missing root is created with mode `0700` (subject to umask) and checked
+after creation as well. Existing root permissions are preserved; upstream
+checks and permission enforcement still apply to the existing or newly created
+`a365` child. Windows behavior and the implicit default resolver are unchanged.
+Creation of missing ancestors follows ordinary filesystem defaults.
+
+These point-in-time checks do not validate ancestor ownership, ACLs, or
+protection against subsequent path replacement. Customers remain responsible
+for provisioning and protecting the entire path; the checks are not a
+confidentiality guarantee.
+
+Storage is initialized lazily on first export. Invalid configuration (including
+blank paths), provider setup failure, or an initialization exception logs an
+error and disables **only offline buffering and replay**; online export continues.
+An explicitly configured path never falls back to another location on failure.
+Explicit disablement bypasses storage initialization altogether.
+
+**Breaking change and existing backlogs (all platforms):** the SQLite format,
+explicit-root layout, and implicit default location have changed. Existing
+`queue.db`, `queue.db-wal`, and `queue.db-shm` files are **not** opened, converted,
+deleted, or automatically migrated. **Required before upgrading:** restore connectivity and
+drain the old queue using the previous SDK and original configuration, then shut
+down all writers cleanly. Verify delivery before retiring the old directory.
+This is a customer-managed upgrade prerequisite, not an SDK migration step.
+If draining cannot be completed, retain a secured, coherent SQLite backup
+(including any necessary WAL/SHM state) for controlled recovery with the previous
+SDK; copying or renaming SQLite files into the new blob cache does not migrate
+them. A detected `queue.db` in an explicitly configured root produces a warning.
+No old implicit temporary/home-directory backlog is discovered automatically.
+
+To locate a previous implicit queue, inspect the old SDK's service environment.
+On Unix, its normal parent was `$XDG_STATE_HOME/a365-durable-queue` when
+`XDG_STATE_HOME` was nonempty, otherwise `~/.local/state/a365-durable-queue`. On
+Windows, it was `%LOCALAPPDATA%\a365-durable-queue`, with a system-temporary
+fallback when `LOCALAPPDATA` was unset. Unix also used a temporary fallback if
+the home directory could not be resolved and `XDG_STATE_HOME` was unset.
+Each queue was in a child named by the first 16 hexadecimal characters of
+`sha256((getpass.getuser() + sys.executable + str(Path.cwd())).encode())`.
+Changes to the service account, Python executable, or working directory can
+leave multiple old queue directories. Identify them before upgrading and use
+the previous SDK in the original environment to drain them; do not assume that
+successful replay of new JSON blobs proves recovery of an old SQLite backlog.
+Reusing an old directory alone does not convert its SQLite records.
 
 ## Auto-Instrumented Libraries
 

CHANGELOG.md

@@ -1,6 +1,42 @@
 # Release History
 # Unreleased
 
+### Breaking Changes
+- Agent365 offline delivery now reuses Azure Monitor's JSON-blob storage provider,
+  file leases, Unix permission handling, and Windows ACL handling instead of a
+  separate SQLite implementation. Explicit roots use an `a365` child;
+  implicit paths use Azure Monitor's resolver with an Agent365 namespace.
+  When both exporters are enabled, configure separate sibling storage roots
+  for independent capacity and replay. Shared or overlapping cache paths are
+  unsupported and are not detected or rejected automatically.
+  This changes cache layout, format, and default locations on all platforms,
+  including Linux and macOS. Drain existing SQLite backlogs with the previous
+  SDK before upgrading; no automatic migration or legacy replay is performed.
+  Provision a private service-account-owned root (normally `0700` on Unix),
+  review ACLs and ancestors, and protect against path replacement. Upstream
+  cache permission enforcement remains enabled.
+- Explicit Agent365 roots on Linux/macOS must not be symlinks, foreign-owned
+  (other than root), or group/world-writable. Invalid roots disable offline
+  delivery without changing their permissions. Missing roots are created with
+  mode `0700` subject to umask and validated before cache initialization.
+
+### Bugs Fixed
+- Invalid Agent365 storage configuration or provider initialization failure logs
+  a nonfatal error and disables only offline buffering/replay, preserving online
+  export. Explicit offline disablement bypasses storage initialization.
+- Malformed durable records with oversized numeric timestamps are logged and
+  discarded without interrupting replay of valid records in the same batch.
+- Normalize configured Agent365 roots with the storage provider's `abspath`
+  rules before validation, so symlinks followed by `..` cannot redirect the
+  checks to a different root. Legacy-backlog warnings use the same root.
+- Serialize Agent365 cache enumeration and lease transitions across Windows
+  threads and processes, including storage maintenance, to prevent concurrent
+  consumers from claiming the same record.
+- Isolate Agent365 storage setup diagnostics from Azure Monitor's process-global
+  customer SDKStats state while preserving upstream permission enforcement.
+- Log blob deletion-check filesystem errors without interrupting replay of
+  other valid Agent365 records in the same batch. Verify deletion with `stat()`
+  so Python 3.14 does not silently treat access failures as missing blobs.
 ### Other Changes
 - Update the `AZURE_OPENAI_API_VERSION` fallback version in integration tests
 

README.md

@@ -138,7 +138,7 @@ string arguments/results remain supported. See the A365 guide for complete examp
 | `a365_exporter_timeout_ms` | `int` | `30000` | Timeout for a single A365 export operation (ms). |
 | `a365_max_export_batch_size` | `int` | `512` | Maximum batch size for a single A365 export operation. |
 | `a365_exporter_disable_offline_storage` | `bool` | `False` | Disable durable offline storage. When `True`, failed exports are not persisted to disk. |
-| `a365_exporter_storage_directory` | `str` | `None` | Custom directory for durable offline storage. Restrict to the service account; payloads may contain prompts or completions when sensitive-data capture is enabled. |
+| `a365_exporter_storage_directory` | `str` | `None` | Root for an `a365` cache using Azure Monitor's offline storage provider and permission handling. Use separate sibling roots for A365 and Azure Monitor; shared or overlapping cache paths are unsupported and are not rejected automatically. `None` uses its default path resolver with an Agent365 namespace. Payloads are unencrypted; protect the root and ancestors. Existing SQLite backlogs require draining before upgrade. See [storage security and migration](A365_DOCUMENTATION.md#durable-delivery-offline-storage). |
 
 > For A365 token resolver patterns, baggage, and scope classes, see the [A365 guide](https://github.com/microsoft/opentelemetry-distro-python/blob/main/A365_DOCUMENTATION.md).
 

src/microsoft/opentelemetry/_distro.py

@@ -209,10 +209,12 @@ def use_microsoft_opentelemetry(**kwargs: object) -> None:  # pylint: disable=to
         failed export payloads are not persisted to disk and at-least-once
         delivery is not guaranteed. Defaults to ``False`` (storage enabled).
     :keyword str a365_exporter_storage_directory:
-        Custom directory for durable offline storage. When ``None``, a
-        platform default path is used. Stored payloads are unencrypted; choose
-        a path that only the service account can read, especially when
-        sensitive-data capture is enabled.
+        Root for an Agent365-only ``a365`` cache using Azure Monitor file
+        storage and permission handling. ``None`` selects Azure Monitor's
+        default path resolver with an Agent365 namespace. Stored payloads
+        are unencrypted; provision private parents for sensitive telemetry.
+        Configure separate sibling roots for Agent365 and Azure Monitor;
+        shared or overlapping cache paths are unsupported and are not rejected.
     :keyword bool enable_console:
         Enable console exporter for traces, metrics, and logs (development
         only).  Mirrors ``ExportTarget.Console`` from the .NET distro.
@@ -536,9 +538,8 @@ def _append_a365_components(
         coerce_storage_directory,
     )
 
-    # Validate/normalize the storage directory up front so an explicitly empty
-    # value raises ValueError instead of being swallowed by the broad handler
-    # below (which would silently fall back to the platform default path).
+    # Defer invalid storage configuration to lazy initialization, so it disables
+    # offline delivery without preventing online export.
     storage_directory_path = coerce_storage_directory(storage_directory)
     _resolve_batch_options(
         max_queue_size=max_queue_size,

src/microsoft/opentelemetry/a365/core/exporters/agent365_exporter.py

@@ -96,7 +96,7 @@ def __init__(
         cluster_category: str = "prod",
         use_s2s_endpoint: bool = False,
         max_payload_bytes: int = DEFAULT_MAX_PAYLOAD_BYTES,
-        storage_directory: Optional[Path] = None,
+        storage_directory: Optional[str | Path] = None,
         enable_durable_delivery: bool = True,
     ):
         if token_resolver is None and contextual_token_resolver is None: