[Nikhil Navakiran (nikhilNava)]

Summary

Replace Agent365's independent SQLite queue and custom filesystem-permission
implementation with the same LocalFileStorage/LocalFileBlob provider used by
Azure Monitor Python. This follows the latest request to have one storage
implementation in this SDK, superseding the earlier customer-managed-only
permission proposal and the initial hardening commit in this PR.

• Reuse upstream atomic JSON-blob writes, file leases, retention, capacity checks,
  maintenance, Unix permissions, Windows ACLs, and the default path resolver.
• Keep Agent365's identity envelope and replay protocol: no persisted bearer
  tokens or fixed URLs; replay resolves fresh credentials and current HTTPS
  endpoints.
• Keep independent storage-directory options, as in the .NET SDK. Configure
  separate sibling roots for Agent365 and Azure Monitor: Agent365 uses
  <Agent365-root>/a365, while Azure Monitor uses its directory directly.
  Shared or overlapping effective cache paths are unsupported and are not
  detected or rejected automatically. Default Agent365 paths use upstream
  _get_storage_directory("agent365").
• Log initialization/configuration failures and disable only buffering/replay.
  Online export continues, and explicit disablement bypasses initialization.
• Remove the Agent365-specific chmod/chown, ancestor traversal, and native ACL
  implementation rather than maintaining two permission-management systems.
  Do not modify or weaken the upstream provider.
• Validate the explicitly configured Unix root before initializing the provider
  on its a365 child: reject symlinks, foreign ownership (except root), and
  group/world write access without permission repair. Create missing roots with
  0700 subject to umask and validate them after creation. Upstream still checks
  and secures the existing or newly created cache child.
• Detect unreadable blobs without deleting possibly valid telemetry: the
  upstream reader conflates malformed JSON and transient read failures.
• Log and drop malformed envelopes with oversized numeric timestamps without
  interrupting valid records already claimed or remaining in the same batch.
• Update configuration docs, security responsibilities, changelog and regression
  tests; add real Ubuntu/macOS/Windows filesystem CI on Python 3.10.

Source comparison

The repository supports Python >=3.10. pyproject.toml declares
azure-monitor-opentelemetry-exporter~=1.0.0b58; uv.lock resolves 1.0.0b58.
No dependency manifest, lockfile, or upstream exporter source is changed.

Inspected the actual b58 release source at Azure/azure-sdk-for-python commit
83907dc9ed2eb8c742b353b5f2cf276e039026da:

• _storage.py:
  LocalFileBlob.put creates exclusive temporary files with mode 0600 and
  renames them; leases are encoded by rename. LocalFileStorage has 50 MiB /
  48-hour defaults, a maintenance task, and permission setup.
• export/_base.py:
  _get_storage_directory uses the system temporary directory and a stable hash
  of namespace/instrumentation key, user, process name, and application directory.

Unlike the customer-managed approach in
microsoft/opentelemetry-distro-dotnet#179, the Python b58 provider creates
directories, validates final-directory ownership, and applies 0700 with
fchmod on Unix. It uses icacls on Windows. These language-specific upstream
defaults remain enabled; this PR does not claim Python and .NET behave alike.

Azure Monitor passes its configured directory directly to LocalFileStorage;
Agent365 passes <configured-root>/a365. Upstream therefore checks the child,
not Agent365's configured root. The additional read-only root validation closes
that initialization gap without adding another storage backend or changing
upstream permission handling. Windows and the default resolver are unchanged.

Separate exporter roots

Configure the exporters independently with sibling roots:

use_microsoft_opentelemetry(
    enable_a365=True,
    a365_enable_observability_exporter=True,
    enable_azure_monitor=True,
    a365_exporter_storage_directory=r"C:\MyApp\Telemetry\Agent365",
    azure_monitor_exporter_storage_directory=r"C:\MyApp\Telemetry\AzureMonitor",
)

Their effective cache directories are
C:\MyApp\Telemetry\Agent365\a365 and
C:\MyApp\Telemetry\AzureMonitor. Unix deployments can use private sibling
roots under /var/lib/my-agent/telemetry instead.

The earlier same-root isolation claim was too broad. Azure Monitor recursively
counts nested files toward its capacity, and identical effective cache paths
can let either exporter claim or delete the other's incompatible records.
This revision documents separate roots as the supported configuration and
adds real-filesystem regressions for independent capacity, replay, and deletion.
It does not add cross-exporter overlap validation, change defaults, alter Unix
permission policy, or introduce another storage backend. Customers must keep
the effective paths separate, including filesystem aliases.

Breaking change / operational guidance

The format, configured-root layout, and default locations change on all
platforms, including Unix. Existing SQLite databases and sidecars are neither
opened nor converted. Drain old backlogs with the previous SDK and original
configuration before upgrade, then stop writers cleanly. If draining is
impossible, retain a secured, coherent SQLite backup for controlled recovery
with that SDK. Moving a SQLite database into the blob cache is not migration.
A queue.db in an explicit root produces a warning; old implicit backlogs are
not automatically searched or migrated.

Explicit Unix roots that are symlinks, foreign-owned (except root), or writable
by group/others now disable offline delivery. Existing customer root permissions
are never repaired.

Payloads remain unencrypted. Normally provision a private service-account-owned
0700 root, review ACLs and ancestors, and prevent path replacement. Shared roots
such as /tmp are unsafe. Provider permission setup is not a guarantee about
all ancestors or future replacement. Its capacity check is upstream's soft
pre-write limit, and operating-system cleanup of a default temporary cache can
remove queued data.

Verification

Deletion-check and umask follow-up

Commit af9756a catches/logs OSError from the post-delete existence check
and returns False without interrupting replay batches. Four regressions
failed before the fix and now pass. The ownership test preserves the observed
original directory mode, with explicit existing-root umask 022/077 cases.

Fresh Windows Python 3.11 validation explicitly verified the imported exporter
version 1.0.0b58 and its _storage.py path in the isolated installation:
269 passed, 18 skipped, 16 subtests passed, with unhandled-thread warnings
treated as errors. Black, Pylint, Mypy, and diff checks pass. POSIX umask cases
are skipped locally and covered by the existing Linux/macOS CI jobs.

Earlier local version-label correction: import-path checking found the
previous environment's installed b58 metadata was shadowed by the system b57
Azure namespace. Earlier local b58 labels below were therefore inaccurate;
the b57 and b58 _storage.py sources are byte-identical, but the fresh run
above verifies actual b58 imports. Separately recorded CI results are unaffected.

Windows concurrency and SDKStats follow-up

Commit 478e396 adds per-cache Windows named-mutex coordination across
threads/processes for enumeration, leases, maintenance, release and deletion.
It also isolates Agent365's setup-error callbacks using an invocation-local
copy of the upstream permission-check function's globals, without changing
its permission implementation or mutating Azure Monitor SDKStats state.

Actual b58 regression tests reproduced duplicate/missing claims with eight
concurrent consumers and setup-state pollution for EACCES/EROFS before the fix.
The one/200-record thread/process cases, claim-limit handoff, mutex-setup
failure, and Monitor diagnostic-isolation cases now pass.

Final local validation: Windows Python 3.11, exporter 1.0.0b58:
265 passed, 17 skipped, 16 subtests passed, treating unhandled-thread
warnings as errors. Black, Pylint, Mypy, and diff checks pass. The local Docker
daemon is unavailable; fresh Linux/macOS results must come from the existing
cross-platform CI job. No dependency manifests or lockfiles were changed.

Merge and root-normalization follow-up

Merged latest main a841401 in 4cf47cc, preserving the Agent365 changelog
entries and upstream integration-test version updates. Follow-up eeb7dbf
normalizes explicit roots with os.path.abspath before validation, matching
the provider, and uses the normalized root for legacy-backlog warnings too.
This fixes the Path.absolute()/provider mismatch for symlinks followed by ...

Windows Python 3.11 with the existing b57 provider: 255 passed, 17 skipped,
16 subtests passed, treating unhandled-thread warnings as errors. Black,
Pylint, Mypy, and diff checks pass. The cross-platform normalization regression
failed before the fix and passes afterward. Two additional POSIX cases cover
private/writable roots on opposite sides of a symlink-plus-.. path, unchanged
permissions, legacy-backlog detection, and restart/replay.
Those POSIX cases were skipped on Windows; the local Docker daemon is unavailable,
so no fresh Linux execution is claimed. They are included in the existing b58
cross-platform storage CI job. Previous-head CI results below are not represented
as validation of the current head.

Separate-root follow-up

Separate-root follow-up validation on Windows Python 3.11 with the existing
b57 provider: 254 passed, 15 skipped, 16 subtests passed, with unhandled-thread
warnings treated as errors. The capacity regression first reproduced the
shared-root failure, then passed with sibling roots. The new cases fill each
cache to its configured limit and verify the other cache can still write and
replay. Black, Pylint on the changed tests, and git diff --check passed.
The unchanged cross-platform CI job pins b58; its new-head results are separate
from the previous-head results recorded below.

Previous-head validation

Focused command (host paths use platform-appropriate separators):

python -m pytest tests/a365/test_persistent_storage.py tests/a365/test_exporter.py tests/a365/test_durable_restart.py tests/a365/test_handler.py tests/a365/test_replay_coordinator.py tests/test_distro.py -q --tb=short

• Windows Python 3.11: 252 passed, 15 skipped, 16 subtests passed, including
  a run treating unhandled-thread warnings as errors. This existing host
  environment has exporter b57; it is not represented as a fresh lockfile install.
• Real Linux Python 3.10 container, non-root user, actual b58 wheel:
  266 passed, 1 skipped, 16 subtests passed. Filesystem tests use the
  container's Linux filesystem, not a Windows bind-mounted cache.
• Real Linux Python 3.11 / b58: 254 passed, 1 skipped, 16 subtests passed
  before the additional read-failure regression; final cross-platform CI covers
  the final version.
• Earlier broader Windows Agent365 suite plus distro tests, before the
  timestamp-overflow regressions:
  763 passed, 9 skipped, 53 subtests passed.
• Earlier broader Linux Python 3.11 / b58 Agent365 suite plus distro tests,
  before the timestamp-overflow regressions:
  735 passed, 8 skipped, 53 subtests passed.
  Both broader runs reported a batch-processor condition-lock thread warning.
  The identical warning was reproduced on unchanged main
  (752 passed, 8 skipped, 45 subtests passed), confirming it predates this
  change. The focused final run is warning-clean with thread warnings treated
  as errors.
• Black, Pylint (implementation and changed tests), Mypy (implementation and
  changed tests), and git diff --check pass.
• Regression red/green verified for provider reuse, retaining records after
  transient upstream read failures, and oversized positive/negative timestamp
  integers placed between valid records in the same replay batch. Review
  feedback is addressed in 32d98bf; the overflow cases failed before the fix.
• Root regressions reproduced 8 failures before the fix on real Linux:
  unsafe roots were accepted and new roots inherited permissive umasks.
  Coverage now includes configured-root symlinks, existing/new foreign-owned
  roots, group/world-write rejection without repair, private creation under
  umasks 000/022/077, restart/replay, unchanged parent permissions, and
  logged offline-only disablement with successful online sends. Ownership is
  simulated through lstat metadata; permissions, symlinks, and replay use real
  files. Existing child-symlink and upstream permission/Windows tests remain.
• Initial Windows CI exposed a
  test-ordering assumption: consecutive writes can share a coarse clock tick,
  making random filename suffixes decide ordering. A deterministic probe
  reproduced the failure; the test now controls timestamps and suffixes to
  force its intended poison-first ordering without sleeps. Adapter behavior
  did not require changing.

Previous-head CI at 2a356c74869b02aa92695e2e35d7c644492bd78d.
The branch includes main dfd55172ad4276350c6fe149f29c86542894b525
via merge without rewriting the PR history:

Real filesystem job	Python / exporter	Result
Ubuntu	3.10 / 1.0.0b58	266 passed, 1 skipped, 16 subtests passed
macOS	3.10 / 1.0.0b58	266 passed, 1 skipped, 16 subtests passed
Windows	3.10 / 1.0.0b58	252 passed, 15 skipped, 16 subtests passed

All checks passed at that previous head: the platform jobs, full Ubuntu Python 3.10–3.14 suites,
Black/Pylint/Mypy, integration tests, documentation, wheel build, changelog,
benchmark workflow, and CLA.
Final validation run
and benchmark run.

Linux dependency installation in Docker hit a PyPI TLS handshake failure.
Verified wheels were downloaded through host TLS and installed offline in the
container, including Linux-native psutil/cryptography/cffi. The actual imported
b58 provider path and version were confirmed.

Limitations: reuse depends on Azure Monitor's internal provider API, covered
by focused compatibility tests. Cache permission handling and default paths
remain upstream's. Configured-root validation is point-in-time; it does not
validate ancestors/ACLs or prevent later path replacement. Customers must protect
the entire path and configure non-overlapping exporter caches. Shared or
overlapping cache paths are unsupported and are not rejected automatically.
No live service transmission is used in these tests; sends are
simulated while persistence/restart/replay uses real files.

[github-actions]

Performance comparison

Threshold: regressions >15.0% on gating scenarios fail the build. Higher ops/s is better; positive Δ means the PR is slower.

Scenario	Gating	Baseline (ops/s)	Candidate (ops/s)	Δ %	Status
azure_monitor_log	yes	18,705.6	18,392.2	+1.70%	✅
azure_monitor_span	yes	154,750.9	154,750.9	-0.00%	✅
otel_log	no	21,912.5	22,043.9	-0.60%	✅
otel_span	no	40,215.6	40,394.2	-0.44%	✅

[Copilot]

🟡 Changes recommended

Safe search-only ancestors currently disable durable delivery, and new file-validation checks need regression coverage.

2 open issues

• Use search-only ancestor access to support 0711 directory traversal · New
• Add regression tests for unsafe queue files and fail-closed behavior · New

What changed in this PR

Hardens Agent365’s durable queue storage against unsafe Unix permissions and paths while preserving Windows ACL handling.

Changes:

• Validates directory ancestry and database files before queue operations.
• Adds filesystem security and restart regressions.
• Documents stricter storage requirements and failure behavior.

File	Description
tests/​a365/​test_persistent_storage.py	Adds permission, symlink, replacement, and restart tests.
tests/​a365/​test_exporter.py	Tests initialization failure without storage fallback.
src/​microsoft/​opentelemetry/​a365/​core/​exporters/​persistent_storage.py	Adds secure directory traversal and runtime validation.
README.md	Summarizes existing-directory requirements.
A365_DOCUMENTATION.md	Explains storage protections and trust boundaries.

_{Review effort: Balanced}

[Copilot]

🟡 Changes recommended

Oversized timestamps can interrupt replay, and a regression test depends on nondeterministic blob ordering.

2 open issues

• Catch OverflowError for oversized timestamps in claim() · New
• Claim both blobs to validate records regardless of filename order · New

2 resolved since last review

• Add regression tests for unsafe queue files and fail-closed behavior
• Use search-only ancestor access to support 0711 directory traversal

_{Review effort: Balanced}

[Copilot]

🔵 Needs a closer look

The previously reported timestamp-overflow validation issue remains unresolved and can interrupt replay batches.

1 open issue

• Catch OverflowError for oversized timestamps in claim()

1 resolved since last review

• Claim both blobs to validate records regardless of filename order

_{Review effort: Balanced}

[Copilot]

🔵 Needs a closer look

The breaking persistence migration and reliance on private upstream storage APIs warrant final human review.

No open issues 🎉

1 resolved since last review

• Catch OverflowError for oversized timestamps in claim()

_{Review effort: Balanced}

[Copilot]

🔵 Needs a closer look

The breaking persistence-format change and cross-platform permission and lease behavior warrant final human review.

No open issues 🎉

_{Review effort: Balanced}

[Copilot]

🟢 Approval recommended

The migration to Azure Monitor file storage has no active review issues.

No open issues 🎉

1 previously missed issue

In code that hasn't changed since last review

Treat filesystem errors as failed deletion

src/​microsoft/​opentelemetry/​a365/​core/​exporters/​persistent_storage.py:305

On Python 3.14, Path.exists() returns False for permission and other filesystem errors, so a failed upstream deletion can be reported as successful and its claim handle discarded while the blob remains. Use lstat() and treat only FileNotFoundError as confirmed deletion, logging other errors. Update the regressions to fail the stat operation rather than mock exists() to raise.

_{Review effort: Balanced}

[Copilot]

🟡 Changes recommended

On Python 3.14, Path.exists() suppresses filesystem errors, so deletion can falsely report success and discard an inaccessible blob’s replay handle without logging the failure.

1 open issue

• Use stat() to detect deletion errors instead of Path.exists() · New

_{Review effort: Balanced}

[Copilot]

🔵 Needs a closer look

On supported Python 3.14, Path.exists() can suppress filesystem errors and falsely signal successful deletion, discarding claim handles while blobs remain and risking duplicate delivery.

No open issues 🎉

1 resolved since last review

• Use stat() to detect deletion errors instead of Path.exists()

_{Review effort: Balanced}

[Copilot]

🔵 Needs a closer look

Security-sensitive permission changes and Windows cross-process mutex coordination require owner review and current-head cross-platform validation, as the linked successful CI run covers an older revision.

No open issues 🎉

_{Review effort: Balanced}

[Copilot]

🔵 Needs a closer look

Security-sensitive storage and permission changes plus native Windows cross-process locking require storage/security-owner review and current-head cross-platform runtime validation beyond the static inspection performed.

No open issues 🎉

_{Review effort: Balanced}