feat(OP-3108): Initialize OpenTelemetry in foundry-python-core

[fubezz]

What this does

Gives every Foundry service OpenTelemetry support (traces, metrics, and optionally logs) via a single core version bump — no per-service instrumentation code required. Once a service turns it on, its telemetry lands in the internal Tempo/Loki/Prometheus stack over OTLP/gRPC, via ADR 0006.

Part of OP-3108. This is the client/SDK side; the gateway it talks to is gitops#3459, and the one-line template wiring (now largely unnecessary — see below) is foundry-python#385.

How it's shaped

Follows the same pattern already used for Sentry: a pydantic-settings-driven enabled flag, a single otel_initialize() entry point called from boot(), and graceful no-ops when the SDK isn't installed or isn't configured.

• Traces, metrics, logs are independently toggleable. Traces and metrics default on once the master switch is on; logs default off — different volume/cost profile, so it shouldn't be forced on every service that just wants tracing.
• Endpoint, service name, and certificate come from the standard, unprefixed OTel env vars (OTEL_EXPORTER_OTLP_ENDPOINT, OTEL_SERVICE_NAME, OTEL_EXPORTER_OTLP_CERTIFICATE) rather than reinventing project-prefixed ones, so services already familiar with the OTel spec don't need to learn a second set of names.
• Logs: this project uses loguru, not stdlib logging, and loguru has no first-party OTel integration. Added a sink that converts each loguru record into a stdlib LogRecord and forwards it to OTel's own LoggingHandler, reusing the SDK's conversion/trace-correlation logic instead of reimplementing OTLP log export.
• Default instrumentors, with full override: HTTPXClientInstrumentor (so an outbound httpx call to another Foundry service continues the caller's trace instead of starting a new, disconnected one) and SQLAlchemyInstrumentor (every Foundry service gets a SQLAlchemy-backed database layer, so DB spans are as much a baseline expectation as HTTP ones). Both are optional-dependency-gated and only applied once a real TracerProvider is actually registered — a project can pass its own list, or [] to opt out entirely.
• FastAPI instrumentation now happens automatically too — init_api() (the function that actually constructs the app instance, and always runs after boot()) instruments the returned app and every versioned sub-app before returning, and skips it entirely when no real TracerProvider is registered (mirroring the guard the other instrumentors use, so a service with OTel disabled doesn't pay for OpenTelemetryMiddleware on every request for nothing). This closes the original "known follow-up" below — outside of the foundry-python template, nothing extra needs to be called.
• TLS: OTEL_EXPORTER_OTLP_CERTIFICATE defaults to a combined CA bundle (certifi's system roots + the fleet's internal aignx-ca-root-authority), since the gateway's TLS-facing endpoint is signed by that internal CA — bundling both keeps a service that points at a public vendor instead working too.
• boot() logs whether it actually initialized — otel: on/off alongside the existing sentry: on/off, so you can tell from the boot log whether telemetry is actually flowing without checking env vars by hand.
• Guards throughout against boot() running twice in the same process (tests, a subprocess that inherited parent state) — each provider no-ops if a real one is already registered, instead of stacking a second exporter on top.

Verified end-to-end

Beyond unit/integration tests, this was smoke-tested against the real sandbox OTLP gateway from a Docker container: TLS handshake succeeds via the combined CA bundle (where a raw curl with only system roots fails), and both SpanExportResult.SUCCESS and LogRecordExportResult.SUCCESS come back from the gateway.

Test plan

• mise run lint — ruff, pyrefly, deptry all pass.
• mise run test_unit — 346 unit tests passing, otel.py at 100% coverage.
• mise run test_integration
• mise run pre_commit_run_all
• Manual smoke test against the real sandbox OTLP gateway (TLS + span/log export, see above).

🤖 Generated with Claude Code

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ All tests successful. No failed tests found.

Files with missing lines	Coverage Δ
src/aignostics_foundry_core/api/core.py	93.33% <100.00%> (+0.17%)	⬆️
src/aignostics_foundry_core/boot.py	79.01% <100.00%> (+0.53%)	⬆️
src/aignostics_foundry_core/otel.py	100.00% <100.00%> (ø)

[fubezz]

Notes from testing this against a locally-generated Foundry service (OP-3108, sandbox). A few design points worth considering before merge, none blocking:

1. otel_initialize() has no equivalent to SENTRY_INTEGRATIONS

sentry_initialize() takes an explicit integrations: list[Integration] | None built by the generated project's own constants.py (try/except-imported per optional dep, e.g. FastApiIntegration, TyperIntegration). otel_initialize() has no equivalent — it only sets up the bare TracerProvider/MeterProvider/LoggerProvider. Everything downstream of the inbound HTTP request (DB queries, outbound httpx calls, Redis, etc.) is invisible unless something calls the relevant opentelemetry-instrumentation-* package's .instrument().

Most of those packages subclass a common BaseInstrumentor with a no-argument .instrument() — genuinely Sentry-integration-shaped, so the same pattern would fit:

# otel.py
def otel_initialize(*, context=None, instrumentors: list[BaseInstrumentor] | None = None) -> bool:
    ...
    for instrumentor in instrumentors or []:
        instrumentor.instrument()

with a project's constants.py building OTEL_INSTRUMENTORS = [HTTPXClientInstrumentor(), SQLAlchemyInstrumentor()] the same way it builds SENTRY_INTEGRATIONS today.

Concretely, HTTPXClientInstrumentor is the one I'd prioritize: without it, an outbound httpx call to another Foundry service doesn't get the W3C traceparent header injected, so the callee's span starts a new disconnected trace instead of a child — which undercuts the actual point of this ADR (getting Cloud Run into the same distributed-tracing picture as everything else).

FastAPIInstrumentor stays exactly as it is (instrument_fastapi(app), called separately from Service.api()) — it's the odd one out because ASGI middleware has to wrap the live app instance, which doesn't exist yet at boot() time. That's a real API shape difference, not an oversight, so full parity with the Sentry list isn't achievable — only partial.

Separately, worth a doc note (in otel_initialize()'s docstring or the ADR's consequences section): manual/business-level spans and custom metrics already work today, no signature change needed — trace.get_tracer(__name__) / metrics.get_meter(__name__) just work anywhere once otel_initialize() has set the global providers.

2. boot() silently discards the init result

otel_initialize() and sentry_initialize() both return bool (whether they actually initialized vs. no-op'd because the master switch was off or the endpoint/DSN was missing), but boot() doesn't do anything with either return value. Today there's no log line telling you at runtime whether OTel/Sentry actually came up. Appending something like , otel: {on/off}, sentry: {on/off} to the existing _log_boot_message() call would close that gap cheaply, using data that's already computed.

3. Considered and rejected: master enabled switch as a boot() parameter

To make it "visible in code" rather than purely env-driven. Checked SentrySettings.enabled — it's the exact same env-only pattern as OTelSettings.enabled (no boot() param for it either), so promoting only OTel's switch to a boot() arg would break symmetry with Sentry rather than restore it, and risks a second source of truth (hardcode True in generated code but still gate on env, or duplicate the env read outside OTelSettings). Point 2 above (log the actual init result) gets the same visibility without the architecture change.

[aig-hannes]

Looks good to me

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
100.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud