mcp(observability): PostHog error tracking, tracing, and usage telemetry at every dispatch chokepoint — redacted, gated, registry-driven

[JSONbored]

Context

Part of #9515; the MCP lane of the PostHog consolidation (#8286). metagraphed's MCP observability is the reference implementation (its src/usage-telemetry.ts + the callTool dispatch wrapper on main):

• One chokepoint. Every tools/call passes through one dispatch function; telemetry wraps there once instead of instrumenting ~150 handlers.
• Two usage events with two shapes: usage_event — minimal own-telemetry (tool name, ok, duration, and an errorCode drawn from a small developer-defined literal set, never a free-form message or arguments); and $mcp_tool_call — PostHog's MCP Analytics event family (their fix(engine): fold idea hints/constraints into the first draft's own explicit criteria #7737), which does include call arguments/results but only after a dedicated redact-and-size-cap step.
• Error tracking via raw $exception capture (their Register loopover_get_outcome_calibration as a local stdio MCP tool #7758): manual $exception_list payloads (stack frames parsed + frame-capped, fingerprint grouped by error code), replacing Sentry at every capture site.
• No posthog-node in the Worker — raw fetch to PostHog's capture endpoint (SDK batching/flush semantics don't survive isolates and it costs ~40 KiB against the bundle budget); safe no-op when the token is unset; never throws.
• Args-safety is tested, not assumed (their mcp-server-trace-span-args-safety / dispatch-error-capture test files).

LoopOver today: the remote server has src/mcp/telemetry.ts (custom mcp_tool_call, env-gated) and route-level error middleware (createWorkerPostHogErrorMiddleware, src/api/routes.ts:1165) that does not cover MCP dispatch; the stdio package has lib/telemetry.ts (opt-in, posthog-node, 4-property allowlist, per-call flush); the miner has a Sentry seam (lib/sentry.ts) slated for the PostHog swap (#8292) plus lib/posthog.ts. Three implementations, three property lists, no exception capture on MCP tool dispatch, no tracing.

Requirements

1. One telemetry module driven by the contract registry, consumed by all three servers at their dispatch chokepoints (remote toolResult/register wrapper; stdio wrapStdioToolHandler; a new miner dispatch wrapper). Per call it emits, subject to each surface's gates: the minimal usage event and the $mcp_tool_call MCP-Analytics event. Tool name, category, ok, duration_ms, and error_code come from the registry + the schema-described error envelope (contract(remote): migrate the remote MCP server's tools to @loopover/contract — typed handlers, real output schemas #9518) — error_code is a closed literal set, never free text.
2. Redaction + size caps before $mcp_tool_call: a single recordMcpToolCall-style function owns argument/result inclusion — redaction via the existing scrub helpers, hard byte caps, and a per-tool opt-out for tools whose inputs are inherently sensitive (config contents, log tails from orb(mcp): ORB management tool family — instance doctor/status/logs, queue+kill-switch ops, fleet + hosted-tenant administration #9522: args/results excluded entirely for those, asserted by test).
3. Exception capture on dispatch: every thrown (non-envelope) error in any server's tool dispatch is captured as a PostHog $exception with mcp_tool + error_code grouping properties — Worker/self-host via the fetch-based capture (no posthog-node in the Worker bundle; respect the existing bundle budget), stdio/miner through their gated clients. The miner's capture rides PostHog error tracking for the miner CLI (opt-in seam swap; egress-allowlist + env-reference coupling) #8292's seam swap and its egress-allowlist + env-reference requirements (the firewall must allow the PostHog host only when telemetry is enabled).
4. Tracing: an OTel span per tool call on the remote/self-host path (the self-host stack already ships src/selfhost/otel.ts + Tempo): span name mcp.tool/<name>, attributes limited to tool name, category, ok, error_code, duration — never arguments (args-safety test, metagraphed-style). Stdio/miner emit no spans (no collector), only events.
5. Structured logging at dispatch: one log line per failed call (level error for envelope errors, warn for declined elicitation) with the same closed property set, flowing into the existing Loki pipeline on self-host; success calls log at debug only. No payload content in logs.
6. Gates unchanged and documented: stdio stays double-gated opt-in; remote stays env-gated; miner stays opt-in per PostHog error tracking for the miner CLI (opt-in seam swap; egress-allowlist + env-reference coupling) #8292. All gate env vars appear in the generated env references (selfhost:env-reference, miner:env-reference). Everything is a safe no-op unset and never throws into the tool path.
7. Single-sourced property allowlist (shared constant per contract: generate every derived surface — typed stdio client, CLI from one spec table, tool-reference docs, miner dedup, UI z.infer types #9521's telemetry item) with a meta-test asserting no telemetry payload key exists outside it, and a FORBIDDEN_CONTENT-style test asserting no secret-shaped value can reach any telemetry sink from any tool's inputs/outputs.
8. Dashboards: update/extend the MCP analytics: dashboards and insights over the existing mcp_tool_call PostHog event #8606 MCP analytics dashboards to the new event shapes (tool × category × error_code breakdowns, per-server split), recorded there.
9. Full branch-counted coverage, including the never-throws guarantees and both sides of every gate.

Non-goals

• Sentry decommission mechanics beyond the MCP surfaces (Sentry decommission: gated cutover checklist across all six surfaces #8298 owns the cutover checklist).
• Product analytics outside MCP dispatch; PostHog LLM-analytics instrumentation of the reviewers (separate concern).

Deliverables

• Registry-driven telemetry module wired at all three dispatch chokepoints
• $mcp_tool_call with redaction/size caps + sensitive-tool opt-outs; minimal usage event with closed error codes
• $exception capture on dispatch across all servers (miner via the PostHog error tracking for the miner CLI (opt-in seam swap; egress-allowlist + env-reference coupling) #8292 seam)
• OTel spans on remote/self-host with args-safety tests
• Structured dispatch logging; allowlist meta-test; forbidden-content telemetry test
• Env-reference entries + MCP analytics: dashboards and insights over the existing mcp_tool_call PostHog event #8606 dashboard update

Expected outcome

Every MCP tool call on every server produces consistent, privacy-safe telemetry: usage analytics by tool/category/error-code, PostHog error tracking with grouped exceptions, and traces on the self-host/remote path — with argument safety proven by tests, all gates honored, and zero hand-duplicated property lists.

References

Part of #9515. Blocked by #9518 (registry + error envelopes). Advances #8286 / #8294 / #8606; the miner half rides #8292. Pattern source: metagraphed src/usage-telemetry.ts + its dispatch wrapper (#7737 / #7758 there).

[linear-code]

JSO-6581