diff --git a/AGENTS.md b/AGENTS.md index bc09b6b..257cd90 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -14,6 +14,17 @@ for a marked rules block. - Hooks: add a hook or a client in `hooks/hook-hosts.mjs`, never in a `hooks.json`. A new client also needs its expected output in `test/hooks.test.mjs`. +- Telemetry: `hooks/telemetry-contract.mjs` defines every event, property, + and allowed value; change them there only. Client-specific hook input mapping + lives only in `hooks/telemetry-adapters/.mjs`. Wire a client through + `HOSTS.telemetry` in `hooks/hook-hosts.mjs` (generated manifests follow + that source). `telemetry.mjs` prints nothing; only `telemetry-send.mjs` may + touch the network (a test enforces both). `TELEMETRY_ENABLED` in + `hooks/telemetry-config.mjs` stays `false` without separate approval; while + false, `npm run generate` writes no telemetry hooks. Reporting and maintained + docs for exports live in `scripts/telemetry-report.mjs` and + `docs/telemetry.md`. `test/telemetry-report.test.mjs` cross-checks adapters + when `hooks/telemetry-adapters/.mjs` exists on the branch. - Codex hooks are blocked upstream ([docs/install/codex.md](docs/install/codex.md)). Don't remove the root `$schema` to force them; that breaks Agent Plugins conformance. diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index 207d7a3..eadc0c7 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -85,6 +85,12 @@ success or narrating tool internals. The Arcade MCP server is the canonical place to record request, authentication, tool-discovery, tool-call, and completion outcomes. This package does not ask a -model to self-report tokens, turns, or success, and it ships no telemetry hook. -If a host-specific hook later adds supplemental signals, it must be explicit, -opt-in, and documented as non-portable. +model to self-report tokens, turns, or success. + +Optional client hooks in `hooks/telemetry-adapters/` can send supplemental, +scoped usage events when `TELEMETRY_ENABLED` is `true` and the user has not +opted out. **This build keeps `TELEMETRY_ENABLED` false**, so no telemetry +hooks are generated and nothing is sent. See [docs/telemetry.md](docs/telemetry.md) +for the contract, opt-outs, local state, and `scripts/telemetry-report.mjs` +for aggregating exports. Gateway telemetry remains authoritative; plugin events +do not establish task success. diff --git a/README.md b/README.md index e72108b..f444dd3 100644 --- a/README.md +++ b/README.md @@ -69,6 +69,17 @@ anything is sent, created, or deleted. - Privacy: tasks run through Arcade's hosted gateway and the apps you connect — [privacy policy](https://www.arcade.dev/privacy-policy). +## Plugin usage events (off in this build) + +Claude Code and Copilot CLI include telemetry adapters, but **`TELEMETRY_ENABLED` +is `false`**: this package sends no usage events and generates no telemetry +hooks in client manifests. [docs/telemetry.md](docs/telemetry.md) describes +the event contract, what would be stored locally when enabled, opt-outs, and how +to aggregate exports with `scripts/telemetry-report.mjs`. The Arcade MCP +gateway remains the canonical source for request and tool-call telemetry. +Adapter behavior is covered by in-process tests when those files are present; +see [docs/telemetry.md](docs/telemetry.md) for what CI does and does not exercise. + ## Develop Agents editing this repo should read [AGENTS.md](AGENTS.md). diff --git a/docs/install/claude-code.md b/docs/install/claude-code.md index b8b4bfa..e0678b5 100644 --- a/docs/install/claude-code.md +++ b/docs/install/claude-code.md @@ -37,6 +37,19 @@ If your host has other Arcade MCP connectors too, Claude may pick the wrong one connected and prefer disabling other Arcade connectors while testing this plugin. +## Telemetry + +Telemetry is **off** in this build (`TELEMETRY_ENABLED` is `false`): no usage +events are sent and generated manifests include no telemetry hooks. When +telemetry is enabled, the adapter in `hooks/telemetry-adapters/claude-code.mjs` +classifies prompts, records `PreToolUse` on the plugin and claude.ai Arcade +prefixes, and stores scope under `CLAUDE_PLUGIN_DATA`. See +[telemetry.md](../telemetry.md) for opt-outs (`ARCADE_PLUGIN_TELEMETRY=0`, +`DISABLE_TELEMETRY`, `CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC`) and limits. + +**Repo coverage:** in-process hook tests and `claude plugin validate --strict` +(**2.1.258**). Not live sessions, IDE extensions, desktop Code tab, or Cowork. + ## First steps - "What's on my calendar tomorrow?" diff --git a/docs/install/copilot.md b/docs/install/copilot.md index 542ae24..6ad5682 100644 --- a/docs/install/copilot.md +++ b/docs/install/copilot.md @@ -26,6 +26,19 @@ from config files, so there's no per-prompt reminder here. VS Code reads the same file but can't run Agent Plugins hook commands yet, so it relies on the skills. +## Telemetry + +Telemetry is **off** in this build: generated manifests include no telemetry +hooks and nothing is sent. When enabled, `hooks/telemetry-adapters/copilot-cli.mjs` +records MCP tools as `-`, operator stops with `subagent_session`, +and session-scoped prompt state under `COPILOT_PLUGIN_DATA` (no `turn`, no +`PreToolUse`, no built-in CLI/web events). See [telemetry.md](../telemetry.md). + +**Repo coverage:** in-process fixtures and `npm run verify:copilot` (**1.0.88**). +Not live sessions, Windows PowerShell hook commands, or VS Code agent sessions. +Shared `com.github.copilot/hooks/hooks.json` uses `runOnlyIfScriptExists` so VS +Code exits quietly without a plugin path. + ## First steps - "What's on my calendar tomorrow?" diff --git a/docs/install/vscode.md b/docs/install/vscode.md index 35e824b..b57c9ed 100644 --- a/docs/install/vscode.md +++ b/docs/install/vscode.md @@ -17,6 +17,9 @@ VS Code loads root `plugin.json` as an Agent Plugin: 2 skills, the gateway, and `arcade-operator` from `com.github.copilot/agents/`. It does not read the `.cursor-plugin/` adapter. No lifecycle hooks: VS Code doesn't yet run Agent Plugins hook commands with the plugin's path or pass their output to the model. +Copilot hook commands check that the script exists at the plugin path and exit +when the path is missing; this package has no validated VS Code telemetry flow, +and nothing is sent from VS Code. If you already installed the plugin via Copilot CLI, VS Code may auto-discover it from `~/.copilot/installed-plugins/`. Install in one place. diff --git a/docs/support-matrix.md b/docs/support-matrix.md index c64034c..7e26775 100644 --- a/docs/support-matrix.md +++ b/docs/support-matrix.md @@ -27,6 +27,19 @@ session-start text doesn't reach the main conversation. The files are `clients/cursor/hooks/hooks.json` (Cursor CLI), and `com.github.copilot/hooks/hooks.json` (Copilot CLI). +Telemetry adapters are separate from routing. **`TELEMETRY_ENABLED` is false** +in this build, so manifests include no telemetry hooks and nothing is sent. +When enabled, wiring comes from `hooks/hook-hosts.mjs`: + +| Client adapter | Telemetry events (when enabled) | +| --- | --- | +| Claude Code | `UserPromptSubmit`, `PreToolUse` on `^(?:mcp__plugin_arcade_arcade__\|mcp__claude_ai_arcade__)`, MCP `PostToolUse` / `PostToolUseFailure`, built-in `WebFetch` / `WebSearch` / listed `Bash` CLIs, Arcade operator `SubagentStop` | +| Copilot CLI | `UserPromptSubmit`, MCP `PostToolUse` / `PostToolUseFailure` (`-`), Arcade operator `SubagentStop` with `subagent_session` (no `PreToolUse`, no `turn`, no built-in tools) | + +Session start clears local prompt scope through the routing hook; it sends no +telemetry event. No other adapter in this package sends telemetry. See +[what's sent and its limits](telemetry.md). + ¹ The Cursor IDE (3.21.18) lists the commands on the plugin page but not in the `/` menu. Other plugins' commands don't appear there either. ² The IDE (3.21.18) and Cloud Agents don't run plugin hooks, so the @@ -37,10 +50,12 @@ load the always-apply rule. ⁴ Cowork runs the prompt and subagent hooks but doesn't add the session-start text, so its main conversation gets the short reminder and the skill, not the full rules. -⁵ Copilot CLI drops the output of prompt hooks from config files, so it gets -session and subagent hooks only. +⁵ Copilot CLI drops the output of **routing** prompt hooks from config files, +so the main conversation gets session and subagent hooks only. Telemetry still +hooks `UserPromptSubmit` when enabled; `reminder_sent` is always `false` there. ⁶ VS Code reads `com.github.copilot/hooks/hooks.json` but doesn't expand -`${PLUGIN_ROOT}` for Agent Plugins hooks or pass their output to the model yet. +`${PLUGIN_ROOT}` for Agent Plugins hooks or pass their output to the model yet; +`runOnlyIfScriptExists` makes those commands no-ops without a plugin path. ⁷ Blocked upstream; see [codex.md](install/codex.md). Skills are `try-arcade` and `scale-arcade`. The subagent is diff --git a/docs/telemetry.md b/docs/telemetry.md new file mode 100644 index 0000000..95240b6 --- /dev/null +++ b/docs/telemetry.md @@ -0,0 +1,387 @@ +# Plugin telemetry + +**Telemetry is off in this build.** The plugin sends no usage events, and +`npm run generate` writes no telemetry hooks into any client manifest. +`TELEMETRY_ENABLED` in `hooks/telemetry-config.mjs` stays `false` until +collection is separately approved after ingestion, opt-outs, privacy, and +distribution requirements are verified. + +The sections below describe what **would** be sent when telemetry is turned on, +how to read exported events, and how that relates to gateway telemetry and +offline routing evaluation. Nothing here implies that this package currently +transmits data. + +When enabled, Claude Code and Copilot CLI hooks would inspect prompts locally +in each session to recognize app-related work. Relevance state belongs to that +session; it does not carry between sessions. Prompts classified as unrelated +would send no event. Direct Arcade tool calls would remain observable even +without a classified prompt. Alternative MCP, CLI, and web tools would send +events only during app-related work. + +Events would carry hashed session IDs, with no ID lasting across sessions. They +would exclude prompt text, commands, app data, names, email addresses, and Arcade +account IDs. These observations do not establish task success or whether +Arcade was needed. Install and signed-in-user counts require gateway data. + +## Prompt scope + +A prompt matching the local app classifier or mentioning Arcade would open a +30-minute observation period for its session. A short explicit confirmation such +as “yes, send it” would continue that period without extending its expiry. An +unrelated substantive prompt would close it. Background task notifications would +leave the current period unchanged. Expired, absent, or invalid state would +produce no alternative-tool telemetry. + +`could_use_arcade` and `service_hints` describe keywords in the current prompt, +not the preceding task. A confirmation reply can therefore send a scoped prompt +event with `could_use_arcade: false` and no service hints. Keyword matching can +misclassify prompts; the labeled evaluation measures that limitation separately. + +Session starts send no usage event. Only the Arcade operator's stop reports +send subagent events. The routing reminder goes on every prompt except short +acknowledgements and background task results, whether or not telemetry is on. + +## Opt-outs when telemetry is on + +While `TELEMETRY_ENABLED` is `false`, these switches are inert but documented +for a future enabled build. They are implemented in `hooks/telemetry-run.mjs` +and each client's adapter in `hooks/telemetry-adapters/`. + +Set `ARCADE_PLUGIN_TELEMETRY=0` in your environment, or in Claude Code's +`settings.json`: + +```json +{ "env": { "ARCADE_PLUGIN_TELEMETRY": "0" } } +``` + +`false`, `off`, and `no` also work. Telemetry is also off when `DO_NOT_TRACK` +is set to any non-empty value other than `0`, `false`, `off`, or `no`. In +Claude Code, `DISABLE_TELEMETRY` and `CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC` +opt out when set to **any** non-empty value (including `0` and `false`), matching +Claude's own telemetry switches. + +With telemetry opted out, the client still invokes its configured Node hooks. +The telemetry hook exits before classifying the prompt or storing state, and the +routing hook still adds its reminder. An environment variable cannot remove +hooks from the manifest. + +In Copilot CLI, set `ARCADE_PLUGIN_TELEMETRY=0` in your shell before starting +`copilot`. `COPILOT_OFFLINE` uses the same `0` / `false` / `off` / `no` semantics +as `ARCADE_PLUGIN_TELEMETRY`; any other non-empty value turns telemetry off +(along with other Copilot network activity). + +For testing an enabled build, `ARCADE_PLUGIN_TELEMETRY_HOST` sends events to a +different host. + +## Where it runs + +The Claude Code and Copilot CLI adapters are the only ones wired for telemetry +in `hooks/hook-hosts.mjs` (`hooks/telemetry-adapters/claude-code.mjs` and +`hooks/telemetry-adapters/copilot-cli.mjs`). Claude maps MCP tools under +`mcp__plugin____` (this plugin's gateway) and +`mcp__claude_ai_arcade__` (the claude.ai Arcade connection). Attempt events +(`PreToolUse`) are emitted only for those two prefixes. Copilot maps MCP tools +as `-` (for example `arcade-Arcade_SelectTools` when the MCP +server name matches `mcp.json`). Copilot sends no `turn`, no `PreToolUse` +attempt events, and no built-in CLI or web tool events. + +Claude ties alternative-tool observations to the hashed `turn` for the +current `prompt_id` (`requiresTurn: true`). Copilot keeps prompt scope per +`session` without a prompt ID (`requiresTurn: false`). + +Local state uses `CLAUDE_PLUGIN_DATA` (Claude Code) and `COPILOT_PLUGIN_DATA` +(Copilot CLI) for `arcade-used` and `prompt-scope/` files. + +Copilot's manifest is shared with VS Code. Hook commands use +`runOnlyIfScriptExists`: each command checks that its script exists at +`${PLUGIN_ROOT}` and exits quietly when the path is missing, so VS Code does +not run telemetry today. Cursor is not wired up; its hook input can include the +user's email. claude.ai, ChatGPT, Codex, and OpenCode don't run telemetry hooks +from this package. + +### What CI exercises (when adapters are present) + +| Adapter | Tested in this repo | Not covered here | +| --- | --- | --- | +| Claude Code | In-process hook captures, contract tests, `scripts/telemetry-report.mjs` cross-check, `claude plugin validate --strict` on **2.1.258** | Live user sessions, IDE extensions, desktop Code tab, Cowork delivery and opt-out | +| Copilot CLI | In-process captures, fixture JSON, report cross-check, `npm run verify:copilot` on **1.0.88** | Live sessions, Windows PowerShell hook commands, VS Code agent sessions | + +Those checks do not prove end-user transmission while `TELEMETRY_ENABLED` is +`false`; they validate the adapters and reporting math for an enabled build. + +## What is stored on your machine + +When telemetry is enabled, each client's plugin data folder contains its own +`arcade-used` flag, readable only by you. It holds `true` after an Arcade call +succeeds in that client plugin installation and supplies `arcade_used_before`. + +Prompt relevance state lives in `prompt-scope/.json` in +the same folder. It contains a relevance boolean, expiry timestamp, and optional +hashed prompt ID that ties tool calls to their prompt. It contains no prompt text, +commands, tool arguments, or service content. Expired state cannot authorize +observation. The next prompt-state write removes expired or malformed entries; +at most 256 session state files are retained. Session start clears that +session's state, except after Claude Code compacts the conversation. Claude +alternative-tool observations require a matching hashed prompt ID; an absent +prompt ID cannot authorize those observations. + +- Claude Code: `~/.claude/plugins/data//` +- Copilot CLI: `~/.copilot/plugin-data/<…>/` + +If the client supplies no plugin data folder, the plugin sends nothing. + +## What is sent + + +Every event has these properties: + +| Property | Value | +| --- | --- | +| `distinct_id` | the same value as `session` | +| `session` | `sha256(session_id)`, first 16 hex characters, where `session_id` is the client's random ID for the session | +| `turn` | `sha256(session_id + ":" + prompt_id)`, first 16 hex characters; Claude Code only, because Copilot CLI has no prompt ID | +| `arcade_used_before` | whether an Arcade tool call had succeeded for this client plugin installation before this event (from the `arcade-used` file) | +| `host` | `claude-code` \| `copilot-cli` | +| `telemetry_version` | `2`, the scoped event contract; earlier events have no version | +| `plugin_version` | from `VERSION` | +| `os` | `darwin` \| `linux` \| `win32` \| `other` | +| `$process_person_profile` | `false` | +| `$geoip_disable` | `true` | +| `$ip` | `0.0.0.0`, so PostHog stores this instead of your real IP address | + +Events and their extra properties: + +| Event | When | Extra properties | +| --- | --- | --- | +| `Plugin prompt submitted` | UserPromptSubmit, only for locally classified app work and short confirmations of that work; background task results are excluded. In Copilot CLI a relevant subagent prompt also sends it | `could_use_arcade`: boolean, a local keyword guess (see below). `service_hints`: service categories the prompt mentions. `reminder_sent`: boolean, whether the routing reminder was added (always `false` in Copilot CLI). | +| `Plugin tool attempted` | PreToolUse, in Claude Code, before an MCP call through this plugin's Arcade gateway or the claude.ai Arcade connection; an attempt does not show whether the tool finished | `server`: `arcade` (this plugin's gateway) \| `other_arcade` (the claude.ai Arcade connection). `tool`: only for `arcade` and `other_arcade`: an exact gateway tool name, `app_tool` for a recognized service category, or `other`; app tool names are never sent. `service`: the service category, when the tool, the app tool passed to `Arcade_UseTool`, or the server name matches one. | +| `Plugin tool called` | PostToolUse, on Arcade tools, or alternative MCP tools while the current turn concerns app work | `server`: `arcade` (this plugin's gateway) \| `other_arcade` (another connection exposing Arcade's gateway tools) \| `other`. `tool`: only for `arcade` and `other_arcade`: an exact gateway tool name, `app_tool` for a recognized service category, or `other`; app tool names are never sent. `service`: the service category, when the tool, the app tool passed to `Arcade_UseTool`, or the server name matches one. `auth_needed`: only for `System_ManageAuthorization`: whether its answer says a service still needs sign-in. | +| `Plugin tool failed` | PostToolUseFailure, on Arcade tools, or alternative MCP tools while the current turn concerns app work | `server`: `arcade` (this plugin's gateway) \| `other_arcade` (another connection exposing Arcade's gateway tools) \| `other`. `tool`: only for `arcade` and `other_arcade`: an exact gateway tool name, `app_tool` for a recognized service category, or `other`; app tool names are never sent. `service`: the service category, when the tool, the app tool passed to `Arcade_UseTool`, or the server name matches one. `failure_kind`: picked on your machine from the error message; the message is not sent: `auth_required` \| `session_expired` \| `unreachable` \| `timeout` \| `http_error` \| `interrupted` \| `tool_error`. | +| `Plugin built-in tool called` | PostToolUse, Claude Code only, on `WebFetch` and `WebSearch`, and on `Bash` commands that run `gh` \| `glab` \| `curl` \| `wget` \| `http` \| `osascript`; only while the current turn concerns app work | `tool`: `Bash` \| `WebFetch` \| `WebSearch`. `cli`: only for `Bash`: the program the command runs, `gh` \| `glab` \| `curl` \| `wget` \| `http` \| `osascript`. `service`: the program's service category, only for `gh` \| `glab`: `code_hosting`. | +| `Plugin built-in tool failed` | PostToolUseFailure, Claude Code only, on `WebFetch` and `WebSearch`, and on `Bash` commands that run `gh` \| `glab` \| `curl` \| `wget` \| `http` \| `osascript`; only while the current turn concerns app work | `tool`: `Bash` \| `WebFetch` \| `WebSearch`. `cli`: only for `Bash`: the program the command runs, `gh` \| `glab` \| `curl` \| `wget` \| `http` \| `osascript`. `service`: the program's service category, only for `gh` \| `glab`: `code_hosting`. | +| `Plugin subagent stopped` | SubagentStop, only for arcade-operator | `agent`: `arcade-operator`. `status`: only for `arcade-operator`: the status line of its final report, `completed` \| `needs_auth` \| `needs_confirmation` \| `needs_clarification` \| `failed` \| `unknown`. `subagent_session`: `sha256(agent_id)`, first 16 hex characters. In Copilot CLI the subagent's own events carry this as `session`. | + +Service categories: `email`, `calendar`, `chat`, `issues`, `docs`, `meetings`, `crm`, `code_hosting`, `analytics`, `storage`. + + +`could_use_arcade` is a local keyword guess (in +`hooks/telemetry-classify.mjs`) at whether the prompt is a task Arcade could +do: email, calendar, chat, and the other categories above. + +`failure_kind`, `auth_needed`, and `cli` are values from fixed lists, picked +on your machine. The error text, the tool output, and the command they are +picked from are never sent. During an app-related observation period, a Bash +command sends an event only when one of its +commands starts with a listed program; commands that call a program by its +full path, such as `/opt/homebrew/bin/gh`, are not counted. A command that runs +two listed programs, such as `gh … && curl …`, sends one event for each, so +count CLI use by turn, not by event. Claude Code starts these hooks only for +commands that match (the hook `if` field, Claude Code 2.1.246 and later). + +## Never sent + +- prompt text, or any text you or the model wrote +- tool inputs, tool outputs, or error messages +- commands, their arguments, URLs, and search queries +- file paths, the working directory, or transcript paths +- names of MCP servers other than Arcade's, or their tool names +- names of subagents other than Arcade's +- your email, username, hostname, repository, or Arcade account + +The plugin drops any property not listed on this page before sending. + +PostHog would see the IP address the request comes from, like any web request, +but would not store it: every event sets `$ip` to `0.0.0.0`, and location +lookup is off. + +## Gateway telemetry + +The Arcade MCP gateway (`https://api.arcade.dev/mcp/arcade`) remains the +canonical source for request, authentication, discovery, tool-call, and +completion telemetry. Plugin events are supplemental observations from client +hooks. They do not replace gateway records and cannot prove task success on +their own. + +## Reading the numbers + +Export PostHog rows in the shape `{ event, distinct_id, properties, timestamp? }` +and aggregate them locally: + +```bash +node scripts/telemetry-report.mjs path/to/export.jsonl +``` + +The script prints JSON with counts, denominators, and a `limits` list. Before +validating, it ignores export envelope fields outside +`{ event, distinct_id, properties }` (such as `timestamp` and `uuid`) and +PostHog-added properties whose names start with `$` but are not on the contract +for that event. Every other property must match the contract; leaked hook +fields such as `prompt` or `cwd` make a row invalid. It counts invalid rows and +legacy rows (no `telemetry_version`) separately, and excludes both from grouped +counts. Groups are split by `host`, `plugin_version`, and +`telemetry_version`. Claude Code and Copilot CLI are never combined into one +denominator. If the file is not valid JSON or JSON Lines, the script exits +nonzero with a short message naming the failing line, prints no report, and +does not echo the file's contents. + +These events measure what plugin hooks observed, not whether Arcade was needed +or whether the user's task succeeded. `could_use_arcade` and `service_hints` +come from a local keyword classifier. A flagged prompt is a candidate for +review, not a confirmed opportunity. Prompt events are selected by relevance, +so they cannot measure the share of all prompts needing Arcade; unrelated +prompts are deliberately absent. Use labeled task evaluations to score routing, and compare the +plugin with a baseline or variant before attributing a change to it. +Neither host emits a task ID. + +Keep the observation stages separate: + +| Stage | Evidence | What it establishes | +| --- | --- | --- | +| Tool attempt (Claude Code) | `Plugin tool attempted` on PreToolUse | The model invoked a tool. An attempt alone has no observed outcome. | +| Gateway discovery or selection | `Plugin tool called` or `Plugin tool failed` for `Arcade_ListApps` or `Arcade_SelectTools` | The discovery or selection call completed or failed; a successful selection is not an app action. | +| Authorization check | `System_ManageAuthorization`, reported separately | `auth_needed: true` means its answer said sign-in was needed. A check alone says nothing about app use; `false` does not prove every app is connected. | +| App action | `Arcade_UseTool` or `app_tool`, split by `Plugin tool called` and `Plugin tool failed` | The hook observed a tool completion or failure. Tools with an unrecognized service category are `other` and cannot be assigned to this stage. Neither outcome proves the user's task succeeded. | +| No Arcade call observed | A prompt with no Arcade tool event in the observable group | The hooks saw no call. This is not a routing miss without an independently labeled need and complete tool visibility. | + +Count each unit once at each stage, and show the denominator, host, date range, +plugin version, telemetry version, and observation coverage beside every rate. +`telemetry_version: 2` identifies scoped events; a missing value identifies the +legacy contract. Keep those populations separate, even at the same plugin +version. Keep the number of +observed prompt units and sessions visible even when a chart has no app actions. +Do not extrapolate rates from a test sample or telemetry-enabled sessions to all users. +Do not mix Claude turns with Copilot session counts in one rate. + +Report field meanings (from `scripts/telemetry-report.mjs`): + +- `observed_relevant_turns` / `observed_relevant_sessions`: distinct Claude + `turn` values or Copilot `session` values with a `Plugin prompt submitted` + event, after Copilot operator-linked child sessions are excluded from the + session denominator. +- `tool_only_turns`: Claude turns with tool events but no prompt event in that + turn (reported outside the turn denominator). +- `attempt_observed_outcome_unknown`: Claude turns with a `Plugin tool + attempted` on a `server` category (`arcade` or `other_arcade`) but no `Plugin + tool called` or `Plugin tool failed` on that same category in the same turn. + An outcome on one category never settles an attempt on the other. + `other_arcade` pools every other Arcade connection, and one outcome settles + every attempt on its category in the turn, so the check cannot pair a result + with a particular connection or call. +- `no_call_observed`: relevant turns or sessions with no Arcade MCP tool event + (`Plugin tool attempted`, `Plugin tool called`, or `Plugin tool failed` on + `server: arcade` or `other_arcade`). +- `parent_attribution_unknown_sessions`: Copilot sessions with a prompt event + that are not operator-linked children and have no operator stop on that + session to confirm parent/child linkage. +- Stage counts use the denominators above; they are not paired attempt/result + metrics and do not imply recall, precision, or task success. + +### Claude Code turns + +The denominator is distinct `turn` values with a `Plugin prompt submitted` +event. This is a count of observed relevant turns, not all prompts or tasks. +Exclude tool-only turns: Claude Code filters background task results +from prompt events. Group tool events with the same `turn`, including an +arcade-operator's events, and report sessions and turns separately. A short +follow-up such as “yes, send it” is a separate observed turn while scope is +active, even when its keyword flag is false. Turn counts do not describe whole +tasks. Report tool-only turns separately from this denominator. + +`server: other_arcade` means the hook observed another connection's Arcade +gateway tool. Results recognize exact gateway tool names on arbitrary MCP +server aliases; attempts recognize only the two configured lowercase Arcade +prefixes. UUID or capitalized aliases can therefore have results without an +attempt event. Direct app tools on an unidentified connection may be classified +as `other`. Gateway names are a recognition heuristic, not verified server +provenance. + +Events contain no tool-call ID. Multiple calls in the same turn cannot be +paired individually, even when their tool categories match. Count observed +events or turn-level stages; do not present a per-attempt completion rate. + +A tool invocation is not always followed by a PostToolUse or +PostToolUseFailure event. A `Plugin tool attempted` event can show the +invocation, but only `Plugin tool called` or `Plugin tool failed` records its +outcome. Count an attempt without either outcome as **attempt +observed, outcome unknown**, not app action success. It does not set +`arcade_used_before`. Label flagged turns with no Arcade tool event **no call +observed**, not **missed**. A direct app tool on another gateway may appear +as `server: other`, without an identifiable Arcade call. + +A built-in CLI or web event is a tool observation, not evidence of fallback. +Only call it a possible fallback after linking it to a labeled Arcade-eligible +task and establishing that it served the same request. +`arcade_used_before: false` says no Arcade call has previously succeeded for +that client plugin installation; it does not describe another client on the +same machine or prove that the gateway or a particular app was unconnected. + +### Copilot CLI sessions + +Copilot CLI supplies no `prompt_id` or `turn`. Count distinct sessions with a +scoped `Plugin prompt submitted` event as **observed relevant sessions**. Report +multiple-prompt sessions separately. This is not a count of all sessions, user +tasks, or turns. Session starts are not transmitted and cannot define roots. + +An Arcade operator's stop report carries `subagent_session`, equal to that +operator's own event `session`. Exclude known linked operator prompts from the +parent-session denominator and include their tool events with the parent. +Other subagents do not send stop reports, so their prompt sessions cannot be +reliably distinguished from roots. Report that parent attribution is unknown +rather than labeling every unlinked session as a root. + +Detached event delivery can change arrival order. A missing operator stop or a +rolling-window boundary can also remove a parent link. Do not reconstruct +per-prompt outcomes from timestamps alone or link resumed tasks across sessions. + +Copilot names MCP tools `-` with no plugin prefix, so an MCP +server named `arcade` counts as `server: arcade` even if this plugin did not +install it. Copilot sends no built-in CLI or web events; its `arcade-operator` +can report `status: unknown`. Do not infer a fallback or task result from +either absence. + +### Failure and outcome limits + +`failure_kind` is a local classifier of error text, not a server error code. +An app sign-in failure can be `tool_error` when its wording does not match the +classifier; bad input, rate limits, and upstream errors can also get that +value. + +`auth_needed: true` on an authorization check reports a service still needing +sign-in; operator `status: needs_auth` is the model's report, not a verified +server status. `auth_required`, `session_expired`, `timeout`, and `unreachable` +match known client or MCP SDK error wording; other wording can classify +differently. Copilot does not report interrupted calls, so its `interrupted` +category is empty. + +Operator status describes its report, not the parent task's result. The hooks +do not send final answers or satisfaction signals. Task success, true routing +misses, and improvement from this plugin require labeled evaluations outside +this telemetry. + +## Routing evaluation + +Evaluate classification and client routing on controlled, labeled cases separately +from production usage events. Include implicit app requests, contextual follow-ups, +and local coding tasks that name an app. Fixture accuracy does not establish +real-user routing quality. + +The existing [Tool Recommendation Cost Eval](https://github.com/ArcadeAI/tool-recommendation-cost-eval) +provides an offline cross-client harness and published results. Record the plugin +SHA and client configuration when evaluating the plugin; Tool Recommendation +on/off results do not establish the plugin's effect on routing or cost. Plugin +telemetry and that harness are complementary; neither replaces the other. + +## Before turning it on + +Separate approval is required before setting `TELEMETRY_ENABLED` to `true` and +shipping telemetry hooks. Confirm, outside this repository: + +- user-facing disclosure and privacy policy coverage for plugin usage events +- analytics retention and processor handling for the PostHog project +- verified ingestion per client and `telemetry_version` for the build under review +- opt-out switches verified on each supported client surface +- applicable store or distribution decisions for builds that transmit data + +These are release acceptance requirements; the current code and CI do not +enforce them. A repository install can consume a branch before a GitHub release +is tagged, so a later release PR is not a transmission gate. diff --git a/scripts/generate-manifests.mjs b/scripts/generate-manifests.mjs index bb1a89a..d047de2 100644 --- a/scripts/generate-manifests.mjs +++ b/scripts/generate-manifests.mjs @@ -11,6 +11,7 @@ import { SKILL_RULES, } from "../hooks/routing-guidance.mjs"; import { HOOK_TIMEOUT_SEC, HOOKS, HOSTS } from "../hooks/hook-hosts.mjs"; +import { fillTelemetryTables, TELEMETRY_DOC } from "./telemetry-docs.mjs"; const ROOT = resolve(dirname(fileURLToPath(import.meta.url)), ".."); @@ -26,6 +27,9 @@ export const FILES_WITH_GENERATED_RULES = { "skills/try-arcade/SKILL.md": SKILL_RULES, }; +/** Hand-written files whose event tables are generated from hooks/telemetry-contract.mjs. */ +export const FILES_WITH_GENERATED_TABLES = [TELEMETRY_DOC]; + /** Generated copy → the file it is copied from. */ export const COPIED_FILES = { // Each skill folder has to work on its own. @@ -63,6 +67,7 @@ export const FILE_SOURCES = { "com.github.copilot/agents/arcade-operator.agent.md": [OPERATOR, "hooks/routing-guidance.mjs"], [OPERATOR]: ["hooks/routing-guidance.mjs"], "skills/try-arcade/SKILL.md": ["hooks/routing-guidance.mjs"], + [TELEMETRY_DOC]: ["hooks/telemetry-contract.mjs"], }; /** Formats an array of source paths as a human-readable list. */ @@ -89,6 +94,9 @@ const outOfDateError = (path) => { if (path in FILES_WITH_GENERATED_RULES) { return `${path}: the rules block is out of date. Run npm run generate. If you edited the block by hand, make the change in hooks/routing-guidance.mjs instead.`; } + if (FILES_WITH_GENERATED_TABLES.includes(path)) { + return `${path}: the telemetry tables are out of date. Run npm run generate. If you edited the tables by hand, make the change in hooks/telemetry-contract.mjs instead.`; + } const sources = FILE_SOURCES[path]; const joined = joinSources(sources); if (path in COPIED_FILES) { @@ -268,6 +276,10 @@ const buildFiles = (root) => { files.set(path, fillRulesBlock(readText(root, path), rules, path)); } + for (const path of FILES_WITH_GENERATED_TABLES) { + files.set(path, fillTelemetryTables(readText(root, path))); + } + // Every output must have a source entry so error messages can name it. for (const path of files.keys()) { requireSources(path, FILE_SOURCES); diff --git a/scripts/telemetry-docs.mjs b/scripts/telemetry-docs.mjs new file mode 100644 index 0000000..dd88f28 --- /dev/null +++ b/scripts/telemetry-docs.mjs @@ -0,0 +1,59 @@ +// @ts-check +/** Writes the event tables in docs/telemetry.md from hooks/telemetry-contract.mjs. */ + +import { COMMON_PROPERTIES, EVENTS, SERVICE_CATEGORIES } from "../hooks/telemetry-contract.mjs"; + +export const TELEMETRY_DOC = "docs/telemetry.md"; +export const TELEMETRY_BLOCK_BEGIN = + ""; +export const TELEMETRY_BLOCK_END = ""; + +const code = (/** @type {string} */ text) => `\`${text}\``; + +export const buildTelemetryTables = () => { + const common = [ + "| Property | Value |", + "| --- | --- |", + "| `distinct_id` | the same value as `session` |", + ...Object.entries(COMMON_PROPERTIES).map(([key, { doc }]) => `| ${code(key)} | ${doc} |`), + ]; + const events = [ + "| Event | When | Extra properties |", + "| --- | --- | --- |", + ...Object.entries(EVENTS).map(([name, spec]) => { + const when = spec.when ? `${spec.hook}, ${spec.when}` : spec.hook; + const extras = Object.entries(spec.properties) + .map(([key, { doc }]) => `${code(key)}: ${doc}.`) + .join(" "); + return `| ${code(name)} | ${when} | ${extras} |`; + }), + ]; + return [ + "Every event has these properties:", + "", + ...common, + "", + "Events and their extra properties:", + "", + ...events, + "", + `Service categories: ${SERVICE_CATEGORIES.map(code).join(", ")}.`, + ].join("\n"); +}; + +/** + * Replaces the generated block in docs/telemetry.md. + * @param {string} text + */ +export const fillTelemetryTables = (text) => { + const begin = text.indexOf(TELEMETRY_BLOCK_BEGIN); + const end = text.indexOf(TELEMETRY_BLOCK_END); + if (begin === -1 || end === -1 || end < begin) { + throw new Error(`${TELEMETRY_DOC} is missing the generated telemetry block markers`); + } + return ( + text.slice(0, begin + TELEMETRY_BLOCK_BEGIN.length) + + `\n${buildTelemetryTables()}\n` + + text.slice(end) + ); +}; diff --git a/scripts/telemetry-report.mjs b/scripts/telemetry-report.mjs new file mode 100644 index 0000000..93a676a --- /dev/null +++ b/scripts/telemetry-report.mjs @@ -0,0 +1,449 @@ +// @ts-check +/** Aggregate exported PostHog plugin telemetry rows into count-only reports. */ + +import { readFileSync } from "node:fs"; +import { resolve } from "node:path"; +import { fileURLToPath } from "node:url"; +import Ajv2020Module from "ajv/dist/2020.js"; +import { allowedProperties, EVENTS, eventSchema } from "../hooks/telemetry-contract.mjs"; + +const Ajv2020 = /** @type {new (options?: object) => import("ajv").default} */ ( + /** @type {any} */ (Ajv2020Module).default ?? Ajv2020Module +); + +const KNOWN_EVENTS = new Set(Object.keys(EVENTS)); + +const DISCOVERY_TOOLS = new Set(["Arcade_ListApps", "Arcade_SelectTools"]); +const MCP_EVENTS = new Set(["Plugin tool attempted", "Plugin tool called", "Plugin tool failed"]); +const OUTCOME_EVENTS = new Set(["Plugin tool called", "Plugin tool failed"]); +const APP_ACTION_TOOLS = new Set(["Arcade_UseTool", "app_tool"]); + +const validateRow = new Ajv2020({ allErrors: true }).compile(eventSchema()); + +/** @param {unknown} row */ +const isValidRow = (row) => validateRow(row) === true; + +/** + * Drops export envelope fields and PostHog `$` noise; other properties must match the contract. + * @param {unknown} raw + * @returns {{ event: string, distinct_id: string, properties: Record } | null} + */ +export const normalizeExportRow = (raw) => { + if (!raw || typeof raw !== "object") return null; + const record = /** @type {Record} */ (raw); + const { event, distinct_id, properties } = record; + if (typeof event !== "string" || typeof distinct_id !== "string") return null; + if (!properties || typeof properties !== "object" || Array.isArray(properties)) return null; + if (!KNOWN_EVENTS.has(event)) return null; + const props = /** @type {Record} */ (properties); + const allowed = new Set(allowedProperties(event)); + /** @type {Record} */ + const trimmed = { ...props }; + for (const key of Object.keys(trimmed)) { + if (key.startsWith("$") && !allowed.has(key)) delete trimmed[key]; + } + return { event, distinct_id, properties: trimmed }; +}; + +/** @param {unknown} row */ +const isLegacyRow = (row) => { + if (!row || typeof row !== "object") return false; + const record = /** @type {{ event?: string, properties?: Record }} */ (row); + return ( + typeof record.event === "string" && + KNOWN_EVENTS.has(record.event) && + record.properties && + typeof record.properties === "object" && + record.properties.telemetry_version === undefined + ); +}; + +/** + * @param {Record} properties + * @returns {{ host: string, plugin_version: string, telemetry_version: number | "legacy" }} + */ +const groupKey = (properties) => ({ + host: String(properties.host), + plugin_version: String(properties.plugin_version), + telemetry_version: + properties.telemetry_version === undefined ? "legacy" : Number(properties.telemetry_version), +}); + +/** @param {string} server */ +const isArcadeServer = (server) => server === "arcade" || server === "other_arcade"; + +/** + * @param {{ event: string, properties: Record }} row + */ +const isArcadeMcpRow = (row) => + MCP_EVENTS.has(row.event) && isArcadeServer(String(row.properties.server ?? "")); + +/** + * @param {{ event: string, properties: Record }} row + */ +const toolName = (row) => (typeof row.properties.tool === "string" ? row.properties.tool : ""); + +/** + * @param {{ event: string, properties: Record }} row + */ +const isDiscoveryRow = (row) => + OUTCOME_EVENTS.has(row.event) && DISCOVERY_TOOLS.has(toolName(row)); + +/** + * @param {{ event: string, properties: Record }} row + */ +const isAuthRow = (row) => + OUTCOME_EVENTS.has(row.event) && toolName(row) === "System_ManageAuthorization"; + +/** + * @param {{ event: string, properties: Record }} row + */ +const isAppActionCalledRow = (row) => + row.event === "Plugin tool called" && APP_ACTION_TOOLS.has(toolName(row)); + +/** + * @param {{ event: string, properties: Record }} row + */ +const isAppActionFailedRow = (row) => + row.event === "Plugin tool failed" && APP_ACTION_TOOLS.has(toolName(row)); + +/** + * @param {Set} turnIds + * @param {Map }[]>} byTurn + */ +const countTurnStages = (turnIds, byTurn) => { + let gateway = 0; + let authNeededTrue = 0; + let authNeededFalse = 0; + let appCalled = 0; + let appFailed = 0; + let attemptOutcomeUnknown = 0; + let noCallObserved = 0; + + for (const turn of turnIds) { + const rows = byTurn.get(turn) ?? []; + const arcadeRows = rows.filter(isArcadeMcpRow); + if (gatewayRow(rows)) gateway += 1; + if (authRow(rows, true)) authNeededTrue += 1; + if (authRow(rows, false)) authNeededFalse += 1; + if (rows.some(isAppActionCalledRow)) appCalled += 1; + if (rows.some(isAppActionFailedRow)) appFailed += 1; + if (attemptWithoutOutcome(arcadeRows)) attemptOutcomeUnknown += 1; + if (arcadeRows.length === 0) noCallObserved += 1; + } + + return { + gateway_discovery_or_selection: { count: gateway, denominator: "observed_relevant_turns" }, + authorization_check_auth_needed_true: { count: authNeededTrue, denominator: "observed_relevant_turns" }, + authorization_check_auth_needed_false: { count: authNeededFalse, denominator: "observed_relevant_turns" }, + app_action_called: { count: appCalled, denominator: "observed_relevant_turns" }, + app_action_failed: { count: appFailed, denominator: "observed_relevant_turns" }, + attempt_observed_outcome_unknown: { count: attemptOutcomeUnknown, denominator: "observed_relevant_turns" }, + no_call_observed: { count: noCallObserved, denominator: "observed_relevant_turns" }, + }; +}; + +/** @param {{ event: string, properties: Record }[]} rows */ +const gatewayRow = (rows) => rows.some(isDiscoveryRow); + +/** + * @param {{ event: string, properties: Record }[]} rows + * @param {boolean} needed + */ +const authRow = (rows, needed) => + rows.some( + (row) => isAuthRow(row) && row.properties.auth_needed === needed, + ); + +/** + * An outcome only settles attempts on its own `server` category, so an `other_arcade` + * result cannot hide an `arcade` attempt. Within a category the check is turn-level. + * @param {{ event: string, properties: Record }[]} arcadeRows + */ +const attemptWithoutOutcome = (arcadeRows) => + ["arcade", "other_arcade"].some((server) => { + const onServer = arcadeRows.filter((row) => row.properties.server === server); + const hasAttempt = onServer.some((row) => row.event === "Plugin tool attempted"); + const hasOutcome = onServer.some((row) => OUTCOME_EVENTS.has(row.event)); + return hasAttempt && !hasOutcome; + }); + +/** + * @param {{ event: string, properties: Record }[]} rows + * @param {string} eventName + */ +const builtinCounts = (rows, eventName) => { + /** @type {Record>} */ + const out = {}; + for (const row of rows) { + if (row.event !== eventName) continue; + const tool = String(row.properties.tool ?? "unknown"); + if (!out[tool]) out[tool] = {}; + const cli = row.properties.cli; + const key = typeof cli === "string" ? cli : "_"; + out[tool][key] = (out[tool][key] ?? 0) + 1; + } + return out; +}; + +/** + * @param {{ event: string, properties: Record }[]} rows + */ +const failureKindCounts = (rows) => { + /** @type {Record} */ + const out = {}; + for (const row of rows) { + if (row.event !== "Plugin tool failed") continue; + const kind = String(row.properties.failure_kind ?? "unknown"); + out[kind] = (out[kind] ?? 0) + 1; + } + return out; +}; + +/** + * @param {{ event: string, properties: Record }[]} rows + */ +const operatorStatusCounts = (rows) => { + /** @type {Record} */ + const out = {}; + for (const row of rows) { + if (row.event !== "Plugin subagent stopped") continue; + const status = String(row.properties.status ?? "unknown"); + out[status] = (out[status] ?? 0) + 1; + } + return out; +}; + +/** + * @param {{ event: string, properties: Record }[]} rows + */ +const buildClaudeGroup = (rows) => { + const prompts = rows.filter((row) => row.event === "Plugin prompt submitted"); + const relevantTurns = new Set( + prompts.map((row) => String(row.properties.turn ?? "")).filter(Boolean), + ); + + /** @type {Map }[]>} */ + const byTurn = new Map(); + for (const row of rows) { + const turn = row.properties.turn; + if (typeof turn !== "string" || !turn) continue; + const list = byTurn.get(turn) ?? []; + list.push(row); + byTurn.set(turn, list); + } + + const toolOnlyTurns = [...byTurn.keys()].filter( + (turn) => !relevantTurns.has(turn) && (byTurn.get(turn) ?? []).some((row) => row.event !== "Plugin prompt submitted"), + ).length; + + return { + unit: "turn", + denominators: { observed_relevant_turns: relevantTurns.size }, + tool_only_turns: toolOnlyTurns, + stages: countTurnStages(relevantTurns, byTurn), + builtin_tools_called: builtinCounts(rows, "Plugin built-in tool called"), + builtin_tools_failed: builtinCounts(rows, "Plugin built-in tool failed"), + failure_kinds: failureKindCounts(rows), + operator_stop_status: operatorStatusCounts(rows), + }; +}; + +/** + * @param {{ event: string, properties: Record }[]} rows + */ +const buildCopilotGroup = (rows) => { + const prompts = rows.filter((row) => row.event === "Plugin prompt submitted"); + const promptSessions = new Set(prompts.map((row) => String(row.properties.session))); + + /** @type {Map} */ + const childToParent = new Map(); + const linkedChildren = new Set(); + for (const row of rows) { + if (row.event !== "Plugin subagent stopped") continue; + const parent = String(row.properties.session); + const child = row.properties.subagent_session; + if (typeof child === "string" && child) { + childToParent.set(child, parent); + linkedChildren.add(child); + } + } + + const denominatorSessions = new Set( + [...promptSessions].filter((session) => !linkedChildren.has(session)), + ); + + const multiPromptSessions = [...denominatorSessions].filter((session) => { + const count = prompts.filter((row) => String(row.properties.session) === session).length; + return count > 1; + }).length; + + const parentSessionsWithStop = new Set( + rows + .filter((row) => row.event === "Plugin subagent stopped" && row.properties.subagent_session) + .map((row) => String(row.properties.session)), + ); + const parentAttributionUnknown = [...denominatorSessions].filter( + (session) => !parentSessionsWithStop.has(session), + ).length; + + /** @type {Map }[]>} */ + const bySession = new Map(); + /** @param {{ event: string, properties: Record }} row */ + const attributeSession = (row) => { + const session = String(row.properties.session); + if (linkedChildren.has(session)) { + const parent = childToParent.get(session); + return parent ?? session; + } + return session; + }; + + for (const row of rows) { + const session = attributeSession(row); + const list = bySession.get(session) ?? []; + list.push(row); + bySession.set(session, list); + } + + let gateway = 0; + let authNeededTrue = 0; + let authNeededFalse = 0; + let appCalled = 0; + let appFailed = 0; + let noCallObserved = 0; + + for (const session of denominatorSessions) { + const sessionRows = bySession.get(session) ?? []; + const arcadeRows = sessionRows.filter(isArcadeMcpRow); + if (gatewayRow(sessionRows)) gateway += 1; + if (authRow(sessionRows, true)) authNeededTrue += 1; + if (authRow(sessionRows, false)) authNeededFalse += 1; + if (sessionRows.some(isAppActionCalledRow)) appCalled += 1; + if (sessionRows.some(isAppActionFailedRow)) appFailed += 1; + if (arcadeRows.length === 0) noCallObserved += 1; + } + + return { + unit: "session", + denominators: { observed_relevant_sessions: denominatorSessions.size }, + operator_linked_sessions_excluded: linkedChildren.size, + multi_prompt_sessions: multiPromptSessions, + parent_attribution_unknown_sessions: parentAttributionUnknown, + stages: { + gateway_discovery_or_selection: { count: gateway, denominator: "observed_relevant_sessions" }, + authorization_check_auth_needed_true: { count: authNeededTrue, denominator: "observed_relevant_sessions" }, + authorization_check_auth_needed_false: { count: authNeededFalse, denominator: "observed_relevant_sessions" }, + app_action_called: { count: appCalled, denominator: "observed_relevant_sessions" }, + app_action_failed: { count: appFailed, denominator: "observed_relevant_sessions" }, + no_call_observed: { count: noCallObserved, denominator: "observed_relevant_sessions" }, + }, + failure_kinds: failureKindCounts(rows), + operator_stop_status: operatorStatusCounts(rows), + }; +}; + +const REPORT_LIMITS = [ + "Counts describe plugin hook observations only, not task outcomes or whether Arcade was needed.", + "Prompt events are limited to locally classified app-related work; unrelated prompts are absent.", + "Claude Code turn counts and Copilot CLI session counts must not be combined into one rate.", + "Copilot CLI does not emit tool-attempt events; attempt stages apply only to Claude Code.", + "Multiple tool calls in one turn or session cannot be paired without a tool-call ID.", + "Attempt outcomes are checked per server category; other_arcade pools every other Arcade connection, so an outcome there cannot be tied to the connection or call that was attempted.", + "The Arcade MCP gateway remains the canonical source for request, auth, discovery, and tool-call telemetry.", +]; + +/** + * @param {unknown[]} events + */ +export const buildReport = (events) => { + let invalid = 0; + let legacy = 0; + /** @type {Map }[]>} */ + const groups = new Map(); + + for (const raw of events) { + const row = normalizeExportRow(raw); + if (row === null) { + invalid += 1; + continue; + } + if (isLegacyRow(row)) { + legacy += 1; + continue; + } + if (!isValidRow(row)) { + invalid += 1; + continue; + } + const key = JSON.stringify(groupKey(row.properties)); + const list = groups.get(key) ?? []; + list.push(row); + groups.set(key, list); + } + + /** @type {object[]} */ + const reportGroups = []; + for (const [key, rows] of [...groups.entries()].sort(([a], [b]) => a.localeCompare(b))) { + const meta = groupKey(rows[0].properties); + const body = + meta.host === "claude-code" + ? buildClaudeGroup(rows) + : meta.host === "copilot-cli" + ? buildCopilotGroup(rows) + : { unit: "unknown", denominators: {}, stages: {} }; + reportGroups.push({ ...meta, ...body }); + } + + return { + excluded: { invalid, legacy }, + groups: reportGroups, + limits: REPORT_LIMITS, + }; +}; + +/** Thrown for unparseable input; the message never contains the input text. */ +export class ExportParseError extends Error {} + +/** @param {string} text @param {string} where */ +const parseJson = (text, where) => { + try { + return JSON.parse(text); + } catch { + throw new ExportParseError(`${where} is not valid JSON`); + } +}; + +/** + * @param {string} text + * @returns {unknown[]} + */ +export const parseExportedEvents = (text) => { + const trimmed = text.trim(); + if (!trimmed) return []; + if (trimmed.startsWith("[")) return parseJson(trimmed, "the array"); + return trimmed + .split("\n") + .map((line, index) => ({ line: line.trim(), number: index + 1 })) + .filter(({ line }) => line) + .map(({ line, number }) => parseJson(line, `line ${number}`)); +}; + +if (process.argv[1] && resolve(process.argv[1]) === fileURLToPath(import.meta.url)) { + const file = process.argv[2]; + if (!file) { + console.error("usage: node scripts/telemetry-report.mjs "); + process.exit(1); + } + /** @type {unknown[]} */ + let events; + try { + events = parseExportedEvents(readFileSync(file, "utf8")); + } catch (error) { + // Error text from JSON.parse or the runtime can quote the input, so print only our own message. + const reason = error instanceof ExportParseError ? error.message : "the file could not be read"; + console.error(`telemetry-report: ${reason}; no report was produced`); + process.exit(1); + } + console.log(JSON.stringify(buildReport(events), null, 2)); +} diff --git a/test/fixtures/telemetry-report/events.jsonl b/test/fixtures/telemetry-report/events.jsonl new file mode 100644 index 0000000..b4b983e --- /dev/null +++ b/test/fixtures/telemetry-report/events.jsonl @@ -0,0 +1,21 @@ +{"event":"Plugin prompt submitted","distinct_id":"aaaaaaaaaaaaaaaa","properties":{"session":"aaaaaaaaaaaaaaaa","turn":"1111111111111111","host":"claude-code","plugin_version":"0.2.0","telemetry_version":2,"os":"linux","arcade_used_before":false,"$process_person_profile":false,"$geoip_disable":true,"$ip":"0.0.0.0","could_use_arcade":true,"service_hints":["email"],"reminder_sent":true}} +{"event":"Plugin tool called","distinct_id":"aaaaaaaaaaaaaaaa","properties":{"session":"aaaaaaaaaaaaaaaa","turn":"1111111111111111","host":"claude-code","plugin_version":"0.2.0","telemetry_version":2,"os":"linux","arcade_used_before":false,"$process_person_profile":false,"$geoip_disable":true,"$ip":"0.0.0.0","server":"arcade","tool":"Arcade_SelectTools"}} +{"event":"Plugin tool called","distinct_id":"aaaaaaaaaaaaaaaa","properties":{"session":"aaaaaaaaaaaaaaaa","turn":"1111111111111111","host":"claude-code","plugin_version":"0.2.0","telemetry_version":2,"os":"linux","arcade_used_before":false,"$process_person_profile":false,"$geoip_disable":true,"$ip":"0.0.0.0","server":"arcade","tool":"System_ManageAuthorization","auth_needed":false}} +{"event":"Plugin tool called","distinct_id":"aaaaaaaaaaaaaaaa","properties":{"session":"aaaaaaaaaaaaaaaa","turn":"1111111111111111","host":"claude-code","plugin_version":"0.2.0","telemetry_version":2,"os":"linux","arcade_used_before":true,"$process_person_profile":false,"$geoip_disable":true,"$ip":"0.0.0.0","server":"arcade","tool":"app_tool","service":"email"}} +{"event":"Plugin built-in tool called","distinct_id":"aaaaaaaaaaaaaaaa","properties":{"session":"aaaaaaaaaaaaaaaa","turn":"1111111111111111","host":"claude-code","plugin_version":"0.2.0","telemetry_version":2,"os":"linux","arcade_used_before":true,"$process_person_profile":false,"$geoip_disable":true,"$ip":"0.0.0.0","tool":"Bash","cli":"gh","service":"code_hosting"}} +{"event":"Plugin prompt submitted","distinct_id":"aaaaaaaaaaaaaaaa","properties":{"session":"aaaaaaaaaaaaaaaa","turn":"2222222222222222","host":"claude-code","plugin_version":"0.2.0","telemetry_version":2,"os":"linux","arcade_used_before":true,"$process_person_profile":false,"$geoip_disable":true,"$ip":"0.0.0.0","could_use_arcade":true,"service_hints":[],"reminder_sent":true}} +{"event":"Plugin tool attempted","distinct_id":"aaaaaaaaaaaaaaaa","properties":{"session":"aaaaaaaaaaaaaaaa","turn":"2222222222222222","host":"claude-code","plugin_version":"0.2.0","telemetry_version":2,"os":"linux","arcade_used_before":true,"$process_person_profile":false,"$geoip_disable":true,"$ip":"0.0.0.0","server":"arcade","tool":"Arcade_UseTool"}} +{"event":"Plugin tool failed","distinct_id":"aaaaaaaaaaaaaaaa","properties":{"session":"aaaaaaaaaaaaaaaa","turn":"1111111111111111","host":"claude-code","plugin_version":"0.2.0","telemetry_version":2,"os":"linux","arcade_used_before":true,"$process_person_profile":false,"$geoip_disable":true,"$ip":"0.0.0.0","server":"arcade","tool":"app_tool","service":"chat","failure_kind":"timeout"}} +{"event":"Plugin prompt submitted","distinct_id":"aaaaaaaaaaaaaaaa","properties":{"session":"aaaaaaaaaaaaaaaa","turn":"3333333333333333","host":"claude-code","plugin_version":"0.2.0","telemetry_version":2,"os":"linux","arcade_used_before":true,"$process_person_profile":false,"$geoip_disable":true,"$ip":"0.0.0.0","could_use_arcade":false,"service_hints":[],"reminder_sent":true}} +{"event":"Plugin tool called","distinct_id":"aaaaaaaaaaaaaaaa","properties":{"session":"aaaaaaaaaaaaaaaa","turn":"4444444444444444","host":"claude-code","plugin_version":"0.2.0","telemetry_version":2,"os":"linux","arcade_used_before":true,"$process_person_profile":false,"$geoip_disable":true,"$ip":"0.0.0.0","server":"arcade","tool":"Arcade_ListApps"}} +{"event":"Plugin subagent stopped","distinct_id":"aaaaaaaaaaaaaaaa","properties":{"session":"aaaaaaaaaaaaaaaa","host":"claude-code","plugin_version":"0.2.0","telemetry_version":2,"os":"linux","arcade_used_before":true,"$process_person_profile":false,"$geoip_disable":true,"$ip":"0.0.0.0","agent":"arcade-operator","status":"completed"}} +{"event":"Plugin prompt submitted","distinct_id":"bbbbbbbbbbbbbbbb","properties":{"session":"bbbbbbbbbbbbbbbb","host":"copilot-cli","plugin_version":"0.2.0","telemetry_version":2,"os":"linux","arcade_used_before":false,"$process_person_profile":false,"$geoip_disable":true,"$ip":"0.0.0.0","could_use_arcade":true,"service_hints":["issues"],"reminder_sent":false}} +{"event":"Plugin tool called","distinct_id":"bbbbbbbbbbbbbbbb","properties":{"session":"bbbbbbbbbbbbbbbb","host":"copilot-cli","plugin_version":"0.2.0","telemetry_version":2,"os":"linux","arcade_used_before":false,"$process_person_profile":false,"$geoip_disable":true,"$ip":"0.0.0.0","server":"arcade","tool":"Arcade_UseTool","service":"issues"}} +{"event":"Plugin subagent stopped","distinct_id":"bbbbbbbbbbbbbbbb","properties":{"session":"bbbbbbbbbbbbbbbb","host":"copilot-cli","plugin_version":"0.2.0","telemetry_version":2,"os":"linux","arcade_used_before":false,"$process_person_profile":false,"$geoip_disable":true,"$ip":"0.0.0.0","agent":"arcade-operator","status":"needs_auth","subagent_session":"cccccccccccccccc"}} +{"event":"Plugin prompt submitted","distinct_id":"cccccccccccccccc","properties":{"session":"cccccccccccccccc","host":"copilot-cli","plugin_version":"0.2.0","telemetry_version":2,"os":"linux","arcade_used_before":false,"$process_person_profile":false,"$geoip_disable":true,"$ip":"0.0.0.0","could_use_arcade":true,"service_hints":[],"reminder_sent":false}} +{"event":"Plugin tool called","distinct_id":"cccccccccccccccc","properties":{"session":"cccccccccccccccc","host":"copilot-cli","plugin_version":"0.2.0","telemetry_version":2,"os":"linux","arcade_used_before":false,"$process_person_profile":false,"$geoip_disable":true,"$ip":"0.0.0.0","server":"arcade","tool":"Arcade_SelectTools"}} +{"event":"Plugin prompt submitted","distinct_id":"dddddddddddddddd","properties":{"session":"dddddddddddddddd","host":"copilot-cli","plugin_version":"0.2.0","telemetry_version":2,"os":"linux","arcade_used_before":false,"$process_person_profile":false,"$geoip_disable":true,"$ip":"0.0.0.0","could_use_arcade":true,"service_hints":["docs"],"reminder_sent":false}} +{"event":"Plugin prompt submitted","distinct_id":"dddddddddddddddd","properties":{"session":"dddddddddddddddd","host":"copilot-cli","plugin_version":"0.2.0","telemetry_version":2,"os":"linux","arcade_used_before":false,"$process_person_profile":false,"$geoip_disable":true,"$ip":"0.0.0.0","could_use_arcade":false,"service_hints":[],"reminder_sent":false}} +{"event":"Plugin prompt submitted","distinct_id":"eeeeeeeeeeeeeeee","properties":{"session":"eeeeeeeeeeeeeeee","host":"copilot-cli","plugin_version":"0.2.0","telemetry_version":2,"os":"linux","arcade_used_before":false,"$process_person_profile":false,"$geoip_disable":true,"$ip":"0.0.0.0","could_use_arcade":true,"service_hints":["calendar"],"reminder_sent":false}} +{"event":"Plugin prompt submitted","distinct_id":"aaaaaaaaaaaaaaaa","properties":{"session":"aaaaaaaaaaaaaaaa","host":"claude-code","plugin_version":"0.2.0","os":"linux","arcade_used_before":false,"$process_person_profile":false,"$geoip_disable":true,"$ip":"0.0.0.0","could_use_arcade":true,"service_hints":[],"reminder_sent":true}} +{"event":"not-a-contract-event","distinct_id":"ffffffffffffffff","properties":{"session":"ffffffffffffffff","host":"claude-code","plugin_version":"0.2.0","telemetry_version":2,"os":"linux","arcade_used_before":false,"$process_person_profile":false,"$geoip_disable":true,"$ip":"0.0.0.0"}} diff --git a/test/fixtures/telemetry-report/expected-report.json b/test/fixtures/telemetry-report/expected-report.json new file mode 100644 index 0000000..05662f6 --- /dev/null +++ b/test/fixtures/telemetry-report/expected-report.json @@ -0,0 +1,111 @@ +{ + "excluded": { + "invalid": 1, + "legacy": 1 + }, + "groups": [ + { + "host": "claude-code", + "plugin_version": "0.2.0", + "telemetry_version": 2, + "unit": "turn", + "denominators": { + "observed_relevant_turns": 3 + }, + "tool_only_turns": 1, + "stages": { + "gateway_discovery_or_selection": { + "count": 1, + "denominator": "observed_relevant_turns" + }, + "authorization_check_auth_needed_true": { + "count": 0, + "denominator": "observed_relevant_turns" + }, + "authorization_check_auth_needed_false": { + "count": 1, + "denominator": "observed_relevant_turns" + }, + "app_action_called": { + "count": 1, + "denominator": "observed_relevant_turns" + }, + "app_action_failed": { + "count": 1, + "denominator": "observed_relevant_turns" + }, + "attempt_observed_outcome_unknown": { + "count": 1, + "denominator": "observed_relevant_turns" + }, + "no_call_observed": { + "count": 1, + "denominator": "observed_relevant_turns" + } + }, + "builtin_tools_called": { + "Bash": { + "gh": 1 + } + }, + "builtin_tools_failed": {}, + "failure_kinds": { + "timeout": 1 + }, + "operator_stop_status": { + "completed": 1 + } + }, + { + "host": "copilot-cli", + "plugin_version": "0.2.0", + "telemetry_version": 2, + "unit": "session", + "denominators": { + "observed_relevant_sessions": 3 + }, + "operator_linked_sessions_excluded": 1, + "multi_prompt_sessions": 1, + "parent_attribution_unknown_sessions": 2, + "stages": { + "gateway_discovery_or_selection": { + "count": 1, + "denominator": "observed_relevant_sessions" + }, + "authorization_check_auth_needed_true": { + "count": 0, + "denominator": "observed_relevant_sessions" + }, + "authorization_check_auth_needed_false": { + "count": 0, + "denominator": "observed_relevant_sessions" + }, + "app_action_called": { + "count": 1, + "denominator": "observed_relevant_sessions" + }, + "app_action_failed": { + "count": 0, + "denominator": "observed_relevant_sessions" + }, + "no_call_observed": { + "count": 2, + "denominator": "observed_relevant_sessions" + } + }, + "failure_kinds": {}, + "operator_stop_status": { + "needs_auth": 1 + } + } + ], + "limits": [ + "Counts describe plugin hook observations only, not task outcomes or whether Arcade was needed.", + "Prompt events are limited to locally classified app-related work; unrelated prompts are absent.", + "Claude Code turn counts and Copilot CLI session counts must not be combined into one rate.", + "Copilot CLI does not emit tool-attempt events; attempt stages apply only to Claude Code.", + "Multiple tool calls in one turn or session cannot be paired without a tool-call ID.", + "Attempt outcomes are checked per server category; other_arcade pools every other Arcade connection, so an outcome there cannot be tied to the connection or call that was attempted.", + "The Arcade MCP gateway remains the canonical source for request, auth, discovery, and tool-call telemetry." + ] +} diff --git a/test/generate-manifests.test.mjs b/test/generate-manifests.test.mjs index dc237f1..85140f6 100644 --- a/test/generate-manifests.test.mjs +++ b/test/generate-manifests.test.mjs @@ -5,6 +5,7 @@ import { test } from "node:test"; import { FILE_SOURCES, FILES_WITH_GENERATED_RULES, + FILES_WITH_GENERATED_TABLES, generateManifests, requireSources, } from "../scripts/generate-manifests.mjs"; @@ -24,6 +25,7 @@ const EXPECTED_SOURCES = { "com.github.copilot/agents/arcade-operator.agent.md": ["agents/arcade-operator.agent.md", "hooks/routing-guidance.mjs"], "agents/arcade-operator.agent.md": ["hooks/routing-guidance.mjs"], "skills/try-arcade/SKILL.md": ["hooks/routing-guidance.mjs"], + "docs/telemetry.md": ["hooks/telemetry-contract.mjs"], }; test("FILE_SOURCES lists the expected sources for every generated path", () => { @@ -48,6 +50,9 @@ test("check mode fails with a source-naming error when a generated file is hand- if (path in FILES_WITH_GENERATED_RULES) { // Edit inside the block so the rules-block path is specifically tested. writeFileSync(fullPath, readFileSync(fullPath, "utf8").replace("Gateway: Arcade is connected", "Gateway: Any server is connected")); + } else if (FILES_WITH_GENERATED_TABLES.includes(path)) { + // Only the tables are generated, so the edit has to be inside them. + writeFileSync(fullPath, readFileSync(fullPath, "utf8").replace("Service categories:", "Service kinds:")); } else { writeFileSync(fullPath, `${readFileSync(fullPath, "utf8")} `); } diff --git a/test/helpers.mjs b/test/helpers.mjs index ea8bf25..68befd5 100644 --- a/test/helpers.mjs +++ b/test/helpers.mjs @@ -3,7 +3,7 @@ import { cpSync, mkdtempSync, readFileSync } from "node:fs"; import { tmpdir } from "node:os"; import path from "node:path"; import { fileURLToPath } from "node:url"; -import { COPIED_FILES, FILES_WITH_GENERATED_RULES } from "../scripts/generate-manifests.mjs"; +import { COPIED_FILES, FILES_WITH_GENERATED_RULES, FILES_WITH_GENERATED_TABLES } from "../scripts/generate-manifests.mjs"; export const ROOT = path.resolve(path.dirname(fileURLToPath(import.meta.url)), ".."); @@ -19,7 +19,7 @@ export const runHook = (script, input = {}, args = []) => /** A temp copy of the generator's source files, for tests that write. */ export const makeFixture = () => { const root = mkdtempSync(path.join(tmpdir(), "arcade-plugin-")); - const sources = ["VERSION", "plugin.json", "mcp.json", ...Object.keys(FILES_WITH_GENERATED_RULES), ...Object.values(COPIED_FILES).flat()]; + const sources = ["VERSION", "plugin.json", "mcp.json", ...Object.keys(FILES_WITH_GENERATED_RULES), ...FILES_WITH_GENERATED_TABLES, ...Object.values(COPIED_FILES).flat()]; for (const file of sources) { cpSync(path.join(ROOT, file), path.join(root, file)); } diff --git a/test/telemetry-docs.test.mjs b/test/telemetry-docs.test.mjs new file mode 100644 index 0000000..e955f15 --- /dev/null +++ b/test/telemetry-docs.test.mjs @@ -0,0 +1,29 @@ +// @ts-check + +import assert from "node:assert/strict"; +import { readFileSync } from "node:fs"; +import path from "node:path"; +import { test } from "node:test"; +import { + buildTelemetryTables, + fillTelemetryTables, + TELEMETRY_BLOCK_BEGIN, + TELEMETRY_BLOCK_END, +} from "../scripts/telemetry-docs.mjs"; +import { ROOT } from "./helpers.mjs"; + +test("buildTelemetryTables lists every contract event and common property", () => { + const tables = buildTelemetryTables(); + assert.match(tables, /Plugin prompt submitted/); + assert.match(tables, /telemetry_version/); + assert.match(tables, /Service categories:/); +}); + +test("fillTelemetryTables replaces only the generated block", () => { + const docPath = path.join(ROOT, "docs/telemetry.md"); + const source = readFileSync(docPath, "utf8"); + const filled = fillTelemetryTables(source); + assert.ok(filled.includes(TELEMETRY_BLOCK_BEGIN)); + assert.ok(filled.includes(TELEMETRY_BLOCK_END)); + assert.ok(filled.includes(buildTelemetryTables().split("\n")[0])); +}); diff --git a/test/telemetry-report.test.mjs b/test/telemetry-report.test.mjs new file mode 100644 index 0000000..2e9ee44 --- /dev/null +++ b/test/telemetry-report.test.mjs @@ -0,0 +1,369 @@ +// @ts-check + +import assert from "node:assert/strict"; +import { existsSync, readFileSync, writeFileSync } from "node:fs"; +import { spawnSync } from "node:child_process"; +import path from "node:path"; +import { test } from "node:test"; +import { TELEMETRY_HOSTS } from "../hooks/telemetry-contract.mjs"; +import { loadTelemetryAdapter } from "../hooks/telemetry-adapter.mjs"; +import { buildReport, normalizeExportRow, parseExportedEvents } from "../scripts/telemetry-report.mjs"; +import { assertMatchesContract, captureTelemetry, hookInput, tempDataDir } from "./telemetry-helpers.mjs"; +import { ROOT } from "./helpers.mjs"; + +const FIXTURE_DIR = path.join(ROOT, "test/fixtures/telemetry-report"); +const FORBIDDEN_KEY = /recall|precision|success.?rate|task.?success|routing.?miss|matched.?attempt/i; + +const loadFixture = () => { + const events = parseExportedEvents(readFileSync(path.join(FIXTURE_DIR, "events.jsonl"), "utf8")); + for (const event of events) { + if (event.event?.startsWith("Plugin") && event.properties?.telemetry_version === 2) { + assertMatchesContract(event); + } + } + const expected = JSON.parse(readFileSync(path.join(FIXTURE_DIR, "expected-report.json"), "utf8")); + return { events, expected }; +}; + +/** @param {unknown} value @param {string[]} keys */ +const collectKeys = (value, keys = []) => { + if (value && typeof value === "object") { + if (Array.isArray(value)) { + for (const item of value) collectKeys(item, keys); + } else { + for (const [key, child] of Object.entries(value)) { + keys.push(key); + collectKeys(child, keys); + } + } + } + return keys; +}; + +test("buildReport matches the telemetry-report fixture", () => { + const { events, expected } = loadFixture(); + assert.deepEqual(buildReport(events), expected); +}); + +test("buildReport strips PostHog export fields before validating", () => { + const { events, expected } = loadFixture(); + const noisy = events.map((row) => { + if (!row || typeof row !== "object" || !row.properties) return row; + return { + ...row, + timestamp: "2026-01-01T00:00:00.000Z", + uuid: "550e8400-e29b-41d4-a716-446655440000", + properties: { + ...row.properties, + $lib: "posthog-node", + $lib_version: "5.0.0", + }, + }; + }); + for (const row of noisy) { + if (row?.event?.startsWith("Plugin")) { + const trimmed = normalizeExportRow(row); + assert.ok(trimmed); + assert.equal(trimmed.properties.$lib, undefined); + } + } + assert.deepEqual(buildReport(noisy), expected); +}); + +test("buildReport rejects leaked hook fields in export properties", () => { + const { events } = loadFixture(); + const sample = events.find( + (row) => row.event === "Plugin prompt submitted" && row.properties?.telemetry_version === 2, + ); + assert.ok(sample); + const withPrompt = { ...sample, properties: { ...sample.properties, prompt: "secret text" } }; + const withCwd = { ...sample, properties: { ...sample.properties, cwd: "/Users/secret" } }; + assert.equal(buildReport([withPrompt]).excluded.invalid, 1); + assert.equal(buildReport([withCwd]).excluded.invalid, 1); +}); + +test("CLI prints the same JSON as buildReport", () => { + const { events, expected } = loadFixture(); + const file = path.join(FIXTURE_DIR, "events.jsonl"); + const result = spawnSync(process.execPath, [path.join(ROOT, "scripts/telemetry-report.mjs"), file], { + encoding: "utf8", + }); + assert.equal(result.status, 0, result.stderr); + assert.deepEqual(JSON.parse(result.stdout), expected); + assert.deepEqual(buildReport(events).excluded, { invalid: 1, legacy: 1 }); +}); + +const claudeRow = (event, extra) => ({ + event, + distinct_id: "aaaaaaaaaaaaaaaa", + properties: { + session: "aaaaaaaaaaaaaaaa", + turn: "1111111111111111", + host: "claude-code", + plugin_version: "0.2.0", + telemetry_version: 2, + os: "linux", + arcade_used_before: false, + $process_person_profile: false, + $geoip_disable: true, + $ip: "0.0.0.0", + ...extra, + }, +}); + +const unknownOutcomeCount = (rows) => { + for (const row of rows) assertMatchesContract(row); + const report = buildReport(rows); + assert.deepEqual(report.excluded, { invalid: 0, legacy: 0 }); + return report.groups[0].stages.attempt_observed_outcome_unknown.count; +}; + +test("an outcome on one Arcade server category does not settle an attempt on the other", () => { + const prompt = claudeRow("Plugin prompt submitted", { could_use_arcade: true, service_hints: ["email"], reminder_sent: false }); + const attempt = (server) => claudeRow("Plugin tool attempted", { server, tool: "Arcade_UseTool" }); + const called = (server) => claudeRow("Plugin tool called", { server, tool: "Arcade_SelectTools" }); + const failed = (server) => claudeRow("Plugin tool failed", { server, tool: "Arcade_UseTool", failure_kind: "timeout" }); + + assert.equal(unknownOutcomeCount([prompt, attempt("arcade"), called("other_arcade")]), 1); + assert.equal(unknownOutcomeCount([prompt, attempt("other_arcade"), failed("arcade")]), 1); + assert.equal(unknownOutcomeCount([prompt, attempt("arcade"), called("arcade")]), 0); + assert.equal(unknownOutcomeCount([prompt, attempt("other_arcade"), failed("other_arcade")]), 0); + assert.equal(unknownOutcomeCount([prompt, attempt("arcade"), attempt("other_arcade"), called("arcade")]), 1); + // Turn-level only: one outcome on a category settles every attempt on that category. + assert.equal(unknownOutcomeCount([prompt, attempt("arcade"), attempt("arcade"), called("arcade")]), 0); +}); + +test("report limits state that other_arcade pools connections and calls are not paired", () => { + const limits = buildReport([]).limits.join("\n"); + assert.match(limits, /other_arcade.*pools/i); + assert.match(limits, /cannot be paired/i); +}); + +test("CLI rejects malformed JSON without echoing the input", (t) => { + const marker = "SYNTHETIC-PRIVATE-MARKER-7f3a"; + const dir = tempDataDir(); + const valid = readFileSync(path.join(FIXTURE_DIR, "events.jsonl"), "utf8").split("\n")[0]; + const inputs = { + "lines.jsonl": `${valid}\n{"event":"Plugin prompt submitted","properties":{"prompt":"${marker}"\n`, + "array.json": `[{"event":"${marker}",}]`, + "bare.jsonl": `${marker} not json`, + }; + for (const [name, contents] of Object.entries(inputs)) { + const file = path.join(dir, name); + writeFileSync(file, contents); + const result = spawnSync(process.execPath, [path.join(ROOT, "scripts/telemetry-report.mjs"), file], { + encoding: "utf8", + }); + t.diagnostic(`${name}: ${result.stderr.trim()}`); + assert.notEqual(result.status, 0, name); + assert.equal(result.stdout, "", name); + assert.match(result.stderr, /^telemetry-report: /, name); + for (const stream of [result.stdout, result.stderr]) { + assert.equal(stream.includes(marker), false, `${name}: ${stream}`); + assert.equal(stream.includes("Plugin prompt submitted"), false, `${name}: ${stream}`); + } + } +}); + +test("report keys avoid forbidden metric names and hosts stay separate", () => { + const { events } = loadFixture(); + const report = buildReport(events); + for (const key of collectKeys(report)) { + assert.doesNotMatch(key, FORBIDDEN_KEY, key); + } + const hosts = report.groups.map((group) => group.host); + assert.deepEqual(new Set(hosts), new Set(hosts)); + for (const group of report.groups) { + if (group.host === "copilot-cli") { + assert.equal(group.stages.attempt_observed_outcome_unknown, undefined); + } + if (group.host === "claude-code") { + assert.ok(group.stages.attempt_observed_outcome_unknown); + } + } +}); + +const authStatusResponse = (statuses) => [ + { + type: "text", + text: JSON.stringify({ + message: statuses.includes("authorization_required") ? "Not yet authorized." : "All authorized.", + providers: statuses.map((status, index) => ({ provider: `provider${index}`, status })), + }), + }, +]; + +const claudeArcadePrefix = () => { + const plugin = JSON.parse(readFileSync(path.join(ROOT, "plugin.json"), "utf8")); + const [server] = Object.keys(JSON.parse(readFileSync(path.join(ROOT, "mcp.json"), "utf8")).mcpServers); + return `mcp__plugin_${plugin.name}_${server}__`; +}; + +const COPILOT_SESSION = "dd3beb80-4471-4513-99a4-a57d3d7c08df"; +const COPILOT_SUBAGENT = "57946be7-1a73-40ca-a042-abb0f68d9445"; +const COPILOT_OPERATOR = "arcade:arcade-operator"; + +/** @param {import("../hooks/telemetry-adapter.mjs").TelemetryAdapter} adapter */ +const runClaudeCrossCheck = async (adapter) => { + const dataDir = tempDataDir(); + const prefix = claudeArcadePrefix(); + const session = "cross-check-claude-session"; + /** @type {object[]} */ + const events = []; + const cap = async (fields, argv = []) => { + const { sent } = await captureTelemetry({ + adapter, + dataDir, + input: hookInput({ session_id: session, ...fields }), + argv, + }); + events.push(...sent); + }; + const p1 = "turn-one-11111111"; + const p2 = "turn-two-22222222"; + const p3 = "turn-three-33333333"; + const p4 = "turn-four-44444444"; + await cap({ hook_event_name: "UserPromptSubmit", prompt: "What is on my calendar tomorrow?", prompt_id: p1 }); + await cap({ hook_event_name: "PostToolUse", tool_name: `${prefix}Arcade_SelectTools`, prompt_id: p1 }); + await cap({ + hook_event_name: "PostToolUse", + tool_name: `${prefix}System_ManageAuthorization`, + tool_response: authStatusResponse(["authorized"]), + prompt_id: p1, + }); + await cap({ + hook_event_name: "PostToolUse", + tool_name: `${prefix}Arcade_UseTool`, + tool_input: { tool_name: "Gmail.ListEmails" }, + prompt_id: p1, + }); + await cap({ + hook_event_name: "PostToolUseFailure", + tool_name: `${prefix}Slack_SendMessage`, + error: "rate limited", + prompt_id: p1, + }); + await cap({ hook_event_name: "UserPromptSubmit", prompt: "Fix the parser in src/main.ts", prompt_id: p2 }); + await cap({ + hook_event_name: "PostToolUse", + tool_name: "mcp__granola__Granola_ListMeetings", + prompt_id: p2, + }); + await cap({ hook_event_name: "UserPromptSubmit", prompt: "What is on my calendar tomorrow?", prompt_id: p3 }); + await cap({ + hook_event_name: "PreToolUse", + tool_name: `${prefix}Arcade_UseTool`, + tool_input: { tool_name: "Gmail.ListEmails" }, + prompt_id: p3, + }); + await cap({ hook_event_name: "UserPromptSubmit", prompt: "Summarize unread email from this week", prompt_id: p4 }); + return events; +}; + +/** @param {import("../hooks/telemetry-adapter.mjs").TelemetryAdapter} adapter */ +const runCopilotCrossCheck = async (adapter) => { + const dataDir = tempDataDir(); + /** @type {object[]} */ + const events = []; + const cap = async (fields) => { + const { sent } = await captureTelemetry({ adapter, dataDir, input: fields }); + events.push(...sent); + }; + const parent = (fields) => ({ + session_id: COPILOT_SESSION, + timestamp: "2026-09-24T21:27:02.743Z", + cwd: "/Users/someone/private-repo", + ...fields, + }); + const child = (fields) => parent({ session_id: COPILOT_SUBAGENT, ...fields }); + await cap(parent({ hook_event_name: "UserPromptSubmit", prompt: "What is on my calendar tomorrow?" })); + await cap(parent({ hook_event_name: "PostToolUse", tool_name: "arcade-Arcade_SelectTools", tool_input: {}, tool_result: { result_type: "success", text_result_for_llm: "[]" } })); + await cap(parent({ + hook_event_name: "PostToolUse", + tool_name: "arcade-System_ManageAuthorization", + tool_input: {}, + tool_result: { result_type: "success", text_result_for_llm: '{"providers":[{"status":"authorized"}]}' }, + })); + await cap(parent({ + hook_event_name: "PostToolUse", + tool_name: "arcade-Arcade_UseTool", + tool_input: { tool_name: "Gmail.ListEmails" }, + tool_result: { result_type: "success", text_result_for_llm: "ok" }, + })); + await cap(parent({ + hook_event_name: "PostToolUseFailure", + tool_name: "arcade-Slack_SendMessage", + tool_input: {}, + error: "MCP server 'arcade': Something went wrong in the upstream service", + })); + await cap(parent({ hook_event_name: "UserPromptSubmit", prompt: "Fix the parser in src/main.ts" })); + await cap(parent({ + hook_event_name: "PostToolUse", + tool_name: "granola-Granola_ListMeetings", + tool_input: {}, + tool_result: { result_type: "success", text_result_for_llm: "[]" }, + })); + await cap(parent({ + hook_event_name: "SubagentStop", + agent_type: COPILOT_OPERATOR, + agent_id: COPILOT_SUBAGENT, + last_assistant_message: "status: needs_auth\nsummary: sign in", + })); + await cap(child({ hook_event_name: "UserPromptSubmit", prompt: "What is on my calendar tomorrow?" })); + return events; +}; + +/** @param {object} group */ +const claudeCrossCheckExpectation = (group) => { + assert.equal(group.unit, "turn"); + assert.deepEqual(group.denominators, { observed_relevant_turns: 3 }); + assert.equal(group.tool_only_turns, 0); + assert.deepEqual(group.stages, { + gateway_discovery_or_selection: { count: 1, denominator: "observed_relevant_turns" }, + authorization_check_auth_needed_true: { count: 0, denominator: "observed_relevant_turns" }, + authorization_check_auth_needed_false: { count: 1, denominator: "observed_relevant_turns" }, + app_action_called: { count: 1, denominator: "observed_relevant_turns" }, + app_action_failed: { count: 1, denominator: "observed_relevant_turns" }, + attempt_observed_outcome_unknown: { count: 1, denominator: "observed_relevant_turns" }, + no_call_observed: { count: 1, denominator: "observed_relevant_turns" }, + }); +}; + +/** @param {object} group */ +const copilotCrossCheckExpectation = (group) => { + assert.equal(group.unit, "session"); + assert.deepEqual(group.denominators, { observed_relevant_sessions: 1 }); + assert.equal(group.operator_linked_sessions_excluded, 1); + assert.equal(group.multi_prompt_sessions, 0); + assert.equal(group.parent_attribution_unknown_sessions, 0); + assert.deepEqual(group.stages, { + gateway_discovery_or_selection: { count: 1, denominator: "observed_relevant_sessions" }, + authorization_check_auth_needed_true: { count: 0, denominator: "observed_relevant_sessions" }, + authorization_check_auth_needed_false: { count: 1, denominator: "observed_relevant_sessions" }, + app_action_called: { count: 1, denominator: "observed_relevant_sessions" }, + app_action_failed: { count: 1, denominator: "observed_relevant_sessions" }, + no_call_observed: { count: 0, denominator: "observed_relevant_sessions" }, + }); + assert.equal(group.stages.attempt_observed_outcome_unknown, undefined); + assert.deepEqual(group.operator_stop_status, { needs_auth: 1 }); +}; + +for (const host of TELEMETRY_HOSTS) { + test(`adapter cross-check buildReport counts for ${host}`, async (t) => { + const adapterPath = path.join(ROOT, "hooks/telemetry-adapters", `${host}.mjs`); + if (!existsSync(adapterPath)) { + t.skip(`${host} adapter not in this branch`); + return; + } + const adapter = await loadTelemetryAdapter(host); + const events = host === "claude-code" ? await runClaudeCrossCheck(adapter) : await runCopilotCrossCheck(adapter); + for (const event of events) assertMatchesContract(event); + const report = buildReport(events); + assert.deepEqual(report.excluded, { invalid: 0, legacy: 0 }); + assert.equal(report.groups.length, 1); + const group = report.groups[0]; + assert.equal(group.host, host); + if (host === "claude-code") claudeCrossCheckExpectation(group); + else copilotCrossCheckExpectation(group); + }); +}