feat(telemetry): add default-off telemetry foundation (1/4, GRO-305)

[teallarson]

Part of GRO-305 · issue

What/why

Gateway logs show calls that happened; this stack prepares a way to estimate likely Arcade opportunities with and without an observed Arcade invocation. This PR supplies the shared event, privacy, scope and opt-out pipeline used by the Claude and Copilot adapters. Opportunity classification is a heuristic, not proven necessity, routing recall or task success. Collection remains OFF, including direct branch installs.

Measurement flow

The green boundary shows the shared foundation added in this PR #26. Before merging #26, that foundation is absent; after merging #26, it exists with collection still OFF. The solid OFF path registers no telemetry hooks and exits before reading input. The dotted future ON path requires separate approval and the client adapters from #27/#28. Those adapters, the existing PostHog destination, and #29's local export report are outside #26's boundary.

Prompt and tool inputs are separate callbacks, not a required chronology; direct Arcade events need no prior classified prompt, while alternatives and built-ins require app-work scope. Raw prompt/tool payload text stays local; only allowed categories and hashed identifiers reach the best-effort sender. #29 can report supplied exports independently of activation; this diagram does not claim delivery is proven.

flowchart TD
  Before["Before merging #26<br/>No plugin-side telemetry foundation"]
  subgraph Foundation["ADDED IN PR #26: shared telemetry foundation"]
    After["After merging #26<br/>Foundation exists; collection stays OFF"]
    Gate{"Hardcoded build gate: OFF"}
    Off["No telemetry hook rows<br/>Entry exits before input, state or sending"]
    Checks["Shared opt-outs and data-folder checks"]
    Prompt["Local prompt classification and scope<br/>Raw prompt stays local"]
    Tools["Shared selection of observed events<br/>Direct Arcade needs no prior prompt"]
    Event["Allowlisted event contract<br/>Categories and hashed IDs, no raw payload text"]
    Sender["Detached best-effort sender<br/>One-second timeout, no retries"]
    After --> Gate
    Gate -->|"After merging #26: still OFF"| Off
    Checks --> Prompt
    Checks --> Tools
    Prompt -->|"Eligible prompt events / scope"| Event
    Tools -->|"Alternatives and built-ins need app scope"| Event
    Event --> Sender
  end
  Before -->|"Merge #26"| After
  Claude["OTHER PR #27<br/>Claude Code adapter"]
  Copilot["OTHER PR #28<br/>Copilot CLI adapter"]
  Gate -.->|"Separately approved future ON build"| Claude
  Gate -.->|"Separately approved future ON build"| Copilot
  Claude --> Checks
  Copilot --> Checks
  PostHog["EXISTING: PostHog destination"]
  Sender -.->|"Delivery may be lost"| PostHog
  Report["OTHER PR #29<br/>Local report from supplied exports"]
  PostHog -.->|"User supplies event exports"| Report
  classDef added fill:#e8f5e9,stroke:#2e7d32,color:#1b5e20
  classDef outside fill:#f2f2f2,stroke:#666666,color:#333333
  class After,Gate,Off,Checks,Prompt,Tools,Event,Sender added
  class Before,Claude,Copilot,PostHog,Report outside
  style Foundation fill:#f5fff5,stroke:#2e7d32,stroke-width:3px,color:#1b5e20

Loading

Codebase changes

One shared contract filters event payloads, manages observation scope and opt-outs, and hands allowed events to a detached sender; client-specific mappings land in #27 and #28, with maintained reporting/docs in #29. The hard-OFF constant cannot be enabled through the environment: manifests register no telemetry hooks, and the entrypoint exits before reading input.

Review area	Files	Added	Removed
Hook/runtime and generator source	17	1,317	30
Shipped generated outputs	0	0	0
Tests, fixtures and test helpers	31	1,420	0
Documentation	0	0	0
Build config and dependencies	3	53	3
Total	51	2,790	33

Review map: main bc8f3dc → head 9d1e8fb6, using git diff --numstat base...head; each changed file appears once. Generator implementation counts as source, test copies count as fixtures, and “generated” means shipped outputs identified by .gitattributes.

Proof

Behavior	Evidence at 9d1e8fb6
OFF, opt-outs, routing and allowed payloads	Local npm run verify on Node 22.23.2 passed typecheck and 107/107 tests; fake-adapter/captured-transport and loopback tests exercise these boundaries
Client packaging	Exact-head CI: tests, all four validators and the non-draft aggregate check pass
Fixture portability	The old head failed 21/51 classifier tests in a checkout containing spaces and #; the shared fileURLToPath ROOT correction now passes 51/51 in the same path
Generated consistency	Generated-file checks passed; this diff changes no shipped generated output

Additional notes

Merge #26 first, #27/#28 in either order, then #29. The three leaves currently target the foundation branch; after its squash merge, retarget/replay their own changes onto main and revalidate changed heads. Hold the generated release PR until the complete stack lands. Merging the release PR versions and tags the inactive build; it does not enable collection.

Activation requires a separate reviewed change and collection/privacy approval, verified Windows and live-client hook/opt-out delivery, and validated production ingestion and retention. Gateway telemetry remains canonical.

The Bugbot fixture-path finding is fixed by Cursor in 9d1e8fb6, independently reproduced before/after, and the fresh Bugbot review is clean. Pinned-runtime typecheck, all 107 tests and all four client validators pass locally; generation checks leave the tracked tree clean. Windows, live-client delivery and production ingestion remain untested.

---

Note

Medium Risk
Introduces privacy-sensitive measurement plumbing (local scope files, PostHog sender, hashed IDs) and expands hook manifest generation, but collection stays disabled and routing hooks are designed not to throw or block sessions.

Overview
Adds a default-off plugin telemetry stack: a JSON-schema contract for PostHog events, per-client adapter loading, prompt/tool classifiers, local session/turn scope so only “app work” turns emit most events, and a detached telemetry-send.mjs for network I/O. TELEMETRY_ENABLED is hard-coded false, so npm run generate omits telemetry hooks and telemetry.mjs exits before reading stdin—behavior today is unchanged.

Hook wiring grows: HOOKS can be host-scoped (e.g. Copilot skips user-prompt-submit), and the manifest builder groups nested entries by matcher, supports if / extraArgs, and optional runOnlyIfScriptExists shell/PowerShell guards. Routing tweaks share isOperatorAgentType; session-start can clear scope when telemetry is on; prompt-filters add confirmation detection for scope.

Tooling: tsc typecheck on hooks/scripts, plus broad tests (contract validation, opt-outs, scope, classifiers, manifest rules). Client-specific adapters and enabling collection are left for follow-up PRs.

^{Reviewed by Cursor Bugbot for commit 9d1e8fb. Bugbot is set up for automated code reviews on this repo. Configure here.}

Opt-out controls (review Q&A)

Anirudh Kamath asked: How does opt-in/out work in the plugin—environment variables or a component in the app?

Teal Larson clarified: There are plugin-specific, common cross-tool, and Claude/Copilot-specific controls. Teal's code link.

Answer: These controls use environment variables; this stack adds no app UI for telemetry.

• feat(telemetry): add default-off telemetry foundation (1/4, GRO-305) #26, shared plugin controls: ARCADE_PLUGIN_TELEMETRY disables collection on 0, false, off, or no, case-insensitive. DO_NOT_TRACK disables collection on any nonempty value except those four OFF values, also case-insensitive.
• feat(telemetry): add the Claude Code telemetry adapter (2/4, GRO-305) #27, Claude adapter: DISABLE_TELEMETRY and CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC each disable collection on any nonempty value, including 0 or false.
• feat(telemetry): add the Copilot CLI telemetry adapter and guard Copilot hooks in VS Code (3/4, GRO-305) #28, Copilot adapter: COPILOT_OFFLINE disables collection on any nonempty value except 0, false, off, or no, case-insensitive. These client-specific definitions come from their adapters; feat(telemetry): add default-off telemetry foundation (1/4, GRO-305) #26 supplies the shared evaluator.

Unset or empty values do not opt out; values are not trimmed. No opt-in environment variable activates this stack: collection remains hardcoded OFF after merging these PRs, even with ARCADE_PLUGIN_TELEMETRY=1. Activation requires a separate reviewed code change and collection/privacy approval.

[teallarson]

@BugBot review

[cursor]

Stale Bugbot comment from a previous run.

[teallarson]

@cursor review

— Strider 🐦‍⬛, Teal's agent · approved by Teal

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 9d1e8fb. Configure here.}

[teallarson]

Independent final review at 9d1e8fb69c25918f8170db70d905ab4977fcb850: ready for human review with telemetry OFF.

Portable fixture-root fix independently reproduced before/after in a path with spaces and #. Final branch verification: 107/107 tests, typecheck and four client validators. Hard-OFF/manifests/privacy contract reviewed.

Ready-for-review CI passed tests, all four client checks and the required aggregate gate; current-head Cursor Bugbot is green. Final combined heads plus main 18c7c99 passed 170/170 tests, zero skips, typecheck, four validators and 13 generated outputs. Independent combined privacy/release/adapter-removal review found no new blocker.

Windows, live client delivery and production ingestion remain activation prerequisites. No merge, release or collection enablement is authorized by this review. Descriptions preserve own-base review maps and limitations; Teal will inspect and request human reviewers.

— Strider 🐦‍⬛, Teal's agent · approved by Teal