feat(telemetry): add telemetry reporting and document the default-off telemetry (4/4, GRO-305)

[teallarson]

Linear ticket: GRO-305

What/why

The local reporter compares selected app-related work with observed Arcade activity: Claude by turn and Copilot by session. This prepares an estimate of likely opportunity with or without invocation, complementing gateway logs that cannot reveal uncalled opportunities. Classification and delivery gaps mean ‘no call observed’ does not prove a missed necessary call. This PR also documents the proposed collection contract; collection remains OFF.

Report flow

The reporter operates on supplied exports, independently of collection. Opportunity means locally classified app-related work. Here, invocation evidence includes attempts and failures: no_call_observed means zero Arcade attempt/call/failure rows, not the inverse of app_action_called. Claude attempts may have unknown outcomes; Copilot has no attempt hook. Missing delivery can look like absence, and Claude turns must not be combined with Copilot sessions into one rate.

flowchart TD
  Export["Exported observations<br/>Current OFF build collects none"]
  Export --> Validate["Normalize and validate contract<br/>Exclude invalid and legacy rows"]
  Validate --> Groups["Separate host, plugin and telemetry versions"]
  Groups --> Claude["Claude: relevant prompt turns"]
  Groups --> Copilot["Copilot: relevant prompt sessions<br/>Direct operator-child attribution"]
  Claude --> Compare["Compare within each unit<br/>Report separate, overlapping stage counts"]
  Copilot --> Compare
  Compare --> Seen["Opportunity + observed invocation evidence<br/>Arcade attempt, call or failure"]
  Compare --> None["Opportunity + no observed invocation<br/>Zero Arcade attempt, call or failure rows"]
  Seen -.-> Limits["Opportunity is heuristic; delivery may be missing<br/>No proven necessity, recall or task success"]
  None -.-> Limits

Loading

Codebase changes

scripts/telemetry-report.mjs validates export rows against the shared contract, ignores export metadata and PostHog $ properties, and groups observations by host/plugin/telemetry version. It keeps unknown outcomes separate for arcade and pooled other_arcade, and malformed input exits nonzero with a sanitized diagnostic that never echoes the row. Fixture tests cover counts, leaked fields, cross-category outcomes and real-adapter cross-checks; the two adapter checks skip on this isolated branch and run on the combined stack. Maintained docs describe the inactive build, client coverage, privacy and measurement limits; contract-derived tables remain generated.

Review area	Files	Added	Removed
Reporter, docs generator and generator registration	3	520	0
Tests, fixtures and test helpers	6	537	2
Maintained documentation and repository guidance	8	465	6
Total	17	1,522	8

Review map uses git diff --numstat foundation...head, excluding foundation changes; docs/telemetry.md includes generated tables inside a maintained document.

Proof

Behavior	Evidence at 49932242
Counts, contract and privacy	Branch verify: 117 passed, 2 expected adapter skips; typecheck and all four client validators pass. Fresh independent Node 22.23.2 review: 15 focused tests, outcome-order/empty-input probes and sanitized-error checks pass
Adapter integration	Both real-adapter cross-checks pass in final heads plus main 18c7c99: 170/170 tests, zero skips, typecheck/four validators and generation checks, with Node 22.23.2 and macOS pwsh 7.6.6
Generation and CI	Generation leaves no diff; exact-head CI including the non-draft aggregate check, and fresh Bugbot pass

Additional notes

Merge #26 first, #27/#28 in either order, then #29. The three leaves currently target the foundation branch; after its squash merge, retarget/replay their own changes onto main and revalidate changed heads. Hold the generated release PR until the complete stack lands. Merging the release PR versions and tags the inactive build; it does not enable collection.

Activation requires a separate reviewed change and collection/privacy approval, verified Windows and live-client hook/opt-out delivery, and validated production ingestion and retention. Gateway telemetry remains canonical.

Gateway telemetry remains canonical; plugin observations supplement the offline routing evaluation and cannot establish per-call pairing or outcomes within pooled connections. The current acceptance checklist remains at PR13's pinned head, with an earlier version on backup/pr13-ffc22e8-20261006; one-off logs/checklists stay outside this product diff. Accepted minor limitation: leading blank lines can offset the JSONL error line number, while input stays private and no report is produced. Windows/live-client delivery and production ingestion are untested; the old integration ref af9dd52 is historical and final validation uses a fresh unpushed combined tree.

Linear Issue: GRO-305

  

[teallarson]

Independent review of the current telemetry-OFF draft. Findings below remain valid at this exact head; no implementation, approval, merge or collection enablement is implied.

— Strider 🐦‍⬛, Teal's agent · approved by Teal

[teallarson]

@cursor review

— Strider 🐦‍⬛, Teal's agent · approved by Teal

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 4993224. Configure here.}

[teallarson]

Independent final review at 4993224250fb25ca14e5506011ec7968fa69b823: ready for human review with telemetry OFF.

Independent report/privacy review: 15 focused tests; isolated-branch adapter skips run successfully in the combined stack. All three earlier findings are corrected. Accepted nonblocking limitation: leading blank JSONL lines can offset the diagnostic physical line number; raw input remains private.

Ready-for-review CI passed tests, all four client checks and the required aggregate gate; current-head Cursor Bugbot is green. Final combined heads plus main 18c7c99 passed 170/170 tests, zero skips, typecheck, four validators and 13 generated outputs. Independent combined privacy/release/adapter-removal review found no new blocker.

Windows, live client delivery and production ingestion remain activation prerequisites. No merge, release or collection enablement is authorized by this review. Descriptions preserve own-base review maps and limitations; Teal will inspect and request human reviewers.

— Strider 🐦‍⬛, Teal's agent · approved by Teal