Skip to content

perf(telemetry): reuse opened project, cache device ID, throttle uploads - #3021

Merged
mikeland73 merged 2 commits into
mainfrom
mikeland73/perf-telemetry
Oct 26, 2026
Merged

mikeland73 merged 2 commits into
mainfrom
mikeland73/perf-telemetry

Conversation

@mikeland73

Copy link
Copy Markdown
Collaborator

Summary

Implements items 2–4 of specs/perf-improvements.md, which together remove fixed telemetry costs from every command (including every devbox global shellenv and every direnv reload).

2. Don't re-open the project after every command

  • The telemetry post-run hook returns immediately when telemetry is disabled (DO_NOT_TRACK, or builds without keys). Before, it still ran devbox.Open.
  • devbox.Open now records the project it returned (devbox.LastOpened()), and telemetry reads packages and the nixpkgs commit from that project instead of opening it again (which re-parsed devbox.json/devbox.lock and reloaded plugins).
  • That also gets rid of the --config bug: the old code looked the flag up on the root command, so it never found it.
  • Behavior change: commands that never open a project (e.g. version, search) now report no packages. Before, they reported whatever project was in the cwd.

3. Cache the device ID

  • The salted machineid.ProtectedID is written to $XDG_STATE_HOME/devbox/telemetry/device-id and read from there on later runs. This skips ioreg on macOS (~20ms).

4. Throttle devbox upload-telemetry

  • The uploader is spawned at most once every 30 minutes. In between, events stay buffered on disk as they do today. It still runs right away when an error was reported (so Sentry errors arrive promptly and the Sentry transport's 30-event buffer isn't exceeded) or when 50+ events are buffered.
  • The last upload time goes in $XDG_STATE_HOME/devbox/telemetry/last-upload before the child starts, so commands running at the same time don't all spawn one. A timestamp in the future is ignored.
  • Sentry events now set Timestamp when they're created, since they may be uploaded later.

Testing

  • go test ./internal/telemetry/ ./internal/boxcli/midcobra/: new tests for the device ID cache (hit and miss) and shouldUpload (never uploaded, recent, stale, future timestamp, error, threshold). Updated the getPackagesAndCommitHash test.
  • golangci-lint run on the changed packages: 0 issues.
  • Timing on macOS, with binaries built with fake telemetry keys and network blocked by a proxy, averaged over 20 runs in this repo:
Command main this PR
devbox version 106.6ms 85.8ms
devbox list 106.1ms 86.3ms

Most of the remaining cost is nix --version (~75ms), which newTrackMessage calls through nix.Version(). Item 1 (a separate PR) caches it.

🤖 Generated with Claude Code

- Skip the telemetry post-run hook entirely when telemetry is disabled.
- Reuse the project the command already opened instead of calling
  devbox.Open a second time after every command. This also drops the
  broken --config lookup, which was done on the root command and so never
  found the flag.
- Cache the device ID in the XDG state dir. Computing it runs ioreg on
  macOS (~20ms).
- Only spawn the `upload-telemetry` process at most every 30 minutes,
  unless an error was reported or 50+ events are buffered. Sentry events
  now record their own timestamp since they may be uploaded later.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
mikeland73 added a commit that referenced this pull request Oct 25, 2026
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@mikeland73

Copy link
Copy Markdown
Collaborator Author

@claude ptal

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@claude

claude Bot commented Oct 25, 2026 •

Copy link
Copy Markdown

PR #3021 was updated after the trigger (2026-10-25T21:46:34Z >= 2026-10-25T21:44:01Z). Mention Claude again to run on the latest version.

@mikeland73

Copy link
Copy Markdown
Collaborator Author

@claude ptal

@claude

claude Bot commented Oct 25, 2026 •

Copy link
Copy Markdown

Claude finished @mikeland73's task in 23s —— View job


Review of #3021

I read the diff and traced the changed code paths. I didn't run the tests or the linter. The PR description says both pass, but I haven't confirmed that. I found no blocking issues, only a few small points below.

What looks right

  • postRun returns early when telemetry is disabled, and defer telemetry.Stop() is registered before that return. Stop already no-ops when !started, so this is safe.
  • Setting started = false before the upload check in Stop is fine. It also covers the case where shouldUpload returns false, so no upload runs later in the process.
  • shouldUpload fails open. A missing, unparsable or future last-upload file means upload now. The age < 0 check handles clock changes.
  • Segment events already set Timestamp: time.Now() at creation (segment.go:52). Adding it to Sentry events is the right fix for throttled uploads.
  • writeStateFile is a good extraction. It removes the duplicated mkdir-and-retry logic.
  • Dropping the --config lookup is correct. The old code read the flag off the root command, so it never found it.

Points to consider (non-blocking)

  1. Failed uploads aren't retried for 30 minutes. last-upload is written before the child starts. If the upload fails (offline, for example), events stay buffered but won't be retried until the next interval or until 50 events accumulate. That seems an acceptable tradeoff and matches the "avoid concurrent spawns" goal. A one-line comment saying so would help the next reader.
  2. LastOpened() keeps only the last project opened. If a command opens more than one project (for example a global project and a local one), telemetry reports the last. That is probably fine, but the doc comment could say so explicitly.
  3. Cached device ID is never invalidated. The cache holds the machine-ID hash. If the file is copied between machines, such as in a baked image or a synced home directory, devices will share an ID. This is an edge case. It could be handled by caching only when the state dir is local, or ignored.
  4. bufferedEventCount counts directory entries. Stray non-event files in devbox/segment or devbox/sentry would inflate the count. This is unlikely and harmless.
  5. Tests. The new tests cover the cache hit and miss and the shouldUpload cases listed in the description. Consider one test that Stop writes last-upload only when it actually spawns. That is optional and awkward to do without stubbing exec.

Verdict: looks good to me. The behavior change (commands that open no project report no packages) is clearly documented in the PR description.
· branch mikeland73/perf-telemetry

@mikeland73
mikeland73 merged commit a0b1cae into main Oct 26, 2026
28 checks passed
@mikeland73
mikeland73 deleted the mikeland73/perf-telemetry branch October 26, 2026 00:40
mikeland73 added a commit that referenced this pull request Oct 26, 2026
…#3022)

## Summary

Implements item 5 of `specs/perf-improvements.md`.

direnv runs `devbox generate direnv --print-envrc` on every reload. It
only prints a fixed `.envrc` template, but it still:
- inherited `ensureNixInstalled` from `generate`'s `PersistentPreRunE`,
which runs `nix --version`, and
- recorded a telemetry event, which also spawned the `devbox
upload-telemetry` process.

Changes:
- `generate`'s `PersistentPreRunE` skips the nix check when
`--print-envrc` is set. The root uses `cobra.EnableTraverseRunHooks`, so
overriding the hook on the `direnv` subcommand wouldn't stop the
parent's hook from running. That's why the check goes in the parent.
- New `midcobra.SkipTelemetry(cmd)` lets a command opt out of being
recorded for the current run. The telemetry post-run hook checks for it.
The `--print-envrc` path calls it.

## Testing

- New `TestGenerateDirenvPrintEnvrcSkipsNixCheck` runs the command
through `RootCmd()` and checks the output and that `nix.Ensured()` stays
false. Without the fix it fails.
- New `TestSkipTelemetry`.
- `go test ./internal/boxcli/...` passes. `golangci-lint run
./internal/boxcli/...` reports 0 issues.
- Output is byte-for-byte the same as on `main`.
- Timing for `generate direnv --print-envrc` on macOS, averaged over 20
runs. Binaries were built with fake telemetry keys and network was
blocked by a proxy.

| | main | this PR |
|---|---|---|
| telemetry on | 131.0ms | 33.7ms |
| `DO_NOT_TRACK=1` | 92.7ms | 10.6ms |

With telemetry on, almost all of the remaining ~23ms is
`telemetry.Start()` running `ioreg` for the machine ID. #3021 caches
that.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

1 participant