Skip to content

feat: CLI usage telemetry with coding agent detection - #318

Open
tkkhq wants to merge 28 commits into
mainfrom
feat/agent-detection
Open

tkkhq wants to merge 28 commits into
mainfrom
feat/agent-detection

Conversation

@tkkhq

@tkkhq tkkhq commented Oct 15, 2026 •

Copy link
Copy Markdown
Contributor

Tracking

Why

  • we can't see which commands people run, where they fail, or which coding agent ran them, so we can't tell how agents use Volcano or where they get stuck

Summary

  • add internal/agent: detect the coding agent from env vars in this order: named AI_AGENT/AGENT, harness markers (CLAUDECODE, CODEX_THREAD_ID, GEMINI_CLI, CURSOR_AGENT, OPENCODE_CLIENT, ...), VOLCANO_AGENT, bare truthy AI_AGENT/AGENT (unknown), then host-terminal markers (CURSOR_TRACE_ID, REPL_ID); exact name matches only, canonical names, values sanitized to a header-safe charset capped at 64 chars
  • send X-Volcano-Agent, -Source, -Raw, -Markers on every API request from versionProtocolDoer, and nothing when no agent is detected, telemetry is off, or the first-run notice has not been shown yet
  • add internal/telemetry: one cli_command_invoked event per run with command path, names of flags set, selected project ID, resource names and IDs, CLI version, OS, arch, CPU count, CI vendor, stdin TTY, agent name/source/markers, Volcano plugin installed + version for that agent (version sent only when it is semver-shaped), exit code, duration; every failed run adds error kind, HTTP status, API error code, and the error message
  • take resource names only from positional args under an allowlisted usage-line placeholder (<database>, <name-or-id>, <bucket>, ...) and from the name, function, database, project, topic, bucket flags, keyed by placeholder or flag name; paths, queries, payloads, and sandbox commands are never sent, and values that don't look like a name or look like a secret are dropped
  • before sending an error message, replace the home dir with ~, strip ANSI escapes and control characters, collapse whitespace, then redact bearer tokens, JWTs, pk-/pt-/sk-/ak- keys, AWS key IDs, URL user info, URL query strings, password=-style pairs (including prefixed keys such as DB_PASSWORD=), and 40+ char opaque runs, and cut to 500 runes; the API applies the same steps again, and its docs/internal/guides/cli-telemetry.md is the wire contract
  • add internal/sensitive to mark errors whose text can carry file contents or application data; telemetry replaces that part with [redacted] while the terminal still shows it; used for volcano-config.yaml YAML/interpolation/decode errors, env file parse errors (godotenv quotes the file), and a function's passthrough error body in volcano functions invoke
  • send the event from a detached volcano telemetry flush process to POST {apiURL}/cli/telemetry with the saved login token when present, so commands never wait on the network
  • the notice points to volcano telemetry --help, which ships in the binary, since docs caches synced before this release lack cli/telemetry.md
  • add setup.PluginStatus to answer "is the Volcano plugin installed for this harness" from local files, without the harness presence probe
  • add opt-out: volcano telemetry disable|enable|status and VOLCANO_TELEMETRY_DISABLED=1; VOLCANO_TELEMETRY_DEBUG=1 prints the event instead of sending (off unless set); volcano telemetry commands are never recorded, so an opt-out isn't itself reported
  • print a one-time notice on the first recorded run and send nothing on that run, so users can opt out before any event leaves the machine; save the state before printing, so a read-only home (sandboxed agents) shows no notice instead of repeating it
  • store device id, opt-out, and notice state in a telemetry block of ~/.volcano/config.json, updated through config.Update so a first run can't overwrite a concurrent volcano telemetry disable
  • a process that started before the first-run notice never sends, even when a concurrent process showed the notice first
  • read the setting once per process: the agent headers and the event follow the read at start, and the flush doesn't recheck, so a change applies from the next command
  • change volcano logout to clear the token, user ID, current project, and function aliases instead of deleting config.json, so the telemetry setting and docs source survive a logout; the file is still removed when nothing else is in it, or when it can't be parsed (so logout still removes the token from a damaged file)
  • never record shell completion (__complete, which runs on every Tab with stderr discarded, and volcano completion <shell>, which rc files run on every new shell) or --offline runs, checking raw args too so a flag parse error can't bypass it
  • run the detached flush from the temp dir so it never holds the caller's working directory open
  • record from run() after the exit code is known, using ExecuteC for the executed command; telemetry never changes the exit code or output beyond the one-time notice
  • turn telemetry off in the API and local-mode e2e env helpers and the CI smoke step
  • treat COPILOT_MODEL/COPILOT_ALLOW_ALL as weak signals, since people export them in their own profile; map pi to its volcano setup harness
  • add docs/telemetry.md and docs/agent-detection.md, linked from docs/README.md and docs/setup.md
  • the agent name list lives in three places and must change together: internal/agent aliases and markers here, allowedAgents in volcano-hosting internal/productanalytics/allowlist.go, and the harness list in volcano-skills AGENTS.md
  • merge order: the CLI posts to /cli/telemetry, which needs Kong/volcano-hosting#1919 deployed first; before that the flush gets a 404 in the background and nothing breaks
  • defer a first-run consent prompt: telemetry is opt-out like the Vercel and Next.js CLIs; whether EU ePrivacy rules require opt-in consent for reading local env vars, and how GDPR applies to resource names and error messages that may carry personal data, are open questions for legal

Verification

  • make lint
  • go test ./...
  • go test -race ./internal/agent/ ./internal/api/
  • ran the built binary with a fresh HOME against a local sink: first run printed the notice once and sent nothing, each later run posted one event to /cli/telemetry with the expected command, flags, agent, exit code, and error kind; a failing cloud functions get hello carried the project ID, {"name-or-id": "hello"}, and the error message; volcano telemetry status/disable sent nothing; DO_NOT_TRACK=1 no longer opts out; after volcano telemetry disable nothing was sent; volcano logout kept the telemetry block in config.json
  • ran the built binary with a read-only HOME (no notice on repeated runs) and __complete (no state written)
  • ran volcano projects list twice with a fresh HOME against a local server: the first run sent no X-Volcano-Agent header and no event; the second sent both
  • ran the built binary against a local header-echo server: Claude Code shell, CODEX_THREAD_ID + CURSOR_TRACE_ID, VOLCANO_AGENT only, a CRLF injection attempt in AI_AGENT, and a clean env (no headers)

@tkkhq
tkkhq requested a review from a team as a code owner October 15, 2026 02:35
Copilot AI balanced review requested due to automatic review settings October 15, 2026 02:35

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@github-actions

github-actions Bot commented Oct 15, 2026 •

Copy link
Copy Markdown

AI review: 5 P0-P2 finding(s)

Reviewed 04c6e6b. Run log

Adds default-on CLI telemetry, agent attribution, and persistent telemetry settings across logout. The main risks are unintended disclosure of local data and incorrect project attribution.

🟠 P1

  • internal/telemetry/telemetry.go:367 Exclude database execution errors from telemetry text

🟡 P2

  • internal/telemetry/telemetry.go:365 Suppress error codes carried by sensitive errors
  • internal/projectconfig/manifest.go:512 Mark manifest validation errors sensitive too
  • internal/telemetry/fields.go:69 Exclude function paths from collected resource names
  • internal/telemetry/telemetry.go:346 Use the local command's project context

⚪ P3

  • docs/agent-detection.md:13 Published doc says the CLI does not read files or send command arguments, but telemetry now does both
  • docs/telemetry.md:50 Resource example uses a placeholder that volcano databases get does not have

Not checked: The AI review read the diff, the code, and the PR's text; it ran no tests or builds and read nothing outside this repository.

  • Whether volcano-hosting's POST /cli/telemetry (Kong/volcano-hosting#1919) validates and re-redacts every field as the docs claim, and is deployed before this CLI release.
  • Whether the detached flush process behaves correctly on Windows (DETACHED_PROCESS with a stdin pipe) and inside agent sandboxes that restrict process spawning or network access.
  • Whether the PostHog agent allowlist in volcano-hosting and the harness list in volcano-skills match the canonical names in internal/agent.
  • Verify Kong/volcano-hosting#1919 is deployed and its telemetry schema, agent allowlist, and server-side redaction match this client.

Push a fix, or have a collaborator with write access (including the PR author) add the ai-review-override label to accept 04c6e6b as is. An override also needs a human review.

Human review: needed

  • Should the CLI send project IDs, resource names, and error text to analytics by default (opt-out)? The PR says the ePrivacy and GDPR questions are still open with legal.
  • Should the CLI deliberately ignore DO_NOT_TRACK=1, which users set to opt out of CLI telemetry? TestRecord_FalsyEnvDoesNotOptOut asserts it is ignored.
  • Is it acceptable that volcano logout no longer deletes ~/.volcano/config.json and that its output text changes? Scripts that check for the file's absence after logout will see different behavior.
  • Is it OK that device_id survives logout? It is sent in logged-in events, so later events stay joinable to the account, yet telemetry.md says 'linked only to the machine ID'.
  • Should internal/telemetry send account-linked resource identifiers and error text by default, using a one-time notice and opt-out?

Decide the points above: approve to accept them, or request changes.

Suggested reviewer: subnetmarco (frequent committer to docs/README.md and projectconfig, suited to judge user-facing data-collection policy). Any collaborator with write access other than the author can approve.

Comment thread docs/agent-detection.md Outdated
| Claude Code | `CLAUDECODE`, `CLAUDE_CODE_ENTRYPOINT` |
| Codex | `CODEX_THREAD_ID`, `CODEX_SANDBOX`, `CODEX_CI` |
| Gemini CLI | `GEMINI_CLI` |
| Cursor | `CURSOR_EXTENSION_HOST_ROLE=agent-exec` |

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚪ P3: Docs list Cursor before Cursor CLI, but the code checks Cursor CLI first

The doc says "If several are set, the first row in this table wins" and lists Cursor (CURSOR_EXTENSION_HOST_ROLE=agent-exec) above Cursor CLI (CURSOR_AGENT). In internal/agent/agent.go:92-93, markers puts CURSOR_AGENT → cursor-cli first. Running cursor-agent from a Cursor agent-exec terminal sets both variables, so the CLI reports cursor-cli while the published doc says cursor. Fix: swap the two table rows so they match the code.

Found by claude-opus-5-5

Comment thread internal/agent/agent.go Outdated
{env: "ANTIGRAVITY_AGENT", agent: "antigravity"},
// COPILOT_GITHUB_TOKEN is left out on purpose: people export it in their
// own shell profile, so it does not show an agent ran the command.
{env: "COPILOT_MODEL", agent: "github-copilot"},

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚪ P3: COPILOT_MODEL and COPILOT_ALLOW_ALL may be shell-profile settings, not proof an agent ran

The code skips COPILOT_GITHUB_TOKEN because people export it in their shell profile. As far as I know, Copilot CLI also documents COPILOT_MODEL (choose the model) and COPILOT_ALLOW_ALL (auto-approve tools) as settings users export themselves. If so, a user with export COPILOT_MODEL=... in .zshrc gets every command they type tagged github-copilot with source marker. Because step-2 markers outrank VOLCANO_AGENT, an agent that identifies itself only through VOLCANO_AGENT would also be reported as Copilot. Consider dropping these two markers, or treating them as a host-level signal, unless Copilot sets them in its own command shells.

Found by claude-opus-5-5

Comment thread docs/setup.md Outdated

## Agent detection

Once set up, the CLI reports which agent ran each command to the Volcano API.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚪ P3: "Once set up" suggests reporting depends on volcano setup

The new section says "Once set up, the CLI reports which agent ran each command." Detection actually runs on every API request whatever the setup state: detectAgent in versionProtocolDoer reads only environment variables and never checks whether volcano setup ran. A user who skips setup could wrongly think nothing is reported. Fix: say the CLI reports the agent on every API request whether or not volcano setup has run.

Found by claude-opus-5-5

Comment thread docs/agent-detection.md Outdated
| Claude Code | `CLAUDECODE`, `CLAUDE_CODE_ENTRYPOINT` |
| Codex | `CODEX_THREAD_ID`, `CODEX_SANDBOX`, `CODEX_CI` |
| Gemini CLI | `GEMINI_CLI` |
| Cursor | `CURSOR_EXTENSION_HOST_ROLE=agent-exec` |

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 P2: Cursor marker precedence is documented in the wrong order

When both CURSOR_EXTENSION_HOST_ROLE=agent-exec and CURSOR_AGENT=1 are set without an earlier match, this table promises cursor because the first row wins. However, internal/agent/agent.go checks CURSOR_AGENT first and reports cursor-cli. Swap these rows to match the implemented precedence.

Found by gpt-6.1-sol

Comment thread docs/agent-detection.md Outdated
| `X-Volcano-Agent-Raw` | `claude-code_2-1-299_agent` | The original value of `AI_AGENT`, `AGENT`, or `VOLCANO_AGENT`, when it differs from the name |
| `X-Volcano-Agent-Markers` | `claude-code,cursor` | Every agent whose step 2 or step 5 variables are set. More than one means one agent ran inside another, for example Claude Code run from a Cursor terminal |

Values are limited to letters, digits, and `._-/@+:`, and to 64 characters.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 P2: The stated header limits do not apply to the markers header

setAgentHeaders in internal/api/version_protocol.go joins marker names with commas without sanitizing or truncating the combined value. Consequently, even the documented claude-code,cursor example contains an excluded character, and sufficiently many markers can exceed 64 characters. Scope this statement to the environment-derived name and raw fields, and describe the marker list separately.

Found by gpt-6.1-sol

Comment thread docs/agent-detection.md
| Augment | `AUGMENT_AGENT` |
| Antigravity | `ANTIGRAVITY_AGENT` |
| GitHub Copilot | `COPILOT_MODEL`, `COPILOT_ALLOW_ALL` |

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚪ P3: Docs table lists Cursor before Cursor CLI, but the code checks them the other way round

The doc says "If several are set, the first row in this table wins" and puts CURSOR_EXTENSION_HOST_ROLE=agent-exec (Cursor) above CURSOR_AGENT (Cursor CLI). In internal/agent/agent.go the markers slice puts CURSOR_AGENT first. So when both are set, the CLI reports cursor-cli, while the published doc says cursor. Fix: swap the two table rows, or swap the two marker entries, so the doc and code agree.

Found by claude-opus-5-5

@tkkhq tkkhq changed the title feat: report which coding agent ran each CLI command feat: CLI usage telemetry with coding agent detection Oct 15, 2026
@github-actions
github-actions Bot requested a review from subnetmarco October 15, 2026 03:54
Comment thread internal/telemetry/telemetry.go Outdated

func printNotice(w io.Writer) {
fmt.Fprintln(w, "Note: the Volcano CLI collects usage data: the command and flag names you use (never their values), CLI version, platform, and which coding agent ran it. This helps us improve the CLI.")
fmt.Fprintf(w, "Turn it off with `volcano telemetry disable` or %s=1. Details: `volcano docs get telemetry`.\n", EnvDisabled)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 P2: First-run notice points users to volcano docs get telemetry, which cannot resolve

docs.Service.Get reads the cached snapshot path exactly as given (snap.readFile(docPath) after safeRelPath). It adds no .md extension and does no fuzzy lookup, and doc IDs look like cli/deploy.md. AGENTS.md says this repo's docs/ syncs under /cli, so the new page will be at cli/telemetry.md. Running the exact command the notice prints fails with a doc-not-found error, at the one moment a user is told where to read about data collection. Change the hint to volcano docs get cli/telemetry.md, or point to the published URL.

Found by claude-opus-5-5

Comment thread internal/telemetry/telemetry.go Outdated
// swallowed, since telemetry must not change what the CLI does.
func (r Recorder) Record(inv Invocation) {
r = r.withDefaults()
if inv.Command != nil && inv.Command.Annotations[SkipAnnotation] != "" {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 P2: Every shell-completion request is recorded, spawns a flush process, and can silently use up the first-run notice

Only commands annotated with SkipAnnotation are skipped. Cobra's hidden __complete command (added by ExecuteC because the root does not disable completion) is returned as cmd, so every Tab press records an event. Each one also starts a detached volcano telemetry flush process that makes an HTTP POST. Cobra's bash, zsh, and fish completion scripts run __complete with 2>/dev/null. If a user's first interaction is a Tab completion, printNotice writes to a discarded stderr, NoticeShown is saved as true, and the user never sees the notice. Skip cobra.ShellCompRequestCmd / ShellCompNoDescRequestCmd (and possibly the completion command) in Record.

Found by claude-opus-5-5

Comment thread internal/telemetry/telemetry.go Outdated
if state.DeviceID == "" {
state.DeviceID = uuid.NewString()
}
if !state.NoticeShown && inv.Stderr != nil {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 P2: Notice is printed on every command when ~/.volcano/telemetry.json cannot be written

Record prints the notice and sets NoticeShown, then returns if State.Save fails, so the flag is never saved. When HOME is readable but not writable, every invocation prints the two-line notice to stderr again and sends no event. That covers sandboxed agents (Codex's default workspace-write sandbox, Claude Code's sandbox), read-only containers, and Nix-style builds. These are the agent runs this feature targets, and every command's output picks up the noise. Save the state before printing, and print only when the save succeeds (or skip the notice when state cannot be saved).

Found by claude-opus-5-5

Comment thread docs/telemetry.md Outdated
failure (`api`, `network`, `local_not_running`, `usage`, or `other`) and, for an
API error, its HTTP status and error code.

The CLI never sends flag values, command arguments, file contents, project or

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚪ P3: Docs say environment variable values are never sent, but free-text agent values are

For an unrecognized AI_AGENT, AGENT, or VOLCANO_AGENT value, canonical returns the sanitized, lowercased value itself. That value goes out as agent in the telemetry event and as X-Volcano-Agent/X-Volcano-Agent-Raw on every API request. AGENT in particular is a generic name a user may have set for something else. The privacy statement "never sends ... environment variable values" is therefore not accurate. Qualify it (e.g. "other than the agent name read from AI_AGENT/AGENT/VOLCANO_AGENT").

Found by claude-opus-5-5

}
return
}
_ = r.Send(payload)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 P2: Honor the docs command's offline guarantee

With telemetry enabled, volcano docs list --offline still starts a background POST to /cli/telemetry, even when the docs command fails because its cache is missing. This breaks the explicit “Never touch the network” contract in internal/cmd/docs/docs.go and docs/docs-search.md. Suppress network telemetry for invocations using this offline mode.

Found by gpt-6.1-sol

Comment thread internal/telemetry/telemetry.go Outdated
printNotice(inv.Stderr)
state.NoticeShown = true
}
if err := r.State.Save(state); err != nil {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 P2: Prevent initialization from overwriting a concurrent opt-out

A first-run recorder can load enabled state, overlap with volcano telemetry disable saving disabled:true, then overwrite that preference with its stale state here. Subsequent invocations resume telemetry despite the successful opt-out. Atomic rename prevents partial JSON but does not protect this read-modify-write sequence; serialize initialization and preference updates with a cross-process lock.

Found by gpt-6.1-sol

case "opencode":
return "opencode"
}
return ""

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 P2: Map the supported pi harness before checking plugin status

After volcano setup --agent pi installs skills into ~/.pi/agent/skills, an invocation with VOLCANO_AGENT=pi reaches this fallback and checks ~/.volcano/skills instead. It therefore reports plugin_installed:false for a valid Pi installation, or reports an unrelated manual installation. Add the pi mapping supported by internal/setup/harness.go.

Found by gpt-6.1-sol

Comment thread docs/telemetry.md

Resource names come from the arguments that name a resource, such as
`<database>` in `volcano databases get <database>`, and from the `--name`,
`--function`, `--database`, `--project`, `--topic`, and `--bucket` flags. Other

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 P2: Telemetry doc says flag values and file paths are never sent, but the error message field carries them

The doc says "Other arguments and flag values, such as file paths, search queries, payloads, and commands run in a sandbox, are never sent." It also says the CLI never sends environment variable values except the agent name. But every failed run sends error_message, and that text often contains exactly those things:

  • pflag errors quote the flag value (invalid argument "..." for "--timeout" flag).
  • unknown command "..." quotes the typed argument.
  • File errors quote the path. internal/telemetry/fields.go leaves / out of the opaque pattern on purpose "so file paths survive", and the tests expect read ~/app/x.yaml: bad indent to be sent.
  • project_id can come straight from VOLCANO_PROJECT_ID.

This page is the user-facing privacy disclosure, so the absolute claims should be narrowed. For example: "except as they appear in an error message, after the redaction above." The project ID from VOLCANO_PROJECT_ID should also be listed.

Found by claude-opus-5-5

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in e72bb43 and 0327620. The doc now says error messages can carry typed input (unknown commands, rejected flag values, file paths) after redaction. It also lists VOLCANO_PROJECT_ID as a source of the project ID and names the parsed files whose text is redacted.

return
}
if state.DeviceID == "" || !state.NoticeShown {
var firstRun bool

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚪ P3: A process that started before the notice was shown can still send an event during parallel first runs

Reporting() uses the state read at process start, so agent headers stay off when NoticeShown was false. Record does not follow that read: it calls initialize() and sends whenever firstRun comes back false.

When an agent runs two volcano commands at once on a fresh machine, process A marks the notice and prints it. Process B started before the notice existed, finds NoticeShown already true, and sends a full event at about the same moment. The user never had a chance to opt out, which breaks the "sends nothing ... before any event is sent" promise in docs/telemetry.md.

Fix: if the start-of-process state() had NoticeShown == false, still call initialize(), but never send from this process.

Found by claude-opus-5-5

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in ff11e45, same change as the P2 above: a process that started before the notice was shown sends nothing.

}
found[key] = value
}
args := cmd.Flags().Args()

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚪ P3: sandboxes exec <cmd> without -- records the sandbox command as session-id

resources() trims positional args at ArgsLenAtDash() only when a -- is present. placeholders() for exec [session-id] -- <command> [args...] gives session-id.

So a common mistake like volcano sandboxes exec "echo hunter2" (no --) sends resources: {"session-id": "echo hunter2"}. The value passes resourceValue, which allows spaces, and redact. The command fails with a usage error, but the event still carries the text. That contradicts the doc's promise that commands run in a sandbox are never sent.

Fix: for usage lines that contain --, skip positional extraction unless ArgsLenAtDash() >= 0, or require ID-shaped placeholders (*-id) to parse as UUIDs.

Found by claude-opus-5-5

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in ff11e45. When the usage line has -- and the user didn't pass one, no positional arguments are read as resources.

Comment thread internal/config/config.go
func ClearLogin() error {
return withLock(func() error {
configPath, err := Path()
cfg, err := load()

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚪ P3: volcano logout now fails on an unparseable config.json and leaves the token on disk

Before, logout called config.Delete(), which removed the file without reading it. So volcano logout was also the way to recover from a corrupt or hand-mangled config.

ClearLogin() now calls load() first. A JSON parse error (from a manual edit, or a crash during the non-atomic os.WriteFile in save()) makes logout fail with failed to clear credentials: failed to parse config file, and user_token stays in the file. Consider falling back to remove() when the config cannot be parsed.

Found by claude-opus-5-5

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in e3ab28d. ClearLogin removes the file when it can't be parsed, so logout still removes the token from a damaged config. Covered by TestLogoutRemovesUnparseableConfig.

h.recorder().Record(Invocation{})
assert.NotContains(t, string(h.sent[0]), "error_")
assert.NotContains(t, string(h.sent[0]), "resources")
assert.NotContains(t, string(h.sent[0]), "project_id")

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚪ P3: TestRecord_SuccessSendsNoErrorFields fails when VOLCANO_PROJECT_ID is set in the environment

build() fills ProjectID from r.cfg.ProjectID(), which reads the real VOLCANO_PROJECT_ID (the in-memory config has IgnoreEnv false). This test asserts the payload has no project_id, so it fails for any developer or CI job that has the variable exported, which is likely for people who use the CLI. TestRecord_ProjectAndResources already calls t.Setenv("VOLCANO_PROJECT_ID", ""); this test should do the same.

Found by claude-opus-5-5

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in ff11e45. The in-memory test config sets IgnoreEnv, so no harness test reads the real VOLCANO_PROJECT_ID.

event.ErrorStatus = api.Status(inv.Err)
event.ErrorCode = api.Code(inv.Err)
home, _ := r.HomeDir()
event.ErrorMessage = errorMessage(inv.Err, home)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟠 P1: Manifest parse errors can upload plaintext secrets

After the notice has been shown, running volcano config deploy with a manifest containing an SMTP password: 'hunter2${' makes internal/projectconfig/interpolate.go return an error quoting that entire value. None of the redaction patterns removes it, so telemetry uploads the plaintext password despite the documented exclusion of file contents and secrets. Suppress secret-bearing parser errors or send allowlisted error summaries instead of arbitrary error text.

Found by gpt-6.1-sol

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in d49c59f. Added internal/sensitive to mark errors whose text can quote file contents. Manifest YAML, interpolation, and decode errors and env file parse errors are marked, and telemetry replaces the marked part with [redacted]. The terminal still shows the full error. feecd6e applies the same marker to a function's passthrough error body in functions invoke.

Comment thread internal/telemetry/telemetry.go Outdated
if err != nil || state.Disabled {
return
}
if firstRun {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 P2: Concurrent first runs can report before the notice

When two processes start without telemetry state, both initially return false from Reporting(). Once the first persists NoticeShown, the second process's initialize() returns firstRun=false, allowing it to upload its event, potentially before the first process prints the notice. This contradicts the startup snapshot and “nothing until your next command” guarantee. Keep invocations that started before the notice from sending, even when initialization finds another process has saved the state.

Found by gpt-6.1-sol

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in ff11e45. A process whose start-of-process state had no notice shown never sends, even when a concurrent process marked the notice first. Covered by TestRecord_ConcurrentFirstRunSendsNothing.

Comment thread internal/telemetry/fields.go Outdated
awsKeyPattern = regexp.MustCompile(`\b(?:AKIA|ASIA)[0-9A-Z]{16}\b`)
userPattern = regexp.MustCompile(`([a-zA-Z][a-zA-Z0-9+.-]*://)[^\s/@]+@`)
queryPattern = regexp.MustCompile(`(https?://[^\s?#]+)\?[^\s#]*`)
pairPattern = regexp.MustCompile(`(?i)\b(password|passwd|pwd|secret|token|api[_-]?key|access[_-]?key)=\S+`)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 P2: Secret-pair redaction misses prefixed names like DB_PASSWORD= and client_secret=

pairPattern puts \b before the key, and _ counts as a word character, so DB_PASSWORD=hunter2, SMTP_PASSWORD=…, MY_API_KEY=… and client_secret=… never match. Only the bare password= / API_KEY= forms in the test match. Short values also slip past opaquePattern (40+ chars). Concrete trigger: an agent or user runs volcano variables deploy DB_PASSWORD=hunter2. deploy is cobra.NoArgs, so the error is unknown command "DB_PASSWORD=hunter2" for "volcano variables deploy", and errorMessage sends it unredacted. That contradicts docs/telemetry.md, which promises to remove values of password=, token=, and similar pairs. The comment says the server applies the same rules, so it would miss this too. Fix: allow an identifier prefix, e.g. (?i)\b[A-Za-z0-9_]*?(password|passwd|pwd|secret|token|api[_-]?key|access[_-]?key)=\S+ (keep the prefix in the replacement), add tests for DB_PASSWORD= and client_secret=, and mirror the change in volcano-hosting.

Found by claude-opus-5-5

Comment thread docs/agent-detection.md
Volcano team which agents people build with. It never changes what a command
does, and it is never used to allow or block a request.

The CLI reads environment variables only. It does not read files, inspect other

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚪ P3: Published doc says the CLI does not read files or send command arguments, but this PR's telemetry does both

agent-detection.md says "The CLI reads environment variables only. It does not read files, inspect other processes, or send command arguments." In this same PR, the usage event reads harness plugin files to report plugin_installed/plugin_version, and it sends resource names taken from positional arguments and flag values (docs/telemetry.md says so). This is a published, privacy-relevant page, so the blanket statement will mislead readers. Scope it to the request headers, e.g. "Detection reads environment variables only, and the headers below carry no command arguments", and point to telemetry.md for what the usage event collects.

Found by claude-opus-5-5

return nil, fmt.Errorf("invalid manifest: %w", sensitive.Wrap(err))
}

if err := manifest.Validate(); err != nil {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 P2: Mark manifest validation failures as sensitive

Parse expands environment references before validation, and validateFunctionKind and validateFunctionVisibility quote rejected values. A misconfigured visibility: ${SMTP_PASSWORD} therefore sends the expanded password in telemetry when it does not match a redaction pattern. Wrap validation errors with sensitive.Wrap, as the surrounding parsing and decoding errors already do.

Found by gpt-6.1-sol

Resources: resources(inv.Command),
}
if r.cfg != nil {
event.ProjectID = r.cfg.ProjectID()

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 P2: Use the command's effective project for telemetry

After selecting a cloud project, running volcano functions get hello targets the local project through internal/cmd/local/local.go's withLocalConfig. The recorder instead reads the saved cloud configuration, attaching that cloud project's ID to local resource names and failures. Pass the effective session project into the invocation, or omit the project ID for local commands.

Found by gpt-6.1-sol

Comment thread docs/telemetry.md
secrets in command arguments.

Resource names come from the arguments that name a resource, such as
`<database>` in `volcano databases get <database>`, and from the `--name`,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚪ P3: Resource example uses a placeholder that volcano databases get does not have

The doc says resource names come from arguments "such as <database> in volcano databases get <database>". The real usage line is get <name> (internal/cmd/databases/get.go), so the event key for that command is name, not database. <database> is the first argument of commands such as databases branch get <database> <branch> and databases backup list <database>. Use one of those, or volcano databases get <name>, so the published example matches what the event actually contains.

Found by claude-opus-5-5

if inv.Err != nil {
event.ErrorKind = errorKind(inv.Err)
event.ErrorStatus = api.Status(inv.Err)
event.ErrorCode = api.Code(inv.Err)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 P2: Suppress error codes carried by sensitive errors

A function returning HTTP 400 with {"error":"failed","code":"private_customer_4411"} produces a sensitive error, but api.Code unwraps it and copies the application's code into telemetry unchanged. Function response content consequently leaves the machine despite the documented promise to retain only its HTTP status. Omit error codes originating inside sensitive errors.

Found by gpt-6.1-sol

break
}
if resourcePlaceholders[placeholder] {
add(placeholder, args[i])

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 P2: Exclude function paths from collected resource names

volcano functions invoke /Users/alice/private-project/hello.js is supported: internal/function/function.go resolves it using the basename. Telemetry instead collects the complete argument under resources.name, because the placeholder is name and the value pattern accepts slashes. This uploads local paths even on successful invocations, contrary to the documented collection policy. Normalize path-valued function identifiers or omit them.

Found by gpt-6.1-sol

@marckong marckong left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review of the CLI side (head 04c6e6b), read alongside Kong/volcano-hosting#1919 and Kong/volcano-skills#62. No blocking findings. I read the core paths (recorder, redaction, agent detection, sender, opt-out commands, logout change) but did not run tests.

[P2] Quoted secret pairs leak part of the value — internal/telemetry/fields.go:159-161
pairPattern ends at \S+, so password='a b' becomes password=[redacted] b' and the tail still leaves the machine. password: x and JSON "token": "x" don't match. The opaque-run rule excludes / and ., so a long base64 secret containing / can split into runs under 40 chars. The API copy is identical, so it won't catch these. Suggest table tests for these shapes in error messages.

[P2] Redaction/allowlists are hand-synced across repos — internal/telemetry/fields.go:155-162
Eight regexes are duplicated with volcano-hosting and the agent list lives in three repos; only prose enforces they match. A shared golden test file would make drift a CI failure.

[P2] First-run notice omits account linkage — internal/telemetry/telemetry.go:452-455
The notice lists what is collected but not that events are attributed to the logged-in account (only volcano telemetry --help and the docs say so), and the device ID survives logout. Since the notice is the only consent point, consider adding account linkage and that error messages can contain typed input.

[P3] Recorder reads the cloud config, not the command's config — internal/telemetry/telemetry.go:143, :346
LoadConfig defaults to config.Load and ProjectID comes from it, while main.go honors deps.ConfigLoader elsewhere. If local-mode commands reach Record, they would report the cloud config's project and post to the cloud API URL. I didn't confirm they do.

Question: does versionProtocolDoer only wrap calls to the Volcano API? X-Volcano-Agent* headers shouldn't reach a customer's function endpoint via functions invoke.

Strengths: conservative consent flow (nothing sent on first run, no send from a process started before the notice), sensitive parser errors blanked, opt-out survives logout with a test.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants