Skip to content

Let users and their coding agents send feedback about Beam - #2088

Closed
mernit wants to merge 1 commit into
mainfrom
eli/agent-feedback
Closed

mernit wants to merge 1 commit into
mainfrom
eli/agent-feedback

Conversation

@mernit

@mernit mernit commented Oct 31, 2026 •

Copy link
Copy Markdown
Contributor

Agents that hit a confusing error, a misleading doc or a missing feature had no way to tell us, so that friction never reached the team. This adds one path for it, shared by the CLI and MCP.

Gateway. POST /api/v1/feedback/:workspaceId stores a report in a new feedback table (migration 064). It then sends a feedback.submitted event to the operator's event callbacks only. Reports are about the platform, so they never go to workspace event history or webhooks.

  • A report has a kind (bug, error_message, missing_capability, docs, performance, praise, other) and a summary (required, at most 300 characters). It can also say whether the problem blocked the task, and carry the command, the error code and message, details, a suggested improvement, the model and the environment. The caller and agent session come from the attribution headers.
  • Before storing, text is scrubbed of the caller's own token, private keys, URL credentials, bearer values, key=value secrets, known token formats (AWS, GitHub, Slack, OpenAI, Stripe, Hugging Face, Google, JWTs) and long random strings. Paths, hex digests, UUIDs, max_tokens and tokenizer survive.
  • Bodies are capped at 64 KiB. Each workspace can send 30 reports an hour, and each agent session (or token, without a session) 10. Past that, the gateway answers 429 and tells the agent to continue. An idempotency key, or the same reporter repeating a summary within a day, returns the earlier report instead of a new one.

MCP. send_feedback posts through the same route, so validation, scrubbing and limits live in one place. It is listed whatever ?features=, ?tools= or ?readonly= a connection sets, and the server instructions say when to use it: when Beam seems at fault or a doc or tool description misled the agent, never for the user's own errors.

CLI. beam feedback -m "..." --kind bug, with --blocked, --details, --suggest, --command, --model and --last-error.

  • The CLI remembers the error that ended its last run, so --last-error can attach it. It keeps the command and option names but never option values.
  • Server failures now get their own INTERNAL code (HTTP 500, gRPC INTERNAL and DATA_LOSS). Their JSON errors carry a feedback field with the command to report them.
  • HTTP 429 now maps to RATE_LIMITED.
  • BETA9_NO_FEEDBACK=1 turns reports off and hides send_feedback from the local MCP bridge.
  • The agent skill gains a Feedback section.

Bumps the SDK to 0.1.293.

Rollout

  1. Migration 064 only adds a table.
  2. Production's gateway callbacks live in the beta9 AWS secret (events.callbacks in config.yaml). If the internal-api callback lists eventTypes, add feedback.submitted to it. Until then, reports are stored in Postgres and go nowhere else.
  3. internal-api posts reports to Slack and tracks them in PostHog: beam-cloud/internal-api#343. Operators without internal-api can point a slack format callback at an incoming webhook with eventTypes: [feedback.submitted].
  4. The CLI command and the bridge changes reach users with the 0.1.293 release.

Worth a look

  • Reports need a workspace token; there is no anonymous path.
  • Restricted tokens can't call send_feedback over MCP, as with every MCP tool, but can still use the REST route and the CLI.
  • The opt-out is an environment variable only; there is no config setting yet.
  • Callers that matched ERROR for server failures now see INTERNAL. gRPC UNKNOWN stays ERROR, because the gateway returns plain Go errors as UNKNOWN.

Testing

  • Go: new tests for the route (scrubbing, validation, duplicates, limits, size cap), the MCP tool against the real route, the repository, and the event sinks. go test ./pkg/... passes apart from two pkg/worker tests: one also fails on main locally (it needs Linux), and the other is timing-sensitive and passes when run alone.
  • Postgres 14: every migration applies to a fresh database, a repeated idempotency key returns the first report, deleting the token nulls token_id, and 064 rolls back and reapplies.
  • SDK: pytest tests gives 858 passed, 2 skipped (871 with Attribute CLI and MCP calls to the coding agent that made them #2087 merged). uv lock --check and ruff check . pass.

Independent of #2087; the two merge cleanly in either order.

Agents that hit a confusing error, a misleading doc or a missing feature
had no way to tell us, so that friction never reached the team. This adds
one path for it, shared by the CLI and MCP.

Gateway:
- POST /api/v1/feedback/:workspaceId stores a report in a new feedback
  table (migration 064) and sends a feedback.submitted event to the
  operator's event callbacks only. Reports are about the platform, so
  they never go to workspace event history or webhooks.
- A report has a kind, a summary (required, at most 300 characters),
  whether it blocked the task, the command, error code and message,
  details, a suggested improvement, the model and the environment. The
  caller and agent session come from the attribution headers.
- Text is scrubbed before it is stored: the caller's own token, private
  keys, URL credentials, bearer values, key=value secrets, known token
  formats and long random strings.
- Bodies are capped at 64 KiB. A workspace can send 30 reports an hour
  and an agent session (else a token) 10; past that the gateway answers
  429 and tells the agent to continue. An idempotency key, or the same
  reporter repeating a summary within a day, returns the earlier report.

MCP: send_feedback posts through the same route and is listed whatever
?features=, ?tools= or ?readonly= a connection sets. The server
instructions say when to use it.

CLI: `feedback -m "..." --kind bug`, with --blocked, --details,
--suggest, --command, --model and --last-error. The CLI remembers the
error that ended its last run, with the command and option names but
never option values, so --last-error can attach it. Server failures now
get their own INTERNAL code (HTTP 500, gRPC INTERNAL and DATA_LOSS), and
their JSON errors carry a `feedback` field with the command to report
them; HTTP 429 maps to RATE_LIMITED. BETA9_NO_FEEDBACK=1 turns reports
off and hides send_feedback from the local MCP bridge. The agent skill
gains a Feedback section.

Bumps the SDK to 0.1.293.

Against Postgres 14, every migration applies to a fresh database, a
repeated idempotency key returns the first report, and 064 rolls back.
`pytest tests`: 858 passed, 2 skipped.

Co-authored-by: Cursor <cursoragent@cursor.com>

@luke-lombardi luke-lombardi left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This does not belong in the OSS layer - we shouldn't have a feedback table in the control plane. cc @mernit. If we need this, the feedback should live in beam-client + internal-api

@mernit

mernit commented Oct 31, 2026

Copy link
Copy Markdown
Contributor Author

Closing: feedback doesn't belong in the open-source control plane. It now lives in Beam's own code: beam feedback in beam-client (beam-cloud/beam-client#102), with storage, scrubbing, limits and Slack/PostHog routing in internal-api (https://github.com/beam-cloud/internal-api/pull/343). Nothing is needed in beta9. Agent attribution stays in #2087.

@mernit mernit closed this Oct 31, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants