Skip to content

feat(observability): enrich Sentry error events with root-cause fingerprinting#3298

Merged
migmartri merged 1 commit into
chainloop-dev:mainfrom
migmartri:feat/sentry-error-enrichment
Jul 19, 2026
Merged

feat(observability): enrich Sentry error events with root-cause fingerprinting#3298
migmartri merged 1 commit into
chainloop-dev:mainfrom
migmartri:feat/sentry-error-enrichment

Conversation

@migmartri

@migmartri migmartri commented Jul 19, 2026

Copy link
Copy Markdown
Member

Problem

All server errors funnel through pkg/servicelogger.LogAndMaskErr, which calls sentry.CaptureException with AttachStacktrace: true. Every event therefore carries the same capture-site stacktrace (servicelogger in LogAndMaskErr) and — because biz errors are wrapped with fmt.Errorf("...: %w") — the same generic outermost type *fmt.wrapError. The issue list is homogeneous and grouping depends on variable message text (IDs, constraint names), while the actually discriminating info (root-cause type, SQLSTATE, gRPC code) sits unindexed in the exception chain.

Change

Adds a BeforeSend hook in pkg/servicelogger, wired into both the controlplane and artifact-cas sentry.Init, that analyzes the captured error's unwrap chain and enriches every outgoing error event:

  • Fingerprint: ["{{ default }}", discriminator] where discriminator is SQLSTATE, gRPC code, or kratos reason. This extends Sentry's stack-based default grouping with structured root-cause data — it preserves call-site separation while subdividing by failure class. Formatted error text is never used for fingerprinting because it embeds request-specific values (emails, UUIDs, paths) that would cause unbounded issue cardinality. Errors without a structured discriminator keep Sentry's default grouping.
  • Tags: error.root_type, error.sqlstate, error.grpc_code, error.kratos_reason, error.chain_depth, error.multi_error — searchable root-cause metadata (types and codes only, never raw messages).
  • Type rewrite: generic wrapper types (*fmt.wrapError, *fmt.withMessage, pkg/errors wrappers) on the primary exception are rewritten to the root-cause Go type, so issue titles read e.g. *pgconn.PgError: creating version: ... instead of *fmt.wrapError.

Chain analysis is a depth- and visit-bounded DFS over both Unwrap() error and Unwrap() []error (matching sentry-go's traversal). It deliberately avoids errors.As (no cycle protection in stdlib), collects discriminators via per-node type assertions, and skips fingerprinting for errors.Join multi-errors, where any single-root key would mislead.

One hook covers all capture paths (LogAndMaskErr, the auditor, future CaptureException calls) with zero callsite changes.

Also removes a pre-existing leading-newline lint violation in app/controlplane/cmd/main.go flagged by the scoped lint run.

Design notes

  • Why {{ default }} + discriminator, not a custom fingerprint: Sentry's default grouping already uses stack traces to separate by call site. Prefixing {{ default }} preserves that while adding a structured subdivision. A fully custom fingerprint would lose Sentry's stack-based grouping improvements and require maintaining grouping logic in code. See Sentry fingerprint rules.
  • Why not normalize message text: wrappers like fmt.Errorf("error finding user %s: %w", email, err) embed request-specific values. Regex scrubbing covers known shapes (emails, UUIDs, numbers) but cannot handle arbitrary %s/%v content (paths, names, opaque tokens). Structured discriminators avoid this entirely.
  • Follow-up: if operation-level grouping is needed beyond call-site + discriminator, a typed operationError wrapper carrying a stable operation constant at the capture boundary would be the right approach — tracked in spec 114 as a deferred item.

Test plan

  • 19 unit subtests in pkg/servicelogger/sentry_enrich_test.go: SQLSTATE/gRPC/kratos/plain/nested/joined/cyclic chains, discriminator precedence, nil status, default-grouping fallbacks, recovered panics (error and non-error), existing-tag preservation, nil event/hint.
  • go test ./pkg/servicelogger/, go build + go vet (controlplane, artifact-cas, pkg), and golangci-lint run on all touched packages: clean.
  • Recommend pointing a dev environment at a staging Sentry project to eyeball the new grouping/titles before release.

🤖 Posted by Maximus bot (Claude Code) on behalf of @migmartri

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 4 files

Reply with feedback, questions, or to request a fix.

Re-trigger cubic

Comment thread pkg/servicelogger/sentry_enrich.go Outdated
@migmartri
migmartri force-pushed the feat/sentry-error-enrichment branch from 422cb49 to 49720d5 Compare July 19, 2026 12:47
@migmartri
migmartri requested a review from a team July 19, 2026 13:15
All server errors funnel through LogAndMaskErr, so every Sentry event
carries the same capture-site stacktrace and a generic *fmt.wrapError
outermost type: issues are indistinguishable.

Add a BeforeSend hook in pkg/servicelogger, wired into both the
controlplane and artifact-cas binaries, that analyzes the captured
error's unwrap chain (depth- and visit-bounded, cycle- and
errors.Join-safe) and enriches the event:

- Fingerprint: ['{{ default }}', discriminator] where discriminator is
  SQLSTATE, gRPC code, or kratos reason — extends Sentry's stack-based
  default grouping with structured root-cause data. Formatted error
  text is never used for fingerprinting (it embeds request-specific
  values). Errors without a discriminator keep default grouping.
- Tags: error.root_type, error.sqlstate, error.grpc_code,
  error.kratos_reason, error.chain_depth, error.multi_error
- Generic wrapper exception types (*fmt.wrapError et al.) on the
  primary exception rewritten to the root-cause type so issue titles
  identify the failure.

One hook covers all capture paths (LogAndMaskErr, auditor, future
CaptureException calls) with zero callsite changes.

Also removes a pre-existing leading-newline lint violation in
app/controlplane/cmd/main.go.

Signed-off-by: Miguel Martinez Trivino <miguel@chainloop.dev>
@migmartri
migmartri force-pushed the feat/sentry-error-enrichment branch from 49720d5 to 18fc01b Compare July 19, 2026 13:17
@migmartri
migmartri merged commit f966f70 into chainloop-dev:main Jul 19, 2026
15 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants