Skip to content

Fix memory Leak: use route template for request-metric path label - #115

Merged
shawnburke merged 1 commit into
mainfrom
fix/prometheus-path-cardinality
Jul 19, 2026
Merged

Fix memory Leak: use route template for request-metric path label#115
shawnburke merged 1 commit into
mainfrom
fix/prometheus-path-cardinality

Conversation

@shawnburke

Copy link
Copy Markdown
Collaborator

Problem

A heap profile (heap1.prof) showed ~820 MB of ~977 MB retained by Prometheus, attributed to two lines in the request middleware:

561 MB  h.requestLatency.WithLabelValues(r.Method, r.URL.Path, status).Observe(...)
259 MB  h.requestCounter.WithLabelValues(r.Method, r.URL.Path, status).Inc()

The root cause is label-cardinality explosion, not a lack of scraping. prometheus/client_golang keeps every metric series in memory in the registry permanently, regardless of whether anything scrapes /metrics — scraping only reads values, it never frees series.

The path label was set to the raw r.URL.Path. The webhook server registers with mux.PathPrefix("/webhook/"), so every /webhook/<id>/... URL matched and created its own permanent counter and histogram (each histogram is ~11 buckets + sum + count + label pairs, hence the large newHistogram / MakeLabelPairs allocations). Unique webhook ids / resource ids → unbounded series growth → the observed ~1 GB.

Fix

Use the matched mux route template for the path label instead of the raw path:

  • /webhook/<any-id>/webhook/ (PathPrefix route)
  • /webhook/{id} variable routes → /webhook/{id}

All concrete URLs collapse to a single bounded series. Falls back to a fixed "<unmatched>" sentinel if no route template is resolvable, so the label can never become unbounded. (Note: mux only runs middleware for matched routes, so 404 scans already never created series.)

This bounds memory for every deployment, not just those that disable metrics.

Tests

Added http_server_metrics_test.go:

  • TestRequestMetricsCollapsePrefixRouteToTemplate — 25 distinct /webhook/<id> requests through a PathPrefix route produce exactly one series labeled /webhook/ on both the counter and the histogram. (Fails before the fix with 25 distinct series.)
  • TestRequestMetricsCollapseVariableRouteToTemplate — same for a /webhook/{id} variable route, collapsing to /webhook/{id}.

Full ./server/http/ package passes; go build ./... and go vet clean.

🤖 Generated with Claude Code

The request middleware used the raw r.URL.Path as the "path" label on the
axon_http_requests counter and axon_http_request_latency_seconds histogram.
Because prometheus/client_golang retains every series in memory permanently
(independent of scraping), routes that carry unique path segments — most
notably the webhook PathPrefix route /webhook/<id>/... — created a new,
never-freed counter + histogram per distinct URL. A heap profile attributed
~820MB / ~977MB to these two lines (newHistogram + MakeLabelPairs).

Use the matched mux route template (e.g. "/webhook/" or "/webhook/{id}")
instead of the raw path so all concrete URLs collapse to a single bounded
series. Falls back to a fixed "<unmatched>" sentinel when no route template
is resolvable, so the label can never become unbounded.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@shawnburke shawnburke changed the title fix(http): use route template for request-metric path label Fix memory Leak: use route template for request-metric path label Jul 18, 2026
@shawnburke
shawnburke requested a review from ashiramin July 18, 2026 13:46
@shawnburke
shawnburke enabled auto-merge (squash) July 18, 2026 13:47
@shawnburke
shawnburke merged commit 47cb6c8 into main Jul 19, 2026
17 checks passed
@shawnburke
shawnburke deleted the fix/prometheus-path-cardinality branch July 19, 2026 02:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants