# Should I add high-cardinality labels like user_id to my Prometheus metrics?

> Keep unbounded labels like user_id off your metrics, restrict labels to bounded dimensions, and take per-customer breakdowns to exemplars or traces.

- Asked: 2026-10-14
- Answered: 2026-10-16
- Asked by: Kaya
- Tags: observability, metrics
- Source: https://muhammetsafak.com/just-ask/add-high-cardinality-labels-like-user-id-prometheus/
- Language: en-US
- Author: Muhammet Şafak

---
**Question:** I monitor a Go API with Prometheus and I want per-customer latency breakdowns; being able to see which tenant is slow at the p95 level would be genuinely useful.

The easiest path I can think of is adding a user_id (or tenant_id) label to my histogram metrics. But I have thousands of customers, and I'm scared of blowing up Prometheus cardinality and the memory/storage cost if I do that. Should I add this label, or should I solve per-customer breakdowns somewhere else?


Short answer: no — don't put user_id (or any unbounded label) on your metrics.

## Short answer

Prometheus stores a separate time series per unique label combination, and per-customer cardinality grows your memory and cost multiplicatively. Leave the customer breakdown to exemplars, traces, or logs; reserve labels for bounded dimensions.

The trap here is thinking "it's just one more label, what's the harm." In Prometheus, cost is measured by active series count, not by number of metrics — and an unbounded label grows that count multiplicatively, not linearly.

## Why

1. **Cardinality = series count.** Each unique label combination is a separate, in-memory time series. The product `endpoint × status × user_id` explodes fast; in the head block each series costs kilobytes of RAM. A few million series will OOM your Prometheus.
2. **Watch histogram cost specifically.** Histograms already multiply: `buckets × label combinations`. A 12-bucket histogram over 50 endpoints is at least 600 series (the `+Inf` bucket, `_sum` and `_count` come on top) before you add anything. Adding user_id on top is catastrophic.

## What to do

1. **Use bounded labels only.** Good labels: `endpoint`, `method`, `status_code`, `region` — low and stable cardinality. Bad ones: `user_id`, `email`, `request_id`, a full URL with IDs in it — those are unbounded, never add them.
2. **Per-customer latency belongs elsewhere.** Options: **exemplars** (attach a trace_id/user_id to a histogram bucket — bounded, one per bucket per scrape), **tracing** ([put user_id as a span attribute in OTel](/just-ask/distributed-tracing-with-opentelemetry-across-laravel-and-go/), sampled to Tempo/Jaeger), or **structured logs** (aggregated in Loki/ClickHouse).
   ```go
   // Histogram with bounded labels + an exemplar for the drill-down:
   timer := prometheus.NewTimer(nil)
   lat := timer.ObserveDuration().Seconds()

   reqDuration.
       With(prometheus.Labels{"endpoint": ep, "status": code}).
       (prometheus.ExemplarObserver).
       ObserveWithExemplar(lat, prometheus.Labels{"trace_id": traceID})
   ```
3. **If you truly need top-N customers, tier it.** A bounded `customer_tier` label (free/pro/enterprise) gives you segment breakdowns cheaply. Or use an allowlist to label only your few heaviest customers and bucket the rest as `other`.
4. **Put up guardrails.** Set `sample_limit` on the scrape, and alert on `prometheus_tsdb_head_series` and `scrape_samples_scraped` so cardinality can't run away silently.
   ```promql
   # Watch cardinality — if this number keeps climbing, a label has escaped:
   topk(10, count by (__name__)({__name__=~".+"}))
   ```

**Bottom line:** personally I'd keep metrics on bounded labels (endpoint/status/tier), attach exemplars for the drill-down, and push real per-customer analytics to logs/ClickHouse. High-cardinality "just in case" labels are exactly how you turn a 4 GB Prometheus into a 40 GB one. Decide first whether your question is a metric or a trace — a per-customer breakdown is almost always a job for tracing.

## Related Reading

- [Should I instrument my services with RED metrics or USE metrics first?](https://muhammetsafak.com/just-ask/instrument-services-red-metrics-use-metrics-first/) — Just Ask
- [How should I carry a correlation ID from the HTTP request all the way into queued jobs?](https://muhammetsafak.com/just-ask/how-should-i-carry-a-correlation-id-from-the-http-request/) — Just Ask
- [How do I set a clear rule for when to log at WARN versus ERROR?](https://muhammetsafak.com/just-ask/how-do-i-set-a-clear-rule-for-when-to-log/) — Just Ask