Should I add high-cardinality labels like user_id to my Prometheus metrics?

Short answer: no — don’t put user_id (or any unbounded label) on your metrics.

Short answer

Prometheus stores a separate time series per unique label combination, and per-customer cardinality grows your memory and cost multiplicatively. Leave the customer breakdown to exemplars, traces, or logs; reserve labels for bounded dimensions.

The trap here is thinking “it’s just one more label, what’s the harm.” In Prometheus, cost is measured by active series count, not by number of metrics — and an unbounded label grows that count multiplicatively, not linearly.

Why

1. Cardinality = series count. Each unique label combination is a separate, in-memory time series. The product endpoint × status × user_id explodes fast; in the head block each series costs kilobytes of RAM. A few million series will OOM your Prometheus.
2. Watch histogram cost specifically. Histograms already multiply: buckets × label combinations. A 12-bucket histogram over 50 endpoints is at least 600 series (the +Inf bucket, _sum and _count come on top) before you add anything. Adding user_id on top is catastrophic.

What to do

1. Use bounded labels only. Good labels: endpoint, method, status_code, region — low and stable cardinality. Bad ones: user_id, email, request_id, a full URL with IDs in it — those are unbounded, never add them.
2. Per-customer latency belongs elsewhere. Options: exemplars (attach a trace_id/user_id to a histogram bucket — bounded, one per bucket per scrape), tracing (put user_id as a span attribute in OTel, sampled to Tempo/Jaeger), or structured logs (aggregated in Loki/ClickHouse).

   // Histogram with bounded labels + an exemplar for the drill-down:
   timer := prometheus.NewTimer(nil)
   lat := timer.ObserveDuration().Seconds()
   
   reqDuration.
       With(prometheus.Labels{"endpoint": ep, "status": code}).
       (prometheus.ExemplarObserver).
       ObserveWithExemplar(lat, prometheus.Labels{"trace_id": traceID})

3. If you truly need top-N customers, tier it. A bounded customer_tier label (free/pro/enterprise) gives you segment breakdowns cheaply. Or use an allowlist to label only your few heaviest customers and bucket the rest as other.
4. Put up guardrails. Set sample_limit on the scrape, and alert on prometheus_tsdb_head_series and scrape_samples_scraped so cardinality can’t run away silently.

   # Watch cardinality — if this number keeps climbing, a label has escaped:
   topk(10, count by (__name__)({__name__=~".+"}))

Bottom line: personally I’d keep metrics on bounded labels (endpoint/status/tier), attach exemplars for the drill-down, and push real per-customer analytics to logs/ClickHouse. High-cardinality “just in case” labels are exactly how you turn a 4 GB Prometheus into a 40 GB one. Decide first whether your question is a metric or a trace — a per-customer breakdown is almost always a job for tracing.