Skip to content
Muhammet Şafak
tr
Answered:

Should I build my alerts on symptoms or on causes?


Question

My team uses Prometheus/Alertmanager for observability, and most of our alerts are tied straight to resource metrics: page when CPU goes over 80%, page when memory hits some threshold. The problem: we get paged at midnight for CPU spikes, but nothing breaks on the user side; the service responds normally. Now everyone has started ignoring the pager, and I'm afraid we'll miss a real incident. Should I build my alerts on symptoms or on causes?

Answer

Short answer

Page on symptoms — that is, on what the user actually feels (latency, error rate, availability).

Context

Causes like CPU are valuable for diagnosis, but they belong on a dashboard and in a ticket, not on the pager. The fatigue you’re seeing is exactly the result of confusing the two.

Why

  1. Page on symptoms that touch the user. Request latency (p99) and error rate (5xx, failed-transaction percentage) — two of Google SRE’s four golden signals (the other two are traffic and saturation). If the user feels nothing, by definition there is no emergency.

  2. Don’t mistake CPU for a fault. High CPU is often healthy; it means you’re using the machine efficiently. Waking someone for a CPU spike that never reached a user only kills trust in the pager — which is precisely what happened.

What to do

  1. Where you can, alert on SLOs and error budget. Instead of “CPU 80% for 5 minutes”, say “we’re burning our error budget at this rate”. A multi-window burn-rate alert catches both a fast collapse and a slow leak, and cuts the noise.

    - alert: HighErrorBurnRate
      expr: |
        sum(rate(http_requests_total{status=~"5.."}[5m]))
          / sum(rate(http_requests_total[5m])) > 0.02
      for: 5m
      labels: { severity: page }
      annotations:
        summary: "5xx rate >2% (users are affected)"
  2. Every page must be actionable and urgent. If the person you wake can’t do anything immediately, or doesn’t need to, it isn’t a page. Every non-actionable alert trains the team to “ignore the pager”.

  3. Don’t throw causes away — relocate them. CPU, memory, disk IO are perfect for diagnosis and capacity planning; keep them on dashboards and as warning-level tickets, not pages. One exception: cases where a cause makes a symptom inevitable. Predictive saturation alerts like “disk will fill in 4 hours” are worth a page, because they warn of a symptom that hasn’t landed yet.

Bottom line: personally I’d reduce the definition of a page to one sentence: “Does this mean a user is hurting now, or will be soon?” If the answer is no, it moves to a dashboard or a ticket. I’d drop the current CPU/memory pages, define a few SLOs per service, restrict paging to symptoms and predictive saturation, and turn the rest into signals you look at during the day. Once trust in the pager is gone it’s hard to earn back; save it for what’s genuinely urgent.

Compiled from questions I have run into in the field or been asked.

Comments

Sign in with your GitHub account to join the discussion. Comments are stored in GitHub Discussions.

More Questions

All questions

Search the site

Start typing to search posts, projects and pages.

Esc to close Powered by Pagefind