Alert Severity and Fatigue Management: On-call and SRE Practice

Alert Severity and Fatigue Management: On-call and SRE Practice

The ultimate paradox of alerting is too many alerts equals no alerts. When an on-call engineer receives dozens of notifications an hour and nine out of ten need no action, the brain starts ignoring every reminder — so when a real incident happens, nobody responds in time. Google's SRE philosophy stresses that the primary goal of being on-call is to make every wake-up worth it. That requires working on three layers at once: severity tiers, notification noise reduction, and the on-call process itself.

Severity tiers: not every problem deserves a page

First, separate "needs human action" from "just needs a glance". A common four-tier model:

The deciding factor is not "is a metric abnormal" but "if I do nothing, what happens to users". Metric-level noise should be filtered at the rule layer (see Prometheus alert rule design), not pushed onto the on-call phone.

The Real Cost of an Alert Storm

A mid-sized e-commerce company reviewed exactly this in 2025. During the early hours of a sales-event warm-up, the payment gateway started timing out against an upstream bank API and fired 3,200 alerts in 40 minutes. The on-call channel flooded, the root-cause alert drowned in the noise, and the whole core team was pulled into the group chat with nobody able to say who owned it. Post-incident analysis showed half the alerts were duplicate instances of one root cause, 20% were baseline noise that had existed for months, and only eight actually needed a human. With grouping and escalation in place, those eight would have been merged into one notification sent only to the payments on-call. The lesson: alert governance is not about receiving fewer messages — it is about someone acting immediately when an incident happens.

Noise reduction: grouping, routing, escalation

Even with good tiers, occasional alert storms still bury the critical signal. Three levers reduce noise:

• Group and aggregate: merge alerts sharing one root cause into a single notification (Alertmanager grouping, inhibition, and silences exist precisely for this).
• Route to the right receiver: different teams and severities flow through different channels, so each alert only disturbs the person responsible.
• Escalate when unacknowledged: auto-escalate if nobody confirms — e.g., one level up after 5 minutes, up the chain after 15 minutes — guaranteeing a backstop without spamming everyone.

With Prometheus + Alertmanager, a production-grade noise-reduction config looks roughly like this:

route:
  group_by: ['alertname', 'cluster']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  routes:
    - matchers: ['severity="page"']
      receiver: oncall-phone
    - matchers: ['severity="ticket"']
      receiver: ticket-queue
      continue: false

group_by merges instances sharing one root cause; group_wait prevents short flapping from firing in bursts; repeat_interval controls how often the same alert repeats instead of hammering you hourly. Combined with overnight on-call, this setup keeps the phone ringing only when a human actually needs to act.

On-call rotations: let the paged person be able to act

No matter how good the tiers, fatigue persists if the on-call person gets an alert but doesn't know what to do. SRE practice closes the loop with a complete rotation cycle:

1. Rotation calendar and handoff: define who is on call, the rotation period, and how handoff happens, so no one is on call indefinitely.
2. Runbooks: every high-frequency alert maps to a runbook — which dashboard to check first, what commands to run, what the usual root cause is — turning "first-time debugging" into "follow the checklist".
3. Post-incident review: hold a postmortem after every real incident, separate human error from system defects, and turn improvements into new runbooks or rules.

A good runbook covers five things: blast radius, diagnostic steps, remediation commands, rollback plan, and when to escalate. Writing those five down is often enough to cut average recovery time in half.

Measure on-call health with data

The effectiveness of alert governance can be observed with a few metrics: alerts per shift (should trend down), page rate (share truly needing immediate response, aiming for 10%-20%), false positive rate (alerts confirmed as "no action needed"), and MTTA/MTTR (time to acknowledge and resolve). If the false positive rate stays high, alert rules need tightening; if the page rate is too low, many "page-level" alerts should be demoted to ticket-level. Spending 30 minutes a week reviewing these numbers beats relying on the feeling that "alerts seem fewer".

16IDC Take

Small teams don't need a heavyweight on-call program like a big company, but two principles are universal: every alert must have a clear next action, or it gets deleted; and make on-call a rotation — even three people rotating significantly reduces individual fatigue. For scope and tooling see the monitoring and alerting guide and deploying Uptime Kuma; for review templates see the incident postmortem template; and calibrate whether alerts align with business goals using SLO/SLI and error budgets. For rule writing and dashboards, start with Prometheus and Grafana basics. See more in the Monitoring & Alerting category.

Reference: Google SRE Workbook, "Being On-Call" https://sre.google/sre-book/being-on-call/; "On-Call Rotations: How Best to Awaken Engineers" https://sre.google/sre-book/oncall/