Alert Design Basics: How to Set Thresholds and Severity So Alerts Are Neither Noisy nor Missed

Alert Design Basics: How to Set Thresholds and Severity So Alerts Are Neither Noisy nor Missed

In the first week after monitoring goes live, the chat group is flooded with alerts; a month later, everyone has learned to "read and ignore". That is the classic symptom of failed alert design: alert fatigue. Conversely, raising thresholds too high to avoid noise means failures surface only after users complain. Good alert design must be both "not noisy" (useful information) and "not missing anything" (critical failures always reach someone). This article covers three things: how to set thresholds, how to assign severity, and how to run the loop — plus a set of rule examples you can copy directly.

1. How to Set Thresholds: Do Not Guess

Set them too low and you get false alarms daily; set them too high and real problems go unnoticed. Three practical methods:

1. Baseline method: run 1–2 weeks of collect-only (no alerts), observe the "normal fluctuation range" of each metric, then set the threshold just outside that range with buffer. For example, if CPU usually swings 20%–40%, a threshold of 80% is reasonable; 45% would alarm every day.
2. Dual-threshold method (warning + critical): for example, disk >80% sends a "warning" notice, while >90% triggers a "critical" alert. This leaves buffer time to handle issues instead of jumping straight to the top severity.
3. Duration method: a single momentary spike above threshold may be jitter; add "sustained for 5 minutes" before triggering. For instance, "CPU >90% for 10 minutes" is far more reliable than "CPU >90%", filtering out most transient spikes.

A principle: an alert should land where "you can do something about it". If receiving an alert leaves you only able to watch and do nothing, that alert is noise.

2. Severity Levels P1–P4: Make "Urgent" Actually Urgent

Mixing all alerts into one level is the same as having no levels. A common breakdown:

The key point: P1/P2 use strong channels like phone/SMS; P3/P4 use weak channels like group messages/email. Do not let "can handle tomorrow" wake up the on-call at 3 a.m. — set P3/P4 notification windows to work hours.

3. Five Actions to Avoid Alert Fatigue

1. Deduplicate: do not repeatedly fire the same alert for one failure. For example, if "error rate >5%" is checked every 5 minutes, it should "silence" after triggering and reset only on recovery, rather than posting 12 messages per hour.
2. Group and route: group alerts by team/system (infrastructure, payment, marketing) and notify only the relevant people — send each alert to whoever owns it.
3. Suppress dependencies: a server going down triggers dozens of service alerts — use "parent alert suppresses child alerts": when a host is unreachable, all service alerts on it are suppressed automatically.
4. Reduce noise by severity: use P1–P4 to decide the channel and audience, per the table above.
5. Review regularly: review alert statistics monthly, delete rules that "never led to action", and fix thresholds that repeatedly false-alarm.

Full alert-fatigue governance and on-call practices are in Alert Fatigue & On-Call Practice.

4. Alert → Notify → Respond: Only a Closed Loop Counts

An alert only has value when it completes the whole loop: "trigger → notify → someone responds → handle → close → review". Implementation points:

1. Trigger: the rule fires, carrying enough context (which host, which metric, current value, time).
2. Notify: use the channel matching the severity (P1 phone/SMS, P3 group message), and write in the message "what it is, how big the impact, where to look first".
3. Respond: someone acknowledges; if not acknowledged in time, escalate automatically (e.g., notify the manager after 15 minutes).
4. Handle: act per the runbook; the incident postmortem template is in Incident Response Playbook.
5. Close and review: close after confirming recovery, then record "why it happened, whether the alert missed anything, whether the threshold was reasonable".

One advanced concept: bind alerts to SLOs (service level objectives) and alert only on "failures that would hurt the SLO", which cuts noise dramatically — see SLO & Error Budget Practice.

5. Alert Rule Examples: Copy Directly

In Prometheus-style syntax (fields vary slightly by platform):

- name: infra-critical
  rules:
    - alert: DiskWillFill
      expr: disk_usage_percent > 90
      for: 5m
      labels: { severity: P2 }
      annotations:
        summary: "Disk usage above 90% (current {{ $value }}%)"
        runbook: "Clean logs or expand capacity; steps in wiki/DiskRunbook"

    - alert: High5xxRate
      expr: rate(http_requests_total{status=~"5.."}[5m]) / rate(http_requests_total[5m]) > 0.05
      for: 3m
      labels: { severity: P1 }
      annotations:
        summary: "5xx error rate above 5%"
        runbook: "Check recent releases and error logs"

    - alert: ApiLatencyP99High
      expr: histogram_quantile(0.99, rate(http_request_duration_seconds_bucket[5m])) > 2
      for: 5m
      labels: { severity: P2 }
      annotations:
        summary: "Endpoint P99 latency above 2 seconds"

The three examples cover capacity (disk), reliability (error rate), and experience (latency). For deeper rule-design methodology see Prometheus Alert Rule Design, and for standing up the full stack see Prometheus + Grafana Basics.

6. Frequently Asked Questions

Q1: We have no dedicated on-call. Do we still need severity levels?
Yes — even more so. Without dedicated on-call, severity decides "should I get up at 3 a.m. for this". P3/P4 during work hours protects sleep without missing real failures.

Q2: How do I tell whether my alert design is healthy?
Watch two numbers: the alert volume trend (should fall or stabilize) and the number of missed incidents (problems users found before you did). The best alerts are "rare enough to remember, useful every time".

Q3: Too many false alarms — should I just raise the thresholds?
Find the root cause of the false alarms first (threshold, collection, or jitter) before deciding. Raising thresholds directly can "raise away" real failures too. Combine the duration and dual-threshold methods instead.

Q4: Where can I find more monitoring & alerting content?
The Monitoring & Alerting category on this site covers monitoring setup, metric systems, SLOs, and more — a complete learning path alongside this article.