Designing Prometheus Alert Rules: PromQL and Alertmanager Routing

The value of monitoring is not "having data" — it is "getting the right alert at the moment you need to act". The Prometheus alerting chain has two layers: alerting rules continuously evaluate PromQL on the Prometheus side and produce alerts; Alertmanager then deduplicates, groups, routes, inhibits, and silences them, and delivers notifications to the correct channel. How well this layer is designed decides whether a team is drowned in alerts or actually gets a good night's sleep.

Write meaningful rules with PromQL

An alerting rule's core is a PromQL expression plus a for duration. The official guidance recommends several principles:

Use for to filter jitter: instant checks like up == 0 are easily tripped by network jitter. Adding for: 5m — "alert only after 5 minutes" — dramatically reduces false positives.
Use rate() rather than raw counters: for counter-type metrics (requests, errors), always compute a rate with rate() or increase(). Comparing raw values is almost meaningless.
Watch ratios and trends: more useful than "error count above X" is "error rate above 1%" or "error rate doubled in the last 15 minutes", which maps closer to user-perceivable failures.

A typical high-quality rule looks like this:

groups:
  - name: api-errors
    rules:
      - alert: HighErrorRate
        expr: |
          sum(rate(http_requests_total{job="api",status=~"5.."}[5m]))
            / sum(rate(http_requests_total{job="api"}[5m])) > 0.05
        for: 10m
        labels:
          severity: page
        annotations:
          summary: "API 5xx error rate above 5%"

There is no universal answer for for, but these starting points work well:

Metric type	Suggested `for`	Why
Node liveness (`up`)	2-5m	Too short trips on network jitter; too long delays response
Rising error rate	5-10m	Need to filter out transient spikes
Resource exhaustion (memory/disk)	10-15m	These usually build up gradually; no need to race the alert

To express something like "the error rate doubled over the last window", combine rate() with a comparison: sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total{status=~"5.."}[15m])) > 2. Trend alerts track user-perceivable change more closely than fixed thresholds and map more naturally onto SLO targets.

Alertmanager: grouping, inhibition, and silences

Prometheus pushes produced alerts to Alertmanager, which does three things:

Grouping: merges similar alerts into a single notification. The official docs cite the example of dozens or hundreds of instances losing database connectivity at once: grouping by cluster and alertname sends one compact notification listing the affected instances instead of hundreds. Grouping is configured by a routing tree.
Inhibition: when a higher-level alert (e.g., an entire cluster is unreachable) is firing, mute other lower-signal alerts that won't help diagnosis, avoiding "hundreds of unrelated alerts firing at once".
Silences: mute alerts for a given time based on matchers, e.g., silencing a service's alerts for 30 minutes during a release window. Configured in the Alertmanager web UI.

An example route that dispatches by severity:

route:
  group_by: ['cluster', 'alertname']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  routes:
    - match:
        severity: page
      receiver: pagerduty-critical
    - match_re:
        severity: warning
      receiver: slack-alerts

Rate limiting and high availability

Alertmanager supports two easily overlooked operational details: the --alerts.per-alertname-limit flag caps active alerts per alertname, preventing an abnormal instance count from flooding receivers; and --cluster-* flags form a high-availability cluster. Importantly, do not load balance between Prometheus and Alertmanager — point Prometheus at the full list of Alertmanager instances.

Labels: the addressing system for alerts

The labels in an alerting rule are not just display info — they determine how alerts get routed, grouped, and inhibited. Add two layers of labels to every rule: business dimensions (such as service, environment, team) for team-based routing, and severity (such as severity: page|warning) for deciding the notification channel and escalation path. Matching these labels in the routing tree with match and match_re gives you fine-grained control over "which team, what level, which channel".

A common pitfall is stuffing dynamic values into the alert name or labels, e.g., putting the instance IP into alertname. That makes one logical rule produce countless "apparently different" alerts, breaking grouping and inhibition. The right pattern is to keep rules templated: dynamic information belongs in annotations for display, static information belongs in labels for addressing.

Another easily overlooked piece is recording rules: precompute expensive expressions reused across many dashboards into new metrics. This lightens query load and keeps alert expressions short and readable. For example, record sum(rate(http_requests_total[5m])) by (service) as job:http_requests:rate5m, then have every dashboard and alert reference that single name.

Debriefing an alert storm

A team shipped a "disk usage > 80%" alert with no for and no grouping by node. During a nightly batch job, 80 machines crossed the threshold at once, and Alertmanager pushed 80 notifications to the on-call phone. By the time the engineer muted the feed, the "primary database connection count spike" alert had been buried in the noise.

Three fixes came out of the postmortem: add for: 15m to the disk alert, group by instance in the route so one notification lists all affected nodes, and give high-priority alerts like "primary database" their own route at a higher precedence. The same scenario never flooded the channel again. The lesson: the number of alerts is not the problem — whether the alert that needs attention is actually seen is.

Reference: Prometheus alerting overview https://prometheus.io/docs/alerting/latest/overview/, recording rules docs https://prometheus.io/docs/prometheus/latest/configuration/recording_rules/

16IDC Take

Polishing alert rules is an ongoing process: after a month in production, demote or delete alerts that never drove action, and turn post-incident reviews into new rules. Pair this with Prometheus + Grafana basics and the Prometheus monitoring setup guide for the base deployment; combine rules with on-call severity in alert fatigue management, and bind alerting to SLOs with SLO/SLI and error budget practice. For Grafana-side features, follow Grafana 13.1 release. See more in the Monitoring & Alerting category.

Source: https://prometheus.io/docs/alerting/latest/alerting_rules/

Designing Prometheus Alert Rules: PromQL and Alertmanager Routing