Prometheus Monitoring System Setup: From Metric Collection to Alert Notifications

Prometheus Monitoring System Setup: From Metric Collection to Alert Notifications

Prometheus is a graduated project of the Cloud Native Computing Foundation (CNCF) and has become the standard tool in the monitoring space. It works on a pull model: monitored targets expose a /metrics endpoint and Prometheus scrapes it on a schedule. Because the monitoring system pulls rather than waiting for targets to push, it fits dynamically scaling environments naturally.

Architecture Overview

Exporter → Prometheus Server → AlertManager → Notification Channels
                                  ↓
                              Grafana (Visualization)

Exporters translate system or application metrics into the Prometheus format; AlertManager deduplicates, groups, and routes alerts to email/IM; Grafana handles visualization. These four layers have clear, separable responsibilities — data collection and storage belong to Prometheus, alerting to AlertManager, and visualization to Grafana — so any layer can be upgraded or swapped independently without starting over.

Metric Types: Four Basics to Know First

For system monitoring you mostly need Counter and Gauge; understanding those two is enough to stand up a usable setup. It also helps to know the naming conventions: Prometheus expects Counters to end in _total (like node_cpu_seconds_total) and unit information to go into the metric name (e.g. _bytes, _seconds), which keeps queries and team collaboration unambiguous.

Installation and Configuration

# docker-compose.yml
version: '3'
services:
  prometheus:
    image: prom/prometheus:latest
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
    ports:
      - "9090:9090"

  grafana:
    image: grafana/grafana:latest
    ports:
      - "3000:3000"
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=admin

  node-exporter:
    image: prom/node-exporter:latest
    ports:
      - "9100:9100"

The default scrape_interval is 15s: the higher the scrape frequency, the smoother the metrics — but also the higher the storage and overhead. For host monitoring, 15s or 30s is usually enough; there is no need to chase higher frequency. Keep the evaluation interval in line with the scrape interval so alerts are computed from fresh data.

Scrape targets and alert-rule files live in prometheus.yml:

global:
  scrape_interval: 15s
rule_files:
  - /etc/prometheus/rules/*.yml
scrape_configs:
  - job_name: node
    static_configs:
      - targets: ["node-exporter:9100"]

Common PromQL Queries

# CPU usage
100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)

# Memory usage
100 * (1 - node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)

# Disk usage
100 * (node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"})

These three cover the essentials — CPU, memory, and disk — and combined with Grafana they form a basic host dashboard.

Alert Rules

groups:
  - name: server
    rules:
      - alert: HighCPUUsage
        expr: 100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 80
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "CPU usage above 80%"

for: 5m means the condition must hold for 5 minutes before firing, which avoids false alerts from momentary spikes. Categorize rules by severity — warning (worth watching) and critical (needs immediate action) — and wire them to AlertManager:

route:
  receiver: "webhook"
  routes:
    - match:
        severity: critical
      receiver: "pager"
receivers:
  - name: "webhook"
    webhook_configs:
      - url: "https://hooks.slack.com/services/xxx"

Grafana Visualization

Grafana's default login is admin/admin. Two recommended steps: import the official Node Exporter Full dashboard (ID 1860), then build a custom dashboard with the PromQL above and add business metrics such as homepage page views, API latency, and error rate. The goal is not many dashboards — it is being able to judge "is the system healthy" in 30 seconds.

A Real-World Scenario

A site wired up this monitoring stack the day before a big promotion. In the early hours of the promotion, the CPU alert (sustained >80% for 5 minutes) fired; the on-call engineer followed the latency spike on the product-page API in Grafana, traced it to slow database queries, and added an index before traffic ramped up. The whole promotion ran without incident. Without monitoring, users would likely have discovered the slow pages before the team did.

Monitoring Launch Checklist

• node-exporter deployed on every server
• Key applications expose custom metrics
• Alerts reach at least one channel that someone actually sees
• Alert thresholds avoid false positives (use a for duration)
• A Grafana dashboard a newcomer can read in 30 seconds

Frequently Asked Questions

The metric data keeps growing and the disk is filling up — what now? Prometheus is a time-series database and data grows continuously. Common fixes: adjust the retention period with --storage.tsdb.retention.time (e.g. keep 15 days and archive history to object storage); cap storage with --storage.tsdb.retention.size; or run business metrics on a separate instance so they do not mix with host metrics.

Alerts fire all night and never stop — how do I tame that? That is an alert storm. Common causes: thresholds set too sensitively, no for duration, and no deduplication. Tame it in this order: add a for duration (at least 1-5 minutes) to every rule; use AlertManager's group_by to merge similar alerts into one; silence known maintenance windows; and periodically delete alert rules nobody acts on.

Is monitoring just CPU and memory enough? No — that only covers the server layer. Business-level metrics matter just as much: API latency, error rates, queue backlog, slow database queries — these are what users actually feel. node-exporter is only the first step; next, have applications expose custom metrics through a Prometheus client library, or use existing integrations (nginx exporter, mysql exporter) to build a three-layer view of host + middleware + business.

Grafana is slow to open — what should I check? First check whether a dashboard loads too many queries or too large a time range at once, then confirm the network latency between Grafana and Prometheus, and enable Grafana caching if needed. Keep the number of dashboards under 10, pin the common ones to the sidebar, and reduce the computation per load.

Reference: Prometheus official docs https://prometheus.io/docs/introduction/overview/
Reference: Grafana official docs https://grafana.com/docs/
Reference: AlertManager configuration https://prometheus.io/docs/alerting/latest/alertmanager/