Implementing SLO/SLI and Error Budget Governance

Implementing SLO/SLI and Error Budget Governance

"99.9% reliability" is meaningless if it's just a number on a metrics wall. Google SRE methodology turns it into an executable management mechanism: define SLIs (service level indicators), set SLOs (service level objectives), convert "availability" into a spendable balance called the error budget, and then decide when to fix failures versus when to ship features.

Step 1: choose the right SLIs

SLIs are "what we actually measure", usually drawn from availability, latency, throughput, and error rate. A good SLI tracks user-perceivable experience: for websites, common denominators are "the share of valid requests that succeed" and "P95 latency". Two points to remember:

• Be precise about the denominator: define exactly which HTTP 4xx/5xx count as "errors" and whether crawlers and health checks are excluded from "valid requests".
• Use ratios, not absolutes: express SLIs as "good events / valid events", which makes downstream SLO math straightforward.

Step 2: set SLOs and compute the error budget

An SLO is a target value for an SLI, e.g., "over the past 30 days, availability SLI ≥ 99.9%". Higher targets cost more, and SRE stresses that there is no 100% SLO — a 99.9% target promises roughly 43 minutes of allowed downtime per month. That "allowed to fail" margin is the error budget.

Error budget = 100% − SLO; e.g., 99.9% means roughly 43 minutes per month of "budget". Its greatest value is giving release cadence a basis for decision: when the budget is healthy, ship features confidently; when it's exhausted, stop releases and fix reliability. Reliability now has a currency to discuss with the business.

Step 3: burn-rate alerting instead of fixed thresholds

Traditional "alert if latency exceeds 500ms" either false-positives at low traffic or misses at high traffic. Google recommends error-budget burn-rate alerting: alert based on how fast the budget is being consumed relative to the 30-day window. A typical two-tier setup:

• Page-level: error rate is on track to exhaust the budget within ~2 hours → notify on-call immediately.
• Ticket-level: projected to exhaust within ~1 day → handle during business hours.

Alerts then naturally tie to "is the service consuming its reliability budget" rather than to instantaneous thresholds, significantly cutting false positives.

How Burn Rate Is Actually Calculated

Burn rate = actual error rate ÷ the error rate the SLO allows. With a 99.9% SLO (0.1% allowed), a 5-minute error rate of 1% gives a burn rate of 10, meaning at this pace the 30-day budget burns out in 3 days (30 ÷ 10). Google recommends "multi-window + multi-tier burn rates" to balance sensitivity against false positives. A commonly used set:

On Prometheus, use recording rules to precompute the error rate and then alert per window, following Prometheus alert rules:

groups:
  - name: slo-burn-rate
    rules:
      - record: job:slo_errors_total:rate5m
        expr: sum(rate(http_errors_total{code=~"5.."}[5m])) by (job)
      - record: job:slo_requests_total:rate5m
        expr: sum(rate(http_requests_total[5m])) by (job)

The dual-window idea: a short window ensures an immediate alert when burn is fast, while the long window suppresses false positives from brief jitter. For example, the page tier requires both the 1-hour and 5-minute windows to fire together, so a momentary spike does not wake anyone but a real outage is never missed.

Implementation checklist

1. Define 1-3 SLIs per core user journey — don't overdo it;
2. Set a quarterly SLO and error budget, and share the budget with the product team;
3. Configure burn-rate alerts, implemented with Prometheus alert rules;
4. Review budget consumption weekly, and revisit targets quarterly.

A Real Decision Scenario

Imagine an e-commerce site with a 99.9% availability SLO, i.e., 43 minutes of error budget per month. A week before the big sales day, burn-rate alerts keep firing and the page-level alert shows only about 4 hours of budget left. The real choice facing the team: ship the new add-to-cart feature as planned, or stop releases and investigate the cache stampede behind the alerts. Under error-budget rules the answer is clear — budget is nearly gone, halt non-essential releases and put reliability above new features. Conversely, if 90% of the budget remains at the start of the month, there is no reason to reject a low-risk release. Turning the "should we ship" argument into the numeric question "how much budget is left" is the greatest management value of SLOs.

A reference SLO set for a content site

For a content website, a pragmatic SLO set might look like this:

Two points: performance SLOs have no "error budget" concept, but they can still have a target-achievement rate; and more SLOs are not better — track only 2-3 per core journey so every target has a corresponding alert and owner.

16IDC Take

The value of an SLO is not "reaching 99.99%" but aligning product and engineering with one metric language. Independent sites can start light: set baselines with the SLO/SLI template, measure availability with synthetic checks (synthetic monitoring in practice), measure performance with RUM (frontend RUM monitoring), and feed burn-rate alerts into your on-call rotation (on-call practice). See more in the Monitoring & Alerting category.

References: https://sre.google/workbook/error-budget-policy/, https://sre.google/sre-book/service-level-objectives/, https://prometheus.io/docs/practices/slo/

Source: https://sre.google/workbook/error-budget-policy/