SLOs: deciding how reliable is reliable enough

Ask any manager how reliable a service should be, and the answer is usually "100%". It sounds right. Who wants downtime?

But 100% is impossible, and chasing it is incredibly expensive. Every extra "nine" of availability costs more than the last. And your users can't tell the difference between 99.99% and 100%, because their own Wi-Fi, phone and ISP fail far more often than that.

Service Level Objectives (SLOs) are a way to decide, on purpose, how reliable is reliable enough.

SLIs, SLOs, error budgets, simply

SLI (Service Level Indicator): a measurement of how your service is doing, from the user's point of view. For example: the percentage of HTTP requests that succeed, or the percentage served in under 300 ms.

SLO (Service Level Objective): a target for that indicator over a time window. For example: 99.9% of requests succeed, measured over 30 days.

Error budget: the amount of failure the SLO allows. With a 99.9% target over 30 days, you can afford 0.1% failed requests, or roughly 43 minutes of full downtime.

SLA (Service Level Agreement): a contract with customers, usually with financial consequences. SLAs should be looser than your internal SLOs, so you notice problems before you owe anyone money.

Why error budgets are so useful

The error budget turns reliability from an argument into a shared number.

• Budget left? The team can ship features, experiment, take reasonable risks.
• Budget almost gone? Slow down, focus on reliability, postpone risky changes.
• Budget exhausted? Freeze risky deployments and fix what's causing the failures.

Product and engineering stop arguing about "should we work on reliability or features?" The budget answers it.

An SLO isn't a promise that nothing breaks. It's an agreement about how much breaking is acceptable, and what we'll do when we go over.

Picking good SLIs

Good SLIs measure what users actually experience:

• Availability: proportion of successful requests (non-5xx responses).
• Latency: proportion of requests faster than a threshold.
• Freshness: for data pipelines, how recent the data is.
• Correctness: for some systems, whether results are right.

Measure as close to the user as possible: at the load balancer or gateway rather than deep inside the service. Avoid SLIs like "CPU below 80%": users don't care about CPU.

Picking a target

Some practical advice:

• Start with your current performance. If you're at 99.5% today, a 99.99% SLO is fantasy. Set something achievable, then tighten it.
• Different services, different targets. Checkout might need 99.95%. An internal report generator might be fine with 99%.
• Consider dependencies. Your service can't be more reliable than the things it depends on, unless you design for their failures.
• Use a 28 or 30-day rolling window to start.

SLO	Allowed downtime per 30 days
99%	about 7 hours 12 minutes
99.5%	about 3 hours 36 minutes
99.9%	about 43 minutes
99.95%	about 22 minutes
99.99%	about 4 minutes

Looking at this table usually ends the "100%" conversation quickly.

Alerting on burn rate

SLOs also make alerting much better. Instead of alerting on CPU or individual errors, alert on how fast you're burning the error budget.

• Fast burn: at the current rate, you'll use the whole month's budget in hours. Page someone now.
• Slow burn: you'll use it up in days. Create a ticket, look at it during working hours.

The Google SRE Workbook recommends multi-window, multi-burn-rate alerts, for example: page if you've burned 2% of the monthly budget in the last hour (and the last five minutes confirm it's still happening). That catches real incidents quickly and ignores short blips.

Tools that help

Writing these Prometheus rules by hand is tedious and error-prone. Tools generate them from a simple SLO definition:

• Sloth: generates Prometheus recording and alerting rules from an SLO spec.
• Pyrra: similar, with a nice UI showing budgets and burn rates.
• OpenSLO: a vendor-neutral specification for defining SLOs as code.
• Most commercial observability platforms (Datadog, Grafana Cloud, Honeycomb, Nobl9 and others) have SLO features built in.

# Sloth example
version: "prometheus/v1"
service: "shop-api"
slos:
  - name: "requests-availability"
    objective: 99.9
    sli:
      events:
        error_query: sum(rate(http_requests_total{job="shop-api",code=~"5.."}[{{.window}}]))
        total_query: sum(rate(http_requests_total{job="shop-api"}[{{.window}}]))
    alerting:
      page_alert:
        labels:
          severity: critical

So, where do I start?

Pick your most important user-facing service. Define one availability SLI and one latency SLI. Set targets based on the last month's real data. Add burn-rate alerts, show the budget on a dashboard, and review it as a team every couple of weeks.

My take: perfection isn't the goal. Knowing exactly how imperfect you can afford to be, and acting on it, is.

— Desly