SRE · commonly confused
Symptom-based vs cause-based alerts
Short answer
Reproduced on Python 3.9.6, . Where the text says “reproduced”, the behaviour was run on that version, not taken from documentation.
First published
On this page · 10 sections
In simple terms
Cause-based alerting is the natural thing to build, because causes are what you can see. CPU is at 90%, the queue has 5,000 items, disk is 80% full, a pod restarted. All easy to measure and easy to threshold.
The trouble is that none of those statements says anything about whether anything is wrong. CPU at 90% might be a service working efficiently. A deep queue might be a batch job doing its job. The unusual thing can be perfectly fine, and a page about it is then noise.
Symptom-based alerting starts from the other end: is anybody actually experiencing a problem? Error rate up, latency past what users tolerate, a queue whose age means work will miss its deadline. Far fewer of these fire, and when one does, something genuinely needs attention.
The reason this change pays off is arithmetic rather than philosophy. Causes outnumber symptoms, because any given symptom has many possible causes, so alerting on causes means alerting on a much bigger set, and a cause can be present without any user noticing.
Words used on this page
- symptom
- What users experience: errors, slowness, work arriving late. In Google's SRE book, the "what's broken".
- cause
- A reason a symptom might happen, such as high CPU or a full queue: the "why". One person's symptom can be another's cause, as with slow database reads seen from the web front end.
- page
- An alert that interrupts a person straight away, by phone or pager, as opposed to a ticket or a dashboard someone reads later.
- actionable
- There is something the person paged can and should do now. Google's SRE book: every page should be actionable.
- SLI
- Service level indicator: a measured number that reflects what users get, such as the share of requests that succeed.
- SLO
- Service level objective: the target for an SLI, such as 99.9% of requests succeeding over 30 days.
- error budget
- The failure an SLO allows: for 99.9%, 0.1% of requests over the window.
- burn rate
- How fast the error budget is being spent compared with the pace that would use it up exactly at the end of the window. A burn rate of 1 spends it all in 30 days; 14.4 spends it in about two days.
- predict_linear
- A Prometheus function that fits a straight line to recent samples and predicts the value a given number of seconds ahead. It turns a disk-space metric into "full within four hours".
What actually happens
Google's SRE book puts the rule for paging this way: "Paging alerts should be aligned with the symptoms that threaten a service's SLOs." In practice, page on what your SLIs measure and let most other metrics be dashboards. If a metric is not part of how you measure whether the service is meeting its objective, it is diagnostic data rather than a paging condition.
This does not mean discarding cause metrics — they are exactly what you need once you are already investigating. It means changing what wakes somebody up. The cause data should be one click from the symptom alert, not the thing that fired it.
The counter-argument is real and worth stating: symptom alerts fire later. A disk filling at a steady rate is a cause you can act on hours before any user notices, and waiting for the symptom means waiting for an outage. So the honest position is that a small number of cause alerts are justified — specifically the ones where the lead time is long and the trajectory is predictable, like disk capacity or certificate expiry.
That is a narrow exception, not a general licence. The test is whether you can name the user-visible failure it prevents and whether the trajectory genuinely gives you useful warning.
The same incident, alerted both ways
# Cause-based — fires often, mostly harmlessly ALERT HighCPU cpu > 90% for 5m ALERT QueueDepth queue_length > 5000 ALERT PodRestarted increase(restarts[10m]) > 0 ALERT DiskAt80 disk_used > 80% # Symptom-based — fires rarely, and means something ALERT ErrorRateHigh rate(5xx) / rate(all) > 0.02 for 5m ALERT LatencyBad p95_latency > 1s for 10m ALERT QueueStale oldest_message_age > 300s # The justified cause exception: long lead time, predictable trajectory ALERT DiskFullSoon predict_linear(disk_free[6h], 4*3600) < 0
The rules are shorthand: in Prometheus each one is a YAML entry with alert, expr and for fields. Note what changed in the disk alert. disk > 80% fires on a state that may be entirely stable; the prediction fires only when the trajectory says you will actually run out — and it tells you roughly when.
Try it yourself
Before you start
- An SLO for the service, or at least an agreed SLI such as the share of requests that fail or are too slow. Without one there is no symptom to alert on.
- Read access to your alerting rules and to the history of pages per rule for the last few weeks, including whether anyone acted on each one.
- A way to try a rule before it can wake anyone: a test route in your alert manager, or
promtool test rulesif you use Prometheus. - Python 3 or a calculator for the arithmetic in the example below; it was reproduced on Python 3.9.6.
Turning an SLO into page thresholds
Reproduced on Python 3.9.6 on macOS, with the time zone set to UTC; it is plain arithmetic, so any Python 3 prints the same. The inputs are the starting points Google's SRE workbook recommends for a 99.9% SLO over 30 days: page when 2% of the error budget is spent in one hour or 5% in six hours, and open a ticket for 10% in three days. They are Google's recommendation, not a standard: the workbook says the right numbers depend on the service.
Command
python3 -c 'print(round(0.02 * 30 * 24 / 1, 1))' python3 -c 'print(round(0.05 * 30 * 24 / 6, 1))' python3 -c 'print(round(0.10 * 30 * 24 / 72, 1))' python3 -c 'print(round(14.4 * (1 - 0.999) * 100, 2), round(6 * (1 - 0.999) * 100, 2))'
Output
$ python3 -c 'print(round(0.02 * 30 * 24 / 1, 1))' 14.4 $ python3 -c 'print(round(0.05 * 30 * 24 / 6, 1))' 6.0 $ python3 -c 'print(round(0.10 * 30 * 24 / 72, 1))' 1.0 $ python3 -c 'print(round(14.4 * (1 - 0.999) * 100, 2), round(6 * (1 - 0.999) * 100, 2))' 1.44 0.6
How to read it
- Burn rate is the share of the budget spent, times the SLO window, divided by the alert window: 2% of a 720-hour budget in one hour is 14.4. At that pace a 30-day budget is gone in about 50 hours.
- 5% in six hours is a burn rate of 6. 10% in three days is 1.0, exactly the pace that spends the whole budget by the end of the window; the workbook makes that a ticket, because there is time to deal with it in working hours.
- Multiply the burn rate by the budget, 0.1% for a 99.9% SLO, to get the error ratio to alert on: page if more than 1.44% of requests failed over the last hour, or more than 0.6% over the last six hours. The workbook writes the first as
> (14.4*0.001). - Both alerts watch a symptom, the share of requests that fail for users, and their size says how urgent it is. The fixed
> 0.02in this page's example is a burn rate of 20 against a 99.9% SLO, chosen by feel rather than from the budget. - The workbook pairs each long window with a short one, 5 minutes for the one-hour alert and 30 minutes for the six-hour one, so the page stops soon after the errors do instead of firing for the rest of the hour.
The mistake this causes
Adding an alert after every incident.
Why people do it It feels like learning, and it is an easy post-incident action item. Something was missed, so we add an alert for the thing that was missed — and each addition is individually defensible.
What you see Alert volume climbing steadily while the actionable rate falls, and nobody able to point at when it became unmanageable because no single addition was wrong. Eventually the rational response to a page is to assume it is noise, and people adopt that response whether or not they say so — which means the next real page is treated the same way.
If it goes wrong
- A symptom alert fires and nobody can tell from the page what is causing it
- Cause metrics are now your investigation tools, so put them one click away: add a link to the service's dependency dashboard in the alert's annotations. Google's SRE book keeps cause-oriented heuristics "as aids to debugging problems", not as pages.
- One failed request pages someone at night
- The service has too little traffic for a ratio to mean much. Google's workbook gives the case of 10 requests an hour, where one failure is a 1,000x burn rate against a 99.9% SLO. Its options: synthetic traffic, combining related low-traffic services into one alert, making a single failure matter less (client retries), or a lower SLO or longer window agreed with the service's owners.
- The burn-rate page never fires, even during a full outage
- Check the thresholds against the SLO. The workbook's example: at a 90% target, a complete outage spends only 1.4% of a 30-day budget per hour, so a page at 2% per hour can never fire. Services with very low or very high targets need their windows tuned.
- One incident produces several pages: the one-hour, six-hour and three-day alerts all fire
- Fast burning satisfies every slower threshold too. Suppress the lower-severity alerts while a higher one is firing (in Prometheus, an Alertmanager inhibition rule), and group related alerts so one problem sends one notification.
How it shows up in production
A shared dependency slows down. Because most alerting is cause-based, every service that touches it fires: CPU alerts from retry loops, queue-depth alerts from backpressure, restart alerts from failing health checks, latency alerts at every layer.
Forty pages arrive in ten minutes describing one problem, and the responder's first job is not diagnosis — it is working out how many distinct incidents they are looking at. That triage is pure cost, created by the alerting rather than by the fault.
With symptom alerting the same incident produces a handful of pages naming user-visible impact, and the cause metrics are all still there to investigate with. Same information, radically different demand on the person holding the pager at 3am.
How to tell which one you are hitting
- Does this alert name something a user would notice?
- If you cannot describe the user-visible impact in one sentence, it is a cause alert. That is not automatically wrong, but it should be a dashboard unless it passes the lead-time test.
- What proportion of these pages led to an action?
- Google's SRE book sets the bar at every page being actionable, and each page that leads to nothing teaches people to ignore the next one. That is worth measuring per rule rather than in aggregate, so you can see which rules produce the noise.
- Would waiting have been worse?
- The question that justifies a cause alert. Disk filling and certificates expiring both fail this in the other direction — waiting for the symptom means waiting for an outage, so the cause alert earns its place.
Security considerations
- Security detection is the stated exception to symptom-only paging. Google's SRE book says never to alert just because something seems a bit weird, unless you are "performing security auditing on very narrowly scoped components". An intrusion can have no user-visible symptom at all, so keep narrow security alerts of their own rather than waiting for an SLO to burn.
- Alert on the monitoring itself. If Prometheus or the alert manager stops, symptom alerts go silent, which looks exactly like a healthy service. Prometheus's guide says to have alerts that monitoring servers and Alertmanagers are running, ideally an end-to-end test that alerts actually get through.
- Certificate expiry is a cause alert worth keeping. An expired certificate fails every connection at once, with no gradual symptom to page on first, and Google's workbook says you might need alerts as you approach exhaustion of a resource with a hard limit.
- Alert text travels: annotations end up in chat channels, email and third-party paging services. Link to dashboards rather than pasting request contents, customer identifiers or tokens into the alert itself.
In an interview
How this gets asked in an interviewpreparation
Usually phrased as “How would you reduce alert fatigue?”
What a strong answer contains Lead with the shift from cause to symptom and explain why it works arithmetically — causes outnumber symptoms, so alerting on causes means alerting on a much larger set that includes states users never notice. Then give the practical rule: alert on your SLIs, keep the rest as dashboards. Naming the justified exception for long-lead-time causes shows you are reasoning rather than reciting.
The follow-up “What about disk filling up?” is the standard test of whether you hold the position dogmatically. The good answer is that predictable trajectories with long lead times are exactly where a cause alert is right, and that the alert should fire on the prediction rather than on a static threshold.
Sources
The behaviour this page describes, as documented by the projects themselves.
Terms used here (3)plain definitions
Short definitions for the vocabulary on this page, in case any of it is new. Each links to the full entry.
- Logging
- The practice of recording events, errors, and informational messages from applications and infrastructure. Centralized logging with tools like the ELK stack or Loki is essential for debugging, auditing, and understanding system behavior in production.
- Observability
- The ability to understand a system's internal state through its external outputs — metrics, logs, and traces. Observability goes beyond traditional monitoring by enabling engineers to ask arbitrary questions about system behavior without deploying new instrumentation.
- Prometheus
- An open-source monitoring and alerting toolkit that collects and stores time-series metrics data. Prometheus uses a pull model to scrape metrics from instrumented targets and provides a powerful query language (PromQL) for analysis and alerting.
Did this get you to an answer?
No text box on purpose — please do not paste production logs anywhere
Something wrong or out of date? Report it